atom

Your Website is Your API: Quick Wins for Government Data

Jeni — Sun, 01 Feb 2009 09:28:57 +0000

This is the talk I prepared for the UKGovWeb Barcamp, in blog form. It’s probably better this way. Most of what’s written here seems blindingly obvious to me, and probably to most readers of this blog, but maybe Google will direct someone here who finds it useful.

Working with public-sector information on the web, one of the things that I take an interest in is making government data freely available for anyone to re-present, mash-up, analyse and generally do whatever they want to do. This post is born out of a feeling that the people who control data don’t realise that the smallest changes can be beneficial: they don’t need to do everything right now, just something.

There are three fundamental things that you need to do:

identify the data that you control
represent that data in a way that people can use
expose the data to the wider world

but you can choose the degree to which you do each of these things.

Identify

Take a look at what data you have some kind of responsibility for or control over. You might be a PDF containing a table of schools in the local area and their intakes over the last couple of years. You might have a spreadsheet of the amount of money assigned to maintaining the playgrounds within the borough. You might have a database of company information. You might have a set of HTML agendas for court cases.

The first step is simply to identify what the information is about. Schools, playgrounds, companies, court cases — each row in your table or spreadsheet or database, or each section in your document will be about something. We call this a resource.

To play nicely with the web, every resource should have an identifier. A Uniform Resource Identifier. A URI. That URI tells us where we can find information about the resource (we’ll get to what those look like later). So your second step is to work out URIs for each of your resources.

Now, there are actually three levels of URIs that you can care about:

identifier URIs
document URIs
representation URIs

You probably already have document and representation URIs on your web server. Representation URIs are URIs for particular formats and languages and views of the information that you make available. Document URIs are typically the same URI without an extension; web servers use content negotiation to work out which representation to serve up when a web browser asks for the page at a particular document URI.

So you already have a URI for the PDF that contains the table of schools, for the Excel spreadsheet about the playgrounds. You already have URIs for the results of a particular query on your database, and of course the HTML pages that you deliver have URIs already. That’s all in place. You don’t want to change it.

But identifier URIs are what are really important when it comes to opening up your data. They shift the focus from the documents that you serve to the resources that they are about. By assigning URIs to resources, you enable other people to talk about them. Even if that’s all you do, you have done good.

For example, if Companies House stated that companies could be referred to using URIs of the form http://www.companieshouse.co.uk/id/company/{registeredNumber} then other people who needed to talk about companies (websites containing customer feedback, monitoring companies going into receivership, displaying stock price information, whatever) could use these URIs whenever they referred to a company. If all websites that make data available about companies point to the same identifier for a company, then it’s possible to pull that data together very easily.

Now the URIs that you use should be short, clean, readable, hackable, hierarchical and so on. If you can, you should use a natural identifier for the resource within the URI for that resource. So URIs for registered companies should use their registered number. URIs for schools should use the school’s unique reference number (URN). URIs for playgrounds could use the name of the playground (scoped within the council responsible for the playground). URIs for court cases should include the court, the year, and the case number. And so on.

Remember as you’re creating these identifier URIs that they are nothing to do with the structure of your website or the user’s experience of navigating through your website. For navigation, you might want to group schools into primary, secondary and sixth-form, but you shouldn’t do that in the identifier URIs. To help decide, imagine someone wanting to construct a URI and the information that they need to do so. If any of the information they need can be derived from other information (as a school’s type can be derived from its URN), leave it out.

When you’re doing this, you might realise that actually you shouldn’t be the one in control of these URIs. If you’re not the one assigning the registered number, URN or case number then there’s probably a higher authority that does assign those (real-world) identifiers. Don’t let that stop you creating URIs — you’ll still find them useful for identifying your information about that particular resource — but do look to see if there are existing URIs that you could point to and reuse whatever scheme they’re using if there are.

Represent

So I said in the last section that assigning URIs to resources was useful. And it is. But it’s even more useful if you provide some kind of response when someone requests those URIs. A request for a URI can be done by a web browser or one of those search-engine-spider-things that crawls the web looking for data. Requests are done on the web using HTTP (hypertext transfer protocol), specifically using a GET request, which means “get this resource”.

When a web server receives a request, it sends back a response. The first part of the response is a status code that tells the browser, spider, or whatever issued the request, generally what kind of response it is. Now when a browser says “get this company” or “get this school” a web server should either respond with a 404 Not Found response or a 303 See Other response.

If the company or school doesn’t exist, a web server should respond with a 404 Not Found response. It’s actually really useful to give appropriate 404 Not Found responses, because it tells whoever made the request that the resource (company/school/playground/court case) doesn’t exist. This can act as simple validation: if I’m building a site that parents can use to rate schools, and a parent enters a URN into a form, I can construct a URI based on that URN, try to GET the information about that school, and if I get a 404 Not Found response then I know that the parent has entered an invalid URN.

If the company or school exists, a web server should respond with a 303 See Other response that points the browser to a document URI that contains information about the company or school. After all, the web server can’t very well deliver the company or school itself into your lap; all it can do is give you information about it. 303 See Other means “if you want information about that, see that other thing over there instead”. The “other thing over there” will be a document of some kind. It might be the PDF that contains information about the school, or the spreadsheet that contains information about the playground.

Simply giving a yes-this-exists or no-this-doesn’t-exist response is useful. Even if that’s all you do, you have done good.

It’s even more useful, though, if you can make the information that you have about the school, playground, company, court case or whatever, available in a format that can be processed by a computer reasonably easily. PDFs are really really hard to extract information from, so do everything you can not to use PDFs. Word documents and Excel spreadsheets are next worse; if you have to use them, keep them really really simple and definitely don’t use Word Art or embed images to display your data.

You should always make your data available in HTML. Try to make it as clean and regular as you can; use microformats to indicate information about people, places and events. If you want to push the boat out, use RDFa to mark up the data in your page even more explicitly.

The great thing about HTML is that it’s human readable as well as (if you do it well) machine readable. You can also make your data available in explicitly machine-readable forms as well if you want: XML, JSON, RDF/XML, whatever floats your boat. If there are already standard formats or ontologies for the kind of data that you’re making available, then use them, certainly, but it’s very likely that there aren’t. And in comparison to the nightmare of extracting anything useful from a PDF, it’s easy to transform between different formats, so you only have to concern yourself with different formats if you want to.

If you do provide multiple formats for your data, you should use server-driven content negotiation to deliver the data in an appropriate format to whatever’s requesting it. So a web browser will request HTML; a semantic web crawler will request RDF/XML; a Javascript program will request JSON and so on. The 200 OK response that the web server sends with your data should include a Content-Location header that gives the representation URI of whichever format is being returned, and a Vary header that tells caches how it’s decided which representation to serve up.

Expose

All the good work identifying resources and representing them comes to naught if you don’t expose it. You can (and should!) tell other people about the URIs that you’ve developed, but the best way to give them exposure is to use them yourself, within your website. Simply using the URIs within your website gives them exposure. Even if that’s all you do, you have done good. People who are interested in linking to you will look at your site and they will learn about your URI scheme from your use of it.

The identifier URIs that you’ve created might not be particularly easy to generate. For example, with the URI scheme that I suggested above for Companies House, unless you happen to know that Tesco Plc’s registered company number is 00445790, you’re not going to be able to get to information about them. So you should have a way of searching based on something that people will know, such as the name of the company. Use an HTML search form that makes GET requests like

http://www.companieshouse.gov.uk/company?name=Tesco Plc

The response should be a 302 Found that redirects (using the Location header) to the true identifier URI for the company (http://www.companieshouse.gov.uk/id/company/00445790). If it’s not possible to identify a single resource from the search string (for example, there are lots of companies with ‘Tesco’ in their name), then the correct response is a 300 Multiple Choices that provides a list of links to the possible URIs (in HTML).

There are other ways to help people find your data. If there aren’t gazillions of resources, you can list the URIs within your sitemap, which will make them discoverable by search engines. You can also list them on web pages and, especially for data that’s constantly updating, in (Atom) feeds which you link to from your HTML pages. Use metadata within the pages and feeds to help the consumers of your data work out what’s relevant to them.

To help even more, slice your Atom feeds into portions that different consumers of your data are going to be interested in. Slice by type, by area, by subject. That way people can stay up to date with just the resources that they’re interested in, and not be bothered with information about those that are irrelevant to them.

That’s It

What I’ve tried to describe here is the minimum that you need to do to help people use the information you have, and some of the other things that you can do to make it even more useful. Here are some things that you shouldn’t do:

don’t wait for someone else to define a URI scheme for the things that you want to talk about
don’t wait for someone else to define an XML schema or RDF ontology for your data
don’t wait until you can find the time and money to do it all “properly”

Just do what you can, now.

The Distributed Web

Jeni — Sun, 11 May 2008 21:07:29 +0000

XTech was subtitled “the mobile web”, but one of the major themes for me was that of the distributed web. The first keynote, by Simon Wardley, gave a vision of a future in which hardware, frameworks and applications are services in the cloud rather than products on machines we own: where we use flickr to store our photographs, Google App Engine to host our applications, and Amazon S3 to store our data. In David Recordon’s keynote (written up by Jeremy Keith), he talked about small, specific services provided by sites that aren’t “destination sites”. The same picture was painted by Gareth Rushgrove in his talk on Design Strategies for a Distributed Web.

So I was surprised at how contentious Steven Pemberton’s talk on Why you should have a Website (thankfully again documented by Jeremy Keith) proved to be. Because to me it seemed to be the logical extension to the distribution of hardware, frameworks and application: the distribution of data. In fact, I’ve written about the same idea myself, as has Leigh Dodds, more recently.

From the session, the main question seems to be “how could we do flickr without them holding our data?” I don’t want to particularly pick on flickr, especially because it’s not one of the worst offenders, but the problem of serving and sharing images does illustrate a whole range of issues, so I will use it as an example. I could just as easily be talking about ancestry.com. The way I see it, you need three levels:

providers which make information available in known formats
user interfaces which provide the end-user with a way to access and manipulate the information
brokers which locate information on the web and provide an aggregated interface

(It occurs to me that this is similar to a model/view/controller architecture: the providers give the model, the user interfaces give the views and the brokers control the flow between the two.)

Where flickr is at the moment is a conglomeration of the three: to have your photo appear on flickr, and to gain the advantages that it gives you in terms of tag-based aggregations and social networking, you have to upload it. They are then the provider of the image+metadata (perhaps the only place it is located on the web), the user interface on the image+metadata (the interface through which the image is annotated), and the broker (they provide keyword-based retrieval, for example).

What would it look like to separate those functions?

First, you, as the owner of the image+metadata, could put your data anywhere: on a home wireless network box, on a webserver hosted by an ISP of your choice, on a site specifically designed for hosting photos. Your data is exposed to the larger web through a standard read/write protocol (I’m betting on AtomPub) that allows you to provide metadata both about resources and the links between resources. The point of it being read/write is that it allows other people to add metadata to or links from your resource to others, such as adding a comment on your image.

Second, an information broker would locate your photos by crawling for them (or perhaps by you submitting the URL somewhere, but mostly that shouldn’t be necessary). There are already information brokers around: Google provides a RESTful API for general search results, as does Yahoo!; at XTech, Richard Cyganiak talked about Sindice, and Aidan Hogan about the Semantic Web Search Engine, both of which crawl for RDF triples and provide an API for querying the results. In an AtomPub-based environment, you’d want an information broker that located Atom feeds and resources, indexed them, and provided an AtomPub-based API for publishers to use.

Third, a user interface would provide an attractive and usable front-end that brought together many different sets of information. For example, flickr might combine your friends feed with an image search to provide a view of images recently made available by your friends. There’s no requirement for your friends to use flickr for this to work: flickr queries a broker for a list of your friends, then queries a broker for images by a particular person, the broker searches its index and points the application to the original resources that are provided by your friends.

A user interface has another role, though: to add to the web. Flickr wants to make it easy to add tags to photos, to create sets and collections that help you navigate your photos, for others to add comments and so on and on. And that’s fine, because AtomPub is a read/write API. To add a tag to a photo, flickr simply edits the resource with PUT. To add a comment, it locates the comment feed (which would be referenced from the entry for the particular image) and POSTs to create a new resource. And everyone can see those changes — the added value that you get from a social network.

None of this is to say that a single application can’t act as provider, broker and publisher at the same time, but I’m certain that users will favour those applications that do all of each role: provide to the whole web, broker the whole web, provide a user interface to the whole web. Flickr is almost there, but it doesn’t do the whole brokering job because it only brokers the data it provides, and therefore it doesn’t provide the whole user interface job.

This distributed web is a clear win, particularly for users, over walled gardens. They can switch from user interface to user interface, even use more than one at a time (perhaps one application is good for browsing while another is good for categorising), without any cost. They can choose who to use to serve their information on the basis of things that matter when you’re serving information (low downtime, backups, security, etc.) rather than on how pretty an interface looks or how much functionality it gives you. On the other side of the equation, applications get to do one thing and do it well.

It seems to me that this is simply how the web works, and the questions we should be asking are about privacy and trust and licensing and revenue models and standards development.

Partial implementations #2: XSLT in Google Search Appliance

Jeni — Fri, 23 Nov 2007 22:22:19 +0000

A Google Search Appliance (GSA) is a box that you plug into your network which crawls and indexes your data, and serves up the results of searches. Search results come in an XML format, and there’s a built in XSLT engine which means you can convert that XML into as many different views as you like. So you can have HTML-based search results, summaries, feeds, and so on.

My task recently was to debug some XSLT that transformed the GSA XML into an Atom feed. Easy enough, right? The GSA XML format is pretty hideous — most of the elements max out at three capital letters in length (whatever happened to human-readability) — but logical enough, and the mapping is hardly complex.

But all was not as it seemed. The GSA’s XSLT implementation is… how can I put this politely?… “non-standard”. This post describes some of the problems and workarounds.

To get the GSA to use your own XSLT, you have to go through its web interface. Basically there’s a form with a text field in which you can type your XSLT. Or you can upload a file that you develop offline. Naturally you’re going to do the latter because it means you can use your favourite editor with helpful things like syntax highlighting and validation-as-you-type, but of course that means switching between web browser windows and your IDE as you develop.

So I upload the transformation, point the browser at a relevant search page, and… oh…

When the GSA doesn’t like the XSLT that you use, you get a really helpful error message. It says:

Internal server error.

So you know that there’s been an error. With the server. Internally.

Back to basics, I thought. Let’s find out what processor the server’s using. Then we can develop on that processor and be pretty sure the resulting XSLT will work. So I load up the default XSLT (which is used to create an HTML result) and add the line

<xsl:value-of select="system-property('xsl:vendor')" />

Save the XSLT, reload the page, and…

Internal server error.

Okaaay… so this is an XSLT processor that doesn’t support the xsl:vendor system property. If it doesn’t support that, I’m going to have to tread carefully. So let’s start with something really simple:

<xsl:stylesheet version="1.0"
   xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:template match="/">
  <xsl:copy-of select="." />
</xsl:template>
</xsl:stylesheet>

Save the XSLT, reload the page, and…

Internal server error.

On a whim, I tried

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
   version="1.0">

<xsl:template match="/">
  <xsl:copy-of select="." />
</xsl:template>
</xsl:stylesheet>

instead. Save the XSLT, reload the page, and… Success!

Can you spot the difference? Yes, that’s right: it’s the order of the XSLT namespace declaration and the version attribute. Namespace declaration first, you’re OK, version first, you’re not.

Okaaay… so this is an XSLT processor that doesn’t support the XML Recommendation (which says that attribute order doesn’t matter). But heck, why split hairs? At least it’s working! Now to create some Atom instead:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
   version="1.0"
   xmlns="http://www.w3.org/2005/Atom">

<xsl:template match="/">
  <feed />
</xsl:template>
</xsl:stylesheet>

Save the XSLT, reload the page, and we’re back to

Internal server error

At least there’s some documentation about this one:

XSL stylesheets that include other files may not be used with the Google search engine. An XSL stylesheet that contains the following tags generates an error result:

<xsl:import>

<xsl:include>

xmlns:

document()

Read that again. Yes, the third bullet point. That’s right, it’s saying that an XSLT that contains a namespace declaration will generate an error result because it “includes other files”.

But, but, but, namespace declarations in XSLT stylesheets (or elsewhere for that matter) do not indicate file inclusion. Namespace URIs are identifiers, not locations. They are strings. They are not resolved. You do not need to be connected to the ‘net to use them.

And how am I supposed to serve an Atom feed, since Atom documents use a namespace? Or XHTML for that matter? Fortunately, the GSA only goes so far in banning namespace declarations: you’re OK as long as you don’t put them on the <xsl:stylesheet> element. Moving it to the <feed> element as in

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
   version="1.0">

<xsl:template match="/">
  <feed xmlns="http://www.w3.org/2005/Atom" />
</xsl:template>
</xsl:stylesheet>

and you’re OK. Of course you have to repeat the namespace declaration in every template so you don’t end up creating elements in no namespace. Tedious, oh so tedious, but workable.

(I have a vague suspicion that the idea behind banning namespace declarations is something to do with certain XSLT processors using namespace URIs to pull in Java classes. But addressing that problem by banning namespace declarations entirely isn’t just throwing the baby out with the bathwater, it’s throwing the whole bathroom suite out of the window. And if you then allow namespace declarations further down the stylesheet, you haven’t actually solved the problem.)

Amazingly enough, given the inauspicious beginning, everything else I tried actually worked. I suspect that it’s some standard XSLT processor underneath with a regex based filter that (among other things) limits what’s allowed in the <xsl:stylesheet> start tag. They probably disallow system-property('xsl:vendor') for security — knowledge is power, after all.

Anyway, my suggestions to others who might want to create a customised XSLT processor:

Use a custom URL resolver to restrict access to documents.
Restrict external function calls using something like the ALLOW_EXTERNAL_FUNCTIONS property in JAXP
Document the restrictions you’re placing on the stylesheets.
Produce meaningful error messages that explain the extra restrictions when they’re broken.

Incomplete implementations #1: Atom in IE7

Jeni — Fri, 23 Nov 2007 20:42:28 +0000

IE7 gives you a really quite nice view of an Atom feed. Take a look at the one for this blog, for example. You can filter by category, sort by date or title or author, and search for particular words or phrases. Pretty neat.

But it’s only a partial implementation. I’ve been having to create some Atom feeds recently, and getting them to display nicely in IE7 has proven a bit tricky. I couldn’t find any documentation about this with a quick google, so thought I’d blog it for future reference.

The main things are:

Do not use a prefix for the Atom namespace. IE7’s Atom support is not namespace aware. (I gather this is a problem in a lot of Atom readers. Sigh.)
If you don’t embed the content in the entry then it will display the summary; to get a clickable link to the content, you must provide a <link> element whose href attribute points to that content. The <link> element mustn’t have any other attributes on it, so don’t add rel="alternate" or type="application/xhtml+xml". Having a src on <content> will not give you a link.

If you want to support viewing the feed in IE6 as well, you need to provide a stylesheet to transform it into something nice. As long as the feed is served as XML, IE6 will interpret an <?xml-stylesheet?> PI correctly, and both IE7 and Firefox will ignore it (which is probably what you want, since both Firefox and IE7 have decent native rendering of Atom).

A sketch: personal APP servers and feed-based web apps

Jeni — Fri, 19 Oct 2007 21:27:58 +0000

OK, so I can’t remain a Luddite for long. What’s a technological solution to the posterity problem, in particular in regard to web applications that tuck away all your data in their databases, just waiting to be forgotten?

Well, what if web applications accepted information as feeds rather than through forms? The original data would be distributed rather than centralised. Web applications would use the web as more than a distribution medium: they would be of the web rather than simply on the web.

How it would work:

Keep your data on your computer (in XML where possible). Use whatever tool you like to create and edit it (by hand, using a dedicated standalone application, using a browser-based application in the manner of TiddlyWiki, or however), in some common markup language.
Serve your data through a web server that supports APP, and provide a feed that exposes the data. I’m not saying that this is easy for Joe Bloggs to do now, but if we’re talking about having web servers on mobile phones then surely it’s not long before having a personal web server is a matter of course, and why not with APP support? Feeds could be generated based on directory structures, or simply created as the main file format for a particular application.
Visit the web application that you want to use and point them at your data feed. They access, store and index your data, and relate it to the other data that they have stored from other people’s feeds. It’s their responsibility to keep their database up to date by doing a regular crawl of the feeds they know about; they don’t have to store all your data, just the bits that enable them to do their job.
Edit your data through the web application’s interface; it can update your data for you using APP. (The web application is a APP client, and you host an APP server.) That can include adding comments or whatever other community-level annotations you might expect.

Why bother?

You have all your data locally (although it might be mirrored elsewhere).
You can edit your data in whatever tool you want to use, but still use a funky “rich internet application” to view it and share it.
You can provide the same data feed to any number of web applications; you’re not locked in.
Because it’s stored locally, your data is available for viewing and editing even when you’re offline.
Because the important parts are cached by a web application, your data is available for viewing and editing even when you’re away from your normal computer.
Your friends can access your data directly in a peer-to-peer network.

The obvious problems:

Setup: Web servers aren’t that easy to set up and maintain at the moment, because they’re designed for the use of sysadmins who are quite happy hacking text-based configuration files. That could change (look at WiFi router setup nowadays compared to how it was a few years ago).
Security: you might not want everyone to be able to access the data you keep on your personal web server. So you’ll need a way of assigning user names and passwords to feeds, and only handing those over to web applications that you trust.
Server load: it’s not so much DoS attacks (which tend to have big juicy targets), but the fact you’re more likely to experience large volumes of requests if, say, a picture you host suddenly gets popular, and might not be able to respond to them, or might have to start paying heavily for the bandwidth. So sites that offer hosting will still be really useful, particularly for media that’s (a) large and (b) likely to be embedded in other people’s pages.
Schemas: you can’t split creation of XML from processing of XML unless you have some kind of mutual understanding about the markup language that’s used. This used to seem like a major stumbling block to me, but in fact people seem to be sensible enough to use standards where there are standards, write stylesheets to convert between languages, extend what exists with what they need in more-or-less acceptable ways and generally muddle through without spending years in meetings thrashing out a single consensual model. And Atom and APP are pretty good at supporting extensions and so on that would make that work.

My hope is that we’ll get round to trying some of these ideas out in our genealogy-based Web 2.0 project.

Web 2.0 Project: Using Atom and XML with Graph Data Structures

Jeni — Sun, 02 Sep 2007 19:57:36 +0000

A Ruby on Rails specialist friend and I are building a Web 2.0 application. I would say it’s “social networking for the dead” except that I doubt that description would be attractive to most people (my ex-Goth defacto being a rare exception), and it can be for the living too. It’s a bit like all those genalogy websites, except that our focus is on people’s social relationships as well as their familial ones.

(I should say that this is all very casual. We’re both fitting it in around our other responsibilities, and are mainly interested in working together, learning new things, and trying out all the best practices that everyone keeps talking about. So don’t think I’m becoming a dotcom entrepreneur or anything. Its got a very Web 2.0 name, and I’m only not telling you in case you start hitting our servers. We’re nowhere near ready for visitors.)

We’re using the Gentech data model as the basis for the application (though I expect that we’ll tweak it a bit). You don’t really need to know anything about it to follow what I’m talking about here. The Gentech data model is very much a relational model. They might call it a logical model, but for anyone who isn’t a database head, it’s a physical model. That’s fine; we’re storing our data in a database, so a relational model for that is great.

In the Rails world, the model that Rails is object-oriented rather than relational. So there’s a certain amount of mapping from the relational world into the OO world, in particular eliding the tables that are created simply for normalisation purposes. Making that mapping is one thing that Rails is very good at, of course.

Then we’re into the worlds that I’m particularly interested in. One of our goals is to use Atom as an API, on the basis that it’s a fairly generic way of packaging things (entries) and lists-of-things (feeds) with a bunch of metadata. Plus, the Atom Publication Protocol shows you how to do RESTful applications right.

The trouble, as others have found is that Atom is designed for a flattish structure, in which you have things, and a list of things. Like blog posts and feeds of posts, or pictures and feeds of pictures. But the model that we’re starting from is relational, or object-oriented, or anyway it’s a graph. And that makes things more complicated.

The first steps are pretty obvious. Objects are equivalent to entries, and lists of objects equivalent to feeds. So every object has its own URL, and every significant feed has its own URL too. There’s the obvious http://www.example.com/people/DarwinC01 for a person, and http://www.example.com/people/ for a feed of people, but also http://www.example.com/people/DarwinC01/events/ for events that are related to a particular person. An entry’s content is an XML document that describes the equivalent object. It has attributes and children to represent the properties from the OO model (columns in the database tables).

Atom defines a bunch of metadata that you can associate with the content in an entry. These are:

id
title
summary (optional, as long as there’s textual or XML content)
updated
published (optional)
category (multiple, optional)
source (optional)
author (multiple, optional as long as there’s a source that specifies one or the entry’s in a feed that specifies one)
contributor (multiple, optional)
link (multiple, optional as long as there’s some content)
rights (optional, defaults to the feed’s rights)
extension elements (optional)

The metadata properties need to be used to indicate who created/updated the object and when. This gets confusing because some of the information in our system is likely to be about content that has authors and publishing dates and so on: the Gentech data model is strong on documenting the sources of information about people you’re reasearching. Even when documenting the source of some information, the Atom metadata should still be metadata about that object in our data model.

The set of Atom metadata does indicate a place where we’re going to want to tweak the Gentech data model though: every object should have metadata associated with it, at the very least an updated date, to populate the Atom metadata fields. Also, we need to identify the property of each object that is used in the <atom:title>, though the title can be something generic if there isn’t an obvious one.

Now the question that’s vexing me: how should we represent relationships to other objects/entries? Let’s take the example of documenting Charles Darwin’s voyage on HMS Beagle. It goes something like this:

<atom:entry>
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      ...
    </passenger>
  </atom:content>
</atom:entry>

The <evr:passenger> element needs to reference a person and a voyage (event), to say that Darwin was a passenger on the voyage.

Here are the options, I think:

Use <atom:link> within the <atom:entry>, with a URL in rel that indicates the kind of relationship

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:link rel="/link-relationships/assertion-persona"
             href="/persona/DarwinC01" />
  <atom:link rel="/link-relationships/assertion-event"
             href="/events/BeagleVoyage" />
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
    </passenger>
  </atom:content>
</atom:entry>

Use extension elements within the <atom:entry>

<atom:entry xml:base="http://www.example.com">
  ...
  <persona href="/persona/DarwinC01" />
  <event href="/events/BeagleVoyage" />
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
    </passenger>
  </atom:content>
</atom:entry>

Use child elements in the object’s XML, referencing the URLs of the related objects

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      <persona href="/persona/DarwinC01" />
      <event href="/events/BeagleVoyage" />
    </passenger>
  </atom:content>
</atom:entry>

Use child elements in the object’s XML, embedding the related objects’ Atom entry or feed

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      <persona>
        <atom:entry>
          ...
          <atom:title>Charles Darwin</atom:title>
          <atom:content>
            <persona>
              <name>Charles Darwin</name>
              ...
            </persona>
          </atom:content>
        </atom:entry>
      </persona>
      <event>
        <atom:entry>
          ...
          <atom:title>Beagle Voyage</atom:title>
          <atom:content>
            <event>
              <name>Beagle Voyage</name>
              <date-range>
                <date>1831-12-27</date>
                <date>1836-10-02</date>
              </date-range>
              ...
            </event>
          </atom:content>
        </atom:entry>
      </event>
    </passenger>
  </atom:content>
</atom:entry>

Use child elements in the object’s XML, embedding the related objects’ XML content

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      <persona>
        <name>Charles Darwin</name>
        ...
      </persona>
      <event>
        <name>Beagle Voyage</name>
        <date-range>
          <date>1831-12-27</date>
          <date>1836-10-02</date>
        </date-range>
        ...
      </event>
    </passenger>
  </atom:content>
</atom:entry>

I don’t think that there’s any point in using an extension element (#2), given that using <atom:link> (#1) situates the information in the same place in a more standard way.

Embedding information (as in #4 and #5) is a good thing because it means fewer requests to the server in order to get some useful information. Providing access to Atom feeds (as in #1, #3 and #4) is a good thing because it means you can get metadata about who created the refenced objects, and additional information about them. So #4 is good, since it does both these things, but I don’t like embedding Atom in the XML because it’s a lot of extra weight in the XML (making it harder to read/process).

In fact, #1, #3 and #5 aren’t mutually exclusive. It’s possible to add relevant <atom:link>s to the metadata, reference the URLs of the other objects and embed their content at the same time:

    <atom:entry xml:base="http://www.example.com">
      ...
      <atom:link rel="/link-relationships/assertion-persona"
                 href="/persona/DarwinC01" />
      <atom:link rel="/link-relationships/assertion-event"
                 href="/events/BeagleVoyage" />
      <atom:content type="application/xml">
        <passenger>
          <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
          <persona src="/persona/DarwinC01">
            <name>Charles Darwin</name>
            ...
          </persona>
          <event src="/events/BeagleVoyage">
            <name>Beagle Voyage</name>
            <date-range>
              <date>1831-12-27</date>
              <date>1836-10-02</date>
            </date-range>
            ...
          </event>
        </passenger>
      </atom:content>
    </atom:entry>

We embed the core information for easy access (#5), reference its original URI for more details (#3), and then we may as well add the <atom:link>s (#1) so that run-of-the-mill Atom readers who have no knowledge about our content can do something useful. We don’t get the metadata embedded in the XML, but it’s retrievable: a client could use the entry as a kind of “low resolution” information set, which they can add to by retrieving the “high resolution” Atom for the referenced objects, via their URLs, as necessary.

The problem with using an embedding method rather than a referencing method is that the object model is a graph, not a hierarchy. So you can’t always embed an object’s XML: sometimes you have to only use a reference (#3 without #5) to avoid getting into an endless loop of repeated information. As a publisher, sometimes you might want to only use a reference, because the information is only tangential to the main subject of the original entry. I’m imagining that we might serve several different Atom entries for the same object, with different amounts of detail. Maybe.

As an author, creating this XML, you can’t include a reference if you’re constructing XML (either in code or by hand) for new objects because they won’t have URLs yet. Therefore, for the purpose of creating objects as defined by the Atom Publishing Protocol, you’ll use embedded XML (#5) with references to existing objects if necessary. The resource returned will include the references for all the created objects. When updating, you’ll want to include as little as possible aside from the updated information, I imagine (small updates being less prone to clashes than large ones).

By the way, I’m using src attributes when the information is embedded and href attributes when the information is purely referenced (or almost purely referenced; the referencing elements might still have some content equivalent to the <atom:title> element, in the interests of presenting a clickable link).

So that’s the plan at the moment, but we’re open to suggestions. Anybody?

URL design for searches and queries

Jeni — Sun, 12 Aug 2007 22:00:00 +0000

Another fascinating post from Bill de hÓra, this time on URL design for resources:

Let’s take editing some resource, like a document, and let’s look at browsers and HTML forms in particular, which don’t a do a good job of allowing you to cleanly affect resource state. What you would like to do in this suboptimal environment is provide an “edit-uri” of some kind. There are basically 5 options for this; here they are going from most to least desirable

Uniform method. Alter the state by sending a PUT to the document’s URL. The edit-uri is the resource URL. URL format: http://example.org/document/xyz

Function passing. Allow the document resource to accept a function as an argument. URL format: http://example.org/document/xyz?f=edit

Surrogate. Create another resource that will accept edits on behalf of the document. URL format: http://example.org/document/xyz/edit

CGI/RPC explicit: send a POST to an “edit-document” script passing the id of the document as a argument. URL format: http://example.org/edit-document?id=xyz

CGI/RPC stateful: send a POST to an “edit-document” script and fetch the id of the document from server state, or a cookie. URL format: http://example.org/edit-document

My current task at work is to look at how to add RDFa to a website that is completely driven by “CGI/RPC explicit” URLs. That includes URLs for the resources themselves, by the way, we’re not even talking about edit URLs here. Take a look at the URL for this page, for example (this isn’t the actual website that I’m working on, but it’s more or less the same in terms of URL design). The URL is

http://www.statutelaw.gov.uk/content.aspx?LegType=All+Legislation&title=wine&Year=2007&searchEnacted=0&extentMatchOnly=0&confersPower=0&blanketAmendment=0&sortAlpha=0&TYPE=QS&PageNumber=1&NavFrom=0&parentActiveTextDocId=3032571&ActiveTextDocId=3032571&filesize=16218

So here I am trying to construct RDF examples, and all the URLs look like this mess. What URI am I supposed to use in RDF to talk about the resource itself, rather than a particular view (table of contents, actual content, etc) of that resource?

In this case, the thing that identifies the resource in the URL is the value of the ActiveTextDocId request parameter: you can do

http://www.statutelaw.gov.uk/content.aspx?ActiveTextDocId=3032571

and see the same legislation; this could be mapped to a resource-oriented URL such as

http://www.statutelaw.gov.uk/legislation/3032571

very easily. It isn’t, but it could.

However, doing that, you do lose some context about what the original search was that led you to this page. In the case of the above URI, the fact that I searched for all 2007 legislation with “wine” in the title gets lost. And this is important because the breadcrumb on the page has to take me back to that original search.

Now, you could argue that this is bad website design: after all, you can navigate back to a search page using *gasp* the Back button, and not doing so just adds unnecessary items to your history. But what about providing previous and next links for navigating through the items found in a search? There, surely, you do need some state information that indicates how we got to this particular item?

Well, no. When you’re navigating through the results of a search, the primary resource that you’re viewing is the collection of items that have been identified by the search. Even if you’re just viewing one of the items in that collection, if the collection still matters then that item should be viewed as just a subresource of the collection.

In this case, the search has three fields — title, year and (legislation) number — so the search URL has three parts after the initial one. The general scheme (using URI template syntax) is

http://www.statutelaw.gov.uk/search/{title}/{year}/{number}

So here, I could use

http://www.statutelaw.gov.uk/search/wine/2007/_

as a URL that would give me a list of all the 2007 legislation with any number whose title contained “wine”, and then

http://www.statutelaw.gov.uk/search/wine/2007/_/1

would show me the first piece of legislation found in the context of that search, with a next button taking me to

http://www.statutelaw.gov.uk/search/wine/2007/_/2

An individual resource itself could occur in many searches, and thus you would get many URLs for the resource in the context of those particular searches, but that’s OK so long as there is one URI that identifies the resource itself. You should link to the actual resource from the page, of course, both in a <link> in the header and a <a> in the body (it feels like there should be a “this is the real resource” value for the rel attribute of <link> or <a>, but I don’t know of one).

The kind of URL above works fine when you have a fixed number of fields for searches, but what if you’re doing more complicated searches: something that requires a proper query language? Well, you can stuff a query language into a URL. See Dare Obasanjo’s comparison of Google and Astoria APIs for queries to see what that looks like.

Alternatively, several years ago Paul Prescod introduced me to the notion that the query itself is a resource — it’s something that you’ll probably want to save and edit — and can be assigned a unique identifier in the same way as other resources. So you visit

http://www.example.com/queries/new

to create a new query, which gets assigned the URL

http://www.example.com/queries/4328

and you can then visit that page to see a list of the results of the query. Unlike with a simple search, the query parameters themselves don’t get used in the URL: they’re stored on the server. So you can’t hack the URL to change the query, but you do have a simple URL that you can easily share with other people if you want to.

XTech 2007: Thursday 17th May Afternoon

Jeni — Sun, 27 May 2007 22:03:24 +0000

UPDATE: Dare Obasanjo has written an interesting critique on using the Atom Publishing Protocol as the basis for general purpose sharing of data in the way that the Google Data API does.

Thursday afternoon had a few really interesting talks. I learned about the Google Data API (no longer called gData); Oracle’s use of XLink to represent relationships between documents, and the requirements that entails; using XSLT to create JSON to use Exhibit widgets; and using XMPP to enhance instant messaging.

Google Data API

Frank Mantek

The Google Data API is the unified API that Google offers to all its services, such as Google Base, Blogger, Google Calendar, Google Spreadsheets and so on.

Frank talked about how awful SOAP/WSDL is, in particular how two services developed in different platforms can’t talk to each other (which one might imagine is rather the point of Web Services). (Later, when challenged by a Microsoft guy about this claim, he revealed that he’d been a major developer of the SOAP/WSDL stuff at Microsoft, so knew exactly what he was talking about from bitter experience.)

So the Google Data API is a RESTful API, using the Atom Publishing Protocol with a few additions:

extra data model
querying
concurrency control
extra authentication

What this basically means is that you can query any of the Google services using HTTP, and get back an Atom document. The URI can contain queries (the precise nature of which depend on the service; Google Base, for example, uses a single URI request parameter that has a complex internal query syntax), and you get back the feed with the items that you’d requested. The Atom items themselves have the basic Atom elements, but then a bunch of service-specific elements that provide the extra information you need.

Listening to this talk I finally got what Tim Bray was talking about at the XML Summer School a couple of years ago: REST gives us verbs and Atom gives us objects and lists of objects. I didn’t get it before, because, after all, aren’t all XML documents objects? But I think the point is that Atom has a lot of the mechanics that you need for talking about objects built into it, and the extensibility necessary for adding your own information to it (which is what each of Google’s services are doing).

The really interesting part of the talk was where Frank started talking about what the problems (still) are. The problems I noted were:

Atom’s verbose
Google have to use <category> to indicate the kind of thing they’re representing (as opposed to using the document element which is what you’d do with normal XML documents)
the rel attribute is too vague
they made up their own markup languages, rather than reusing existing standards
they should be using ETags for concurrency control
they haven’t got any versioning (eek)
incremental updates are a problem; they don’t want to serve the whole Atom feed (to a mobile device) when only a small amount has changed, so what they do is have several feeds, each of which reveals a different part of the information

From Trees to Graphs: Evolving XML for building enterprise applications

Ravi Murthy

Ravi Murthy talked about the provision for defining links between documents in Oracle’s database, and their consequent requirements. Information Oracle’s XML database has a file system abstraction (every XML ‘object’ has a file path) with access control, versioning, metadata and protocol access. Within an XML ‘object’ stored in the database, they use XLink to represent the relationships with other objects. When you export the XML, the XLinks get resolved to create the XML document.

Using XLink to represent relationships between documents brings a whole new set of constraints that you might want to express in a schema language, or annotations that you can use to describe the links (depending on how you look at it):

The type of the linked resource (eg the document element’s name, substitution group or XSD type)
The scope of a particular reference, similar to the scoping of XSD’s identity constraints
That a particular link is acyclic (eg, given an XPath expression, keep evaluating it and make sure you never get back to where you started)
The kind of a link, one of:
- hard: the target of the link must exist, and cannot be deleted while this resource exists (but can be renamed) — these are similar to links in normal databases
- symbolic: trust the file path specified by the link and only resolve it on demand
- weak: like a hard link, except the target can be deleted, in which case the link becomes symbolic
The versioning of a link, whether it points to the “current” version of a resource or a specific version

These extra constraints are expressed as annotations on the definitions of xlink:href attributes in XSD schemas for the documents held in the database.

Ravi also talked a bit about expressing decomposition rules: how an XML document should be shredded when it gets put into the database. They use XPath to specify rules that indicate that particular elements should be placed at a particular filepath.

I was really flattered in the tea break. Chatting with a guy called Phil working at the University of Bath, who politely asked about my presentation, and after I’d explained how it was all to do with overlapping markup and that kind of hard-core theory he said: “You don’t look like a markup geek”. Me: “What, because I’m a girl?”. Him: “No, no, that’s not what I meant. You just look more Web 2.0-ey.” Max was there at the time, and labelled me “the Geekess of XSLT”, which I think clarified things. (Actually most of the people at XTech this year were Web 2.0-ey rather than markup geeks, but I’m glad I looked as though I fitted in.)

XML-powered Exhibit: A Case Study of JSON & XML Coexistence

Chimezie Ogbuji

“What’s Exhibit?” I hear you ask. Or maybe you’re more with-it than I am, but that’s what I was asking. Chimezie never really explained, but I kinda gathered that it’s a funky AJAX toolset for creating views of data by importing scripts and using magical IDs and extension attributes within web pages. The other phrase that Chimezie dropped in was Rich Web Application Backplane, which again I hadn’t heard of. Even having read the W3C Note, I still don’t get it. Ho hum.

Anyway, Chimezie made the point that while entering data using XForms is great, it’s too heavy-weight for viewing that data. Exhibit gives a lot more flexibility (take a look at the US presidents example), which enables users to explore data more freely. In Exhibit pages, you provide a JSON schema for your data, a number of lenses/views/widgets that you can use to view the data, then you embed the widgets in the HTML page and point it at the data source. The JSON schema indicates the type of a particular property (eg “country”), and gives labels for it (including a plural label (“countries”) and a reverse label (“country of”)) that it uses in the widgets.

But that requires JSON, right? Chimezie showed how easy it is (and it’s really easy) to transform data-oriented XML into JSON using XSLT.

You know, there are all these cool ways out there for viewing information, I just wish I had some really meaty data to use them on! Timelines are one thing, but I’d also love to find some data to employ in Gapminder or even in an interface like the one for the music of Philip Glass. Perhaps I should just mine Google Base, but I’d like it to be something personally or collectively useful.

Real-time user-to-user web with Mozilla and XMPP

Massimiliano Mirra

This talk was strong on motivation — the requirement to enhance basic instant messaging functionality — and strong on demonstration, with Massimiliano chatting and playing with a pre-programmed bot, but really weak on the technical details. It was only through the post-talk questions that we learned that what we’d seen was based on XMPP (the Extensible Messaging and Presence Protocol), which allowed DOM events to be passed between clients. Have to read the paper if you want to learn more.

XTech 2007: Thursday 17th May Morning

Jeni — Fri, 25 May 2007 21:34:18 +0000

On Thursday morning, I was down to chair the first session in the “Core Technologies” track. Two interesting papers: one on XForms and one on Google Base. Then I snuck on to the “Applications” track to hear about scientific Wikis and the trials of managing schema repositories.

XForms, REST, XQuery… and skimming

Mark Birbeck

Mark Birbeck, one of the developers of formsPlayer (and an invited expert on the XForms and XHTML WGs), discussed the rationale behind using XForms. The only thing that really stood out for me was the fact that he used an XML document to provide the labels for the form controls (in just the same way as you can use XML documents to provide the data in the form controls). That was quite neat, and made me think of the different requirements of data entry and data presentation: a topic that returned in Chimezie Ogbuji’s talk later that afternoon.

Another theme here, for me, was the use of declarative programming: you write a form, which is just some XML and leave all the technical stuff about submitting a PUT HTTP request to the XForms player. Mark talked about using WebDAV and eXist on the server to store the XML documents, and demonstrated using <oXygen/> to load and save documents. Hmm… I wonder if I should experiment with XForms and that Unicode database browser I was thinking about…

Google Base, a mashups database for the REST of us

Jeffrey Scudder

A very popular, thought-provoking, and slightly disturbing, talk on Google Base. So Google are asking us to upload data on anything (jobs, personals, cars, etc.) into their huge databases. And then they’ll serve us back that information (and other people’s information) in formats such as Atom, RSS and JSON, as well as standard web pages.

The thought-provoking bit, for me, was the fact that they don’t have any particular schema for each of these kinds of items. Now, I come from a knowledge engineering background where we’re very into ontologies and creating conceptual models and all that stuff. But Google don’t bother: you create categories and structure your data the way you want to, and they’ll serve it back in that way. But they look at all the data they have their hands on in order to decide how to display and serve information. So, for example, if I define cars with the property ‘shade’ but a hundred other people define them with the property ‘colour’ then on a feed that includes all our items, we’ll see the ‘colour’ property.

This is a kind of bottom-up ontology design: the properties of an item are the properties that other people think are important about an item. One thing that surprised me was that it looks like it’s not very intelligent yet: simple differences in case (like ‘color’ vs. ‘Color’) don’t seem to be detected, so I guess nothing else is. Time to dig out my old research on automated comparison of ontologies…

The slightly disturbing part? Well, Google are trying to get us to upload our data to their servers. And they’re not putting any limit on how much we upload. One member of the audience asked “What’s in it for you?”; Jeffrey seemed to have a hard time understanding the question and said something like “Better indexed information means we can give you better information”, but that doesn’t really answer the question. Presumably it’s all about being able to advertise to us better: the more data we upload, the more They know about us, the better targeted Their adverts can be.

What I found strange was the idea of uploading data to a central server. Surely the whole point of the web is that I put my data on my machine. I don’t have a problem putting the data together in a nice Atom feed so that Google can index it easily and pointing them at it, but I want to own it, y’know?

By the way, one thing that was apparent to me during this talk was how important it is that web pages look good with large font sizes, not just for people with poor eyesight, but also for when you’re demoing your cool web applications! The Google Base drop-down menus were impossible to see with increased font size because their height is fixed in pixels.

An Augmented Wiki for Interactive Scientific Visualization and Evolutionary Collaboration

Frank Marchese

On to the less well-attended “Applications” track. This talk was about supporting scientists (specifically biochemists) in providing side-by-side visualisation (of complex molecules) and textual analysis. Frank talked about a Wiki in which Jmol Java applets for visualising molecules are arranged side-by-side with standard journal articles. The articles themselves have links in them that animate the Jmol visualisation: highlighting particular groups of atoms, moving it to show a particular view, and so on.

It was kind of neat, as pretty pictures of molecules often are, but I didn’t think the Wikiness of the whole enterprise was really explored: I got the impression that the textual articles were basically static: you could add comments, but not collaboratively create an article about the molecule. Also, the link between the text and the animation of the molecule was through Javascript, as far as I could tell: I’d expect a declarative method of defining animations would make it a lot more accessible.

Real-world metadata registries; sharing concepts, schemas and semantics

Emma Tonkin

This talk took me back to the trials of creation of top-down conceptual models, focusing on the definition of metadata schemas. Unfortunately, there was a lot of philosophy and not many practical guidelines in the talk, and I didn’t get a lot out of it. One thing that Emma touched on, though, was the way that the meaning of a term can change over time, through:

extension or generalisation
narrowing or specialisation
amelioration (when a term gains approval)
deterioration or perjoration (when a term gains disapproval)

The latter two are particularly demonstrated by political correctness, whereby terms like “Eskimo” fall out of favour and “Inuit” becomes more acceptable (all highly culture-specific; see the Wikipedia Eskimo page for more discussion on what term to use).

The advantage of a principled conceptual model is that the concept itself and the term(s) you use for that concept are loosely coupled, so if a given term falls out of favour or becomes inappropriate, you can always decouple it. On the other hand, bottom-up tagging tends (I think) to have a 1:1 relationship between term and concept, so if the use of terminology changes you might be left with inaccurate tagging of legacy data. Maybe.