linked data

What Do URIs Mean Anyway?

Jeni — Tue, 05 Jul 2011 22:06:33 +0000

If you’ve hung around in linked data circles for any amount of time, you’ll probably have come across the httpRange-14 issue. This was an issue placed before the W3C TAG years and years ago which has become a permathread on semantic web and linked data mailing lists. The basic question (or my interpretation of it) is:

Given that URIs can sometimes be used to name things that aren’t on the web (eg the novel Moby Dick) and sometimes things that are (eg the Wikipedia page about Moby Dick), how can you tell, for a given URI, how it’s being used so that you can work out what a statement (say, about its author) means?

One answer is to use a hash URI whenever you want to refer to something that doesn’t live on the web, with the base URI providing information about that thing. For example:

http://en.wikipedia.org/wiki/Moby-Dick is the URI for the Wikipedia page
http://en.wikipedia.org/wiki/Moby-Dick#thing is a URI for the novel itself

The problem some people (including me) have with this is that hash URIs are primarily used to indicate portions of a web page, and using them for things that aren’t page fragments overloads them. It’s also an inflexible method, because the server isn’t told what the fragment identifier is, and therefore it can’t be used as the basis for a redirection, for example.

The 2005 TAG resolution for people who wanted to use separate non-hash URIs, such as [warning, made-up URIs]

http://en.wikipedia.org/wiki/Moby-Dick is the URI for the Wikipedia page
http://wikipedia.org/thing/Moby-Dick is the URI for the novel itself

was:

if you get a 2XX response when you request a URI, that URI refers to a document (the document that you get back)
if you get a 303 response when you request a URI, that URI could refer to anything, and the resource you get by following the redirection describes that thing (hence if a URI should refer to something that isn’t on the web then requests to it should respond with a 303)
if you get a 4XX response when you request a URI, that URI could represent anything

This leads to the 303 pattern described for example within Cool URIs for the Semantic Web; in the example here, the response to http://wikipedia.org/thing/Moby-Dick would be a 303 redirection to http://en.wikipedia.org/wiki/Moby-Dick.

Six years later, we have a lot of experience about this technique of distinguishing between things that are or are not on the web, and it has a bunch of practical limitations.

it requires access to web server configuration (to add 303 redirections) that make life difficult for people without that level of access
URIs for things that aren’t on the web always require two round-trips to get hold of information, as the first always responds with a 303 redirection, which adds server load and slows things down (this is made worse as 303 responses can’t be cached — an oversight in the HTTP spec that I gather is fixed in HTTPbis)
using the 303 pattern requires a level of knowledge and understanding that is beyond most web developers, particularly if they get no benefit from taking care over their use of URIs (for example, Facebook, schema.org and so on all encourage the use of URIs for non-web things without a word about 303 redirections)
even people who do have this knowledge and understanding sometimes find it hard to work out whether a particular thing that they want to talk about is a thing-on-the-web or not and therefore whether the use of a 303 redirection is required
even people who do try to take care in their use of URIs easily make mistakes because we interact with URIs by copy-and-pasting them from browser address bars, and the only URIs that appear there are URIs for things on the web

Basically, while the web architectural principles behind the use of 303 redirections are (arguably!) sound, the collective experience of the past six years indicates that many publishers will not use it because they don’t know to, because they don’t care to, because they make mistakes or because they simply can’t while meeting the other practical constraints of their project.

A number of other approaches have been suggested, before and after the TAG decision, many of which are documented within the draft TAG finding Providing and discovering definitions of URIs.

The first observation that I want to make is that many of the objections to the 303 pattern are about the practicalities of publishers using it. Therefore, any suggestions to provide an alternative technique that involves

introducing new URI schemes (eg tdb)
introducing new HTTP methods (eg MGET)
introducing new HTTP status codes (eg 209)
using particular HTTP headers (eg Link or Content-Location or other specialist headers)

are not going to be widely used for exactly the same reason. I’m not at all persuaded that it’s worth spending time developing them.

My second observation is that there are three questions that are being conflated and we might make more progress if we separated them:

Must publishers provide separate URIs for things-on-the-web and the non-web-things that they describe?
How can you tell what a reference to a particular URI within a piece of data (eg an RDF statement) means?
How can you get from a URI to information about whatever that URI refers to?

Ambiguity in URIs

Both the hash URI pattern and the 303 pattern make the assumption that you need to have separate URIs for things that are not on the web (eg books) and documents on the web about them (eg pages about books). This is useful because it enables people to make separate statements about the author of a book:

<http://wikipedia.org/thing/Moby-Dick> 
  dct:creator <http://wikipedia.org/thing/Herman_Melville> ;
  .

from the authors of the Wikipedia page about that book:

<http://en.wikipedia.org/wiki/Moby-Dick>
  dct:creator 
    <http://wikipedia.org/user/Aristophanes68> ,
    <http://wikipedia.org/user/SporkBot> ,
    <http://wikipedia.org/user/Curb_Chain> ,
    ...
  .

If we only have the URI http://en.wikipedia.org/wiki/Moby-Dick then we run into difficulties interpreting statements made about that URI, and indeed different people might use the URI in different ways, or make some statements that use the URI to mean the novel and some to mean the Wikipedia page.

So there are good reasons to have two separate URIs in these cases.

But the fact is that many publishers currently have a one-URI-fits-all policy. And even if they don’t, people reusing those URIs will often make mistakes and use the wrong one. It would be nice if we could make the world see that this leads to all sorts of logical problems for the Semantic Web, but I just can’t see that happening.

This situation reminds me of one of the central innovations that the web had over previous hypertext systems. There is a great slide) by Dan Connolly which roughly looks like:

Web Semantic Web

Traditional Design hypertext logic/database

+ URIs

- link integrity ?

= viral growth

Are there parts of traditional logic and databases that, if we set them aside, will result in viral growth of the Semantic Web?

	Web	Semantic Web
Traditional Design	hypertext	logic/database
+	URIs
-	link integrity	?
=	viral growth

(By the way, in case my replication of this slide is interpreted incorrectly: I’m certainly not implying that viral growth of the Semantic Web as an end in itself, though I would like to see viral growth in data sharing.)

Dropping the requirement for link integrity, coping with the fact that sometimes links would break, was what made the web work. It would have been simply impossible to build the web as a decentralised system if there had been a requirement for links to always work.

Of course that doesn’t mean that we like it when links get broken. There’s oodles of best practice advice out there on making sure that you retain support for old URIs if you change your web space; we have backup systems in place in the form of web archives so we can work out what was once at the end of a particular URI; and the resolvability of links is something a linter will check about your website.

So it’s not that when he developed the web TimBL rejected entirely the very concept of link integrity, it’s that he recognised that we have to work with the imperfection of the real world. Links break. HTTP copes. Browsers cope. People cope.

The imperfection of the real world as it applies to linked data is that URIs will be used in ambiguous ways. We might not like it; we might write best practice documents that encourage people to have separate URIs for web-thing and non-web-thing, develop tools that help people detect when they’ve used the wrong URI, and so on. But it will still happen, and in my opinion we need to work out how to cope.

In fact, ambiguity in URIs goes much further than just a confusion between the Wikipedia page about Moby Dick and the novel Moby Dick itself. URIs are names, and names are used by different people to mean different things. The same URI might end up meaning:

the Wikipedia page about Moby Dick
the novel Moby Dick
the whale Moby Dick
the story Moby Dick (originally a novel but later adapted as a film)
and so on

Even if the publisher provides a clear and unambiguous definition about what the URI http://en.wikipedia.org/wiki/Moby-Dick means, other people will use it to mean something different because it’s close enough for what they want to say.

So I think the answer to the first question I posed — “Must publishers provide separate URIs for things-on-the-web and the non-web-things that they describe?” — has to be “No, though it is good practice to.” We can fight against ambiguity, but we have to accept that we cannot win.

Disambiguating Statements

As discussed above, in a perfect world, we would have separate URIs for things-on-the-web and non-web-things and any data that we published about Moby Dick would use the URI for the Wikipedia page to talk about things like the licence for that information, or how the information was created (its provenance), and the URI for the novel to talk about things like the licence for the novel and what characters appeared in it.

But the world is not perfect, and we are going to end up with situations where the same URI is used to refer to a whole range of different things. How do we cope?

Well, first let me say that I don’t see people merging data together willy-nilly and hoping to get something useful out of it. URIs give us connection points and RDF gives us a flexible data model, which means that merging data can be easier than the kinds of custom merging that you have to do with CSV and JSON, but I don’t think it can ever remove entirely the requirement for curation. We want to ensure that the need for intervention in merging two datasets is kept to a minimum, but we can’t expect it to be entirely removed.

So with that in mind, there are at least three techniques that can be used to get useful data out of a world in which the same URI is used to mean different things.

One-Step-Removed Properties

The first technique is to interpret particular properties as describing a one-or-more-step-removed relationship between a resource and a value. For example, the bib:author and dct:creator properties would be defined such that the RDF statements

<http://en.wikipedia.org/wiki/Moby-Dick>
  bib:author <http://en.wikipedia.org/wiki/Herman_Melville> ;
  dct:creator <http://en.wikipedia.org/wiki/User:Aristophanes68> ;
  .

would be interpreted as saying

The topic of the page http://en.wikipedia.org/wiki/Moby-Dick was authored by the topic of the page http://en.wikipedia.org/wiki/Herman_Melville. The creator of the page http://en.wikipedia.org/wiki/Moby-Dick is the topic of the page http://en.wikipedia.org/wiki/User:Aristophanes68.

The biggest problem with the global application of this approach is that there are a lot of existing properties defined in vocabularies such as FOAF or Dublin Core that aren’t defined as one-step-removed properties. One publisher might use dct:creator to link to “a page describing the creator of this page” and another might use it to point directly to a (non-web-thing) URI for the creator of the page. So practically, this approach requires the interpretation of properties to be done on a dataset-by-dataset basis. Which leads onto the next approach.

Named Graphs

A second technique would be to make the assumption that within a single dataset, a single URI has a single meaning, but that the meaning may differ between datasets. I suspect that this is true even when publishers attempt to take care about which URI they use, because, like names, the meaning of a URI is slightly different depending on its use.

Re-users of data need to work out whether the way URIs are used in one dataset is close enough to the way they are used in another dataset, to ascertain whether it’s appropriate to simply merge the datasets or whether something slightly more complicated needs to be done to bring the datasets together.

The problem with this approach is that it raises the barrier to joining together graphs: you can’t just bung the data into a triplestore and perform queries on it, you have to work out some kind of mapping between the datasets up front.

Duck Typing

The final technique that I’ll talk about here is to say that different applications need to access different properties, and can ignore any properties that don’t fit with how they want to use the data. It is relatively rarely useful to have generic RDF viewers; people (generally) build applications to answer questions and perform tasks, not to just browse around data.

For example, if a single dataset were to contain:

<http://en.wikipedia.org/wiki/Moby-Dick>
  a bib:Book ;
  bib:author <http://en.wikipedia.org/wiki/Herman_Melville> ;
  a foaf:Document ;
  dct:creator <http://en.wikipedia.org/wiki/User:Aristophanes68> ;
  .

then an application that was interested in gathering data about books would only care about the fact that http://en.wikipedia.org/wiki/Moby-Dick was a book with an author of http://en.wikipedia.org/wiki/Herman_Melville and wouldn’t care about the FOAF or Dublin Core classes or properties associated with the URI. An application that was interested in gathering information about the authorship of documents on the web, on the other hand, might look for the foaf:Document class and Dublin Core properties and ignore everything else.

To me, this approach seems the most promising way of retaining the core benefits of RDF. It seems more robust in the face of user error than the idea of defining one-step-removed properties, and retains the ease of mashing together data from different sources in a way that you wouldn’t get if you had to think about the URI usage within each of the datasets that you want to bring together.

Locating Data From URIs

And so we get to the final question: how should people be able to get from a URI to information about whatever the URI refers to?

I’ve discussed above how I think distinguishing between things-on-the-web and non-web-things has to be seen as a best practice. I think we should continue to recommend the 303 or hash URI methods as the best practice for accessing data from a URI. My reason for this is that introducing yet another method will just makes it harder for publishers to know which method to use when, plus I don’t want to see people who have adopted these techniques in good faith being told that they were doing the wrong thing all along. What I’d like to aim to do is to find a way of fitting these methods into a larger approach.

I also recognise the argument that articulating the relationships between on-the-web and not-on-the-web resources purely through HTTP responses isn’t ideal. It’s useful to have explicit links between resources within the data itself. Within the linked data work that I’ve done for data.gov.uk I’ve tried to adopt a pattern of explicitly using foaf:primaryTopic, foaf:primaryTopicOf and foaf:page to link together the different resources. Other people have suggested the wdrs:describedby property for pointers from a resource to information about that resource; rdfs:isDefinedBy performs a similar function for classes and properties within RDFS.

It would be nice to have one defined property or set of properties to describe these relationships, but we have to recognise that not everyone will use them, so the approach we take has to work when these links aren’t present. The majority of people and sites are going to start off by publishing data about something at a single URI, and simply return data about that thing (a 200 response) when the URI is requested. If they then progress to wanting to have separate URIs for that thing and the page about the thing, or indeed to disambiguate the URI that they’ve used in some other way, we need to make it easy for them to do so.

I think we need two properties: eg:describedBy and eg:couldBe. eg:describedBy describes the link between a resource (of any type) and a document that describes it; eg:couldBe is a disambiguation link that points from a URI to other possible, more precise, URIs.

Then I think we need some rules along the lines of (I don’t pretend these are entirely worked out):

if you get a 303 response redirecting to U' when you fetch a URI U then behave as if the response from U' included the triple U eg:describedBy U'
if the URI U is a hash URI whose base URI is U' then behave as if the response from U' included the triple U eg:describedBy U'
if you get a 2XX response in response to a URI U then:
- if there are multiple triples that match the pattern U eg:describedBy ?page then assume that the document you have is U' where U' eg:couldBe any of the ?pages
- otherwise, if there is a single triple that matches the pattern U eg:describedBy ?page then assume that the document that you have is ?page and it is about U (along with other things, possibly); statements about ?page might include information about the licence or provenance of the returned document
- if there are any triples that match the pattern ?thing eg:describedBy U then assume that the document you have is U and it is about (possibly multiple) ?things
- otherwise, behave as if there is a triple U eg:describedBy U; in this case, U is being used in an ambiguous way

We could go further and say:

if there are two triples that match the pattern U eg:couldBe ?page . ?thing eg:describedBy ?page then assume that the document you have is ?page and it is about ?thing
if there are two triples that match the pattern U eg:couldBe ?thing . ?thing eg:describedBy ?page then assume that the document you have is ?page and it is about ?thing

This way, if someone starts off using U in an ambiguous way, or to mean only the page or only the thing, they can later add eg:describedBy and eg:couldBe statements to disambiguate and add information about the page or thing the page describes.

It’s worth bearing in mind that we shouldn’t just be concerned about locating information about things that aren’t on the web, but about things that are on the web but that cannot have metadata embedded within them. For example, how do we discover the licence associated with a particular image? Although there are methods of embedding metadata within image and other binary formats, such as XMP, it’s still useful to be able to locate metadata about images based on their URI.

With a scheme such as that described above, publishers that used content negotiation to return some data about the image in another format could use eg:describedBy to indicate that the returned document is about the image (or set of images in different formats).

Summary

The summary of my thinking is:

we should learn to cope with ambiguity in URIs
we should not constrain how applications manage that ambiguity, though duck typing seems the most promising approach to me
we should define some specific properties that can be used to disambiguate URIs, describe their defaults with 303s and hash URIs and provide an easy upgrade path as publishers choose to add more specificity

The key will be how we find practical ways to cope with the real, imperfect, fuzzy web of data while providing an evolutionary path to greater clarity and specificity that publishers can take when they see the benefit of doing so.

TAG F2F, June 2011

Jeni — Fri, 17 Jun 2011 10:44:12 +0000

As you may know, I accepted an appointment to the W3C’s Technical Architecture Group earlier this year. Last week was the first face-to-face meeting that I attended, hosted in the Stata Center at MIT. As you can tell from the agenda (which was in fact revised as we went along) it was a packed three days.

What I intend to do here is to briefly report on the major areas that we discussed and give a tiny bit of my own personal take on them. In no way should any of what I write here be judged as revealing the official opinion of the TAG, it’s just me saying what I think, and I’m not going to go into anything in depth because they’re all incredibly gnarly and contentious topics and I’d not only be here all year but also end up in a tar pit.

Role of the TAG

Usefully for me as a newcomer, our first session was about the ongoing role of the TAG. The TAG occupies a unique position within the W3C. According to its charter it was set up

To improve the effectiveness of Working Groups, to reduce misunderstandings and overlapping work, and to improve the consistency of Web technologies developed inside and outside W3C

The TAG ultimately has three routes to do this:

by providing specific advice on issues that are brought to its attention
by writing documents on basic web architecture principles that go through community review, particularly through the general review of the W3C standards track and become Recommendations
by advising the W3C Director (Tim Berners-Lee) about what he should do on the extremely rare occasions when there are issues that he is supposed to adjudicate on

In none of these cases is there anything that binds the people receiving the advice of the TAG, or reading Findings or Recommendations made by the TAG, to accept them or do anything about them. The power and authority of the TAG depends solely on the quality and utility of its arguments, which is how it should be in my opinion.

Client-Side Application State

The first technical session was about client-side application state and was a review of the Identifying Application State draft that T.V. Raman began before he left the TAG and that Ashok Malhotra has been working on since. This should in the next few months or so be published as a TAG Finding (though it is currently on the Recommendation track).

This work is essentially about documenting the different ways in which you can identify application state within a URI, why that’s a useful thing to do, and some of the pitfalls of using hash URIs to do so. Most of the discussion was about details to do with wording within the document. One thing I thought particularly interesting was the point that URI-based application state is relevant in all ‘active content’, not just in HTML; for example, scripting in SVG or in PDFs bring the same considerations.

Buffer Bloat

Over lunch on Monday we listened to and discussed a presentation by Jim Gettys on buffer bloat. Basically (and all the errors here are introduced by me), TCP/IP is designed to route around network blockages, but it can only do so if it detects them quickly. When you have big buffers in place, as in the case of all modern operating systems and hardware, blockages aren’t detected quickly; they’re only detected when the buffers fill up. Then buffers empty and the data has to be sent again. The net result is that connections get really slow, not just for upload or download but for both, not just for you but for everyone using the network.

Jim talked about how this is exacerbated by the large amount of web traffic and the design of HTTP, particularly the lack of use of HTTP pipelining (whereby several HTTP requests and responses are sent over one long-term connection), because it leads to lots of small messages which can’t be handled effectively. There’s lots more about this on his blog.

Jim also talked about the failure of certificate authorities and how we should be looking at distributed protocols using digitally signed data, pointing us in particular to CCNx.

Fragment ID Semantics

First thing Tuesday was a session that I led on fragids, in particular the problems that are arising out of the mime type registration of +xml types (3023bis) clashing with those that are used for, say, images, and what happens when these come together in something like SVG.

The same issues arise whenever you have documents with types that ‘inherit’ fragid semantics from two directions. For example, XHTML documents are XML documents, so constraints on +xml mean you shouldn’t use interpreted fragids (eg hash-bangs) on them, but they are also ‘active content’ which makes interpreted fragids useful. Similarly, in linked data you shouldn’t really use a hash URI to mean a Person with a primary resource that provides as a response an XML document with embedded RDFa, because according to XML fragid semantics, such a URI should point to an XML element.

Basically the use of fragids has grown markedly outside their original scope and these situations aren’t really covered in the specs. I am now tasked to create a document that describes the issues and suggests ways forward. So that will be fun.

Telcon with IAB

The second session on Tuesday was a telcon with the IAB which has a similar role within the IETF as the TAG does within the W3C. This was a bit of a ‘getting to know you’ session, covering the work of the two groups on:

versioning and extensibility
security
privacy, including Do Not Track

and talking about opportunities to meet and work together on various topics like these.

URI Definition Discovery and Metadata Architecture

The afternoon session on Tuesday was spent on Jonathan Rees’s work on the Architecture of the World Wide Semantic Web, which covers, amongst other things, what people in semantic web circles call httpRange-14. At core, this is about the kinds of URIs we can use to refer to real-world things, what the response to HTTP requests on those URIs should be, and how we find out information about these resources.

Jonathan has put together a document called Providing and discovering definitions of URIs which covers the various ways that have been suggested over time, including the 303 method that was recommended by the TAG in 2005 and methods that have been suggested by various people since that time.

It’s clear that the 303 method has lots of practical shortcomings for people deploying linked data, and isn’t the way in which URIs are commonly used by Facebook and schema.org, who don’t currently care about using separate URIs for documents and the things those documents are about. We discussed these alongside concerns that we continue to support people who want to do things like describe the license or provenance of a document (as well as the facts that it contains) and don’t introduce anything that is incompatible with the ways in which people who have been following recommended practice are publishing their linked data. The general mood was that we need to support some kind of ‘punning’, whereby a single URI could be used to refer to both a document and a real-world thing, with different properties being assigned to different ‘views’ of that resource.

Jonathan is going to continue to work on the draft, incorporating some other possible approaches. It’s a very contentious topic within the linked data community. My opinion is while we need to provide some ‘good practice’ guides for linked data publishers, we can’t just stick to a theoretical ideal that experience has shown not to be practical. What I’d hope is that the TAG can help to pull together the various arguments for and against different options, and document whatever approach the wider community supports.

Can publication of hyperlinks cause copyright infringment?

The first session on Wednesday was another session that I led, discussing the Publishing and Linking on the Web draft that Dan Appelquist and I have been working on.

The aim of this document is to explain the tensions between terms that are commonly used in legal documents such as “possession”, “adaptation” and “distribution” and the way that publication works on the web, in which multiple servers may have copies of the same document (because they cache copies to make the ‘net go faster), automated agents may make changes to those documents (such as compressing or resizing documents, or merging Javascript) and people may refer others to those documents through linking.

We’re particularly keen to argue that linking to something is not the same thing as distributing it. The web’s power arises through its links, so it’s important that people are able to link to something without being worried about what happens when/if the domain they link to is taken over by something illegal.

Dan and I are going to continue to work on this document in response to various suggestions around organisation and terminology, with a view to getting some ‘friendly legal experts’ to look it over and then seeking wider review. The intention is for it to eventually become a Recommendation as this will give greater weight to it as a document for a legal audience.

API Minimisation and Client-Side Storage

There were then a couple of short sessions.

Dan talked about API Minimisation, which is the design principle that to increase privacy we should design APIs that enable people requesting information to say exactly what information they need, and return only that rather that everything known about a think. Dan’s put together an draft and should be calling for review for it soon.

Ashok then led discussion on client-side storage and we brainstormed around some of the architectural/design issues about which we might want to write if we were to put together a document. This work is at a very early stage.

TAG Priorities

After lunch, we had a session on TAG priorities where we discussed which of the various pieces of work that we’re doing should receive the most attention and had a quick review of who is doing what within the TAG.

Our basic problem is that a lot of this stuff feels quite urgent, and we want to be responsive, but with only 5-6 of us “actively involved” (which means 1 day/week) in drafting documents, and other TAG duties taking up our time, it feels like we have taken on too much work. Our focus for the next little while is going to be on responding to issues where our lack of response might either hold people up or cause longer term problems (for example the publication of contradictory mime type definitions), which means things like the document on publishing and linking on the web will need to bubble in the background rather than being the focus of activity.

HTML5 Last Call

Our final session, for which we were joined by Philippe Le Hégaret, was on the HTML5 Last Call documents. The TAG has raised various issues over the course of HTML5 development and want to follow up on how those issues have been addressed in the documents. Our role means that we’re responsible for making sure there’s consistency with other specifications, and that there isn’t anything that seems like it’s going to cause problems in the long term.

The part that we spent most discussion time on was the relationship between Microdata and RDFa. We talked about the precedents for having two specifications that do very similar things but with different approaches, such as CSS and XSL, and how this isn’t necessarily a bad thing so long as they don’t contradict each other and people can move between them easily (because they have the same conceptual foundations).

I’m going to save my opinion on this topic for another post. Suffice it to say that microdata and RDFa as currently specified don’t work well with each other but it’s not at all clear what the best path forward is. The TAG decided to recommend that the W3C set up a Task Force to look at what the best way forward might be.

Final Words

If you want links to the minutes of the TAG F2F, they’re available within the agenda or on separate pages for:

If you have anything to say on any of these topics, please send email to the TAG mailing list. Or you could comment here or email me directly if you like. Which leads me on to talking about what I’d like to do in the TAG.

One of the guidance notes for new members to the TAG says:

TAG members are elected or appointed not to represent their individual member organizations, but the Web community as a whole. We try to take that responsibility very seriously.

I do take that responsibility seriously. Web architecture has to be a combination of practice and theory, balancing approaches that work right now with a desire to not break anything long term. I do practical work developing web applications with HTML, CSS, Javascript, XML, RDF, XSLT, XQuery and so on and so on every day, but I know I don’t see all the difficult corners of the open web standard space: no one person can.

I can listen though, so that’s what I will try to do: listen, digest, reflect and act.

But I have limited resources. Unlike most of the members of the TAG, I am not employed by a large organisation that pays me for time I take on the work that I do for the TAG. The W3C kindly paid for my flights to and from F2Fs, but not hotels or expenses. I wouldn’t have taken this on if I wasn’t prepared to shoulder the financial burden, but if there is anyone out there who might sponsor my participation, I’d love to hear from you.

Getting Started with RDF and SPARQL Using Sesame and Python

Jeni — Tue, 25 Jan 2011 17:27:24 +0000

My previous post talked about how to install 4store as a triplestore, and use the Ruby library RDF.rb in order to process RDF extracted from that store. This was a response to Richard Pope’s Linked Data/RDF/SPARQL Documentation Challenge which asks for documentation of how to install a triplestore, load data into it, retrieve it using SPARQL and access the results as native structures using Ruby, Python or PHP.

I quite enjoyed writing the last one, so I thought I’d try again. As before, I am on Mac OS X, but this time I’m going to use Python, which I have not programmed in before. I like a challenge. You might not like the results!

Sesame

This time, I’m going to use Sesame, as I was told by John Sheridan that it was so easy to install that even he, a civil servant, could do it!

Sesame needs a Java servlet container. I’m using Tomcat because I have some experience with it, but you could use something like Jetty if you prefer. I had a bit of trouble getting Tomcat 6 to install, but that might just have been because it has a lot of dependencies and I wasn’t patient enough. It might be worth upgrading your existing ports and getting verbose output so you know there’s activity as you install Tomcat:

$ sudo port upgrade outdated
$ sudo port -v install tomcat6

This installs Tomcat 6 in /opt/local/share/java/tomcat6.

While that’s happening, get Sesame from its download page. I got hold of openrdf-sesame-2.3.2-sdk.tar.gz. The files we actually need are the .wars so we can just extract them and put them in the webapps directory within Tomcat:

$ tar -zxvf openrdf-sesame-2.3.2-sdk.tar.gz openrdf-sesame-2.3.2/war/*.war
$ sudo cp openrdf-sesame-2.3.2/war/*.war /opt/local/share/java/tomcat6/webapps/

Then startup Tomcat:

$ sudo /opt/local/share/java/tomcat6/bin/tomcatctl start

All being well, you should see Tomcat doing some initial setup:

conf_setup.sh: file conf/catalina.policy is missing; copying conf/catalina.policy.sample to its place.
conf_setup.sh: file conf/catalina.properties is missing; copying conf/catalina.properties.sample to its place.
conf_setup.sh: file conf/server.xml is missing; copying conf/server.xml.sample to its place.
conf_setup.sh: file conf/tomcat-users.xml is missing; copying conf/tomcat-users.xml.sample to its place.
conf_setup.sh: file conf/web.xml is missing; copying conf/web.xml.sample to its place.
conf_setup.sh: file conf/setenv.local is missing; copying conf/setenv.local.sample to its place.
Starting Tomcat.... started. (pid 20064)

Now have a look at http://localhost:8080/openrdf-sesame. If you’re like me, you’ll get some error messages because the user that Tomcat is running under (www) isn’t able to create or write to a logging directory that it wants to create, in my case /Users/Jeni/Library/Application Support/Aduna/OpenRDF Sesame/logs. This turns out to be partly caused by permissions issues and partly caused by the spaces in the filename. To get around it, create a data directory for Aduna that doesn’t have spaces in the filename, and change its ownership to www. In my case, I chose /opt/local/var/aduna.

$ sudo mkdir -p /opt/local/var/aduna
$ sudo chown www:www /opt/local/var/aduna

Then edit Tomcat’s setenv.local file which in my environment is at /opt/local/share/java/tomcat6/conf and add a line that sets the info.aduna.platform.appdata.basedir to that directory, like this:

export JAVA_OPTS='-Dinfo.aduna.platform.appdata.basedir=/opt/local/var/aduna'

Restart Tomcat:

$ sudo /opt/local/share/java/tomcat6/bin/tomcatctl restart

Then navigate again to http://localhost:8080/openrdf-sesame and you should see the Welcome page:

As you can see, this recommends using the Workbench for managing the repositories. If you open that, at http://localhost:8080/openrdf-workbench.

We’ll use this Workbench to create a new repository for our data, which I’ll call reference. Click on New Repository from the left hand navigation and fill in the form. I’m just going to use the default in-memory RDF store because I’m only using a little data; the other options (using MySQL or PostgreSQL stores) would be useful if I were creating something more permanent. See the Sesame User Guide for information about those.

So fill in the form to create a new repository with the id reference and whatever title you like:

Click Next and there will be a couple more options to select; I just used the default for these:

Click Create and you will see a summary of the new repository that you’ve created:

Loading Data

I’m going to use the same data as I did before:

http://source.data.gov.uk/data/reference/organogram-co/2010-10-31/index.rdf

You can add data to a Sesame repository in a browser through the Workbench by uploading a file, pointing Sesame at a URL or pasting in some RDF that you want to load. There are also Java bindings for adding data to Sesame. But neither of those are any good to us as we need programmatic access.

So we will use the HTTP method. I want to add some statements to the reference repository in the graph (what Sesame calls “context”) http://source.data.gov.uk/data/reference/organogram-co/2010-10-30, which amounts to an HTTP PUT on the repository’s statements with that context.

Now I don’t know much at all about Python, but it looks as though the built-in library urllib2 doesn’t support PUT and there’s a better HTTP library available in httplib2. MacPorts supports various different packages for httplib2 with different versions of Python. Now there only seems to be a package for rdflib, which we’ll use later, for Python 2.6, so we’ll go for py26-httplib2, which will bring in Python 2.6 with it just in case.

$ sudo port install py26-httplib2

After running this, if you want to actually use it you will need to install the python_select port and choose Python 2.6:

$ sudo port install python_select
$ sudo python_select python26

Then open up another Terminal window or tab (because the change won’t have affected your old one) and check what version of Python you’re running:

$ python --version
Python 2.6.6

With the httplib2 library in place, it’s time for a Python script (load-rdf-into-sesame.py) to do the PUTting:

import urllib
import httplib2

repository = 'reference'
graph      = 'http://source.data.gov.uk/data/reference/organogram-co/2010-06-30'
filename   = '/Users/Jeni/Downloads/index.rdf'

print "Loading %s into %s in Sesame" % (filename, graph)
params = { 'context': '<' + graph + '>' }
endpoint = "http://localhost:8080/openrdf-sesame/repositories/%s/statements?%s" % (repository, urllib.urlencode(params))
data = open(filename, 'r').read()
(response, content) = httplib2.Http().request(endpoint, 'PUT', body=data, headers={ 'content-type': 'application/rdf+xml' })
print "Response %s" % response.status
print content

Run the script from the command line:

$ python load-rdf-into-sesame.py

and you should get just get:

Loading /Users/Jeni/Downloads/index.rdf into http://source.data.gov.uk/data/reference/organogram-co/2010-06-30 in Sesame
Response 204

which isn’t particularly helpful (well, the 204 response tells us it worked), but you can then check http://localhost:8080/openrdf-workbench/repositories/reference/contexts and you should see that there is a new context of http://source.data.gov.uk/data/reference/organogram-co/2010-06-30:

Click on the context and it will take you to a list of (some of) the triples in that graph:

One of the nice things about Sesame is that the Workbench provides you with ways of exploring the data that you have loaded. On the left navigation bar there are ways of listing the types of the entities described in the data:

from which you can find instances of that type, for example of org:Organization:

and then the statements about a particular instance, for example DirectGov:

Running a Query

Onto running a query directly. This is done on Sesame in exactly the same way as it was done on 4store in my last walkthrough: by HTTP POSTing a query to the SPARQL endpoint. Sesame’s page for testing queries on the reference repository is at http://localhost:8080/openrdf-workbench/repositories/reference/query and we’ll use the basic one that lists types of things that are described within the data:

SELECT DISTINCT ?type 
WHERE { 
  ?thing a ?type .
} 
ORDER BY ?type

Paste that into the textarea that’s provided on http://localhost:8080/openrdf-workbench/repositories/reference/query so it looks like:

and you get an HTML page:

That’s nice for humans, but not so good for computers. When we request the results of this query programmatically, we’ll want to make sure that we specifically ask for the query results in either XML or JSON.

I went the XML route last time, so let’s mix it up a bit and try processing the JSON results of a SPARQL query this time, as it’s really easy to access using the json module in Python. So, we need to POST the query, ensuring that we set the Accept header to application/sparql-results+json, and then process the results as JSON. Here is find-rdf-types.py

import urllib
import httplib2
import json

query = 'SELECT DISTINCT ?type WHERE { ?thing a ?type . } ORDER BY ?type'
repository = 'reference'
endpoint = "http://localhost:8080/openrdf-sesame/repositories/%s" % (repository)

print "POSTing SPARQL query to %s" % (endpoint)
params = { 'query': query }
headers = { 
  'content-type': 'application/x-www-form-urlencoded', 
  'accept': 'application/sparql-results+json' 
}
(response, content) = httplib2.Http().request(endpoint, 'POST', urllib.urlencode(params), headers=headers)

print "Response %s" % response.status
results = json.loads(content)
print "\n".join([result['type']['value'] for result in results['results']['bindings']])

Run it:

$ python find-rdf-types.py

and you get:

POSTing SPARQL query to http://localhost:8080/openrdf-sesame/repositories/reference
Response 200
http://purl.org/linked-data/cube#DataSet
http://purl.org/linked-data/cube#DataStructureDefinition
http://purl.org/linked-data/cube#Observation
http://purl.org/net/opmv/ns#Artifact
http://purl.org/net/opmv/ns#Process
http://purl.org/net/opmv/types/google-refine#OperationDescription
http://purl.org/net/opmv/types/google-refine#Process
http://purl.org/net/opmv/types/google-refine#Project
http://rdfs.org/ns/void#Dataset
http://reference.data.gov.uk/def/central-government/AssistantParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/CivilServicePost
http://reference.data.gov.uk/def/central-government/Department
http://reference.data.gov.uk/def/central-government/DeputyDirector
http://reference.data.gov.uk/def/central-government/DeputyParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/Director
http://reference.data.gov.uk/def/central-government/DirectorGeneral
http://reference.data.gov.uk/def/central-government/ParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/PermanentSecretary
http://reference.data.gov.uk/def/central-government/PublicBody
http://reference.data.gov.uk/def/central-government/SeniorAssistantParliamentaryCounsel
http://reference.data.gov.uk/def/intervals/CalendarDay
http://www.w3.org/2000/01/rdf-schema#Class
http://www.w3.org/ns/org#Organization
http://www.w3.org/ns/org#OrganizationalUnit
http://xmlns.com/foaf/0.1/Person

This is the same set of types as that given through the HTML browse interface. Note that the JSON results themselves look like:

{
  "head": {
    "vars": [ "type" ]
  }, 
  "results": {
    "bindings": [
      {
        "type": { "type": "uri", "value": "http:\/\/purl.org\/linked-data\/cube#DataSet" }
      }, 
      {
        "type": { "type": "uri", "value": "http:\/\/purl.org\/linked-data\/cube#DataStructureDefinition" }
      }, 
      {
        "type": { "type": "uri", "value": "http:\/\/purl.org\/linked-data\/cube#Observation" }
      }, 
      ...
    ]
  }
}

Each of the items within the bindings array contains a set of bindings for the variables in the SPARQL query. This closely matches the XML format.

Processing RDF

Now we get onto the part of this where we look at specific libraries for RDF support in Python. The most popular library is rdflib, which you can install using MacPorts:

$ sudo port install py26-rdflib

The SPARQL query we’ll try this time uses a CONSTRUCT query, which creates RDF, rather than a SELECT query, which as we’ve seen can create either XML or JSON. For example, try the query:

PREFIX foaf: <http://xmlns.com/foaf/0.1/>

CONSTRUCT {
  ?person 
    a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
} WHERE { 
  ?person a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
}

This gets all the information in the data about the individuals for whom names have been supplied in the data, as RDF. Again, Sesame will display this as HTML when you try doing it, but you can choose a different format from the drop-down menu at the top of the Query Result display:

When you’re not accessing using a browser, by default Sesame serves up its results in TriG format, which isn’t particularly appropriate for the results of CONSTRUCT queries as we don’t need multiple graphs. We’ll request N-Triples as that’s something rdflib can understand. Sesame 2 uses the content type text/plain for N-Triples, so we can request this type by setting the Accept header:

params = { 'query': query }
headers = { 
  'content-type': 'application/x-www-form-urlencoded', 
  'accept': 'text/plain' 
}
(response, content) = httplib2.Http().request(endpoint, 'POST', urllib.urlencode(params), headers=headers)

We then need to parse this Turtle response into a rdflib.Graph object:

graph = rdflib.ConjunctiveGraph()
graph.parse(rdflib.StringInputSource(content), format="nt")

We then need to get information out of that graph, which rdflib isn’t particularly good at. So let’s use RDFAlchemy instead. That can be installed using easy_install:

$ sudo easy_install-2.6 rdfalchemy

RDFAlchemy can be used to map RDF graphs onto Python object structures in a fairly straight-forward manner. Basically, you define the namespaces of the vocabularies that you want to use, then some classes for the kinds of things that you have in the data, with properties that map onto properties in the RDF. Then you set the rdfSubject.db to the source of the data (which can be an rdflib Graph as above) and can query on it. Here’s an example:

FOAF = rdflib.Namespace('http://xmlns.com/foaf/0.1/')
RDF = rdflib.Namespace('http://www.w3.org/1999/02/22-rdf-syntax-ns#')

class Person(rdfalchemy.rdfSubject):
  rdf_type = FOAF.Person
  name = rdfalchemy.rdfSingle(FOAF.name)
  mbox = rdfalchemy.rdfSingle(FOAF.mbox)

rdfalchemy.rdfSubject.db = graph
stott = Person.get_by(name='Andrew Stott')
print "Andrew Stott's email address: %s" % stott.mbox.n3()

RDFAlchemy adds both get_by() and filter_by() methods on the descriptor classes that you define, to get a single item that matches a query or a list of items, respectively.

The full script for ‘get-names-and-emails.py’ is:

import urllib
import httplib2
import rdflib
import rdfalchemy

query = """PREFIX foaf: <http://xmlns.com/foaf/0.1/>

CONSTRUCT {
  ?person
    a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
} WHERE {
  ?person a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
}"""
repository = 'reference'
endpoint = "http://localhost:8080/openrdf-sesame/repositories/%s" % repository

print "POSTing SPARQL query to %s" % endpoint
params = { 'query': query }
headers = { 
  'content-type': 'application/x-www-form-urlencoded', 
  'accept': 'text/plain' 
}
(response, content) = httplib2.Http().request(endpoint, 'POST', urllib.urlencode(params), headers=headers)
print "Response %s" % response.status

graph = rdflib.ConjunctiveGraph()
graph.parse(rdflib.StringInputSource(content), format="nt")

print "Loaded %d triples" % len(graph)

FOAF = rdflib.Namespace('http://xmlns.com/foaf/0.1/')
RDF = rdflib.Namespace('http://www.w3.org/1999/02/22-rdf-syntax-ns#')

class Person(rdfalchemy.rdfSubject):
  rdf_type = FOAF.Person
  name = rdfalchemy.rdfSingle(FOAF.name)
  mbox = rdfalchemy.rdfSingle(FOAF.mbox)

rdfalchemy.rdfSubject.db = graph
stott = Person.get_by(name='Andrew Stott')
print "Andrew Stott's email address: %s" % stott.mbox.n3()

Run this script with:

$ python get-names-and-emails.py

and you get the result:

No handlers could be found for logger "rdflib.Literal"
POSTing SPARQL query to http://localhost:8080/openrdf-sesame/repositories/reference
Response 200
Loaded 459 triples
Andrew Stott's email address: <mailto:andrew.stott@cabinet-office.gsi.gov.uk>

The first line is apparently a side-effect of rdflib/RDFAlchemy weirdness which you don’t need to worry about. The rest shows that the search was successful; the call to the .n3() call on the email address is only necessary because it is a resource rather than a literal, and therefore doesn’t get converted to a particularly readable string otherwise.

Conclusions

So there you have it, another walkthrough of setting up a local triplestore, loading in data and accessing that data programmatically using SPARQL queries, this time using Sesame and Python rather than 4store and Ruby.

This walkthrough took me a fair bit longer to do than the previous one, for several reasons:

I’ve done almost no previous programming with Python (as you can probably tell), so every little thing took ages to work out — you know you’re in trouble when you’re Googling for string concatenation code! I’m very happy to accept corrections and improvements, which I’ll include in the above.
I spent a lot of time faffing around with different Python versions, opting for the latest and then finding that the libraries that I wanted to use weren’t available for that version and so on. I eventually ended up with Python 2.6; the code above may or may not work with any other versions.
Setting up Sesame 2 was pretty frustrating: first Tomcat wouldn’t work, then Jetty wouldn’t work, and finally I did get Tomcat working and then had the issue with the log directory, as I described above. Once I’d changed the data directory things worked very smoothly.
I thought rdflib was going to be enough to work with RDF in Python, but really it isn’t (if you want to get data out as well as put data in), so I had to find something else.
The documentation for rdflib and RDFAlchemy isn’t as comprehensive as the documentation for RDF.rb, especially if you’re not familiar with Python, so it took me a bit longer to work out how to do things with those particular libraries.
I took a lot more screenshots!

Again, I haven’t followed Richard’s steps to the letter; in particular I haven’t used a package to get data out of (or into) Sesame: I’ve just done it through HTTP calls. I did it this way deliberately because I think it’s a really important feature of triplestores that you can query them through a common interface: SPARQL. It means that you can take the Python code here and use it against 4store or another triplestore with only a change to the value of the endpoint variable, and similarly take the Ruby code from my previous walkthrough and use it against Sesame. Your code is not tied to a particular implementation or API; you “only” have to learn SPARQL and you’re away.

If you prefer something a little more tightly bound, however, RDFAlchemy does have some targeted Sesame support, as does RDF.rb for that matter. These can help with the management of the data within the repository as well as querying it.

Another thing that’s worth pointing out is that 4store and Sesame have completely different (HTTP-based) interfaces for getting data into stores, and that rdflib/RDFAlchemy and RDF.rb have completely different ways of loading data into in-memory graphs, querying it, and getting information from the results, quite aside from the obvious language-based differences that you’d expect.

On the SPARQL side, there are some efforts within the W3C to define a uniform HTTP protocol for managing RDF graphs and of course there’s SPARQL 1.1 Update. There are glimmers of hope for a standard RDF API, as I’ve argued for recently, but I gather that this effort will be focused on client-side developers, ie that it is really a standard RDF API for Javascript, which I think is a wasted opportunity: I would have been faster in this task if I’d been able to use familiar methods, and I wouldn’t have been so dependent on the documentation provided by the author of a particular library.

Anyway, hopefully my tramping this path will make it easier for those who follow.

Getting Started with RDF and SPARQL Using 4store and RDF.rb

Jeni — Sat, 15 Jan 2011 19:17:57 +0000

Updated to include some of Arto Bendicken’s recommendations.

This post is a response to Richard Pope’s Linked Data/RDF/SPARQL Documentation Challenge. In it, he asks for documentation of the following steps:

Install an RDF store from a package management system on a computer running either Apple’s OSX or Ubuntu Desktop.

Install a code library (again from a package management system) for talking to the RDF store in either PHP, Ruby or Python.

Programatically load some real-world data into the RDF datastore using either PHP, Ruby or Python.

Programatically retrieve data from the datastore with SPARQL using using either PHP, Ruby or Python.

Convert retrieved data into an object or datatype that can be used by the chosen programming language (e.g. a Python dictionary).

I’ve been told so many time how RDF sucks for mainstream developers that it was the main point of my TPAC talk late last year. I think that this is a great motivating challenge for improving not only the documentation of how to use RDF stores and libraries but how to improve their generally installability and usability for developers as well.

Anyway, I thought I’d try to get as far as I could to see just how bad things really are. I am on Mac OS X, and I’m going to use Ruby (although I don’t really know it all that well, so please forgive my mistakes). I’ll breeze on through as if everything is hunky dory, but there are some caveats at the end.

4store

I’m going to use 4store because it’s really easy to install on the Mac. If you want to install it on Ubuntu, there’s a package available. For a Mac, it’s a matter of going to the list of Mac downloads, downloading the most recent version, opening the .dmg and installing the 4store application by dragging it into your Applications folder.

When you run the 4store application you get a command line prompt. To set up and start a triplestore called ‘reference’ with a SPARQL endpoint running on port 8000, type the following commands:

$ 4s-backend-setup reference
$ 4s-backend reference
$ 4s-httpd -p 8000 reference

If you then navigate to http://localhost:8000/ you should see the following:

Don’t let the title ‘Not found’ put you off. The fact you get a response means that it’s working.

Loading Data

First, find some data to load. A good place for government RDF data is http://source.data.gov.uk/data/ for example. I downloaded

http://source.data.gov.uk/data/reference/organogram-co/2010-10-31/index.rdf

There are several ways of importing data into 4store using the command line. Yves Raimond has created a Ruby gem for doing so programmatically. There’s also rdf-4store from Fumihiro Kato which ties into the RDF.rb library which I’ll use later on.

However, if you use the SPARQL server then it’s just an HTTP PUT call, which of course you can do in any language you like (every language has support for making HTTP requests, right?) without the need to install any store-specific packages. However, since we’ll be doing a lot of HTTP requests, it’s useful to have a library that can make them simple. There are plenty to choose from for Ruby. I chose rest-client:

$ sudo gem install rest-client

With that, I wrote the following little Ruby script called ‘load-data-into-4store.rb’:

#!/usr/bin/env ruby
require 'rubygems'
require 'rest_client'

filename = '/Users/Jeni/Downloads/index.rdf'
graph    = 'http://source.data.gov.uk/data/reference/organogram-co/2010-06-30'
endpoint = 'http://localhost:8000/data/'

puts "Loading #{filename} into #{graph} in 4store"
response = RestClient.put endpoint + graph, File.read(filename), :content_type => 'application/rdf+xml'
puts "Response #{response.code}: 
#{response.to_str}"

Run the script from the command line:

$ ruby load-rdf-into-4store.rb

and you should get the response:

Sending PUT /data/http://source.data.gov.uk/data/reference/organogram-co/2010-06-30 to localhost:8000
Response 201: 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head><title>201 imported successfully</title></head>
<body><h1>201 imported successfully</h1>
<p>This is a 4store SPARQL server.</p><p>4store v1.0.5</p></body></html>

You can then check http://localhost:8000/status/size/ and you should see that there are now some triples in the store:

Running a Query

The next step is to query that data using SPARQL. Running SPARQL queries is just a matter of HTTP POSTing a query to the SPARQL endpoint. 4store provides a page that you can use to test out queries at http://localhost:8000/test/ so perhaps we should do that before diving into the Ruby code. The easy one to start with is just one that returns a list of the types of things that are described within the data:

SELECT DISTINCT ?type 
WHERE { 
  ?thing a ?type .
} 
ORDER BY ?type

Paste that into the textarea that’s provided on http://localhost:8000/test/ so it looks like:

and you get some XML:

<?xml version="1.0"?>
<sparql xmlns="http://www.w3.org/2005/sparql-results#">
  <head>
    <variable name="type"/>
  </head>
  <results>
    <result>
      <binding name="type"><uri>http://purl.org/linked-data/cube#DataSet</uri></binding>
    </result>
    <result>
      <binding name="type"><uri>http://purl.org/linked-data/cube#DataStructureDefinition</uri></binding>
    </result>
    ...
  </results>
</sparql>

SELECT queries like this one (which are the most common kind of query to run to simply extract data) return SPARQL Query Results XML Format by default, so there’s no need to get hold of a specialised library for processing the results: you just need something to process XML.

For Ruby, I’m choosing Nokogiri as I’ve heard good things about it. To install:

$ sudo port install libxml2 libxslt
$ sudo gem install nokogiri

So now we just need a script that will run this query, process the results as XML, and do something with them. Call it ‘find-rdf-types.rb’:

#!/usr/bin/env ruby
require 'rubygems'
require 'rest_client'
require 'nokogiri'

query = 'SELECT DISTINCT ?type WHERE { ?thing a ?type . } ORDER BY ?type'
endpoint = 'http://localhost:8000/sparql/'

puts "POSTing SPARQL query to #{endpoint}"
response = RestClient.post endpoint, :query => query
puts "Response #{response.code}"
xml = Nokogiri::XML(response.to_str)

xml.xpath('//sparql:binding[@name = "type"]/sparql:uri', 'sparql' => 'http://www.w3.org/2005/sparql-results#').each do |type|
  puts type.content
end

Run it:

$ ruby find-rdf-types.rb

and you get:

POSTing SPARQL query to http://localhost:8000/sparql/
Response 200
http://purl.org/linked-data/cube#DataSet
http://purl.org/linked-data/cube#DataStructureDefinition
http://purl.org/linked-data/cube#Observation
http://purl.org/net/opmv/ns#Artifact
http://purl.org/net/opmv/ns#Process
http://purl.org/net/opmv/types/google-refine#OperationDescription
http://purl.org/net/opmv/types/google-refine#Process
http://purl.org/net/opmv/types/google-refine#Project
http://rdfs.org/ns/void#Dataset
http://reference.data.gov.uk/def/central-government/AssistantParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/CivilServicePost
http://reference.data.gov.uk/def/central-government/Department
http://reference.data.gov.uk/def/central-government/DeputyDirector
http://reference.data.gov.uk/def/central-government/DeputyParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/Director
http://reference.data.gov.uk/def/central-government/DirectorGeneral
http://reference.data.gov.uk/def/central-government/ParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/PermanentSecretary
http://reference.data.gov.uk/def/central-government/PublicBody
http://reference.data.gov.uk/def/central-government/SeniorAssistantParliamentaryCounsel
http://reference.data.gov.uk/def/intervals/CalendarDay
http://www.w3.org/2000/01/rdf-schema#Class
http://www.w3.org/ns/org#Organization
http://www.w3.org/ns/org#OrganizationalUnit
http://xmlns.com/foaf/0.1/Person

So we can see that the dataset contains information that include statistical data using the data cube vocabulary, provenance information using OPMV (Open Provenance Model Vocabulary), some information about organisations using org, some data.gov.uk-specific vocabulary, and people using FOAF.

Processing RDF

Sometimes it can be useful to get non-tabular data out of SPARQL. At that point, rather than using SELECT queries, you will want to use a CONSTRUCT query, which creates RDF. For example, try the query:

PREFIX foaf: <http://xmlns.com/foaf/0.1/>

CONSTRUCT {
  ?person 
    a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
} WHERE { 
  ?person a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
}

This gets all the information in the data about the individuals for whom names have been supplied in the data, as RDF.

Although the response is RDF/XML, you definitely do not want to process it as XML. Instead, you need a proper RDF library. Fortunately, there’s a good one for Ruby in RDF.rb. You can install it and a bunch of extra plugins that make it easy to deal with RDF in all its guises using:

$ sudo gem install linkeddata

This lets us pick out an appropriate parser based on the Content-Type of the response, and load the results of the SPARQL query into an in-memory RDF::Graph:

response = RestClient.post endpoint, :query => query
content_type = response.headers[:content_type][/^[^ ;]+/]
puts "Response #{response.code} type #{content_type}"

graph = RDF::Graph.new
graph << RDF::Reader.for(:content_type => content_type).new(response.to_str)

We can perform subsequent queries over that graph, for example just to extract names and telephone numbers and put them into a dictionary:

query = RDF::Query.new({
  :person => {
    RDF.type  => FOAF.Person,
    FOAF.name => :name,
    FOAF.mbox => :email,
  }
})

people = {}
query.execute(graph).each do |person|
  people[person.name.to_s] = person.email.to_s
end

It’s worth noting that the constants RDF and FOAF are pre-declared by including RDF, and the values that you get back from a query are RDF values, which can be URIs or have datatypes or languages. In the above code I’ve converted them into strings for insertion into the Ruby dictionary.

The full script for ‘get-names-and-emails.rb’ is:

#!/usr/bin/env ruby
require 'rubygems'
require 'rest_client'
require 'linkeddata'

include RDF

query = "PREFIX foaf: <http://xmlns.com/foaf/0.1/>

CONSTRUCT {
  ?person 
    a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
} WHERE { 
  ?person a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
}"
endpoint = 'http://localhost:8000/sparql/'

puts "POSTing SPARQL query to #{endpoint}"
response = RestClient.post endpoint, :query => query
content_type = response.headers[:content_type][/^[^ ;]+/]
puts "Response #{response.code} type #{content_type}"

graph = RDF::Graph.new
graph << RDF::Reader.for(:content_type => content_type).new(response.to_str)

puts "\nLoaded #{graph.count} triples\n"

query = RDF::Query.new({
  :person => {
    RDF.type  => FOAF.Person,
    FOAF.name => :name,
    FOAF.mbox => :email,
  }
})

people = {}
query.execute(graph).each do |person|
  people[person.name.to_s] = person.email.to_s
end
puts "\nCreating directory of #{people.length} people"

stott_email = people['Andrew Stott']
puts "\nAndrew Stott's email address: #{stott_email}"

Run this script with:

$ ruby get-names-and-emails.rb

and you get the result:

POSTing SPARQL query to http://localhost:8000/sparql/
Response 200 type application/rdf+xml

Loaded 459 triples

Creating directory of 75 people

Andrew Stott's email address: mailto:andrew.stott@cabinet-office.gsi.gov.uk

Conclusions and Caveats

So there you have it, a walkthrough of setting up a local triplestore, loading in data and accessing that data programmatically using SPARQL queries.

Now for some caveats. First, you’re bound to have noticed that I having followed Richard’s steps to the letter.

4store wasn’t installed from a package management system. The only packaged triplestore I could locate on MacPorts was Virtuoso (which I’ll come to in a second). I hope that 4store’s installation is simple enough for this slight deviation from the rules not to matter.
I didn’t install a package for specifically talking to 4store in order to load in data, just used HTTP requests. There are client libraries for 4store, but I figure that the HTTP requests are easy enough, and the resulting code more portable into other environments, so I prefer not to use them.

Second, there are a couple of dead ends that I went down that I haven’t written up in the above:

I did spend some time yesterday evening trying to get Virtuoso set up. I managed to get it installed, but loading data into it seemed to require some magic which I couldn’t figure out. So I went to bed instead.
I tried to install and use rdf-raptor in order to parse the RDF/XML that naturally comes out of 4store CONSTRUCT queries, but got a Could not open library 'libraptor' error. I couldn’t find an immediate fix for that, so decided to keep things simple instead and just use plain RDF.rb.

Third, I want to reiterate that there may be better ways of using 4store, rest_client, Nokogiri and RDF.rb, as well as Ruby generally, than those shown above. I don’t claim to be an expert in any of these technologies. If you have suggestions and corrections, I’d encourage you to add a comment and I’ll incorporate them in the text to improve it.

Finally, some general points, because the strong binding of ‘linked data’ and ‘SPARQL’ in Richard’s post bothers me:

It’s not necessary to have a SPARQL endpoint when publishing linked data, nor to run your own triplestore. If you already have a website, you are probably better off generating N-Triples or RDF/XML or Turtle in the same way as you generate HTML or XML or JSON.
It’s not necessary to learn SPARQL to access and use linked data: the whole point is that the data in linked data is available through HTTP access in standard (RDF-based) formats, so you can scrape them using a follow-your-nose approach and store the results however you like.

Having said the above, if you’re collecting linked data from multiple sources with unpredictable content and want to query across it, having a local triplestore is very useful.

I also want to point out that within the linked data we’ve published on data.gov.uk, we’ve made a big effort to make the data available in multiple formats such as JSON, XML and CSV, and through a RESTful, URI-parameter-driven API, precisely to lower the barrier for developers who want to use that information but understandably don’t want to take the time or make the effort to learn the linked data technologies that underly the sites. For those that do, the RDF/XML and Turtle is there as well, and the SPARQL queries that are used to create each page are available to look at, tweak, and learn from. Our hope is that the linked data API that provides access to lists of schools, departments and railway stations can make the linked data learning curve a little less steep.

Priorities for RDF

Jeni — Sun, 28 Nov 2010 21:44:52 +0000

A couple of weeks ago I did a talk at the TPAC Plenary Day about why RDF hasn’t had the uptake that it might and what could be done about it.

I felt quite uncomfortable about doing this for many reasons. The predominant one is that I’m well aware that the world is made by the people who turn up. It is far far easier to snipe from the sidelines than it is to put in the effort to attend telcons and face-to-face meetings, to engage on mailing lists, to write specifications and implementations and tutorials.

On the other hand, what I hope is that the perspective of someone who is outside that process, someone who tries to understand and interpret and use the results of that process, might be valuable. And so I aimed to provide that honestly.

In that spirit, I’m going to put my stake in the ground and say that there are three areas where I think W3C should be concentrating its efforts:

standardising (something like) TriG — Turtle plus named graphs
standardising an API for the RDF data model
standardising a path language for RDF that can be used by that API and others for easy access

and that it should specifically not put its efforts into standardising another syntax for RDF based on JSON.

Format Standardisation

The first point is that I think we need to decide on a single recommended format for RDF.

Fundamentally, unlike XML or JSON, RDF is defined first and foremost as a model rather than as a syntax. That means it can be expressed in a number of syntaxes, the most common of which are RDF/XML, Turtle and N-Triples though of course there’s also RDFa, RDF/JSON, JSON-LD and N3 and if you start factoring in named graphs you can add TriX, TriG and N-Quads to the list.

Except for a few corner cases, it would be perfectly possible to express the same RDF model in any of these syntaxes. Why is this so bad? Surely having choice is a good thing, because publishers can choose an option that fits with their workflows? And aren’t all these formats generated automatically anyway, such that the same data can be provided in many ways with no overhead?

Well, no, there are actually two ways in which having multiple syntaxes makes adoption harder.

First, publishers aren’t always generating data automatically; in a number of cases (which I think and hope will grow) RDF data is being generated just like CSV files are, as static documents which are simply published in the same way as other static documents. In these cases, publishers either have to do the research and make a decision about which format to use, or produce the data in multiple formats. This is a particular challenge when people aren’t convinced they want to generate RDF anyway.

Second, toolsets have to handle producing or consuming multiple formats. That means more code, more testing and more maintenance on both the production and consumption sides of the equation, all of which raise the implementation burden.

Of course it’s natural that during the initial stages of the use of a technology that we should see a variety of patterns of use: there need to be innovations and experiments so that we can find what works and what doesn’t. But as that technology matures, we need to start bedding down some basics. There need to be agreed foundations that, even if imperfect, are solid enough for the majority of us to build upon. And we need to exercise some self-restraint to concentrate on doing that building rather than revisiting those decisions.

We have a number of years of experience now about what formats are easy to understand, to pass around, to create and to process. It is time, I think, to pick one, to get it standardised, to deprecate others and to provide a much cleaner and clearer picture to publishers and consumers.

Of the formats that we have, the one that fits best with the RDF data model and is simplest for humans to understand is Turtle. But it needs to support named graphs, so that it’s possible to share the RDF datasets that are exposed within a SPARQL endpoint, which is why I say W3C should standardise something like TriG.

RDF APIs

The second point is to work on standardising the APIs that are available for developers who work with RDF. Why standardise APIs? Because it would make accessing RDF easier and more predictable for web developers, who often work across multiple languages and platforms. Developers don’t really care about syntax — although having something readable is useful for debugging — they care about the way in which they get to interact with in-memory structures that hold the data.

RDF needs an API that exposes its internal model (of literals and resources and triples and graphs and datasets) in a way that isn’t too onerous for people to use. There are lots and lots of RDF APIs about, within the various parsers that are available for different platforms; the only one that’s approaching a standard is the one embedded within the RDFa API specification. I would like to see that disentangled from RDFa and for it, or something like it, to gain traction amongst the writers of RDF libraries such as the Redland RDF libraries, RDFLib, Moriarty, Reddy, rdfQuery and so on and on.

But having an API for RDF’s data model is not enough.

I think there is a lot that we can learn from XML’s experience here. James Clark’s recent blog post about XML and the web describes what it’s like for developers working with XML compared to JSON:

The fundamental problem is the mismatch between programming language data structures and the XML element/attribute data model of elements. This leaves the developer with three choices, all unappetising:

live with an inconvenient element/attribute representation of the data;

descend into XML Schema hell in the company of your favourite data binding tool;

write reams of code to convert the XML into a convenient data structure.

By contrast with JSON, especially with a dynamic programming language, you can get a reasonable in-memory representation just by calling a library function.

So JSON is popular because accessing information within the JSON is really easy. And that’s for two reasons:

it’s parsed with a single simple function call in a common library
the result of parsing is simple to navigate; typically you can do so using native methods such as dot-notation paths

The first of these is a simple matter of winning hearts and minds. The second is the important one: it’s easy to use because the underlying JSON model fits neatly onto the object-oriented programming paradigm that most developers use.

XML isn’t so popular among web developers because its underlying model doesn’t fit well into most programming languages: it has attributes and mixed content and a whole bunch of other things that don’t map straight-forwardly onto objects-with-properties. Navigating through XML (or HTML) structures using a DOM is tedious and automatic binding mostly doesn’t work.

What about RDF? On the face of it, RDF is a good fit with object-oriented models; they both follow a basic entity-attribute-value approach. However, there are (at least) three things in RDF that do not fit with the object-oriented model:

Properties in RDF are identified through URIs rather than simple names (ie those containing just letters and numbers and underscores). Some programming languages, such as Javascript, let you have properties that aren’t simple names, but you then have to access them through the relatively clunky [] notation rather than dot-notation paths. Properties are first-class objects in RDF with things like labels and ranges and inverses; fitting with standard programming languages here means using reflection and having the ability to annotate fields, and everything gets a bit mind-bending.
Values in RDF often have datatypes or languages associated with them, and the set of datatypes that you can use is completely extensible (and of course datatypes are first-class objects, with their own properties, too). This wouldn’t be so bad except that making every value an object means comparisons with basic strings or numbers won’t generally work.
In RDF, a property can have more than one value (an unordered bag of values, if you like) or it can have a value that is a rdf:List (an ordered sequence of values); it can even have many values which are rdf:Lists. On the other hand, the object-oriented model generally supports values that are arrays (and of course you can have arrays within arrays), which are always ordered. So there is always a choice to be made when mapping from an object-oriented model to RDF, about whether the values should be at the same level or be rdf:Lists.

In other words, just as with XML, there is no straight-forward mapping from RDF to an object structure that developers can immediately use. That doesn’t stop us trying, of course:

There’s the approach that we use within the linked data API and elsewhere, which is to make it easy for data publishers to create simple JSON versions of the RDF they publish. A website-specific configuration file determines what the mapping looks like. URIs of properties get turned into readable names (and provide the map back to the original property URI, so that it’s possible to get more information about the property). Datatypes and languages are ignored by default (but mapped onto structured values if so configured). And the distinction between properties-with-multiple-values and properties-whose-value-is-a-List is ignored. We purposefully lose some of RDF’s expressivity and power in order to gain usability. You can see the result in action at http://education.data.gov.uk/doc/school and http://education.data.gov.uk/doc/department for example.

There’s the approach that Nathan has taken within js3 which is to create libraries that, with a bit of work on the client side, give a way of mapping RDF into an object-oriented structure which is easy to manipulate (or vice versa, create RDF from OO structures). It’s the same basic principle as helping publishers to generate JSON, but the interpretation and mapping is done by the client rather than the publisher. The work that Nathan’s done to manage this in Javascript is very impressive; I don’t know whether the same approach can be mapped to other languages.

But as James intimated, data binding is hairy and scary. Mappings between different data models are always imperfect, lossy, sensitive to what seem like small changes and therefore hard to maintain. I remember nodding along as Mike Kay talked about this at the XML Summer School in relation to the use of XML: the horrors of working with systems in which there three way maps between relational and object-oriented and XML structures, and the relief that comes with working with an XML-only architecture. I suspect this same observation is one of the drivers behind the growth of JSON databases.

On the other hand, you know, perhaps RDF is close enough to the object-oriented model that it won’t be so bad. Perhaps we could find a way to standardise on a method of configuring applications that do the mapping, such as defining short names for properties, describing how to handle objects with datatypes and languages and so on. We have a body of experience that can be learnt from, including the ones above, and perhaps it can be tied into the RDB-to-RDF work too. The biggest challenge, I suspect, will be to create something round-trippable.

Path Languages

The other option that James didn’t mention but that I touched on in my TPAC talk is to learn from how working with HTML and XML has been made easier in libraries such as jQuery or hpricot. These libraries still allow the HTML and XML to be accessed through a DOM, rather than mapping HTML or XML into object structures, but make the lives of developers simpler by supporting querying of the HTML/XML using path languages that are designed to be used to query those kinds of structures. For HTML, that’s CSS; for XML that’s XPath. (It’s the same approach as is used for strings: we use regular expressions for many operations rather than working with them at the character level.) Path languages work over the native model; all that’s offered in the library are functions that take strings (holding the path language) and return objects or values as appropriate.

I don’t know exactly what it looks like, and it might already be out there (the world moves fast and I know I’m not aware of everything), but what I think we need is a path language for navigating around RDF, probably based on SPARQL property paths or the FRESNEL selector language and an API (or APIs) that uses it. For example, something that lets developers use code like:

graph.find("*[foaf:nick = 'web3r']/foaf:name")

to pick values out of an in-memory graph. In my opinion, something like this would be much more likely to bring benefits than a data binding approach.

Why Not RDF in JSON?

What I’ve tried to explain above is firstly that we already have too many syntaxes for RDF, and secondly that the main barrier to developers using RDF is the way in which they are forced to interact with that RDF once they have hold of it, not the syntax itself. The syntax that we use for RDF really doesn’t matter, because developers interact with the in-memory dataset, not directly on the syntax.

Nathan’s recent post on Opening Linked Data, which is worth reading in its entirety, captures the essence of the issue:

You can’t shoe horn RDF in to JSON, no matter how hard you try - well, you can, but you loose all the benefits of JSON in the first place, because the data is RDF, triples and not objects, rdf nodes and not simple values

In other words, using JSON as the basis for an RDF syntax doesn’t actually win you anything in terms of the ease of processing of that RDF. In fact, I’ll go further and say it has exactly the same bad qualities as RDF/XML.

One of the bad things about RDF/XML is that it misleads people into thinking they can use normal XML tooling to process RDF, but XML tooling exposes the XML tree, not the RDF graph that they need. It’s good enough in some circumstances, of course, but it’s not working with RDF as RDF. Similarly, just because you’re using XML tools doesn’t mean RDF/XML is easy to generate; you’re a lot safer to generate correct RDF/XML from an in-memory graph, in the same way as generating XML using string manipulation is harder work than it first appears.

In exactly the same way, I think that a JSON-based syntax for RDF will mislead developers into thinking that they can interpret and generate that JSON like they can normal JSON, and interact with it at that level, when this simply isn’t the case.

The only advantage that I can see for a JSON-based RDF syntax is equivalent to the only advantage of RDF/XML: it is easier to store for people who use JSON databases, just as RDF/XML is easier to store for people who use XML databases. I am not sure that benefit is worth the cost of an additional RDF syntax; isn’t RDF better stored in a triplestore?

Summary

So to reiterate, as far as I’m concerned, W3C and the RDF community should be concentrating on a syntax for RDF that doesn’t come saddled with those kinds of assumptions, which I think is Turtle + graphs; something like TriG. They should be concentrating on developing a standard API for RDF access that has a chance of adoption among the developers of RDF libraries, and on working out what parts of SPARQL and FRESNEL could be used to create a path language that could be reused in several contexts, including within such an API. And these should be done in preference to a RDF syntax in JSON which doesn’t solve the core problems, and in fact just adds another syntax to the mix.

Using Freebase Gridworks to Create Linked Data

Jeni — Sun, 22 Aug 2010 22:23:32 +0000

When we encourage people to put their data on the web as linked data, the biggest question is “How?”. There are so many “How?” questions to answer:

how do we choose what URIs to use for things?
how do we choose what vocabularies to use?
how do we handle changing data?
how do we tell people how the data was created?
how do we publish it?
how will other people know about it?

and, of course:

how do we create it?

Our goal within the linked data part of data.gov.uk (and I know we haven’t achieved it yet) is to both answer these questions and to make the answers as simple as possible. The answers to the questions cannot either require up-front knowledge of all possible types of data that might be published or depend on the availability of linked data for all the things we want to talk about. It cannot require registration at centralised services. It cannot require everyone to do everything in the same way or at the same pace.

We must take adopt an approach that encourages people to make their data available in forms that are easier for other people to pick up and use because they see the benefits for them and their stakeholders and because the effort of doing so is not too high to bear. We must grow, adapt and evolve incrementally. If linked data eventually wins, it will be due to its benefits, not to faith.

Anyway, enough rant. The point of this blog post is to talk about one of the answers to the ‘How do we create it?’ question: using Freebase Gridworks. For those who haven’t encountered it, Gridworks is an incredibly useful application that enables you to easily analyse, clean and manipulate tabular data. In a few steps, it can be used to generated linked datasets which can then be published on the web just like any other file, ready for other people to reuse without jumping through hoops. I’m going to assume that you can download it and install it following the instructions provided on the Gridworks site.

In this post, I’m going to talk about how to use Gridworks to generate linked data, using an example of local government spending data from Windsor and Maidenhead council. Like a good train journey, there’s quite a lot to see along the way.

Note: Many thanks to Dave Reynolds for his work on this data and comments on an earlier version of this post.

Importing Data

The first step is to import the data into Gridworks. If you just take the Windsor & Maidenhead data and import it directly, you’ll get a single not-very-useful column as shown in the following screenshot:

If you look at the spreadsheet in a normal spreadsheet programme then you’ll see why. Like a lot of spreadsheets created by normal people, who want to create something readable by human beings rather than computers, it has some extra lines at the top to explain what the spreadsheet contains, as shown in the following screenshot:

Fortunately, Gridworks lets us easily skip over these first few lines. When you import the data, put the number 1 in the box for “Ignore X initial non-blank lines”, as shown here:

(You need the number 1 because although there are three lines before the table really starts, the second two of those are blank.)

That done, the data should look a lot more useful, as shown in the following screenshot:

Cleaning Data

The next thing to do is to explore the data a bit to get a handle on what’s there and work out whether any cleaning or rationalisation is necessary to improve its quality.

With columns that hold names, such as ‘Directorate’, ‘Service’ or ‘Supplier Name’, you’re looking for slight misspellings caused by bad data entry. Gridworks helps you find these by creating a list of the distinct values for a particular column and telling you how many instances there are of each. Use the arrow at the side of the column name to pull down the menu, then choose Facet > Text Facet to create this list, as shown here:

Once you’ve chosen Text Facet, the list pops up on the left hand side of the window. You can click on these to filter the table to contain just those rows that have that value for that column, but you can then scan through this to spot any places where there looks to be a typo or two entries that should really be the same. For example, the Services list holds both ‘Libraries & Information Services’ and ‘Library & Information Services’, as shown here:

It’s unlikely that there are really two distinct services with such similar names, so we’d like to clean up this data by standardising on one name or another. You can quickly change all occurrences of one value to another using the edit option that appears just to the right of the value when you hover over it. This brings up a dialog that enables you to change all of those values to something else, as shown here:

You can do something similar with numeric columns, such as the ‘Amount excl vat £’ column. This time choose Numeric Facet rather than Text Facet and you’ll get a histogram up as shown here:

This is useful for identifying outliers. If you grab the handle on the left of the histogram and move it to the centre, the rows will get filtered to only those that have an amount within that range. For example, moving it to only show rows between £500,000 and £1,500,000 shows that there are three payments of this size, all made by Children’s Services to Wilmott Dixon Construction Limited, as shown in this screenshot:

Although these values are much higher than most of the others in the spreadsheet, they don’t seem to be errors — I guess a new school was being built or something — so there’s nothing to correct here, but it shows how numeric facets can be used to explore the data.

Another approach to exploring and cleaning the data is to use the clustering algorithms that are built into Gridworks to identify duplicates. To do this, pull down the column menu and this time choose Edit Cells... > Cluster and Edit, as shown in the following screenshot, this time for the ‘Supplier Name’ column:

This brings up a dialog that groups together values that look similar. In this case, ‘Siemens plc’ and ‘Siemens PLC’, as shown in the following screenshot:

You can use this dialog to change all the similar values to a standard one. Check the Merge checkbox for the clusters of values that should be merged, edit the New Cell Value field to whatever standard value you want to adopt, and choose Apply & Re-cluster or simply Apply & Close to make the change.

You will often find that the default clustering algorithm (key collision/fingerprint) doesn’t come up with any clusters as it’s fairly conservative. It’s worth playing around a bit with different algorithms to look for other duplicates by selecting other possibilities from the drop-down menus. For example, choosing the ‘nearest neighbour’ method with the Levenstein distance function and a radius of 2 (edits) results in four possible duplicates within the Suppliers list, as shown here:

If you’re not sure about whether the cluster is due to a typo or not, hover over the row and click on the Browse this cluster link that appears. That will bring up a separate window that will show you just the rows in the cluster, from which you should be able to make a judgement. For example, it’s not clear whether ‘Academia Ltd’ is a typo for ‘Academics Ltd’ but browsing the cluster shows that the Cost Centre codes and the Types of the transactions are completely different for the two Suppliers, so they are probably different.

Deriving Data

The next step is to derive some data from what we have within the spreadsheet. Since our goal is to produce linked data, the kind of derived data that we’re interested in are URIs.

At this point we need to start making decisions about what URIs to use. If you look at the list of spending data from Windsor and Maidenhead, you’ll see that there are a whole bunch of these spreadsheets. It would be really useful if we could tie these spreadsheets together by using the same URIs for the same things across the datasets. For that reason, the only URI that’s going to be local to the dataset is the URI for each line (or data point if you like) itself. On the other hand, most of the things that are named here are going to be local to Windsor & Maidenhead: ‘Abba Cars’ may be sufficient to identify a single company within Windsor & Maidenhead, but certainly wouldn’t be nationwide. So the URIs I’m going to create here are mostly going to be within the www.rbwm.gov.uk domain.

Here’s the table of the columns and the associated URIs that I’m going to use. I should stress that this is just for example purposes, but I’ve used the following principles:

URIs for datasets are just like URIs for any other web document, but shouldn’t have an extension because the data itself should be available in many formats
URIs for real-world things should have /id at the start of the path, and URIs for conceptual things should have /def at the start of their paths; both should result in a 303 redirection to a suitable web page

This is what we’re doing within data.gov.uk, but it’s an important principle of the web that different councils might well choose their own URI schemes, depending on the kind of technology support that they have, without any bad side-effects on the interpretation of the data.

Column	URI pattern
(Dataset)	http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2
(Row/ExpenditureLine)	http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2#{row-number}
(Council)	http://statistics.data.gov.uk/id/local-authority/00ME
Directorate	http://www.rbwm.gov.uk/id/directorate/{directorate-slug}
Updated	http://reference.data.gov.uk/id/day/{date}
TransNo/Payment	http://www.rbwm.gov.uk/id/transaction/{transaction-number}
Service	http://www.rbwm.gov.uk/id/service/{service-slug}
Cost Centre	http://www.rbwm.gov.uk/def/cost-centre/{cost-centre-code}
Supplier Name	http://www.rbwm.gov.uk/id/supplier/{supplier-slug}

As you can see, those of the columns that contain text fields have, as part of their URI, a ‘slug’. This is a shortened, normalised value suitable for putting in a URI: basically ensuring that the string doesn’t contain any punctuation or spaces. For example, ‘Adult & Community Services’ would turn into ‘adult-community-services’.

Our first task will be to create these slugs. To do this, we’ll create a new column based on the existing ones by choosing Edit Column > Add Column Based on This Column ... from the drop-down menu on the appropriate column:

Selecting this will bring up a dialog which will ask you to name the new column and then enter a formula to calculate the new value, as shown here:

The default language for this formula is Gridworks’ own, though there are other options available. To create the slug, we need to:

turn the value to lower case
replace all spaces with hyphens
remove anything that isn’t a letter, number, or hyphen
replace all sequences of two hyphens with a single hyphen

This is done in two steps. The first three steps can be done using the formula:

replace(replace(toLowercase(value), ' ', '-'), /[^-a-z0-9]/, '')

Gridworks helps by listing the original and resulting values for the first several rows of the spreadsheet, so that you can see whether it’s working as expected. When you’re happy, hitting OK creates the new column.

The last step (replacing all sequences of two hyphens with a single hyphen) can be done by editing the cells in the new column. Bring up the Edit Cells... > Transform... dialog using the menu:

and use the formula:

replace(value, '--', '-')

then check the Re-transform until no change checkbox so that any pairs of hyphens are repeatedly replaced with single hyphens, as shown here:

The other tabs in the new column and edit cells dialogs are really helpful. The History tab lets you choose formulae that you’ve used before to use again. This is useful here because we want to create the slugs for the Service and Supplier Name in the same way. The Help tab lists all the functions that you can use within the formula.

Creating the URIs for the columns proceeds in the same way, except this time the formulae are more like:

'http://www.rbwm.gov.uk/id/directorate/' + value

There are two that are slightly different. First, there’s the URI for the date, which needs to be constructed from the date/time value held by Gridworks as follows. We can do this in two stages. First, to construct a new column called ‘Date’ to hold the formatted date:

datePart(value, 'year') + '-' + 
if (datePart(value, 'month') < 9, '0', '') + replace(datePart(value, 'month') + 1, '.0', '') + '-' + 
if (datePart(value, 'day') < 10, '0', '') + datePart(value, 'day')

(note that the datePart() function returns a 0-based count for the month) and then to create the Date URI column based on this:

'http://reference.data.gov.uk/id/day/' + value

Second, there’s the URI for the row (an expenditure line) itself, which needs to be constructed using the row number. It’s useful to construct it as a local URI (ie just the fragment) as this means the same code can be used to construct the column across different datasets, so it’s just:

'#' + rowIndex

Exporting Data

Once the extra columns have been made, it’s time to export data from Gridworks. While Gridworks makes it easy to export to CSV or into Freebase, it’s also possible to export in any format you want using templates. Use the Project menu and choose Export Filtered Rows > Templating ..., as shown in the following screenshot:

Note that this will only export the rows that you currently have selected, so if you want to export everything, make sure that you deselect any facets that you’ve currently got selected.

Choosing the Templating ... option will open up a dialog that you can use to create whatever format you want. The default, as shown in the following screenshot, is JSON.

On the left are four fields:

Prefix is content that’s put at the top of the exported data
Row Template is content that’s generated for each row
Row Separator is content that’s put between each row
Suffix is content that’s put at the bottom of the exported data

One thing to be extremely careful of here is that any changes you made to the fields on the left here will not be saved when the dialog is closed. For that reason, it’s a good idea to create your templates in a separate text file and copy and paste them in. Also note that the sample data on the right is only for the first set of rows, not for the whole spreadsheet.

We’re going to generate Turtle using the template, so the next stage is to work out precisely what Turtle to generate. We’ve been working on small vocabulary for payment data based on the Data Cube vocabulary and that’s what I’ll use here, although it isn’t quite complete and available yet as it will be. We’ll start at the bottom, with the individual rows, and then add extra surrounding information as we go.

Row Template

Within this data, each row corresponds to a payment:ExpenditureLine within the dataset. The expenditure lines can be organised into groups based on the payment:Payment that they’re associated with, which is indicated through the ‘TransNo’ column in the database. Within the payment vocabulary we’re using, we can assign individual expenditure lines to the payment using the payment:expenditureLine property.

The payment:payer of each payment:Payment is Windsor & Maidenhead council. The payment:payee is the ‘Supplier’ listed in the spreadsheet. The payment:date is the ‘Updated’ date.

Each individual line in the spreadsheet is a payment:ExpenditureLine which is associated with one of these payments. The payment:expenditureCode is the ‘Cost Centre’ and the actual payment:amountExcludingVAT is the ‘Amount excl vat £’ value. Some example Turtle for the first line is thus:

<http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2>
  qb:slice <http://www.rbwm.gov.uk/id/transaction/2650750> .

<http://www.rbwm.gov.uk/id/transaction/2650750>
  a payment:Payment , qb:Slice ;
  rdfs:label "Transaction 2650750"@en ;
  qb:sliceStructure payment:payment-slice ;
  payment:transactionReference "2650750" ;
  payment:payer <http://statistics.data.gov.uk/id/local-authority/00ME> ;
  payment:payee <http://www.rbwm.gov.uk/id/supplier/1st-choice-d-b-driveways-limited> ;
  payment:date <http://reference.data.gov.uk/id/day/2010-04-09> ;
  payment:expenditureLine <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2#0> .

<http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2#0>
  a payment:ExpenditureLine , qb:Observation ;
  rdfs:label "Expenditure Line 0"@en ;
  qb:dataSet <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2> ;
  payment:expenditureCode <http://www.rbwm.gov.uk/def/cost-centre/LM05> ;
  payment:amountExcludingVAT 1875.00 .

That’s the basic data for each line, but there’s also some other information which should be brought out for each line:

the name of the payee
the date, year, month and day-of-month for the payment, which may help further analysis of the data
the meaning of the expenditure code (particularly its association to a particular service)

In each of these cases, pulling the information out from each line is going to lead to a lot of repetition, because the same payee, date and so on will be described in multiple lines, but we don’t have any choice and we can tidy it up by removing duplicates afterwards. The Turtle for the first line will look like:

<http://www.rbwm.gov.uk/id/supplier/1st-choice-d-b-driveways-limited>
  a org:Organization ;
  rdfs:label "1st Choice - D B Driveways Limited"@en .

<http://reference.data.gov.uk/id/day/2010-04-09>
  a interval:CalendarDay ;
  rdfs:label "2010-04-09" ;
  time:hasBeginning <http://reference.data.gov.uk/id/gregorian-instant/2010-04-09T00:00:00> ;
  interval:ordinalYear 2010 ;
  interval:ordinalMonthOfYear 4 ;
  interval:ordinalDayOfMonth 9 .

<http://reference.data.gov.uk/id/gregorian-instant/2010-04-09T00:00:00>
  a time:Instant ;
  time:inXSDDateTime "2010-04-09T00:00:00"^^xsd:dateTime .

<http://www.rbwm.gov.uk/def/cost-centre/LM05>
  a rbwm:CostCentre , skos:Concept ;
  rdfs:label "Cost Centre LM05"@en ;
  rbwm:costCentreCode "LM05"^^rbwm:CostCentreCode ;
  rbwm:service <http://www.rbwm.gov.uk/id/service/magnet-leisure-centre> .

<http://www.rbwm.gov.uk/id/service/magnet-leisure-centre>
  a rbwm:Service ;
  rdfs:label "Magnet Leisure Centre"@en ;
  rbwm:providedBy <http://www.rbwm.gov.uk/id/directorate/adult-community-services> .

<http://www.rbwm.gov.uk/id/directorate/adult-community-services>
  a rbwm:Directorate ;
  rdfs:label "Adult & Community Services"@en ;
  org:unitOf <http://statistics.data.gov.uk/id/local-authority/00ME> ;
  rbwm:provides <http://www.rbwm.gov.uk/id/service/magnet-leisure-centre> .

<http://statistics.data.gov.uk/id/local-authority/00ME>
  org:hasUnit <http://www.rbwm.gov.uk/id/directorate/adult-community-services> .

You’ll see that in the last part of this I’ve introduced some properties and classes with a rbwm: prefix. These are for classes and properties that are here in this data, but aren’t part of the payment vocabulary. The basic schema is:

rbwm:CostCentre a rdfs:Class ;
  rdfs:label "Cost Centre"@en ;
  rdfs:comment "A cost centre."@en .

rbwm:Service a rdfs:Class ;
  rdfs:label "Service"@en ;
  rdfs:comment "A service provided by the council."@en .

rbwm:Directorate a rdfs:Class ;
  rdfs:label "Directorate"@en ;
  rdfs:comment "A directorate within the council"@en .

rbwm:service a rdf:Property , owl:ObjectProperty ;
  rdfs:label "Service"@en ;
  rdfs:comment "The service associated with a particular cost centre."@en ;
  rdfs:domain rbwm:CostCentre ;
  rdfs:range rbwm:Service .

rbwm:providedBy a rdf:Property , owl:ObjectProperty ;
  rdfs:label "Provided By"@en ;
  rdfs:comment "The directorate that provides this service."@en ;
  rdfs:domain rbwm:Service ;
  rdfs:range rbwm:Directorate .

rbwm:provides a rdf:Property , owl:ObjectProperty ;
  rdfs:label "Provides"@en ;
  rdfs:comment "A service provided by this directorate."@en ;
  rdfs:domain rbwm:Directorate ;
  rdfs:range rbwm:Service .

rbwm:costCentreCode a rdf:Property , owl:DatatypeProperty ;
  rdfs:label "Cost Centre Code"@en ;
  rdfs:comment "The code of this cost centre."@en ;
  rdfs:domain rbwm:CostCentre ;
  rdfs:range rbwm:CostCentreCode .

rbwm:CostCentreCode a rdfs:Datatype ;
  rdfs:label "Cost Centre Code"@en ;
  rdfs:comment "A cost centre code consisting of two capital letters followed by two digits."@en .

This illustrates how individual councils might extend the information that they make available in RDF without having to seek any kind of prior agreement from anyone else. If, later on, a third party starts to make available ontologies for cost centres, services and directorates, Windsor & Maidenhead could start to link up their RDF with those more widely standardised classes and properties, with appropriate use of rdfs:subClassOf or rdfs:subPropertyOf.

Now we have an idea about what data we can extract for a single row, we can turn this into a Gridworks template. The templates are fairly straight forward. Wherever you want to insert a value from a particular column, you use the syntax ${Column Name}. If you want to do any further processing, you can use the syntax {{Formula}} to insert the result of a calculation.

<http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2>
  qb:slice <${Transaction URI}> .

<${Transaction URI}>
  a payment:Payment , qb:Slice ;
  rdfs:label "Transaction ${TransNo}"@en ;
  qb:sliceStructure payment:payment-slice ;
  payment:transactionReference "${TransNo}" ;
  payment:payer <http://statistics.data.gov.uk/id/local-authority/00ME> ;
  payment:payee <${Supplier URI}> ;
  payment:date <${Date URI}> ;
  payment:expenditureLine <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2${Line URI}> .

<http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2${Line URI}>
  a payment:ExpenditureLine , qb:Observation ;
  rdfs:label "Expenditure Line {{rowIndex}}"@en ;
  qb:dataSet <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2> ;
  payment:expenditureCode <${Cost Centre URI}> ;
  payment:amountExcludingVAT {{cells['Amount excl vat £'].value + 0}} .

Note that the last line here uses the expression cells['Amount excl vat £'].value + 0 in order to ensure that every figure has a decimal place, which makes them into xsd:decimal values within the resulting RDF.

I won’t do the rest of the row template here, though it’s available in full in a separate file.

The other parts of the template are easier to complete. The prefix needs to contain any namespace prefixes that are used within the RDF. It’s also useful to put a base URI here and describe the dataset itself. The RDF for the dataset should contain a number of properties about the dataset as a whole. There are a number of levels at which the dataset can be described:

basic metadata such as its title and the license that it’s available under
statistical metadata including what dimensions it has and how it’s sliced
linked data metadata such as how this dataset links out to other linked datasets

The Turtle for this description is shown here:

<http://www.rbwm.gov.uk/public/finance_supplier_payments>
  a void:Dataset ;
  void:subset <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2> .

<http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2>
  a payment:PaymentDataset , void:Dataset ;
  # basic metadata
  rdfs:label "Windsor & Maidenhead Supplier Payments where charge to specific cost centre is >= £500 for period April 2010 - June 2010"@en ;
  dct:license <http://data.gov.uk/id/licence> ;
  dct:temporal [
    # this time is retrieved from the Last-Modified date on the original spreadsheet
    time:hasBeginning <http://reference.data.gov.uk/id/gregorian-instant/2010-08-02T08:37:02>
  ] ;

  # statistical metadata
  qb:structure payment:payments-with-expenditure-structure ;
  qb:sliceKey payment:payment-slice ;
  payment:currency <http://dbpedia.org/resource/Pound_sterling> ;

  # linked data metadata
  void:exampleResource
    <http://www.rbwm.gov.uk/id/transaction/2650750> ,
    <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2#0> ;
  void:vocabulary payment: , qb: , rbwm: ;
  void:subset [
    a void:Linkset ;
    void:linkPredicate qb:slice ;
    void:subjectsTarget <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2> ;
    void:objectsTarget <http://www.rbwm.gov.uk/id/transaction> ;
  ] , [
    a void:Linkset ;
    void:linkPredicate payment:payer ;
    void:subjectsTarget <http://www.rbwm.gov.uk/id/transaction> ;
    void:objectsTarget <http://statistics.data.gov.uk/id/local-authority> ;
  ] , [
    a void:Linkset ;
    void:linkPredicate payment:payee ;
    void:subjectsTarget <http://www.rbwm.gov.uk/id/transaction> ;
    void:objectsTarget <http://www.rbwm.gov.uk/id/supplier> ;
  ] , [
    a void:Linkset ;
    void:linkPredicate payment:date ;
    void:subjectsTarget <http://www.rbwm.gov.uk/id/transaction> ;
    void:objectsTarget <http://reference.data.gov.uk/id/day> ;
  ] , [
    a void:Linkset ;
    void:linkPredicate payment:expenditureLine ;
    void:subjectsTarget <http://www.rbwm.gov.uk/id/transaction> ;
    void:objectsTarget <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2> ;
  ] , [
    a void:Linkset ;
    void:linkPredicate payment:expenditureCode ;
    void:subjectsTarget <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2> ;
    void:objectsTarget <http://www.rbwm.gov.uk/def/cost-centre> ;
  ] , [
    a void:Linkset ;
    void:linkPredicate rbwm:service ;
    void:subjectsTarget <http://www.rbwm.gov.uk/def/cost-centre> ;
    void:objectsTarget <http://www.rbwm.gov.uk/id/service> ;
  ] , [
    a void:Linkset ;
    void:linkPredicate rbwm:providedBy ;
    void:subjectsTarget <http://www.rbwm.gov.uk/id/service> ;
    void:objectsTarget <http://www.rbwm.gov.uk/id/directorate> ;
  ] , [
    a void:Linkset ;
    void:linkPredicate rbwm:provides ;
    void:subjectsTarget <http://www.rbwm.gov.uk/id/directorate> ;
    void:objectsTarget <http://www.rbwm.gov.uk/id/service> ;
  ] , [
    a void:Linkset ;
    void:linkPredicate org:hasUnit ;
    void:subjectsTarget <http://statistics.data.gov.uk/id/local-authority> ;
    void:objectsTarget <http://www.rbwm.gov.uk/id/directorate> ;
  ] , [
    a void:Linkset ;
    void:linkPredicate org:unitOf ;
    void:subjectsTarget <http://www.rbwm.gov.uk/id/directorate> ;
    void:objectsTarget <http://statistics.data.gov.uk/id/local-authority> ;
  ] .

Provenance

I’ve described here, verbally, exactly what I’ve done in terms of the cleaning of the data, deriving new columns, and the template that I’ve used to create a Turtle rendition of the data in this spreadsheet. One of the things that we’ve worked hard on within data.gov.uk is finding ways of expressing this provenance information in RDF. There are two reasons for this:

Providing provenance increases transparency and enables you to check the processing that the data has been through, increasing your trust in the data.
Describing the process in sufficient detail for you to replicate that process enables you to modify and repeat the process, which both enables you to add value and to apply the same processing to your own situation, thus spreading best practice.

The basic provenance vocabulary that we’re using within data.gov.uk is the Open Provenance Model Vocabulary. This vocabulary talks about Artifacts, Processes that create and use them, and Agents that control those processes. We’ve created an extension of this vocabulary specifically to help describe this kind of scenario, where a spreadsheet is processed using Gridworks and then exported using a template. I’ll put this provenance information in a separate file simply because embedding provenance information, which includes a template, in the template itself gets us into nasty recursion issues.

As well as the template, there are two supplementary artifacts that we need to record the provenance of this data:

the Gridworks project itself
the JSON description of the set of operations performed by Gridworks

The first can be exported using the Project menu. The second is accessed through the Undo/Redo tab as shown in the following screenshot:

This tab shows the actions that have been carried out on the data, and enables you to undo them in sequence. The extract link at the bottom opens up the dialog shown in the following screenshot:

You have to manually copy and paste the JSON description from the right of this dialog into a separate file in order to save it.

We can then start describing the provenance of the RDF; this needs to go in the Turtle file itself. We start by saying that the RDF that we’ve created was created from the Gridworks project and through an extraction operation. A simple link to the spreadsheet that was used as the source of the data also provides a quick link back to the original data:

<http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2>
  a opmv:Artifact ;
  dct:source <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2.xls> ;
  gridworks:wasExportedBy <finance_supplier_payments_2010_q2_provenance#gridworks-export> ;
  gridworks:wasExportedFrom <finance_supplier_payments_2010_q2_project.tar.gz> .

The provenance information then needs to describe the export process:

<#gridworks-export>
  a gridworks:ExportUsingTemplate , opmv:Process ;
  rdfs:label "Process for Exporting Windsor & Maidenhead data as Turtle" ;
  gridworks:project <finance_supplier_payments_2010_q2_project.tar.gz> ;
  gridworks:template <#gridworks-template> .

The project itself was created from the original Excel spreadsheet. The details of how it was generated are through an import that ignored a single non-blank header row and then went through the set of operations described by the JSON.

<finance_supplier_payments_2010_q2_project.tar.gz>
  a gridworks:Project , opmv:Artifact ;
  rdfs:label "Windsor & Maidenhead Supplier Payments April 2010 - June 2010 Gridworks Project"@en ;
  gridworks:wasCreatedFrom <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2.xls> ;
  opmv:wasGeneratedBy <#gridworks-processing> .

<#gridworks-processing>
  a gridworks:Process , opmv:Process ;
  rdfs:label "Processing on the Gridworks Project"@en ;
  common:usedData <http://www.rbwm.gov.uk/public/finance_supplier_payments_2010_q2.xls> ;
  gridworks:ignore 1 ;
  gridworks:operationDescription <finance_supplier_payments_2010_q2_operations.json> .

<finance_supplier_payments_2010_q2_operations.json>
  a gridworks:OperationDescription , opmv:Artifact ;
  rdfs:label "Dump of the Processing carried out by Gridworks on Windsor &amp; Maidenhead Supplier Payments April 2010 - June 2010 data"@en ;
  gridworks:wasExportedFrom <finance_supplier_payments_2010_q2_project.tar.gz> ;
  gridworks:wasExportedBy <#gridworks-operation-description-extraction> .

<#gridworks-operation-description-extraction>
  a gridworks:ExtractOperationDescription , opmv:Process ;
  rdfs:label "Extraction of the operation description from the Windsor &amp; Maidenhead Supplier Payments April 2010 - June 2010 Project from Gridworks"@en ;
  gridworks:project <finance_supplier_payments_2010_q2_project.tar.gz> .

The template is described in terms of the separate parts; in fact it’s useful to use this provenance file as the record of the template that you use, given that Gridworks won’t save the template in the project itself.

<#gridworks-template>
  a gridworks:Template , opmv:Artifact ;
  gridworks:prefix """
...
"""^^xsd:string ;
  gridworks:rowTemplate """
...
"""^^^xsd:string .

Rinse and Repeat

Gridworks makes it easy to repeat a given set of operations on another spreadsheet that follows the same structure. If you download the Windsor and Maidenhead spending data from 2009 Q4 and import it into Gridworks, you’ll see that it uses the same set of columns as the 2010 Q2 data that we’ve been looking at. (Strangely enough, the 2010 Q1 data doesn’t quite follow the same structure as it doesn’t include the ‘TransNo’ column.)

There are a couple of differences:

the ‘Updated’ column isn’t recognised as holding dates on import; you can use Edit Cells... > Transform to change these values into dates using the toDate(value) formula
the ‘Amount excl vat £’ column isn’t recognised as holding numbers on import because the values have commas in them; you can use the formula toNumber(replace(value, ',', '')) to rectify this

You might want to do some more cleaning, for example to check for duplicates, but once that is done, you use the apply link at the bottom of the Undo/Redo tab to apply the JSON operation description that you imported for the previous spreadsheet on this one. The templates require only a little tweaking to give different filenames and labels, but otherwise can be used as-is.

So while the process of cleaning data, deriving values and creating a template for exporting as Turtle is a bit of effort, the likelihood is that you will be able to repeat the same operations on similar data with a minimal amount of work.

Conclusions

Gridworks is a simply amazing tool for data cleansing, analysis and, as we’ve seen, transformation. It’s set to become more so for our purposes in the near future, as it comes to support the mapping of names for things to URIs using configurable reconciliation services (which might allow it to automatically map Government Department names to URIs, for example), and the creation of RDF using a more intuitive and user-friendly approach than the templates that I’ve illustrated here.

Of course there are issues, particularly for UK civil servants who typically have to operate on locked-down machines running IE7 (if they’re lucky). Gridworks also only deals with the fairly simple cases of data that fits in a spreadsheet-like structure, without the complexities of annotations on rows, columns or individual cells that we often see in government data.

Nevertheless, there’s huge potential here to provide a fairly easy route to the publication of linked data for people who are familiar with spreadsheets, in particular one that can be tweaked and extended to allow for the variety and complexity of real-world data.

Distributed Publication and Querying

Jeni — Mon, 22 Mar 2010 21:26:53 +0000

One of the biggest selling points of linked data is that it’s supposed to facilitate web-scale distributed publication of data. Just as with the human web, anyone can publish data at their local site without having to go through any kind of central authority.

Just as with the human web, convergence on particular sets of URIs for particular kinds of things can happen in an evolutionary way: in a blog post I might point to Amazon when I want to talk about a particular book, Wikipedia to define the concepts I mention, people’s blogs or twitter streams when I mention them.

And with everyone using the same terms to talk about the same things, there’s the prospect of being able to easily pull together information from completely different sources to find connections and patterns that we’d never have found otherwise.

What’s been very unclear to me is how this distributed publication of data can be married with the use of SPARQL for querying. After all, SPARQL doesn’t (in its present form) support federated search, so to use SPARQL over all this distributed linked data, it sounds like you really need a central triplestore that contains everything you might want to query.

This post is an attempt to explore this tension, between distributed publication and centralised query, and to try to find a pattern that we might use within the UK government (and potentially more widely, of course) to publish and expose linked data in a queryable way. It’s a bit sketchy, and I’d welcome comments.

Publishing Datasets

First, let’s look at the publication of data. We publish data at the moment in all kinds of ways: embedded tables within PDFs, CSV database dumps, Excel spreadsheets, Word documents, XML, JSON, N3 and so on and on. Each of these documents contains a set of information: a dataset.

Each dataset contains information about a whole load of things, usually real-world things. This is easy to see when you have datasets that contain lots of things of the same type: a spreadsheet might contain information about lots of different local authorities, a database dump about a bunch of schools. In FOAF terms, we’d say that the dataset has each of these things as a topic.

Even datasets that are really about one thing (have, in FOAF terms, a primary topic) contain information about lots of other things. For example, a web page about a hospital might include some level of information about the different departments within the hospital, the strategic health authority that it belongs to, the chief executive and so on. Information that is just about one thing is rarely useful; at the very least, you will want to know the labels of things that it’s related to.

If we move to thinking about linked data, each thing is assigned an HTTP URI. There is then one particular dataset that stands above all the other datasets that contain information about that thing: the dataset in the document that you get when you resolve its URI. The fact that there is this dataset doesn’t alter the fact that there are many many other datasets out there that contain information about the thing. But the dataset that you get at the URI for the thing obviously has a special role.

These datasets — the ones you get at the end of a resource’s URI — are the way in which an organisation can exercise control over the use of URIs minted within their domain. The organisation that controls the URI for a thing determines whether that URI resolves, and what is at the end of the URI. If fifteen different websites all published information about a school consistently using the same URI for that school, anyone could pull that information together into something potentially useful. But if the URI for the school doesn’t actually resolve, then you would have to wonder whether the school actually exists, or if it’s just a figment of the imagination of those fifteen websites: a spoof school.

Also, you’d expect the information that you find at the end of the URI to be correct and up to date. You’d expect it to be reasonably complete as well: to return a bunch of information about the school and pointers to more information about the school. This information is likely to come from a bunch of trusted sources: an integrated view over a collection of other datasets.

Providing SPARQL Endpoints

We’ve established that

anyone can publish information about anything they choose, but that people will have different levels of trust in different sources of information
information about any one thing is seldom useful on its own; the power of the linked data web is the ability to make connections between things

And so on to querying. Linked data can be useful without explicit querying — you can navigate around related sets of information by following links, and pull together information gleaned from different sites — but querying of some kind provides much more potential power and, with a linked data API, the opportunity to provide an easy-to-use web-based API for the data.

SPARQL queries operate over a default graph (or dataset) and a set of supplementary named graphs. For efficiency, these need to be pulled into a single triplestore.

And so we have a quandry. To support queries, we need all the data we might want to query to be pulled into a single triplestore. Given that all data is linked, and all links are potentially interesting, the only answer seems to be to have the whole web of data in a single store. And that kind of centralised solution seems impractical, both in terms of the sheer size of store you’d need and the obvious impact on efficiency of doing so.

Curated Triplestores

I think the answer (for the moment at least) is to forget about querying the entire web of linked data and focus on supporting the easy creation of targeted, curated, triplestores that each incorporate a useful subset of the linked data that’s out there. What subset is useful for a given triplestore is a design question that should be informed by the potential users of that particular service. Larger subsets are likely to locate more cross-connections, but have a performance penalty.

For example, a service that was oriented towards helping local authorities plan their schooling provision might include all the current data about nursery, primary and secondary schools (but not universities or versioned data), information about their administrative district and the district that they appear in (but no extra information about census areas), and those neighbourhood statistics, including historic data, that relate to children and schooling (but not those that relate to care of the elderly, for example).

Another service might include all historic information about schools and universities and historic information about all associated administrative geography, but not include neighbourhood statistics.

Supporting On-Demand Triplestores

In the scenario painted above, each triplestore will include different datasets, brought together for a particular purpose. Imagine a huge warehouse full of boxes, each of which is a particular dataset. Each triplestore will fit together a different set of those boxes. What’s neat about the linked data approach is that the boxes are really easy to bring together: creating a triplestore should just be a matter of selecting which datasets you want to use with little or no hand-crafting of links between them or resolution of naming conflicts.

The challenge from the side of the data publisher is to enable these triplestores to be both created and kept up to date. A data publisher has to:

describe what datasets are available
describe how these link to other potentially interesting datasets, to give hints about where connections might be made
provide a mechanism for getting the current state of all the available datasets (which can obviously be through crawling but could alternatively be through a dump or set of dumps)
provide a mechanism for informing interested parties about new datasets being made available (which could be through routine crawling or through a feed)
provide a mechanism for informing interested parties about when a dataset changes (which could also be through routine crawling or through a feed)

A lot of these problems are solved.

VoiD’s purpose in life is to describe datasets and how they link to each other, and it provides a void:dataDump property that points to a dump of the data. VoiD can describe datasets that are supersets of other datasets, which enables datasets to be grouped together into potentially useful bundles.

Where information needs to be kept up to date, we can use feeds. We need to keep up to date information about the datasets that a publisher makes available, and information about the content of a particular dataset. This can be achieved through a single Atom feed in which each dataset is recorded as an entry, with an <updated> element indicating its last update. Datasets that are removed can be indicated through a deleted-entry element. There is some ongoing work that suggests how to augment voiD with a pointer to such a feed.

As well as pointing to a dataset, and indicating that it has been updated, the Atom feed could contain information about the change itself, represented as a changeset. This could be included as part of the information provided about the new version of the dataset, described in terms of its provenance.

Feeds that were provided in this way could be provided using the normal model, whereby any interested triplestores would regularly check the feed for updates, or using PubSubHubbub in order to push notifications to triplestores. The latter would require triplestore providers to support a service that accepted such notifications, of course.

A triplestore should expose which datasets (and which versions of those datasets) are used within the triplestore. This can be gathered through a SPARQL query to list the available graphs and their metadata, so long as that information is included within the named graphs themselves.

What Should We Do?

How does all this translate into what guidelines we should put into place for UK government publishers and what tools we should provide centrally?

First, we need to recognise the responsibility that comes with the ownership of a URI. Within the UK, we are encouraging people to use URIs of the form:

http://{sector}.data.gov.uk/id/{concept}/{identifier}

to name things like schools and hospitals, with the recognition that information about those things might come from many different public bodies. Someone has to be in charge of that domain: they have to determine which URIs within a particular URI set are resolvable, and what information is provided at the end of each URI. These same sector owners should support easy-to-use APIs based around the particular URI sets that they are responsible for.

The easiest route to supporting the pages, an easy-to-use API, and a SPARQL endpoint for deeper querying is going to be to create a curated triplestore with a linked data API layer over the top. This triplestore will need to be populated with data from multiple datasets, both as separate named graphs (to provide traceability back to the original data) and merged into a default graph that reflects the current state of the world.

The precise datasets that are included within the triplestore will depend on the judgement of the sector owners about both the trustworthiness of the available datasets and their utility. For example, it’s likely that a lot of triplestores will want to include information about administrative geography and perhaps some information about time, simply because everything happens somewhere and sometime.

Second, we need to make this process really easy, through guidelines and tooling.

We encourage the data owners themselves (which are individual public bodies) to publish, along with the datasets themselves:

voiD descriptions of the groups of datasets that they publish
metadata about the individual datasets that they publish (within each dataset itself)
Atom feeds that are updated each time datasets are added, removed or altered, preferably including changeset information
(optionally) dumps of groups of datasets, in NQuads format
(optionally) notifications of changes to the Atom feed to a PubSubHubbub hub

Data owners should be able to split up the datasets that they provide into different groups based on their knowledge of the domain, with the possibility of individual datasets belonging to more than one group.

We then create tooling that can:

enable the sector owners to quickly and easily put together a list of trusted sites from which datasets can be gathered
collect datasets from these sites, either through NQuads dumps or through crawling
merge datasets to create a default current view
put these datasets into a triplestore
keep the triplestore up to date, either through polling feeds or by accepting PubSubHubbub notifications to identify changes, applying those changes, and merging data as required

To facilitate PubSubHubbub use, which supports timely updating of triplestores, we’d need a PubSubHubbub hub. Data owners can inform this hub of updates to their feeds and sector owners can register interest in particular feeds.

These guidelines and tooling are not just useful for sector owners: they are useful for anyone who wants to pull together linked data published in a distributed way across the web. We should expect and encourage multiple stores offering different combinations of datasets and different levels of service. The ones offered centrally, by sector owners, are certainly not the be-all and end-all — in fact we should look on them as a basic level of service, to be superseded by the community.

Translating Existing Models to RDF

Jeni — Sat, 13 Mar 2010 20:35:46 +0000

As we encourage linked data adoption within the UK public sector, something we run into again and again is that (unsurprisingly) particular domain areas have pre-existing standard ways of thinking about the data that they care about. There are existing models, often with multiple serialisations, such as in XML and a text-based form, that are supported by existing tool chains.

In contrast, if there is existing RDF in that domain area, it’s usually been designed by people who are more interested in the RDF than in the domain area, and is thus generally more focused on the goals of the typical casual data re-user rather than the professionals in the area.

To give an example, the international statistics community uses SDMX for representing and exchanging statistics (and a lot more besides; it’s a huge standard). SDMX includes a well-thought through model for statistical datasets and the observations within them, as well as standard concepts for things like gender, age, unit multipliers and so on. By comparison, SCOVO, the main RDF model for representing statistics, barely scratches the surface in comparison.

This isn’t the only example: the INSPIRE Directive defines how geographic information must be made available. GEMINI defines the kind of geospatial metadata that that community cares about. The Open Provenance Model is the result of many contributors from multiple fields, and again has a number of serialisations.

You could view this as a challenge: experts in their domains already have models and serialisations for the data that they care about; how can we persuade them to adopt an RDF model and serialisations instead?

But that’s totally the wrong question. Linked data doesn’t, can’t and won’t replace existing ways of handling data. But it has got some interesting features that can bring great benefit to people who want to publish their data, namely:

web-scale addresses — being able to name and refer to things like individual observations in a statistical hypercube, a particular road junction, or the particular process that led to something being created
annotation — the ability to record metadata about everything that you can name, which is everything!
distributed publication — enabling multiple publishers to control the publication of their data without having to upload it to a central location
links — the joining of information to other information, providing more context, supporting more queries and reducing the requirement for duplication

The question is really about how to enable people to reap these benefits; the answer, because HTTP-based addressing and typed linkage is usually hard to introduce into existing formats, is usually to publish data using an RDF-based model alongside existing formats. This might be done by generating an RDF-based format (such as RDF/XML or Turtle) as an alternative to the standard XML or HTML, accessible via content negotiation, or by providing a GRDDL transformation that maps an XML format into RDF/XML.

Either way, the underlying model needs to be mapped into RDF. We’re furthest down this road with statistical data. I wanted to explore here what it might look like for the Open Provenance Model, building on lessons learned from the statistical domain.

Open Provenance Model

The Open Provenance Model talks about three main nodes:

artifacts, which are the things that are produced or used by processes
processes, which are actions that are performed using or producing artifacts
agents, which are the people or systems that perform actions

and five kinds of edges that can be defined between them:

process A used artifact B
artifact A was generated by process B
process A was controlled by agent B
process A was triggered by process B
artifact A was derived from artifact B

Then things start getting more complicated. OPM indicates that each artifact and agent plays a different role when it is used by, generated by or controls a process. What’s more, each artifact and agent might be involved in the process at different times (though timing information is optional within OPM). And a given provenance graph may contain several accounts of how artifacts, processes and agents fit together.

Existing Mapping to RDF

The OWL ontology for OPM for OPM is a very literal mapping of OPM into RDF. Each of the types of nodes is a separate class, and each of the types of edges is a separate class. Thus, it introduces a lot of n-ary relationships. Take a really simple example of an XML file being transformed into HTML using XSLT. With the OPM ontology, the RDF would look something like:

_:transformation a opm:Process .
<doc.html> a opm:Artifact .
<doc.xml> a opm:Artifact .
<doc.xsl> a opm:Artifact .
_:processor a opm:Agent .
_:Jeni a opm:Agent .

_:stylesheetLink a opm:Used ;
  opm:effect _:transformation ;
  opm:cause <doc.xml> ;
  opm:role eg:xsltSource .

_:sourceLink a opm:Used ;
  opm:effect _:transformation ;
  opm:cause <doc.xsl> ;
  opm:role eg:xsltStylesheet .

_:resultLink a opm:WasGeneratedBy ;
  opm:effect <doc.html> ;
  opm:cause _:transformation ;
  opm:role eg:xsltResult .

_:processorLink a opm:WasControlledBy ;
  opm:effect _:transformation ;
  opm:cause _:processor ;
  opm:role xslt:processor .

_:userLink a opm:WasControlledBy ;
  opm:effect _:transformation ;
  opm:cause _:Jeni ;
  opm:role xslt:user .

_:derivation a opm:WasDerivedFrom ;
  opm:effect <doc.html> ;
  opm:cause <doc.xml> .

xslt:source a opm:Role ;
  opm:value "source" .

xslt:stylesheet a opm:Role ;
  opm:value "stylesheet" .

xslt:result a opm:Role ;
  opm:value "result" .

xslt:processor a opm:Role ;
  opm:value "processor" .

xslt:user a opm:Role ;
  opm:value "user" .

To give you an idea of what this mapping means, if I wanted to work out who created doc.html, I would have to do a query like:

SELECT ?who
WHERE {
  ?generatedBy 
    opm:cause <doc.html> ;
    opm:role xslt:result ;
    opm:effect ?transformation .
  ?controlledBy
    opm:effect ?transformation ;
    opm:role xslt:user ;
    opm:cause ?who .
}

Some Observations

There are two things that I want to pull out about the RDF mapping described above.

it’s incredibly literal; every entity type within the model is mapped onto an RDF class, including the edges, the roles and the accounts (which I didn’t show above)
it doesn’t reuse any existing vocabularies, even when they might help (such as for the ‘value’ of a role, which is really a label)

It reminds me of the mapping of object-oriented or relational data models into each other or into XML, which often result in a god awful mess and people swearing that technology X is goddamned ugly.

The fact is that elegant uses of each modelling paradigm — ones that are easy to understand and efficient to query — always take advantage of the unique features of that paradigm. For example, good XML vocabularies take advantage of the distinctions between attributes and elements, of nesting and hierarchies, and of the ability to hold mixed content.

It’s the same with RDF. There are four features of RDF that I think good vocabularies will take suitable advantage of:

existing vocabularies
inheritance
shortcuts and reasoning
named graphs

Reusing existing vocabularies takes advantage of the ease of bringing together diverse domains within RDF, and it makes data more reusable. For example, an OPM mapping that encourages the reuse of FOAF for people and organisations saves time and effort for the developers of the OPM RDF vocabulary, that they would otherwise have spent modelling the details of agents; and it means that any agents that are described within the description of a piece of provenance are automatically available as agents in the wider FOAF cloud. The same goes for using DOAP to describe software.

By reusing vocabularies, the data isn’t isolated any more, locked within a single context designed for a single use. This is a huge benefit of the linked data approach and it makes sense to leverage it.

Using inheritance means creating general purpose classes and properties and encouraging other people to use rdfs:subClassOf or rdfs:subPropertyOf to specialise them according to their own requirements. Within OPM, the different roles that artifacts and agents might play in a process is a natural fit with either sub-properties or sub-classes, depending on how the edges in the model are represented. For example, rather than

_:stylesheetLink a opm:Used ;
  opm:effect _:transformation ;
  opm:cause <doc.xsl> ;
  opm:role eg:xsltStylesheet .

xslt:stylesheet a opm:Role ;
  opm:value "stylesheet" .

you could generate data that looked like:

_:stylesheetLink a xslt:Stylesheet ;
  opm:effect _:transformation ;
  opm:cause <doc.xsl> .

where xslt:Stylesheet is defined as a subclass of opm:Used.

Inheritance is a basic form of reasoning. In the case of the subclass relationship outlined above, the reasoning is that anything that is a xslt:Stylesheet is also a opm:Used, and thus:

_:stylesheetLink a xslt:Stylesheet .

implies

_:stylesheetLink a xslt:Used .

Taking the scenario where you’re doing native linked data publishing — storing data in a triplestore and then publishing it out from there — you have two choices:

you can store just the basic data, and let the application retrieving it carry out whatever reasoning is necessary to derive the information they need; this limits the size of the triplestore, but can place a large burden on people using it — either they have to be very familiar with the exact choices made in modelling the basic data, or they have to construct complex SPARQL queries that take account of the fact that the data might be modelled in many different ways
you can store not only the basic data but also anything that can be derived from it; this increases the number of triples you have to store, but means that people can query it without having to perform any reasoning themselves

The latter is obviously the more user-friendly approach. (And a triplestore could make it easy by understanding and applying schemas, ontologies and rules as data is loaded in.)

To take a more complex example, provenance could be modelled in a much more direct way, such as:

<doc.html> a opm:Artifact ;
  opm:derivedFrom <doc.xml> ;
  opm:generatedBy [
    xslt:source <doc.xml> ;
    xslt:stylesheet <doc.xsl> ;
    xslt:processor _:processor ;
    xslt:user _:Jeni ;
  ] .

where xslt:source and xslt:stylesheet are sub-properties of a property called opm:used, and xslt:processor and xslt:user are sub-properties of opm:controlledBy. This removes the n-ary properties, which (given the use of inheritance to represent roles) are only actually needed if the model needs to capture the timing of the involvement of particular artifacts or agents within a process, and makes the provenance information much easier to query than before:

SELECT ?who
WHERE {
  <doc.html> opm:generatedBy ?transformation .
  ?transformation xslt:user ?who .
}

But what if we also want to support the more complex, n-ary-relation-based models? We would need to assert, somehow, a rule that said that the presence of a opm:controlledBy relationship from a process to an agent was equivalent to having a opm:WasControlledBy instance with a opm:cause pointing to the agent and an opm:effect pointing to the process. Combine this with xslt:user being sub-property of opm:controlledBy and you have the statement:

_:transformation xslt:user _:Jeni .

implying:

_:transformation opm:controlledBy _:Jeni .

which in turn implies:

[] a opm:WasControlledBy ;
  opm:effect _:transformation ;
  opm:cause _:Jeni .

The same reasoning could be applied in the opposite direction, of course. Part of the definition of the use of OPM in RDF could be that the presence of a opm:WasControlledBy with a opm:cause pointing to an agent and opm:effect pointing to a process implies a opm:controlledBy link between the opm:effect and the opm:cause. Whichever was used in the initial modelling of the data, the same query could be used to query the data (accepting some loss of precision along the way, but if you’re not interesting in timing information then why should you suffer the cost of querying through n-ary relations?).

The final thing that I mentioned above that mappings from existing models to RDF should take advantage of is named graphs. In OPM, the obvious way that named graphs could play a role is in providing support for the different accounts of provenance. Separate named graphs could be used to represent separate accounts, referencing the same artifacts, agents and processes where appropriate. Individually, the graphs can remain simple; together, you have the full power of OPM.

Conclusions

Modelling is a complex design activity, and you’re best off avoiding doing it if you can. That means reusing conceptual models that have been built up for a domain as much as possible and reusing existing vocabularies wherever you can. But you can’t and shouldn’t try to avoid doing design when mapping from a conceptual model to a particular modelling paradigm such as a relational, object-oriented, XML or RDF model.

If you’re mapping to RDF, remember to take advantage of what it’s good at such as web-scale addressing and extensibility, and always bear in mind how easy or difficult your data will be to query. There is no point publishing linked data if it is unusable.

Versioning (UK Government) Linked Data

Jeni — Sat, 27 Feb 2010 22:15:40 +0000

As you probably know, I’ve been working quite a lot recently on the UK government’s use of linked data, and in particular on providing guidance for people who want to publish their data as linked data. One of the things that we need to provide guidance about is how to publish linked data that changes over time. I’ve touched on this topic before but things have progressed now to the stage where we have to make some real, practical, recommendations.

Note: the contents of this post have been greatly informed through discussions with Leigh Dodds, Stuart Williams, Dave Reynolds, Ian Davis and John Sheridan. Ian Davis’ series on representing time in RDF is also well worth a look for a comparison of alternative approaches.

I’ve split this into two parts: versioned information resources (which are pretty easy) and versioned non-information resources (which are pretty hard). For both, we need to

provide some guidance about what the RDF should look like
mint or adopt properties to support that model

Versioned Information Resources

Easy things first. Some of the things that we talk about, such as legislation, are information resources (web documents), and these have different versions. The relevant level of precision for legislation is a day, but this will be different for different kinds of documents — some might change every second, for others an incrementally increasing version number might be more appropriate than a date. A generic pattern for the URIs, based on the design of URI sets for the UK public sector report would be:

http://{sector}.data.gov.uk/doc/{concept}/{identifier}/{version}

For example, the OFSTED report for a particular school based on an inspection carried out in 2009 might be something like:

http://education.data.gov.uk/doc/inspection-report/12345/2009

(There might be sub-versions too, if the inspection report itself goes through a revision process.) The RDF for this document should include links to the previous reports that it replaces, and dates that indicate when it was created and so on:

<http://education.data.gov.uk/doc/inspection-report/12345/2009>
  rdfs:label "2009 Inspection Report for Such-and-Such School"@en ;
  dct:created "2009-10-18"^^xsd:date ;
  dct:replaces <http://education.data.gov.uk/doc/inspection-report/12345/2006> .

It’s also useful to have a URI for unversioned document; this is the same as for the versioned document, but without the version:

http://{sector}.data.gov.uk/doc/{concept}/{identifier}

This document acts as a hub for the various concrete versions of the document:

<http://education.data.gov.uk/doc/inspection-report/12345>
  rdfs:label "Inspection Report for Such-and-Such School"@en ;
  dct:hasVersion
    <http://education.data.gov.uk/doc/inspection-report/12345/2009> ,
    <http://education.data.gov.uk/doc/inspection-report/12345/2006> ,
    <http://education.data.gov.uk/doc/inspection-report/12345/2003> ,
    ... .

<http://education.data.gov.uk/doc/inspection-report/12345/2009>
  rdfs:label "2009 Inspection Report for Such-and-Such School"@en ;
  dct:created "2009-10-18"^^xsd:date ;
  dct:replaces <http://education.data.gov.uk/doc/inspection-report/12345/2006> ;
  dct:isVersionOf <http://education.data.gov.uk/doc/inspection-report/12345> .

<http://education.data.gov.uk/doc/inspection-report/12345/2006>
  rdfs:label "2009 Inspection Report for Such-and-Such School"@en ;
  dct:created "2003-11-23"^^xsd:date ;
  dct:isReplacedBy <http://education.data.gov.uk/doc/inspection-report/12345/2009> ;
  dct:replaces <http://education.data.gov.uk/doc/inspection-report/12345/2003> ;
  dct:isVersionOf <http://education.data.gov.uk/doc/inspection-report/12345> .

It would be expected that people linking to the document would either point to a particular (dated) resource or to the unversioned (hub) document. For example, if someone were talking specifically about the 2006 OFSTED inspection, they would point to the 2006 inspection report; if they were referring to whatever inspection report is current, they’d use the unversioned URI.

Note: Although dct:hasVersion and dct:isVersionOf are sort-of OK here, having a property that points to the current version (ie most recent) version of a resource would be very helpful.

Versioned Non-Information Resources

The harder problem is how we handle changes to non-information resources over time. For example, how do we handle the fact that a school often changes head, sometimes changes name, regularly changes class sizes, rarely changes address and so on? How do we handle the fact that we have legacy statistics about local authorities as they were in 2008, prior to the 2009 reorganisation, and that it’s very likely that these kinds of changes will continue to take place regularly in the future?

Our requirements are:

that the data is easily usable by people who only care about the current state of a resource
that the (current) data remains easily queryable at a SPARQL endpoint
that it’s possible (not necessarily easy) to query historic data
that historic data can be moderately easily retrieved and navigated
that it can represent historical states even when the precise time period is not known
that it can distinguish between a change in the concept and a change in our record of it (e.g. changing the name of a school, versus correcting a typo in the database entry for the school)
that it can trace what the nature or cause of the change was (e.g. redrawing of local authority boundaries)

Statistical Data

To begin our discussion, let’s look at statistical data. Statistical data is data that’s usually numeric and for which we have values that are categorised along multiple dimensions as well as time. School census information is statistical data, for example, because each value is associated with not only the school and the date at which the census was taken but also the age (and gender, but to simplify I’ll pretend just age) of the children being counted. This gives us a set of observations which might each look like:

</data/edubase/census/12345/age/11/2009> 
  a sdmx:Observation ;
  sdmx:dataset </data/edubase> ;
  dct:replaces </data/edubase/census/12345/age/11/2008> ;
  rdf:value 85 ;
  edu:school </id/school/12345> ;
  edu:schoolYear </id/school-year/2009> ;
  sdmx:age 11 .

Note: This is indicative of the vocabulary we might use for statistics; don’t rely on it. If you’re interested in the progress we’re making on modelling statistical datasets using RDF, come and join the publishing statistical data Google Group.

These statistical observations point to the interval that they apply to as a property, with the rdf:value property holding the actual value. The observation won’t change over time (unless it is corrected, which I will come back to), and observations from different times can all remain present within the graph without interacting badly with each other.

This is great because it means that we can make queries that give us time series views over the data. For example, we could define a series for girls aged 11 at this particular school over time something like this:

</data/edubase/census/12345/age/11>
  a sdmx:TimeSeries ;
  edu:school </id/school/12345> ;
  sdmx:age 11 ;
  sdmx:observation
    </data/edubase/census/12345/age/11/gender/F/2009> ,
    </data/edubase/census/12345/age/11/gender/F/2008> ,
    </data/edubase/census/12345/age/11/gender/F/2007> ,
    ... .

and associate this with the school through a specialised property:

</id/school/12345> edu:age11 </data/edubase/census/12345/age/11> .

The fly in the ointment is that data that is purely represented in this way is really hard to query if all you’re actually interested in is the current value for the particular statistic. For example, say that you’ve just moved to an area and are trying desperately to find a school that might have room for your 11-year-old. Given that class sizes are capped at 30, you could look for schools that have a number of 11-year-olds that is not a multiple of 30. If you want to know how many 11 year-olds are currently in a school (according to the most recent measurement), you need a query like:

SELECT ?age11
WHERE {
  </id/school/12345> edu:age11 [
    sdmx:observation ?currentObservation ;
  ]
  OPTIONAL {
    ?futureObservation dct:replaces ?currentObservation .
  }
  FILTER ( !bound(?futureObservation) ) .
  ?currentObservation rdf:value ?age11 .
}

(it’s even more complicated if you don’t have the dct:replaces links!).

How much simpler it would be for people if there was a property that just indicated the current state of the world:

</id/school/12345> edu:currentAge11 85 .

The same argument applies even more strongly for values that we would categorise as reference data, such as the name of a school. Although it would be possible to model all this information using the kind of n-ary relation approach we have to use for statistical observations, it would be both incredibly hard to query and incredibly verbose to do so. Even if n-ary relations are the “correct” way of modelling the changing data, they are impractical for querying.

And, as I hinted, we have to have some way of managing the possibility of statistics themselves being versioned (for example if an error is detected within the statistics). Using n-ary relations to provide the value of an observation gets very complicated very quickly.

So, we have made the decision to use named graphs.

Named Graphs

Named graphs can be used in two ways which are related but need to be thought about slightly differently.

First, we can use a named-graph approach to the publication of RDF. We can describe the same thing within multiple documents; each document can contain different (and contradictory) information, but also metadata about the document that indicates precisely when the information it contains is valid.

Second, we can use a named-graph approach to the representation of RDF within a triple- (or more accurately quad-) store. We can collect together statements that are made at the same time, from the same source, and with the same level of authority into a named graph. These graphs can then be loaded into the store, with the metadata about each graph made explicitly available so that relevant graphs can be selected and queried.

There are two things that are worth noting about this:

Publishing named graphs is relevant however RDF is published. For example, in some linked data publication set-ups, RDF/XML or RDFa might be generated on demand based on an underlying database of some description. In this case, the named graphs for representing data aren’t relevant (the database will presumably capture some provenance and validity information itself that can be exposed within the RDF).
In the case where linked data is published natively (ie stored in a triplestore and exposed as linked data through an API), the two uses of named graphs don’t precisely align with each other. The named graphs that we create when we convert or load data within a triplestore are not (necessarily) the same as the named graphs that we expose when we publish data. What’s important here is
- that the named graphs that we have within the triplestore can feasibly be used (by a publication framework such as the linked data API we’re working on) to create the publication-based named graphs
- that the SPARQL endpoint offered by the triplestore has a default graph which reflects the current state of affairs

Let’s look at these two uses of named graphs in more detail.

Publication of Named Graphs

Our intention is to publish different information about the same resource within different documents (aka named graphs). This approach hooks into the approach for versioning information resources outlined above. A resource is described in a document, and many documents may describe the same resource.

For example, if a school changes its name from “Broadmoor Primary School” to “Wildmoor Heath School” on 1st September 2009, then after 1st September 2009, requesting information about the school at http://education.data.gov.uk/id/school/12345 would result in a 303 See Other redirection to http://education.data.gov.uk/doc/school/12345 which would contain information about the school that is currently relevant:

# Information about the school that is currently relevant
<http://education.data.gov.uk/id/school/12345>
  rdfs:label "Wildmoor Heath School"@en ;
  foaf:isPrimaryTopicOf 
    <http://education.data.gov.uk/doc/school/12345> ,
    <http://education.data.gov.uk/doc/school/12345/2009-09-01> ,
    <http://education.data.gov.uk/doc/school/12345/2001-09-01> ,
    <http://education.data.gov.uk/doc/school/12345/1996-09-01> ,
    ... .

as well as metadata about the document that’s been returned and the “hub” document that lists the alternative versions:

<http://education.data.gov.uk/doc/school/12345>
  rdfs:label "Information about School 123456"@en ;
  foaf:primaryTopic <http://education.data.gov.uk/id/school/12345> ;
  dct:hasVersion
    <http://education.data.gov.uk/doc/school/12345/2009-09-01> ,
    <http://education.data.gov.uk/doc/school/12345/2001-09-01> ,
    <http://education.data.gov.uk/doc/school/12345/1996-09-01> ,
    ... .

<http://education.data.gov.uk/doc/school/12345/2009-09-01>
  rdfs:label "Information about Wildmoor Heath School from 1st Sept 2009"@en ;
  foaf:primaryTopic <http://education.data.gov.uk/id/school/12345> ;
  dct:created "2009-09-01"^^xsd:date ;
  dct:replaces <http://education.data.gov.uk/doc/school/12345/2001-09-01> ;
  dct:isVersionOf <http://education.data.gov.uk/doc/school/12345> .

A request to the replaced document http://education.data.gov.uk/doc/school/12345/2001-09-01 would result in the information that was valid about the school on the 1st September 2001:

<http://education.data.gov.uk/id/school/12345>
  rdfs:label "Broadmoor Primary School"@en ;
  foaf:isPrimaryTopicOf 
    <http://education.data.gov.uk/doc/school/12345> ,
    <http://education.data.gov.uk/doc/school/12345/2009-09-01> ,
    <http://education.data.gov.uk/doc/school/12345/2001-09-01> ,
    <http://education.data.gov.uk/doc/school/12345/1996-09-01> ,
    ... .

and, again, metadata about the document that’s been returned and the “hub” document that lists the alternative versions:

<http://education.data.gov.uk/doc/school/12345>
  rdfs:label "Information about School 123456"@en ;
  foaf:primaryTopic <http://education.data.gov.uk/id/school/12345> ;
  dct:hasVersion
    <http://education.data.gov.uk/doc/school/12345/2009-09-01> ,
    <http://education.data.gov.uk/doc/school/12345/2001-09-01> ,
    <http://education.data.gov.uk/doc/school/12345/1996-09-01> ,
    ... .

<http://education.data.gov.uk/doc/school/12345/2001-09-01>
  rdfs:label "Information about Broadmoor Primary School (2001-2008)"@en ;
  foaf:primaryTopic <http://education.data.gov.uk/id/school/12345> ;
  dct:created "2001-09-01"^^xsd:date ;
  dct:isReplacedBy <http://education.data.gov.uk/doc/school/12345/2009-09-01> ;
  dct:replaces <http://education.data.gov.uk/doc/school/12345/1996-09-01> ;
  dct:isVersionOf <http://education.data.gov.uk/doc/school/12345> .

The statements about http://education.data.gov.uk/id/school/12345 in this second document are inconsistent with the statements retrieved from http://education.data.gov.uk/doc/school/12345 but because they are published within different documents, they should be considered (by anyone retrieving this data) to be different graphs and therefore are allowed to provide different views of the world.

The statements about the named graphs http://education.data.gov.uk/doc/school/12345/2009-09-01 and http://education.data.gov.uk/doc/school/12345/2001-09-01 can include information about the interval during which the content of the document is valid. (We haven’t worked out exactly how to indicate this yet; dct:valid is no good; see later.)

Associated Resources

This story seems fine until you start to look at linked resources. For example, schools may link out to separate resources, particularly when different aspects of a school are likely to change at different rates or come from different sources. A school is unlikely to change its name in the middle of a school year, but may well change some of its staff, and the number of pupils it has, during a year. It’s likely that these separate sets of information will be represented as different resources.

The document published about the school for a particular date will not necessarily include all the details of the linked resource at that point in time. This can make it hard to navigate to the particular version of the linked resource. For example, if a client wants to look at the information about a school at 1st September 2001, they would locate the graph at http://education.data.gov.uk/doc/school/12345/2001-09-01. This might contain:

<http://education.data.gov.uk/id/school/12345>
  rdfs:label "Broadmoor Primary School"@en ;
  edu:staffing <http://education.data.gov.uk/id/school/12345/staff> .

A request to http://education.data.gov.uk/id/school/12345/staff will result in a 303 See Other request to http://education.data.gov.uk/doc/school/12345/staff. This is current information about the staffing, and which will include:

<http://education.data.gov.uk/id/school/12345/staff>
  rdfs:label "Staffing of Wildmoor Heath School"@en ;
  edu:school <http://education.data.gov.uk/id/school/12345> ;
  edu:head ... ;
  edu:deputy ... ;
  ... ;
  foaf:isPrimaryTopicOf
    <http://education.data.gov.uk/doc/school/12345/staff> ,
    <http://education.data.gov.uk/doc/school/12345/staff/2009-09-01> ,
    <http://education.data.gov.uk/doc/school/12345/staff/2009-04-23> ,
    <http://education.data.gov.uk/doc/school/12345/staff/2009-01-01> ,
    ... .

<http://education.data.gov.uk/doc/school/12345/staff>
  rdfs:label "Information about Staffing at School 123456"@en ;
  foaf:primaryTopic <http://education.data.gov.uk/id/school/12345/staff> ;
  dct:hasVersion
    <http://education.data.gov.uk/doc/school/12345/staff/2009-09-01> ,
    <http://education.data.gov.uk/doc/school/12345/staff/2009-04-23> ,
    <http://education.data.gov.uk/doc/school/12345/staff/2009-01-01> ,
    ... .

<http://education.data.gov.uk/doc/school/12345/staff/2009-09-01>
  rdfs:label "Staffing of Wildmoor Heath School in Autumn Term, 2009"@en ;
  foaf:primaryTopic <http://education.data.gov.uk/id/school/12345/staff> ;
  dct:created "2009-09-01"^^xsd:date ;
  dct:isVersionOf <http://education.data.gov.uk/doc/school/12345/staff> ;
  dct:replaces <http://education.data.gov.uk/doc/school/12345/staff/2009-04-23> .

The client then has to work out which of the possible versions of the graph about http://education.data.gov.uk/id/school/12345/staff it should look at to navigate back to the information that’s relevant at 1st September 2001.

There are two techniques that we might use to help address this. One is for the information that’s retrieved at http://education.data.gov.uk/doc/school/12345/2001-09-01 to include some basic information about the linked resource that includes foaf:isPrimaryTopicOf links directly to the relevant versioned document about the linked resource. For example, that document should contain:

<http://education.data.gov.uk/id/school/12345/staff>
  rdfs:label "Staffing of Wildmoor Heath School"@en ;
  foaf:isPrimaryTopicOf <http://education.data.gov.uk/doc/school/12345/staff/2001-09-01> .

These links will have to be generated by the publication framework since they are calculated based on the date of the requested resource.

The other technique is to use HTTP headers to request the applicable date, as suggested by the Memento Experiment. Even with this technique, it’s still useful to have distinct URIs for the individual documents so that they can be pointed to and talked about.

Representation of Data in Named Graphs

Let’s turn to looking at the use of named graphs within a triplestore. In the government case, we’re expecting that information about schools going into a single triplestore is likely to come from multiple sources. Each source may release information at different intervals, with different temporal validity. The data from a single source will over-ride other information from that source over time, but equally data from different sources will be overlapping and contradictory.

To manage this, we split up triples into named graphs based on:

their source
their temporal validity (and their temporal relationship with other graphs)
their authoritativeness

This metadata about the named graph is recorded within the named graph itself, using voiD and other vocabularies.

In more detail:

Named Graphs over Time

Named graphs are expected to occur within a series over time. The triples within one graph will be completely replaced by the triples within another graph. The most recent graph is one that has not yet been replaced. To record this, the graphs should have associated with them:

the dates when the data in the graph is valid (only the start date is really required)
the graph(s) that the graph replaces
the graph(s) that the graph is replaced by
the date when the data in the graph was created

To avoid repetition of data within multiple graphs, graphs should be split up at the level that updates are likely to occur within the source of the data. For example, Edubase holds a database of schools. If the linked data for schools is generated based on dumps of the entire Edubase database, then there would be a separate named graph for each dump of the database. If the linked data is created more dynamically, based on updates at the level of an individual school, say, then there should be a separate series of named graphs for each school. If the updates can occur at an even finer level of granularity (eg at each record within each table within the database), then there can be separate named graphs at that level.

Named Graphs from Different Sources

Information about the same resources will come from different sources, and have gone through different levels of processing to become linked data. To allow us to provide information about the provenance of different triples, separate named graphs should be used for data from different sources. The metadata about a graph should include:

the source of the data (through dct:source)
the provenance of the data (through something more complex, yet to be finalised)

Much of the information about a particular resource will only come from one source. For example, Edubase contains the pupil census for a school while Ofsted provides inspection reports. However, there will be overlaps between the information available from different sources, such as the name and address of the school.

For any given property of a resource (such as the name of the school), there should be one source that is the authoritative source of that information; other sources are considered supplementary. Each source should therefore usually provide two series of named graphs: one of information for which they are considered the authority, and one of information for which they are not. The metadata about the graph should include a property that indicates whether the information it contains is authoritative or not.

Constructing a Graph for a Given Date

It’s extremely useful to be able to create snapshots that contain information that’s current at a particular point in time. The most useful of these is the current graph, which is the one that should be exposed as the default graph in the SPARQL endpoint offered by the triplestore.

The graph can be created by combining:

all the triples from authoritative graphs that are valid at that point in time (eg have a validity date before that point in time, and that are not replaced by a graph whose validity date is also before that point in time)
those triples from supplementary graphs for which there is no existing triple in the graph with the same subject and property

For example, there may be information available about a school from Edubase and from OFSTED, as follows (in TRiG syntax):

# graph containing data from Edubase from 2008-09-01
<http://education.data.gov.uk/data/edubase/12345/2008-09-01/authoritative> {
  <http://education.data.gov.uk/id/school/12345>
    rdfs:label "Broadmoor Primary School"@en ;
    edu:census <http://education.data.gov.uk/id/school/12345/census> .

  <http://education.data.gov.uk/id/school/12345/census>
    ... .

  <http://education.data.gov.uk/data/edubase/12345/2008-09-01/authoritative>
    a void:Dataset ;
    dct:created "2008-09-01"^^xsd:date ;
    dct:replaces <http://education.data.gov.uk/data/edubase/12345/2007-09-01/authoritative> ;
    dct:isReplacedBy <http://education.data.gov.uk/data/edubase/12345/2009-09-01/authoritative> ;
    dct:source <http://www.edubase.gov.uk/> ;
    :authoritative true .

  <http://education.data.gov.uk/data/edubase/12345/2008-09-01>
    a void:Dataset ;
    void:subset <http://education.data.gov.uk/data/edubase/12345/2008-09-01/authoritative> ;
    ... .
}

# graph containing data from Edubase from 2009-09-01; the name of the school 
# has changed (as have) the details of the census
<http://education.data.gov.uk/data/edubase/12345/2009-09-01/authoritative> {
  <http://education.data.gov.uk/id/school/12345>
    rdfs:label "Wildmoor Heath School"@en ;
    edu:census <http://education.data.gov.uk/id/school/12345/census> .

  <http://education.data.gov.uk/id/school/12345/census>
    ... .

  ... metadata about this graph ...
}

# graph containing authoritative data from Ofsted from 2008-03-01
# note that this doesn't include the name of the school
<http://education.data.gov.uk/data/ofsted/12345/2008-03-01/authoritative> {
  <http://education.data.gov.uk/id/school/12345>
    edu:inspection <http://education.data.gov.uk/doc/school/12345/inspection/2008> .

  <http://education.data.gov.uk/doc/school/12345/inspection/2008>
    ... .

  ... metadata about this graph ...
}

# graph containing supplementary data from Ofsted from 2008-03-01
# this includes the name of the school (at the time of the inspection)
<http://education.data.gov.uk/data/ofsted/12345/2008-03-01/supplementary> {
  <http://education.data.gov.uk/id/school/12345>
    rdfs:label "Broadmoor Primary School"@en ;

  ... metadata about this graph ...
}

Note that metadata about each graph is embedded in the graph itself.

In the example above, a graph for 2010-01-01 would contain:

<http://education.data.gov.uk/id/school/12345>
  rdfs:label "Wildmoor Heath School"@en ;
  edu:census <http://education.data.gov.uk/id/school/12345/census> ;
  edu:inspection <http://education.data.gov.uk/doc/school/12345/inspection/2008> .

<http://education.data.gov.uk/id/school/12345/census>
  ... .

<http://education.data.gov.uk/doc/school/12345/inspection/2008>
  ... .

It would not contain the triple:

<http://education.data.gov.uk/id/school/12345>
  rdfs:label "Broadmoor Primary School"@en ;

because this triple is only present in an authoritative form within http://education.data.gov.uk/data/edubase/12345/2008-09-01/authoritative, which is replaced by http://education.data.gov.uk/data/edubase/12345/2009-09-01/authoritative or from http://education.data.gov.uk/data/ofsted/school/12345/2008-03-01/supplementary which is a supplementary graph and can’t override the label provided by the authoritative graph.

Unanswered Questions

There are three gaps within this that need plugging.

First, how should we represent the interval during which a graph is valid? As I’ve indicated above, dct:valid doesn’t cut it because it can’t represent an interval very well (there is a Dublin Core recommended format for representing periods but it’s not going to be easy for people to process). We have work ongoing on defining intervals (by Stuart Williams) and will probably have to mint our own property to indicate the temporal validity of a named graph, given that dct:valid takes a literal rather than a resource.

Second, how should we indicate whether a graph is authoritative or not? Should this be a simple boolean switch (which will make the logic for combining graphs easier, and probably be easiest to assess) or a kind of confidence level, which might allow for missing data better?

Third, how should we represent the events that cause the replacement of one named graph with another? I think that we should be able to use the provenance vocabulary that we end up using to represent these changes, so that it’s possible to indicate whether the new information is the correction of a clerical error or an actual change to the real world thing.

And, we have to try this out. While it looks as if it might work, I won’t be confident until we’ve tried it out with some real data and some real queries. I’m also concerned that while keeping data in separate, annotated, named graphs seems like our best chance of managing versions and tracking provenance, it adds a hurdle onto the generation of linked data that might be too high, particularly for people who are just starting out.

Why Linked Data for data.gov.uk?

Jeni — Tue, 26 Jan 2010 13:10:58 +0000

data.gov.uk was finally launched to the public last week (still in beta, but now a more public beta than the beta that it’s been in for the last few months). It’s a great step forward, and everyone involved should be proud of both the amount of data that’s been made available and the website itself, which (unlike a lot of UK government IT) was developed rapidly by a small team based on open source software (and at low cost).

This is a first step on a long road.

One of the features of the UK Government’s approach to freeing data is the emphasis on using linked data. What I don’t think has really been articulated is either what that means or why we should take this approach. From what I’ve seen, developers seem to think:

linked data is a synonym for turning everything into RDF and putting it in one big triplestore, equivalent to making one big database of government data and therefore prone to exactly the same, well-known and understood problems that government has with creating huge databases
linked data requires everyone to agree to the same model and vocabulary, which means huge efforts in standardisation and ends up with something that suits no one
the UK government will be releasing all their data as linked data immediately, and in no other way
the UK government has been seduced into using linked data by academics who don’t understand anything about how the web or the real world works
the UK government has been seduced into using linked data by big businesses who stand to make a pretty penny providing services to departments that are forced to publish their data in this way

None of these are true. In fact, the UK government is committed to publishing data as linked data because they are convinced it is the best approach available for publishing data in a hugely diverse and distributed environment, in a gradual and sustainable way.

Why?

Because linked data is just a term for how to publish data on the web while working with the web. And the web is the best architecture we know for publishing information in a hugely diverse and distributed environment, in a gradual and sustainable way.

If you’re a web developer, you already know that the best APIs are RESTful APIs. That argument has been won. It means:

using (HTTP) URIs to identify resources: naming things with URIs rather than actions on those things (which are carried out using the standard set of HTTP verbs)
recognising the distinction between resources and representations of those resources: the same URI might return a different representation of the resource, such as HTML or XML or JSON
returning self-descriptive messages: being able to process representations in a manner that is obvious from the mime type
hypermedia as the engine of application state: being able to locate additional resources through the use of (typed) links

Linked data is about following these rules for publishing data. It is about using URIs to identify things, providing information at the end of those URIs that is self-descriptive, and linking those things to other things through typed links.

One of the features of this approach is that it doesn’t require any big bangs. No one planned the web: sat down and mapped out each page and its precise relations to every other page, in advance. It grew, and evolved, and continues to grow and evolve every day. It grows through individuals and institutions publishing information for their own reasons and linking to other people who have published information for their own reasons, and, because we have some fundamental standards that clients and servers understand, it All Just Works.

Standards

Did you notice how I slipped in the “because we have some fundamental standards that clients and servers understand”? One standard is obviously HTTP: that controls how clients and servers can talk to each other: it allows clients to request pages and servers to respond. Another standard is HTML: that enables browsers to display information in ways that people can understand it, and (crucially) has a known set of semantics that browsers can use to tell when something is a link, which people can navigate to find more information.

For linked data, there are two crucial standards: RDF and SPARQL. Yes, I know what you’re thinking, because believe me two years ago that would have been my reaction too, but let me explain why.

There’s one way in which publishing data isn’t like publishing documents: its model. Documents are made up of paragraphs and headings and lists and tables and so on. Data is made up of… what? Well, at its most basic, it’s things that have properties which have values. We might call the things objects or entities, and call some of the properties relations. We might even call them records with columns and values and foreign keys. But however you term them, for better or worse, we do tend to think about data in this way: thing, property, value.

So if we are going to publish data on the web, we need a standard way of expressing the data so that a client receiving the data can work out what’s a thing, what’s a property, what’s a value. And, because this is the web, what’s a link. This is the fundamental standard we need, and this is what RDF gives.

RDF is actually a model rather than a syntax. It’s a bit like the split between the DOM and HTML or XHTML. The DOM tells the browser how to render the page: the HTML or XHTML is just a syntax which the browser is able to convert into a DOM that it displays. We could imagine browsers converting wiki syntax into a DOM. Or creating a DOM based on XML and XSLT, which of course they all do.

So, RDF is like the DOM, with varying representations of RDF (XML-based, text-based, JSON-based, even HTML-based) that can be used to pass to the client the underlying model of things and properties and values (some of which are links). What the client does then is its business: clients that retrieve data aren’t browsers — they’re not all going to display the data, use the same parts of the data, or otherwise process it in the same way — but they can pull out the things, properties and values, and know which are links, and this data structure will often, with a good RDF library, map on to some natural structure within whatever programming language is being used, and make the programmer’s job easier.

Vocabularies

What we don’t want to have to define are standard ways of expressing particular data (such as data about a school) because different individuals and organisations will have completely different ways of thinking about a particular thing. A school itself will have information about uniform and open days; OFSTED about performance; Edubase about administration and pupil numbers; the PTA about after-school activities. Expecting everyone to adopt a particular standard vocabulary for describing a school is as futile as expecting everyone to adopt exactly the same page layout within their web pages, and exactly the same class names in their CSS.

But we don’t want to rule out opportunistic alignments where individuals or organisations, for whatever reason, do want to use the same vocabularies. Look at what’s happened with classes in HTML. There is absolutely no constraint on what classes people use in their HTML. But there are clusters of web pages that use some of the same classes. Websites that use microformats. Websites that adopt a particular CSS framework. Importantly, though, even where some classes are shared, it doesn’t mean that all classes are shared: adoption of a particular microformat or CSS framework doesn’t limit the rest of the page.

RDF has this balance between allowing individuals and organisations complete freedom in how they describe their information and the opportunity to share and reuse parts of vocabularies in a mix-and-match way. This is so important in a government context because (with all due respect to civil servants) we really want to avoid a situation where we have to get lots of civil servants from multiple agencies into the same room to come up with the single government-approved way of describing a school. We can all imagine how long that would take.

The other thing about RDF that really helps here is that it’s easy to align vocabularies if you want to, post-hoc. RDFS and OWL define properties that you can use to assert that this property is really the same as that property, or that anything with a value for this property has the same value for that other property. This lowers the risk for organisations who are starting to publish using RDF, because it means that if a new vocabulary comes along they can opportunistically match their existing vocabulary with the new one. It enables organisations to tweak existing vocabularies to suit their purposes, by creating specialised versions of established properties.

So the linked data web is designed to grow and evolve in exactly the same way as the human web has grown and evolve. It grows through people adding links to existing data. It grows through people creating their own vocabularies. And it evolves as links break and reform, and vocabularies combine and diverge. It is complex and messy and self-organising.

Layers

The cornerstone of the great, messy, web is the URI. URIs have two important roles:

they identify things - If two sets of data use the same URI then it’s dead easy to work out when they are talking about the same thing, for example to bring together the information published by a school with its OFSTED report with its pupil census. Spread this around to five, ten, twenty datasets from different places all using the same identifier for the school, and you have huge pool of information. And the great thing about RDF (because they also use URIs to identify properties) is that those datasets can be combined automatically without worrying about clashes, rather than through painstaking developer effort.
they provide somewhere to look for information - This is the point of using HTTP URIs, because that look-up is as simple as retrieving a document from the web. This enables programmatic, on-demand, access to the information. Developers don’t have to download huge database dumps when all they are interested in is a small fraction of that data.

But we know that of course sometimes developers do want to download huge database dumps. So we need URIs for those dumps, and ways to associate metadata with them, and ways to search them. Adopting linked data doesn’t preclude providing sets of data in larger lumps. In fact, what’s needed are ways of creating those larger datasets by bringing together the more granular linked data into lists and graphs; this is essentially what SPARQL does.

We also know that there’s a trade-off to be made between the power of URIs and the simplicity of using short, unqualified names, particularly when it comes to naming schema-level entities such as properties or classes. Most mashups that we see at the moment bring together just a few datasets, making it easy for developers to scan for naming clashes, or examine values to work out whether a particular property contains a link or not. This is the 80% of the use of data on the web that can be addressed by the 20% solution of the kind of JSON and plain old XML you see in most APIs.

But publishing with RDF can be the basis of these kinds of simple APIs, and still address the hard 20% that we will encounter quickly as we mash more data together. Any data munger knows that the main challenge of making data available in an easily accessible way is cleaning, tidying, modelling and restructuring. If that’s done into RDF then creating simple JSON, XML and even CSV is really easy. Creating middle-ware that will make the creation of these basic APIs really easy must be the top priority of this linked data effort.

Reality Check

So it’s all good, right?

No, of course it’s not all good. Just as in the early days of the human web, we face huge challenges simply getting tooling to a level where it’s easy (really easy) for government departments and local authorities to publish data as RDF and for the consumers of the data to use it. We have some patterns for publishing linked data, but, as in the early days of the human web, there’s still a lot we don’t know about the best way to make data usable by third parties.

It’s worth noting that the main challenges we face are ones that are common to all attempts to make data both open and reusable. How do we easily create structured and reusable data from presentation-oriented Excel or (worse) PDFs? How do we handle changes over time, and record the provenance of the information that we provide? How to we represent statistical hypercubes? Or location information? These are things that we will only learn by trying things out.

In the end, though, the best evidence we have for how the web of linked data will progress is the evidence of how things were for the human web. It is hard to be an early adopter, both for social reasons and technological reasons. Nothing will happen overnight, but gradually there will be network effects: more shared URIs, more shared vocabularies, making it both easier to adopt and more beneficial for everyone.

Is this a kind of faith? Maybe. I believe in the web.