rdf

Microdata + RDF

Jeni — Sun, 31 Jul 2011 19:55:44 +0000

As part of the ongoing discussion about how to reconcile RDFa and microdata (if at all), Nathan Rixham has put together a suggested Microdata RDFa Merge which brings together parts of microdata and parts of RDFa, creating a completely new set of attributes, but a parsing model that more or less follows microdata’s.

I want here to put forward another possibility to the debate. I should say that this is just some noodling on my part as a way of exploring options, not any kind of official position on the behalf of the W3C or the TAG or any other body that you might associate me with, nor even a decided position on my part.

Simplifying RDFa

As I’ve said before, RDFa, in my experience, is complicated not primarily because of the whole namespaces/CURIEs issue but because its processing model tries to be too clever. RDFa was designed to largely fit in with existing markup and turn it into embedded data “just” by adding a few attributes here and there. Thus a simple image like:

<img src="photo1.jpg">

is first marked up to indicate that it’s an image:

<img src="photo1.jpg" typeof="foaf:Image">

then to provide its license:

<img src="photo1.jpg" typeof="foaf:Image"
  rel="license" resource="http://creativecommons.org/licenses/by/2.0/">

and finally to add a title:

<img src="photo1.jpg" typeof="foaf:Image"
  rel="license" resource="http://creativecommons.org/licenses/by/2.0/"
  property="dc:title" content="A Pretty Picture">

all by adding attributes to the one <img> element. The trouble with this approach is that the rules about how statements are made become extremely complex, dependent on context (eg what other attributes are present, what the parent element has on it, what content it has) and default in ways that are hard to remember.

Even having written an RDFa parser, having written code to mark up documents with RDFa, having taught it, I still cannot write RDFa past a trivial example and be 100% sure that it will produce what I was aiming to produce.

If we were to look at really simplifying RDFa, rather than making cosmetic changes, we need to address this complexity. It would certainly mean backwards-incompatible changes, such as dropping the use of particular attributes and revising the way the processing model works, such that future RDFa processors couldn’t be used on RDFa 1.0. There are two possible ways of approaching this:

retaining some backwards compatibility, and aiming for a simplified subset of RDFa 1.0 such that RDFa 1.0 processor will still get the intended triples out of data marked up with RDFa 1.1
dropping backwards compatibility entirely and using completely different attributes, essentially creating a new language

I do not know which of these routes is the best one to take.

My instinct is that the first will be hard to do. For example, there are already certain simplifications in RDFa 1.1 — such as assuming an element with no datatype attribute is giving a string value rather than looking to see if there are any non-text-nodes in the content of the element — which lead to markup that will not be processed correctly by RDFa 1.0 processors. Perhaps that could be addressed by rewriting history: creating a RDFa 1.0 Second Edition that includes any changes that are needed to make a simple subset viable.

What I want to explore here is what the second route — using entirely different attributes from those currently used in RDFa 1.0 — might mean. I think that in this case the substantial difference between microdata and this new language would be support for that much-derided requirement: decentralised extensibility.

Adding Decentralised Extensibility to Microdata

As I discussed earlier in the week, microdata is simply not designed for use in a web where publishers might want to use multiple vocabularies to mark up the same thing for different consumers. This focus is very probably the right one for the majority of uses, where publishers address single consumers or everyone has standardised on a single vocabulary. It’s certainly an assumption that keeps the markup simple.

However, there is a larger data web out there. It’s not just browsers and search engines who might look for and process data embedded within a page. Unlike with HTML, those few, large consumers don’t have to understand a particular vocabulary for other consumers to get valuable information from it. If you operate in a world of multiple consumers with different requirements, you need decentralised extensibility. And support for decentralised extensibility is RDF’s niche as a data model, its unique selling point.

Given that a new language would have to use a different processing model from RDFa 1.0, I would suggest that it simply uses microdata’s as a starting point. Using attributes from RDFa 1.0 would only cause conflicts with RDFa 1.0 processors. Microdata processing is there, already defined, already implemented. It isn’t going to go away. And you know, it’s pretty good.

The ‘new language’ would then not so much a ‘new language’ as an enhancement on something that already exists. It would be a set of additions that augment the data that is generated from normal microdata processing with a few extra features that are useful in a world where there are multiple vocabularies for the same domain, where publishers have to provide data to multiple consumers, where an RDF view of data is useful. Call it microdata+RDF.

So what would we need to add? Well, there are three things, I think, that make microdata hard to use in a decentralised world, and make it hard to generate good RDF from microdata markup:

lack of support for multiple types
scoping of properties by type
lack of datatypes

We would need to find a way to add these for use within the RDF extracted from the microdata markup such that a basic microdata parser would still generate the same JSON, and such that microdata’s DOM API would work as specified in the microdata spec. So we can’t change the types of values that are possible in microdata’s attributes or how they’re interpreted in the DOM API.

Multiple Types

Because of the restrictions I just mentioned in not touching microdata itself, we can’t simply make itemtype take multiple URLs. We could rely on itemprop="http://www.w3.org/1999/02/22-rdf-syntax-ns#type" as a mechanism of providing types for use by RDF processors, but I think that the types of something is such a fundamental property that it makes sense to have a dedicated attribute.

I suggest itemclass. It would only be allowed on elements with an itemscope attribute and would take a space-separated set of values in exactly the same way as the itemprop attribute. The values would be turned into URIs in the same way as for the itemprop attribute, which I’ll describe below.

Microdata+RDF would add a method to the existing microdata DOM API to enable people to access items by class rather than their single type. So:

document . getItemsByClass( classes )
Returns a NodeList of the elements in the Document that create items, that are not 
part of other items, and that have one or more of the types or classes given in the 
argument.

The classes argument is interpreted as a space-separated list of classes.

Note that for simplicity, because they are interpreted in the same way within the RDF model, this returns items whose itemtype is listed in the argument list of classes as well as those whose itemclass is listed.

Within the DOM API, the itemClass IDL attribute on HTML elements would reflect the itemclass attribute.

The itemclass attribute would be ignored for the purpose of creating JSON from microdata, and only be used when creating RDF.

An example would be:

<li class="conference" itemscope itemid="/2011/oscon/"
    itemtype="http://schema.org/Event"
    itemclass="http://microformats.org/profile/hcalendar#vevent /vocab/Conference">
  ...
</li>

The JSON generated from this would look like:

{
  "type": "http://schema.org/Event" ,
  "id": "http://lanyrd.com/2011/oscon/",
  "properties": {}
}

The RDF would look like:

<http://lanyrd.com/2011/oscon>
  a <http://schema.org/Event> ,
    <http://microformats.org/profile/hcalendar#vevent> ,
    <http://lanyrd.com/vocab/Conference> ;
  .

Disambiguating Properties

To work with the RDF model, properties have to have URIs. We need to have a way of easily creating the URIs for the short-name properties without people changing their existing microdata markup.

Note: I’ve substantially revised this section following discussion with Philip Jägenstedt. Old text is struck through, new text underlined.

The way that this is done in RDFa 1.1 is through a vocab attribute, which provides a URI prefix that is concatenated to any short-name properties or types. ~~We could use the same approach here, but call the attribute itemvocab to fit in with the general method of naming attributes in microdata.~~ Using this with microdata would be tedious for users however, and it would be easy for the itemtype and itemvocab to get out of sync in weird ways.

itemvocab would only be allowed on elements with an itemscope. The scope of itemvocab would be limited to the item itself, so that it’s not forgotten when it’s needed, particularly in copy-and-paste scenarios. However, to make it easier to use I think it should probably be given a default value if it isn’t present, as follows:

Instead, the vocabulary for the properties could be identified as follows:

set vocab to the itemtype of the item if it is present, and the URL of the document if not
use a substring of vocab:
1. if vocab contains a #, the substring of vocab up to and including the #
2. otherwise, the substring of vocab up to and including its final /

For example, if you have:

<li class="conference" itemscope itemid="/2011/oscon/"
    itemtype="http://schema.org/Event"
    itemclass="http://microformats.org/profile/hcalendar#vevent /vocab/Conference">
  ...
</li>

then ~~itemvocab~~ the item vocabulary would default to http://schema.org/.

There could be an extra restriction that if itemtype is specified, itemvocab must be in the same domain as that type; that could help prevent the weird situation where in the generated RDF the properties would be interpreted as being in a completely different vocabulary from the itemtype.

~~Within the DOM API, the itemVocab IDL attribute on HTML elements would reflect the itemvocab attribute.~~

Note: the following example has been altered in place.

For example, take the following markup:

<li class="conference" itemscope itemid="/2011/oscon/"
    itemtype="http://schema.org/Event" 
    itemclass="SocialEvent BusinessEvent EducationEvent">
  <h3>
    <a itemprop="url" href="/2011/oscon/">
      <span itemprop="name">OSCON 2011</span>
    </a>
  </h3>
  <p itemprop="location" itemscope itemid="/places/portland/"
     itemtype="http://schema.org/Place">
    <span itemprop="name"><a href="/places/usa/">United States</a> / <a itemprop="url" href="/places/portland/">Portland</a></span>
  </p>
  <p class="date">
    <time itemprop="startDate" datetime="2011-07-25">25th</time>–
    <time itemprop="endDate" datetime="2011-07-29">29th July 2011</time>
  </p>
  ...
</li>

The vocabulary for the <li> element defaults to http://schema.org/ based on the value of the itemtype. The short-named properties and classes within that item are turned into URIs by pre-pending http://schema.org/ to their name. Similarly, the properties on the nested http://schema.org/Place are pre-pended with http://schema.org/Place/. The resulting RDF would be:

@prefix s: <http://schema.org/>

</2011/oscon/>
  a s:Event ,
    s:SocialEvent ,
    s:BusinessEvent ,
    s:EducationEvent ;
  s:url <http://lanyrd.com/2011/oscon/> ;
  s:name "OSCON 2011" ;
  s:location </places/portland/> ;
  s:startDate "2011-07-25"^^xsd:date ;
  s:endDate "2011-07-29"^^xsd:date ;
  .

</places/portland/>
  a s:Place ;
  s:url <http://lanyrd.com/places/portland/> ;
  s:name "United States / Portland" ;
  .

Note: see below for how the values are created in this example.

The JSON would be just the same as from a standard microdata processor; there’s no mapping to URIs for that output:

{
  "type": "http://schema.org/Event",
  "id": "http://lanyrd.com/2011/oscon/",
  "properties": {
    "url": [
      "http://lanyrd.com/2011/oscon/"
    ],
    "name": [
      "OSCON 2011"
    ],
    "location": [
      {
        "type": "http://schema.org/Place",
        "id": "http://lanyrd.com/places/portland/",
        "properties": {
          "name": [
            "United States / Portland"
          ],
          "url": [
            "http://lanyrd.com/places/portland/"
          ]
        }
      }
    ],
    "startDate": [
      "2011-07-25"
    ],
    "endDate": [
      "2011-07-29"
    ]
  }
}

Adding Datatypes

How to manage datatypes in RDF generated from microdata is something where the best approach is not at all clear to me. A couple of years ago I talked about some frustrations with RDF datatyping, and datatypes in RDF still frustrate me by being hard to use in sensible ways throughout the RDF toolchain. Nevertheless, it’s what we have.

The possibilities I can see for microdata+RDF are:

Use plain literals for everything, including URIs, equivalent to using strings as microdata does. This makes things simple for the publisher and keeps the markup in the page clean, but makes it difficult for consumers who are using RDF toolchains: they will usually have to do some kind of processing of the RDF generated from microdata+RDF to add appropriate datatypes to the values. There are two issues with this approach:
- I have a feeling that microdata+RDF processors will make up their own rules to add datatypes to the data extracted from a page (using rules like those described below and/or sniffing of values and/or using information from known built-in vocabularies), in an effort to add value for their users. But if different processors do that in different ways, we have an interoperability problem.
- In some vocabularies, the datatype of a value is not derivable from the property. The most important/common example of this is skos:notation, which uses values with different datatypes to supply different identifiers from different identification schemes for a given concept.
Assign datatypes based on the element type in the HTML. If the property value has come from a URL attribute, assume that it’s a resource rather than a literal; if the element is a <time> element, work out the datatype based on the syntax of the datetime attribute; otherwise assume it’s a string and give it a language in the case that one is specified. This gives some information but leads to a somewhat strange situation where you can mark up something as a date/time but not as a number.
Supplement the processing described in 2. with some basic datatype sniffing. Basically, if the value looks like a number or a boolean value then assign it a numeric or boolean datatype based on its syntax. This could reuse the rules for recognising different literals from Turtle. This wouldn’t be perfect; in particular, it would guess that strings that consist purely of numbers such as zip codes were numbers. I’m inclined not to go down this path.
Supplement the processing described in 2. with a itemvaltype attribute that takes a token from the list of built-in XML Schema Datatypes or the token ‘literal’. The ‘literal’ token would be used to override the normal processing of URL attributes in the case where those really should be literals rather than resources. In this design, it would be easy to create literals using one of the most usual datatypes, but not possible to use datatypes that are specific to a given vocabulary.
Supplement the processing described in 4. by allowing the itemvaltype to take either a token or a URL. The thing I don’t like about this design is that the token would be interpreted as being within the XML Schema Datatypes vocabulary rather than the vocabulary specified for itemvocab (used for tokens in itemprop and itemclass). This seems like it might turn into a source of confusion, but if we went the other way and had itemvaltype being interpreted based on itemvocab, it would be harder to give a value the more common datatypes such as numbers and boolean values.

My inclination, somewhat reluctantly as it’s the most complex, would be to use the last of these, because it provides for decentralised extensibility of datatypes, and support for decentralised extensibility is the core aim of these extensions. In other words, have a itemvaltype attribute that can hold either a token, which must be one of literal or the local name of an XML Schema datatype, or a URL. On a <time> element, this would default to the appropriate type based on the syntax of the value of the datetime attribute.

To be conformant, the itemvaltype would have to be an allowed value type for the properties given in itemprop and the value of the property must be a legal value for the datatype. (In keeping with the style of the microdata specification, the mechanisms for working out what value types are allowed and what the legal values are for non-XML Schema datatypes would be left undefined — a consuming application would look at the definition of the vocabulary.)

Within the DOM API, the itemValType IDL attribute on HTML elements would reflect the itemvaltype attribute. The value of itemvaltype wouldn’t change the types of the values returned by element.itemValue or in the JSON mapping from microdata; it would purely be used when generating RDF from that data.

For example, if someone started with some markup like:

<div itemscope itemtype="http://schema.org/AggregateOffer">
  Priced from: <span itemprop="lowPrice">$35</span>
  <span itemprop="offerCount">1938</span> tickets left
</div>

it might be supplemented with some type information like:

<div itemscope itemtype="http://schema.org/AggregateOffer">
  Priced from: <span itemprop="lowPrice" itemvaltype="http://schema.org/Price">$35</span>
  <span itemprop="offerCount" itemvaltype="integer">1938</span> tickets left
</div>

which would generate RDF like:

@prefix s: <http://schema.org/>

[] a s:AggregateOffer ;
  s:lowPrice "$35"^^s:Price ;
  s:offerCount 1938 ;
  .

(Note: Here I’m assuming that schema.org defines a http://schema.org/Price datatype which includes a currency and a number. They don’t currently.)

The JSON would still be:

{
  "type": "http://schema.org/AggregateOffer",
  "properties": {
    "lowPrice": [
      "$35"
    ],
    "offerCount": [
      "1938"
    ]
  }
}

Non-Additions

When I wrote a couple of years ago about what microdata can’t do, one of the things that I identified was not being able to express XML Literals. Having thought about this more, what’s actually missing isn’t to do with RDF, but is the ability to use the innerHTML of an element to provide a value for a property rather than its textContent.

For example, the description of an event might run over several paragraphs, or even in a single paragraph include other markup such as emphasised text, ruby markup, or links to additional information. People who are working from the DOM API can capture this information when they need it by getting the innerHTML of the element rather than its itemValue, but in the JSON mapping, the value is always the itemValue — the text content of the element.

So this is a general microdata simplifying limitation. I’d argue that we shouldn’t add any special handling to plug this hole at the microdata+RDF level. If it turns out that having values that contain markup is useful then it will be added to microdata, and the microdata+RDF mapping would then be extended to create rdf:XMLLiterals or HTML literals (for which there is no defined datatype in RDF at the moment) for such values.

Similarly, I haven’t said anything in this post about providing machine-readable values to override the text content of an element. There is an open bug about whether and how that capability might be added to HTML/microdata. I happen to think that it’s useful, but that utility isn’t limited to RDF processing. Whichever route is chosen there, I think it’s important to keep the property values used by basic microdata and microdata+RDF aligned.

Summary

To summarise, one direction that we could take in aligning microdata and RDFa would be to define an extension to microdata to add support for decentralised extensibility and the RDF data model. I think that would entail adding attributes such as:

itemclass to make it easy to define multiple types for an item
itemvocab and some default processing to provide nice mappings for short-name properties into URIs
itemvaltype and some default processing to assign datatypes to values

For publishers and consumers, a single language with optional extensions greatly simplifies the use of embedded data. Property names don’t have to be repeated or balancing acts made between different processing models.

RDFa proponents get a syntax that can be used to generate a natural RDF model against which they can build RDF-oriented APIs and map to other formats such as JSON-LD.

For microdata proponents, this approach doesn’t pollute microdata with requirements that they see as superfluous, and doesn’t change the behaviour of core microdata processors. Browsers, search engines and other consumers can continue to use the JSON output and only those who really want to support RDF need to do so.

I’m sure that there are things that I’ve missed in my outline above, issues that I haven’t thought of. But if there is to be any kind of convergence between microdata/RDFa, this layered approach seems to me to be the kind of convergence that is most likely to eventually result in one language for embedding data in HTML rather than two or three.

Note: if you prefer to comment on Google+, please add your comment to my announcement post there

My Experience of Web Standards

Jeni — Sun, 24 Jul 2011 16:24:00 +0000

One of the things that’s been niggling at the back of my mind since the schema.org announcement is how small a role search engine results plays in the wider data sharing efforts that I’m more familiar with in my work on legislation.gov.uk, and more generally how my day job experience differs from (what seem to be) more common experiences of development on the web. In this post, I’m going to talk about that experience, and about the particular problems that I see with the coexistence of microdata and RDFa as a result.

My day job (the one I actually get paid for) is web development. The site I spend most of my time and effort on is legislation.gov.uk. This deals with complex content (UK legislation) that has to be presented in multiple formats (users love PDFs of legislation). Our aim is to make the data as reusable as possible by third parties through good, RESTful, web architecture, and we want to use open standards and open source technologies as part of the UK government’s general strategy.

legislation.gov.uk is not a global website like Amazon or eBay, but it’s not small either: it covers 60,000 changing items of legislation, providing point-in-time views for many of them, and with more added every day. It’s one of the top ten most used UK Government websites, with 2 million visits (about 10-12 million page views) each month and typically about 120 requests/second during the active times of the day. Legislation might sound like a highly specialist interest, but if you search for legislation.gov.uk on Twitter you’ll see it being referenced over and over by people who want to share what the law says.

I do not by any means claim that my experience is representative of the wider web. I know that there are large numbers of sites that deal only in data, not documents, and certainly not documents with the kind of rich semantic structure that legislation has. I offer the following discussion as a data point, partly because I can’t quite believe that legislation.gov.uk is completely unique in its requirements and partly because obviously my perspective on a bunch of issues arises from this experience.

Technology Stacks

Legislation items are complex, semi-structured documents. Their natural fit is XML (well, that’s not quite true — their natural fit would be something that allowed overlapping markup — but XML is the closest that we have). So we store it in XML in a native XML database and we use an XML toolset to query it (XQuery) and transform it (XSLT) into various formats including rendering it as PDF (through XSL-FO).

Our next step for the development of the site involves looking at legislative effects. These form a graph: one item of legislation affects other items of legislation which may in turn affect other items and so on. There are all sorts of other links between items of legislation in terms of commencements, conferred powers and so on. Particularly because we already have well-thought-through URIs for legislation, the natural fit is to use RDF to represent this graph. We already offer a SPARQL endpoint for accessing some aspects of our data, but we expect to expand and develop this over the next few months and to use it as a layer under the website and exposed for reusers, in much the same way as we use the XML database.

As a government site, we have fairly strict limits on what we can do within our web pages: we have to make sure that they’re accessible by everyone who wants to view them. We aren’t able to use technologies that are only available in the latest browsers, but that’s OK because with the kind of content we deal with, we don’t have to do anything fancy anyway. So we use pretty basic HTML and CSS and Javascript, because that’s how you deliver content to end-users on the web (as well as exposing the underlying XML and RDF, to enable others to reuse the data).

In other words, we use three web stacks for delivering legislation.gov.uk:

the XML stack, which is great for single-source publishing of documents that have more semantic structures than those supported by HTML
the RDF stack, which is well-suited for metadata about things that are identified by URIs
the HTML stack, which is absolutely necessary for delivering human-accessible content on the web

What bemuses me, because of this experience, is that sometimes it appears that the narrative around these technologies is framed in terms of an exclusive choice between them. For example, @mattur asked:

It is as if, if you use XML you cannot appreciate the utility of error-handling in HTML; or if you use RDF you cannot understand the need to represent documents in XML; or if you want to utilise HTML fully, you cannot adopt RDF’s view of data on the web. That’s simply not my experience. They each have their role on the web; supporting the use of one does not necessitate rejecting the use of the others.

It’s interesting that some of the standards that are most reviled are those that arise at the intersections, where it appears that one technology is trying to encroach on the space of another:

XHTML at the border of XML and HTML
RDF/XML at the border of RDF and XML
RDFa at the border of all three

At the same time, within legislation.gov.uk, we publish XHTML (because it’s the natural output from an XML toolchain) and create and process RDF/XML (because it gives us access to that data from within the XML toolchain). We use a small bit of RDFa in the XHTML to indicate the rights under which our information is avaialble, and don’t yet, but are thinking about using RDFa to mark up non-document semantics within our XML (to enable the XML markup to focus on the document structures that it’s good at). For all their imperfections, these intersection technologies are useful for managing cross-overs; the problems arise when they overstep their remit and people start to think that all HTML must be XHTML or all XML must be RDF/XML or all RDF must be RDFa.

Sharing Scenarios

The second thing that I wanted to explore is the experience from legislation.gov.uk of what it’s like to be a publisher who actively wants to share their data. We need to operate simultaneously at three levels in our data sharing efforts.

Large-Scale Consumer-Driven Data Sharing

The first target for our data sharing efforts are the search engines. Obviously we’re not selling anything, but we want people to be able to locate legislation easily when they want it, and we want people who have done the search to be able to see some information about the legislation so that they know that they’ve located the right item.

This is large-scale consumer (search engine) driven data sharing, typified by schema.org and Facebook’s Open Graph Protocol (OGP). There are a few very big data consumers (Google, Microsoft, Yahoo!, Facebook etc) who need to consume data from large numbers of data providers. These consumers obviously can’t understand everything, so they determine and document what syntaxes and vocabularies they do understand and expect publishers to follow.

The benefits that publishers get from a particular consumer determines which syntax/vocabulary they use; publishers who are particularly keen to show up prettily within search results will target schema.org whereas those who want to be sharable within Facebook will target OGP. Many publishers will want to target both. There is probably a driver towards eventual convergence:

publishers might push back about inserting two lots of very similar data in their pages
consumers might want to include data from publishers who haven’t specifically targeted them

although there’s likely to be a period where they coexist, much as there was for VHS and Betamax (and V2000, I know, dad) during the early days of video players.

As I discussed previously, these large-scale consumers will be driven by the data that they find in the wild, in all its messy variety. They get relatively little benefit directly from using a generic syntax, as they are really interested in only a few, pretty generic, vocabularies for which they have hardwired processing. Indirectly, adopting a generic syntax has benefits in that publishers might find it easier to find tools that enable them to generate it, tutorials about how to use it, and feel that they aren’t being quite as locked in to something proprietary. However, rejecting data that isn’t marked up properly using that syntax has no benefit for consumers except in so far as it makes them feel that they are being good community members.

This is the pattern we see with schema.org (which accepts microdata but, based on its documentation, won’t reject data that isn’t fully compliant with it) and with OGP (which accepts a subset of RDFa but doesn’t reject data that hasn’t got prefixes properly bound, for example).

Another point to mention is that there is very little trust in this scenario. The communication between consumers and publishers is very limited, and the consumers will want to protect themselves against accidental or malicious errors that are evident in mismatches between explicit metadata and that which is parsed from the visible content of the page.

The parallels to HTML and browser vendors are very strong in this type of data sharing.

Small-Scale Consumer-Driven Data Sharing

A second type of data sharing is again driven by consumers, but this time at a lot smaller and more specialised scale. For legislation.gov.uk, these are services such as GLIN, which is a global legislation registry. Other examples are the recent work that we’ve done to publish UK Government organograms or Chris Taggart’s Open Election Data project. In these cases, there’s a single, relatively small and specialised consumer and a small number of publishers which are closely coordinated together.

As in the large-scale case, the consumer ultimately determines the syntax/vocabulary that it recognises, and communicates that to the publishers. However, small-scale consumers typically have close coordination with the publishers, which has a number of side-effects:

consumers may be more able to both apply pressure to and help publishers to do well in their markup
publishers have the opportunity to feed back directly to the consumer any suggestions that they have about changes to the syntax/vocabulary
publishers are likely to gain an immediate and tangible benefit from their cooperation, such as visualisations of their data that they otherwise wouldn’t have seen

Another noteworthy point about small-scale consumers is that they’re unlikely to have the engineering capability to create a custom parser for a particular syntax, but will instead want to use something off-the-shelf to extract data from pages and into their own backend systems. This, coupled with the closer coordination with publishers, means that they’re much more likely to stick to a specification, assuming that the off-the-shelf tools do.

Publisher-Driven Data Sharing

The final type of data sharing is driven by publishers. At legislation.gov.uk, we’re motivated to make our data available for reuse for transparency/accountability reasons (to help citizens understand the law), efficiency reasons (to help parliament and government departments to publish new legislation better) and economic reasons (to foster innovation in legal publishing). We don’t have any individual consumers in mind when we publish our data, but have found that simply by publishing it well, we foster reuse.

In this case, we as publishers are highly motivated to ensure that the data we publish is easily parsed with something off-the-shelf, since that lowers the barrier for potential consumers. Publishers like us are very likely to have unique, specialised, content and need to use a vocabulary that fits closely to our internal data structures as this lowers implementation cost. Consumers can also trust publishers like us: we simply have no motivation to lie in the data that we provide for reuse.

Mixed Markup

As I’ve outlined above, publishers like legislation.gov.uk need to target several potential consumers at the same time:

large-scale consumers such as search engines
small-scale consumers that provide us with a useful service
specialist consumers that are interested specifically in our data

We cannot use a single vocabulary for all these different purposes. (Well, we could write our own vocabulary and describe mappings to other vocabularies using RDFS, but search engines wouldn’t read it.)

We must therefore use a mix of vocabularies:

generic vocabularies about things that search engines care about
specialised vocabularies for particular small consumers
site-specific vocabularies for sharing our unique data

It’s repetitive, but it’s manageable so long as we have a syntax that enables us to say that an item of legislation is a http://scheme.org/CreativeWork and a http://purl.org/dc/dcmitype/Text and a http://www.legislation.gov.uk/def/legislation/Legislation and allows us to give multiple properties the same value.

The way things are going at the moment, we might well end up having to use multiple syntaxes on the same page, as some consumers understand microdata, others consume RDFa, and still others will parse microformats. This leads to more repetition: adding itemprop for microdata, property for RDFa and specialised class attributes for microformats. But worse (much worse), each of the syntaxes uses a different parsing model to create an entity-property-value structure, so not only do we have to learn substantially different markup patterns but our pages quickly become some kind of hideous polyglot mess trying to balance between them.

Looking Forward

As I said at the start, I’m fairly sure that my experience at legislation.gov.uk isn’t representative of the wider web, but I don’t have a clear idea about just how unrepresentative it is, in terms of technology use or motivations around data sharing. When I read my twitter stream or blogs, there’s a massive sampling bias, both in terms of who I follow and what I read, but also about who talks about what they’re doing. (I’m reminded of Jeff Atwood’s post on the Two Types of Programmers: the vast majority of web developers don’t make a noise about what they do.)

Taking part in web standardisation today often feels like being part on an ongoing cold war between distinct camps rather than a community working towards common aims. The underlying question seems to be “who’s side are you on?” Every decision and activity is cast as a victory or defeat. Time is wasted on attack and defence, or on raking over past slights and stupidities, rather than on progress. Valid criticism from outside a group cannot be listened to for fear of giving ground, cannot be made within a group where it seems like betrayal.

It is the Robbers Cave Experiment played out in web standards. As a psychologist, I find it fascinating. As a developer, and particularly one who doesn’t self-identify with any single group, it is frustrating. As a TAG member, trying to work for the longer-term good of the web, it is worrying, because situations of intergroup conflict lead to groupthink and non-optimal solutions.

As I described above, a non-optimal outcome seems to be the most likely result of the particular microdata vs RDFa conflict for us at legislation.gov.uk. While I know we are not generally representative, I believe that it will be similarly bad for other developers: publishers, consumers and tool implementers.

This is a problem for all who want to foster data sharing on the web using open standards; it is not one that any one group can fix on their own. It’s my hope that a balanced task force of individuals with a variety of experience and backgrounds can provide a focus for us all to work together to solve it. If we can’t, then we have let our prejudice and bias overcome our judgement.

Getting Started with RDF and SPARQL Using Sesame and Python

Jeni — Tue, 25 Jan 2011 17:27:24 +0000

My previous post talked about how to install 4store as a triplestore, and use the Ruby library RDF.rb in order to process RDF extracted from that store. This was a response to Richard Pope’s Linked Data/RDF/SPARQL Documentation Challenge which asks for documentation of how to install a triplestore, load data into it, retrieve it using SPARQL and access the results as native structures using Ruby, Python or PHP.

I quite enjoyed writing the last one, so I thought I’d try again. As before, I am on Mac OS X, but this time I’m going to use Python, which I have not programmed in before. I like a challenge. You might not like the results!

Sesame

This time, I’m going to use Sesame, as I was told by John Sheridan that it was so easy to install that even he, a civil servant, could do it!

Sesame needs a Java servlet container. I’m using Tomcat because I have some experience with it, but you could use something like Jetty if you prefer. I had a bit of trouble getting Tomcat 6 to install, but that might just have been because it has a lot of dependencies and I wasn’t patient enough. It might be worth upgrading your existing ports and getting verbose output so you know there’s activity as you install Tomcat:

$ sudo port upgrade outdated
$ sudo port -v install tomcat6

This installs Tomcat 6 in /opt/local/share/java/tomcat6.

While that’s happening, get Sesame from its download page. I got hold of openrdf-sesame-2.3.2-sdk.tar.gz. The files we actually need are the .wars so we can just extract them and put them in the webapps directory within Tomcat:

$ tar -zxvf openrdf-sesame-2.3.2-sdk.tar.gz openrdf-sesame-2.3.2/war/*.war
$ sudo cp openrdf-sesame-2.3.2/war/*.war /opt/local/share/java/tomcat6/webapps/

Then startup Tomcat:

$ sudo /opt/local/share/java/tomcat6/bin/tomcatctl start

All being well, you should see Tomcat doing some initial setup:

conf_setup.sh: file conf/catalina.policy is missing; copying conf/catalina.policy.sample to its place.
conf_setup.sh: file conf/catalina.properties is missing; copying conf/catalina.properties.sample to its place.
conf_setup.sh: file conf/server.xml is missing; copying conf/server.xml.sample to its place.
conf_setup.sh: file conf/tomcat-users.xml is missing; copying conf/tomcat-users.xml.sample to its place.
conf_setup.sh: file conf/web.xml is missing; copying conf/web.xml.sample to its place.
conf_setup.sh: file conf/setenv.local is missing; copying conf/setenv.local.sample to its place.
Starting Tomcat.... started. (pid 20064)

Now have a look at http://localhost:8080/openrdf-sesame. If you’re like me, you’ll get some error messages because the user that Tomcat is running under (www) isn’t able to create or write to a logging directory that it wants to create, in my case /Users/Jeni/Library/Application Support/Aduna/OpenRDF Sesame/logs. This turns out to be partly caused by permissions issues and partly caused by the spaces in the filename. To get around it, create a data directory for Aduna that doesn’t have spaces in the filename, and change its ownership to www. In my case, I chose /opt/local/var/aduna.

$ sudo mkdir -p /opt/local/var/aduna
$ sudo chown www:www /opt/local/var/aduna

Then edit Tomcat’s setenv.local file which in my environment is at /opt/local/share/java/tomcat6/conf and add a line that sets the info.aduna.platform.appdata.basedir to that directory, like this:

export JAVA_OPTS='-Dinfo.aduna.platform.appdata.basedir=/opt/local/var/aduna'

Restart Tomcat:

$ sudo /opt/local/share/java/tomcat6/bin/tomcatctl restart

Then navigate again to http://localhost:8080/openrdf-sesame and you should see the Welcome page:

As you can see, this recommends using the Workbench for managing the repositories. If you open that, at http://localhost:8080/openrdf-workbench.

We’ll use this Workbench to create a new repository for our data, which I’ll call reference. Click on New Repository from the left hand navigation and fill in the form. I’m just going to use the default in-memory RDF store because I’m only using a little data; the other options (using MySQL or PostgreSQL stores) would be useful if I were creating something more permanent. See the Sesame User Guide for information about those.

So fill in the form to create a new repository with the id reference and whatever title you like:

Click Next and there will be a couple more options to select; I just used the default for these:

Click Create and you will see a summary of the new repository that you’ve created:

Loading Data

I’m going to use the same data as I did before:

http://source.data.gov.uk/data/reference/organogram-co/2010-10-31/index.rdf

You can add data to a Sesame repository in a browser through the Workbench by uploading a file, pointing Sesame at a URL or pasting in some RDF that you want to load. There are also Java bindings for adding data to Sesame. But neither of those are any good to us as we need programmatic access.

So we will use the HTTP method. I want to add some statements to the reference repository in the graph (what Sesame calls “context”) http://source.data.gov.uk/data/reference/organogram-co/2010-10-30, which amounts to an HTTP PUT on the repository’s statements with that context.

Now I don’t know much at all about Python, but it looks as though the built-in library urllib2 doesn’t support PUT and there’s a better HTTP library available in httplib2. MacPorts supports various different packages for httplib2 with different versions of Python. Now there only seems to be a package for rdflib, which we’ll use later, for Python 2.6, so we’ll go for py26-httplib2, which will bring in Python 2.6 with it just in case.

$ sudo port install py26-httplib2

After running this, if you want to actually use it you will need to install the python_select port and choose Python 2.6:

$ sudo port install python_select
$ sudo python_select python26

Then open up another Terminal window or tab (because the change won’t have affected your old one) and check what version of Python you’re running:

$ python --version
Python 2.6.6

With the httplib2 library in place, it’s time for a Python script (load-rdf-into-sesame.py) to do the PUTting:

import urllib
import httplib2

repository = 'reference'
graph      = 'http://source.data.gov.uk/data/reference/organogram-co/2010-06-30'
filename   = '/Users/Jeni/Downloads/index.rdf'

print "Loading %s into %s in Sesame" % (filename, graph)
params = { 'context': '<' + graph + '>' }
endpoint = "http://localhost:8080/openrdf-sesame/repositories/%s/statements?%s" % (repository, urllib.urlencode(params))
data = open(filename, 'r').read()
(response, content) = httplib2.Http().request(endpoint, 'PUT', body=data, headers={ 'content-type': 'application/rdf+xml' })
print "Response %s" % response.status
print content

Run the script from the command line:

$ python load-rdf-into-sesame.py

and you should get just get:

Loading /Users/Jeni/Downloads/index.rdf into http://source.data.gov.uk/data/reference/organogram-co/2010-06-30 in Sesame
Response 204

which isn’t particularly helpful (well, the 204 response tells us it worked), but you can then check http://localhost:8080/openrdf-workbench/repositories/reference/contexts and you should see that there is a new context of http://source.data.gov.uk/data/reference/organogram-co/2010-06-30:

Click on the context and it will take you to a list of (some of) the triples in that graph:

One of the nice things about Sesame is that the Workbench provides you with ways of exploring the data that you have loaded. On the left navigation bar there are ways of listing the types of the entities described in the data:

from which you can find instances of that type, for example of org:Organization:

and then the statements about a particular instance, for example DirectGov:

Running a Query

Onto running a query directly. This is done on Sesame in exactly the same way as it was done on 4store in my last walkthrough: by HTTP POSTing a query to the SPARQL endpoint. Sesame’s page for testing queries on the reference repository is at http://localhost:8080/openrdf-workbench/repositories/reference/query and we’ll use the basic one that lists types of things that are described within the data:

SELECT DISTINCT ?type 
WHERE { 
  ?thing a ?type .
} 
ORDER BY ?type

Paste that into the textarea that’s provided on http://localhost:8080/openrdf-workbench/repositories/reference/query so it looks like:

and you get an HTML page:

That’s nice for humans, but not so good for computers. When we request the results of this query programmatically, we’ll want to make sure that we specifically ask for the query results in either XML or JSON.

I went the XML route last time, so let’s mix it up a bit and try processing the JSON results of a SPARQL query this time, as it’s really easy to access using the json module in Python. So, we need to POST the query, ensuring that we set the Accept header to application/sparql-results+json, and then process the results as JSON. Here is find-rdf-types.py

import urllib
import httplib2
import json

query = 'SELECT DISTINCT ?type WHERE { ?thing a ?type . } ORDER BY ?type'
repository = 'reference'
endpoint = "http://localhost:8080/openrdf-sesame/repositories/%s" % (repository)

print "POSTing SPARQL query to %s" % (endpoint)
params = { 'query': query }
headers = { 
  'content-type': 'application/x-www-form-urlencoded', 
  'accept': 'application/sparql-results+json' 
}
(response, content) = httplib2.Http().request(endpoint, 'POST', urllib.urlencode(params), headers=headers)

print "Response %s" % response.status
results = json.loads(content)
print "\n".join([result['type']['value'] for result in results['results']['bindings']])

Run it:

$ python find-rdf-types.py

and you get:

POSTing SPARQL query to http://localhost:8080/openrdf-sesame/repositories/reference
Response 200
http://purl.org/linked-data/cube#DataSet
http://purl.org/linked-data/cube#DataStructureDefinition
http://purl.org/linked-data/cube#Observation
http://purl.org/net/opmv/ns#Artifact
http://purl.org/net/opmv/ns#Process
http://purl.org/net/opmv/types/google-refine#OperationDescription
http://purl.org/net/opmv/types/google-refine#Process
http://purl.org/net/opmv/types/google-refine#Project
http://rdfs.org/ns/void#Dataset
http://reference.data.gov.uk/def/central-government/AssistantParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/CivilServicePost
http://reference.data.gov.uk/def/central-government/Department
http://reference.data.gov.uk/def/central-government/DeputyDirector
http://reference.data.gov.uk/def/central-government/DeputyParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/Director
http://reference.data.gov.uk/def/central-government/DirectorGeneral
http://reference.data.gov.uk/def/central-government/ParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/PermanentSecretary
http://reference.data.gov.uk/def/central-government/PublicBody
http://reference.data.gov.uk/def/central-government/SeniorAssistantParliamentaryCounsel
http://reference.data.gov.uk/def/intervals/CalendarDay
http://www.w3.org/2000/01/rdf-schema#Class
http://www.w3.org/ns/org#Organization
http://www.w3.org/ns/org#OrganizationalUnit
http://xmlns.com/foaf/0.1/Person

This is the same set of types as that given through the HTML browse interface. Note that the JSON results themselves look like:

{
  "head": {
    "vars": [ "type" ]
  }, 
  "results": {
    "bindings": [
      {
        "type": { "type": "uri", "value": "http:\/\/purl.org\/linked-data\/cube#DataSet" }
      }, 
      {
        "type": { "type": "uri", "value": "http:\/\/purl.org\/linked-data\/cube#DataStructureDefinition" }
      }, 
      {
        "type": { "type": "uri", "value": "http:\/\/purl.org\/linked-data\/cube#Observation" }
      }, 
      ...
    ]
  }
}

Each of the items within the bindings array contains a set of bindings for the variables in the SPARQL query. This closely matches the XML format.

Processing RDF

Now we get onto the part of this where we look at specific libraries for RDF support in Python. The most popular library is rdflib, which you can install using MacPorts:

$ sudo port install py26-rdflib

The SPARQL query we’ll try this time uses a CONSTRUCT query, which creates RDF, rather than a SELECT query, which as we’ve seen can create either XML or JSON. For example, try the query:

PREFIX foaf: <http://xmlns.com/foaf/0.1/>

CONSTRUCT {
  ?person 
    a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
} WHERE { 
  ?person a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
}

This gets all the information in the data about the individuals for whom names have been supplied in the data, as RDF. Again, Sesame will display this as HTML when you try doing it, but you can choose a different format from the drop-down menu at the top of the Query Result display:

When you’re not accessing using a browser, by default Sesame serves up its results in TriG format, which isn’t particularly appropriate for the results of CONSTRUCT queries as we don’t need multiple graphs. We’ll request N-Triples as that’s something rdflib can understand. Sesame 2 uses the content type text/plain for N-Triples, so we can request this type by setting the Accept header:

params = { 'query': query }
headers = { 
  'content-type': 'application/x-www-form-urlencoded', 
  'accept': 'text/plain' 
}
(response, content) = httplib2.Http().request(endpoint, 'POST', urllib.urlencode(params), headers=headers)

We then need to parse this Turtle response into a rdflib.Graph object:

graph = rdflib.ConjunctiveGraph()
graph.parse(rdflib.StringInputSource(content), format="nt")

We then need to get information out of that graph, which rdflib isn’t particularly good at. So let’s use RDFAlchemy instead. That can be installed using easy_install:

$ sudo easy_install-2.6 rdfalchemy

RDFAlchemy can be used to map RDF graphs onto Python object structures in a fairly straight-forward manner. Basically, you define the namespaces of the vocabularies that you want to use, then some classes for the kinds of things that you have in the data, with properties that map onto properties in the RDF. Then you set the rdfSubject.db to the source of the data (which can be an rdflib Graph as above) and can query on it. Here’s an example:

FOAF = rdflib.Namespace('http://xmlns.com/foaf/0.1/')
RDF = rdflib.Namespace('http://www.w3.org/1999/02/22-rdf-syntax-ns#')

class Person(rdfalchemy.rdfSubject):
  rdf_type = FOAF.Person
  name = rdfalchemy.rdfSingle(FOAF.name)
  mbox = rdfalchemy.rdfSingle(FOAF.mbox)

rdfalchemy.rdfSubject.db = graph
stott = Person.get_by(name='Andrew Stott')
print "Andrew Stott's email address: %s" % stott.mbox.n3()

RDFAlchemy adds both get_by() and filter_by() methods on the descriptor classes that you define, to get a single item that matches a query or a list of items, respectively.

The full script for ‘get-names-and-emails.py’ is:

import urllib
import httplib2
import rdflib
import rdfalchemy

query = """PREFIX foaf: <http://xmlns.com/foaf/0.1/>

CONSTRUCT {
  ?person
    a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
} WHERE {
  ?person a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
}"""
repository = 'reference'
endpoint = "http://localhost:8080/openrdf-sesame/repositories/%s" % repository

print "POSTing SPARQL query to %s" % endpoint
params = { 'query': query }
headers = { 
  'content-type': 'application/x-www-form-urlencoded', 
  'accept': 'text/plain' 
}
(response, content) = httplib2.Http().request(endpoint, 'POST', urllib.urlencode(params), headers=headers)
print "Response %s" % response.status

graph = rdflib.ConjunctiveGraph()
graph.parse(rdflib.StringInputSource(content), format="nt")

print "Loaded %d triples" % len(graph)

FOAF = rdflib.Namespace('http://xmlns.com/foaf/0.1/')
RDF = rdflib.Namespace('http://www.w3.org/1999/02/22-rdf-syntax-ns#')

class Person(rdfalchemy.rdfSubject):
  rdf_type = FOAF.Person
  name = rdfalchemy.rdfSingle(FOAF.name)
  mbox = rdfalchemy.rdfSingle(FOAF.mbox)

rdfalchemy.rdfSubject.db = graph
stott = Person.get_by(name='Andrew Stott')
print "Andrew Stott's email address: %s" % stott.mbox.n3()

Run this script with:

$ python get-names-and-emails.py

and you get the result:

No handlers could be found for logger "rdflib.Literal"
POSTing SPARQL query to http://localhost:8080/openrdf-sesame/repositories/reference
Response 200
Loaded 459 triples
Andrew Stott's email address: <mailto:andrew.stott@cabinet-office.gsi.gov.uk>

The first line is apparently a side-effect of rdflib/RDFAlchemy weirdness which you don’t need to worry about. The rest shows that the search was successful; the call to the .n3() call on the email address is only necessary because it is a resource rather than a literal, and therefore doesn’t get converted to a particularly readable string otherwise.

Conclusions

So there you have it, another walkthrough of setting up a local triplestore, loading in data and accessing that data programmatically using SPARQL queries, this time using Sesame and Python rather than 4store and Ruby.

This walkthrough took me a fair bit longer to do than the previous one, for several reasons:

I’ve done almost no previous programming with Python (as you can probably tell), so every little thing took ages to work out — you know you’re in trouble when you’re Googling for string concatenation code! I’m very happy to accept corrections and improvements, which I’ll include in the above.
I spent a lot of time faffing around with different Python versions, opting for the latest and then finding that the libraries that I wanted to use weren’t available for that version and so on. I eventually ended up with Python 2.6; the code above may or may not work with any other versions.
Setting up Sesame 2 was pretty frustrating: first Tomcat wouldn’t work, then Jetty wouldn’t work, and finally I did get Tomcat working and then had the issue with the log directory, as I described above. Once I’d changed the data directory things worked very smoothly.
I thought rdflib was going to be enough to work with RDF in Python, but really it isn’t (if you want to get data out as well as put data in), so I had to find something else.
The documentation for rdflib and RDFAlchemy isn’t as comprehensive as the documentation for RDF.rb, especially if you’re not familiar with Python, so it took me a bit longer to work out how to do things with those particular libraries.
I took a lot more screenshots!

Again, I haven’t followed Richard’s steps to the letter; in particular I haven’t used a package to get data out of (or into) Sesame: I’ve just done it through HTTP calls. I did it this way deliberately because I think it’s a really important feature of triplestores that you can query them through a common interface: SPARQL. It means that you can take the Python code here and use it against 4store or another triplestore with only a change to the value of the endpoint variable, and similarly take the Ruby code from my previous walkthrough and use it against Sesame. Your code is not tied to a particular implementation or API; you “only” have to learn SPARQL and you’re away.

If you prefer something a little more tightly bound, however, RDFAlchemy does have some targeted Sesame support, as does RDF.rb for that matter. These can help with the management of the data within the repository as well as querying it.

Another thing that’s worth pointing out is that 4store and Sesame have completely different (HTTP-based) interfaces for getting data into stores, and that rdflib/RDFAlchemy and RDF.rb have completely different ways of loading data into in-memory graphs, querying it, and getting information from the results, quite aside from the obvious language-based differences that you’d expect.

On the SPARQL side, there are some efforts within the W3C to define a uniform HTTP protocol for managing RDF graphs and of course there’s SPARQL 1.1 Update. There are glimmers of hope for a standard RDF API, as I’ve argued for recently, but I gather that this effort will be focused on client-side developers, ie that it is really a standard RDF API for Javascript, which I think is a wasted opportunity: I would have been faster in this task if I’d been able to use familiar methods, and I wouldn’t have been so dependent on the documentation provided by the author of a particular library.

Anyway, hopefully my tramping this path will make it easier for those who follow.

Getting Started with RDF and SPARQL Using 4store and RDF.rb

Jeni — Sat, 15 Jan 2011 19:17:57 +0000

Updated to include some of Arto Bendicken’s recommendations.

This post is a response to Richard Pope’s Linked Data/RDF/SPARQL Documentation Challenge. In it, he asks for documentation of the following steps:

Install an RDF store from a package management system on a computer running either Apple’s OSX or Ubuntu Desktop.

Install a code library (again from a package management system) for talking to the RDF store in either PHP, Ruby or Python.

Programatically load some real-world data into the RDF datastore using either PHP, Ruby or Python.

Programatically retrieve data from the datastore with SPARQL using using either PHP, Ruby or Python.

Convert retrieved data into an object or datatype that can be used by the chosen programming language (e.g. a Python dictionary).

I’ve been told so many time how RDF sucks for mainstream developers that it was the main point of my TPAC talk late last year. I think that this is a great motivating challenge for improving not only the documentation of how to use RDF stores and libraries but how to improve their generally installability and usability for developers as well.

Anyway, I thought I’d try to get as far as I could to see just how bad things really are. I am on Mac OS X, and I’m going to use Ruby (although I don’t really know it all that well, so please forgive my mistakes). I’ll breeze on through as if everything is hunky dory, but there are some caveats at the end.

4store

I’m going to use 4store because it’s really easy to install on the Mac. If you want to install it on Ubuntu, there’s a package available. For a Mac, it’s a matter of going to the list of Mac downloads, downloading the most recent version, opening the .dmg and installing the 4store application by dragging it into your Applications folder.

When you run the 4store application you get a command line prompt. To set up and start a triplestore called ‘reference’ with a SPARQL endpoint running on port 8000, type the following commands:

$ 4s-backend-setup reference
$ 4s-backend reference
$ 4s-httpd -p 8000 reference

If you then navigate to http://localhost:8000/ you should see the following:

Don’t let the title ‘Not found’ put you off. The fact you get a response means that it’s working.

Loading Data

First, find some data to load. A good place for government RDF data is http://source.data.gov.uk/data/ for example. I downloaded

http://source.data.gov.uk/data/reference/organogram-co/2010-10-31/index.rdf

There are several ways of importing data into 4store using the command line. Yves Raimond has created a Ruby gem for doing so programmatically. There’s also rdf-4store from Fumihiro Kato which ties into the RDF.rb library which I’ll use later on.

However, if you use the SPARQL server then it’s just an HTTP PUT call, which of course you can do in any language you like (every language has support for making HTTP requests, right?) without the need to install any store-specific packages. However, since we’ll be doing a lot of HTTP requests, it’s useful to have a library that can make them simple. There are plenty to choose from for Ruby. I chose rest-client:

$ sudo gem install rest-client

With that, I wrote the following little Ruby script called ‘load-data-into-4store.rb’:

#!/usr/bin/env ruby
require 'rubygems'
require 'rest_client'

filename = '/Users/Jeni/Downloads/index.rdf'
graph    = 'http://source.data.gov.uk/data/reference/organogram-co/2010-06-30'
endpoint = 'http://localhost:8000/data/'

puts "Loading #{filename} into #{graph} in 4store"
response = RestClient.put endpoint + graph, File.read(filename), :content_type => 'application/rdf+xml'
puts "Response #{response.code}: 
#{response.to_str}"

Run the script from the command line:

$ ruby load-rdf-into-4store.rb

and you should get the response:

Sending PUT /data/http://source.data.gov.uk/data/reference/organogram-co/2010-06-30 to localhost:8000
Response 201: 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head><title>201 imported successfully</title></head>
<body><h1>201 imported successfully</h1>
<p>This is a 4store SPARQL server.</p><p>4store v1.0.5</p></body></html>

You can then check http://localhost:8000/status/size/ and you should see that there are now some triples in the store:

Running a Query

The next step is to query that data using SPARQL. Running SPARQL queries is just a matter of HTTP POSTing a query to the SPARQL endpoint. 4store provides a page that you can use to test out queries at http://localhost:8000/test/ so perhaps we should do that before diving into the Ruby code. The easy one to start with is just one that returns a list of the types of things that are described within the data:

SELECT DISTINCT ?type 
WHERE { 
  ?thing a ?type .
} 
ORDER BY ?type

Paste that into the textarea that’s provided on http://localhost:8000/test/ so it looks like:

and you get some XML:

<?xml version="1.0"?>
<sparql xmlns="http://www.w3.org/2005/sparql-results#">
  <head>
    <variable name="type"/>
  </head>
  <results>
    <result>
      <binding name="type"><uri>http://purl.org/linked-data/cube#DataSet</uri></binding>
    </result>
    <result>
      <binding name="type"><uri>http://purl.org/linked-data/cube#DataStructureDefinition</uri></binding>
    </result>
    ...
  </results>
</sparql>

SELECT queries like this one (which are the most common kind of query to run to simply extract data) return SPARQL Query Results XML Format by default, so there’s no need to get hold of a specialised library for processing the results: you just need something to process XML.

For Ruby, I’m choosing Nokogiri as I’ve heard good things about it. To install:

$ sudo port install libxml2 libxslt
$ sudo gem install nokogiri

So now we just need a script that will run this query, process the results as XML, and do something with them. Call it ‘find-rdf-types.rb’:

#!/usr/bin/env ruby
require 'rubygems'
require 'rest_client'
require 'nokogiri'

query = 'SELECT DISTINCT ?type WHERE { ?thing a ?type . } ORDER BY ?type'
endpoint = 'http://localhost:8000/sparql/'

puts "POSTing SPARQL query to #{endpoint}"
response = RestClient.post endpoint, :query => query
puts "Response #{response.code}"
xml = Nokogiri::XML(response.to_str)

xml.xpath('//sparql:binding[@name = "type"]/sparql:uri', 'sparql' => 'http://www.w3.org/2005/sparql-results#').each do |type|
  puts type.content
end

Run it:

$ ruby find-rdf-types.rb

and you get:

POSTing SPARQL query to http://localhost:8000/sparql/
Response 200
http://purl.org/linked-data/cube#DataSet
http://purl.org/linked-data/cube#DataStructureDefinition
http://purl.org/linked-data/cube#Observation
http://purl.org/net/opmv/ns#Artifact
http://purl.org/net/opmv/ns#Process
http://purl.org/net/opmv/types/google-refine#OperationDescription
http://purl.org/net/opmv/types/google-refine#Process
http://purl.org/net/opmv/types/google-refine#Project
http://rdfs.org/ns/void#Dataset
http://reference.data.gov.uk/def/central-government/AssistantParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/CivilServicePost
http://reference.data.gov.uk/def/central-government/Department
http://reference.data.gov.uk/def/central-government/DeputyDirector
http://reference.data.gov.uk/def/central-government/DeputyParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/Director
http://reference.data.gov.uk/def/central-government/DirectorGeneral
http://reference.data.gov.uk/def/central-government/ParliamentaryCounsel
http://reference.data.gov.uk/def/central-government/PermanentSecretary
http://reference.data.gov.uk/def/central-government/PublicBody
http://reference.data.gov.uk/def/central-government/SeniorAssistantParliamentaryCounsel
http://reference.data.gov.uk/def/intervals/CalendarDay
http://www.w3.org/2000/01/rdf-schema#Class
http://www.w3.org/ns/org#Organization
http://www.w3.org/ns/org#OrganizationalUnit
http://xmlns.com/foaf/0.1/Person

So we can see that the dataset contains information that include statistical data using the data cube vocabulary, provenance information using OPMV (Open Provenance Model Vocabulary), some information about organisations using org, some data.gov.uk-specific vocabulary, and people using FOAF.

Processing RDF

Sometimes it can be useful to get non-tabular data out of SPARQL. At that point, rather than using SELECT queries, you will want to use a CONSTRUCT query, which creates RDF. For example, try the query:

PREFIX foaf: <http://xmlns.com/foaf/0.1/>

CONSTRUCT {
  ?person 
    a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
} WHERE { 
  ?person a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
}

This gets all the information in the data about the individuals for whom names have been supplied in the data, as RDF.

Although the response is RDF/XML, you definitely do not want to process it as XML. Instead, you need a proper RDF library. Fortunately, there’s a good one for Ruby in RDF.rb. You can install it and a bunch of extra plugins that make it easy to deal with RDF in all its guises using:

$ sudo gem install linkeddata

This lets us pick out an appropriate parser based on the Content-Type of the response, and load the results of the SPARQL query into an in-memory RDF::Graph:

response = RestClient.post endpoint, :query => query
content_type = response.headers[:content_type][/^[^ ;]+/]
puts "Response #{response.code} type #{content_type}"

graph = RDF::Graph.new
graph << RDF::Reader.for(:content_type => content_type).new(response.to_str)

We can perform subsequent queries over that graph, for example just to extract names and telephone numbers and put them into a dictionary:

query = RDF::Query.new({
  :person => {
    RDF.type  => FOAF.Person,
    FOAF.name => :name,
    FOAF.mbox => :email,
  }
})

people = {}
query.execute(graph).each do |person|
  people[person.name.to_s] = person.email.to_s
end

It’s worth noting that the constants RDF and FOAF are pre-declared by including RDF, and the values that you get back from a query are RDF values, which can be URIs or have datatypes or languages. In the above code I’ve converted them into strings for insertion into the Ruby dictionary.

The full script for ‘get-names-and-emails.rb’ is:

#!/usr/bin/env ruby
require 'rubygems'
require 'rest_client'
require 'linkeddata'

include RDF

query = "PREFIX foaf: <http://xmlns.com/foaf/0.1/>

CONSTRUCT {
  ?person 
    a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
} WHERE { 
  ?person a foaf:Person ;
    foaf:name ?name ;
    ?prop ?value .
}"
endpoint = 'http://localhost:8000/sparql/'

puts "POSTing SPARQL query to #{endpoint}"
response = RestClient.post endpoint, :query => query
content_type = response.headers[:content_type][/^[^ ;]+/]
puts "Response #{response.code} type #{content_type}"

graph = RDF::Graph.new
graph << RDF::Reader.for(:content_type => content_type).new(response.to_str)

puts "\nLoaded #{graph.count} triples\n"

query = RDF::Query.new({
  :person => {
    RDF.type  => FOAF.Person,
    FOAF.name => :name,
    FOAF.mbox => :email,
  }
})

people = {}
query.execute(graph).each do |person|
  people[person.name.to_s] = person.email.to_s
end
puts "\nCreating directory of #{people.length} people"

stott_email = people['Andrew Stott']
puts "\nAndrew Stott's email address: #{stott_email}"

Run this script with:

$ ruby get-names-and-emails.rb

and you get the result:

POSTing SPARQL query to http://localhost:8000/sparql/
Response 200 type application/rdf+xml

Loaded 459 triples

Creating directory of 75 people

Andrew Stott's email address: mailto:andrew.stott@cabinet-office.gsi.gov.uk

Conclusions and Caveats

So there you have it, a walkthrough of setting up a local triplestore, loading in data and accessing that data programmatically using SPARQL queries.

Now for some caveats. First, you’re bound to have noticed that I having followed Richard’s steps to the letter.

4store wasn’t installed from a package management system. The only packaged triplestore I could locate on MacPorts was Virtuoso (which I’ll come to in a second). I hope that 4store’s installation is simple enough for this slight deviation from the rules not to matter.
I didn’t install a package for specifically talking to 4store in order to load in data, just used HTTP requests. There are client libraries for 4store, but I figure that the HTTP requests are easy enough, and the resulting code more portable into other environments, so I prefer not to use them.

Second, there are a couple of dead ends that I went down that I haven’t written up in the above:

I did spend some time yesterday evening trying to get Virtuoso set up. I managed to get it installed, but loading data into it seemed to require some magic which I couldn’t figure out. So I went to bed instead.
I tried to install and use rdf-raptor in order to parse the RDF/XML that naturally comes out of 4store CONSTRUCT queries, but got a Could not open library 'libraptor' error. I couldn’t find an immediate fix for that, so decided to keep things simple instead and just use plain RDF.rb.

Third, I want to reiterate that there may be better ways of using 4store, rest_client, Nokogiri and RDF.rb, as well as Ruby generally, than those shown above. I don’t claim to be an expert in any of these technologies. If you have suggestions and corrections, I’d encourage you to add a comment and I’ll incorporate them in the text to improve it.

Finally, some general points, because the strong binding of ‘linked data’ and ‘SPARQL’ in Richard’s post bothers me:

It’s not necessary to have a SPARQL endpoint when publishing linked data, nor to run your own triplestore. If you already have a website, you are probably better off generating N-Triples or RDF/XML or Turtle in the same way as you generate HTML or XML or JSON.
It’s not necessary to learn SPARQL to access and use linked data: the whole point is that the data in linked data is available through HTTP access in standard (RDF-based) formats, so you can scrape them using a follow-your-nose approach and store the results however you like.

Having said the above, if you’re collecting linked data from multiple sources with unpredictable content and want to query across it, having a local triplestore is very useful.

I also want to point out that within the linked data we’ve published on data.gov.uk, we’ve made a big effort to make the data available in multiple formats such as JSON, XML and CSV, and through a RESTful, URI-parameter-driven API, precisely to lower the barrier for developers who want to use that information but understandably don’t want to take the time or make the effort to learn the linked data technologies that underly the sites. For those that do, the RDF/XML and Turtle is there as well, and the SPARQL queries that are used to create each page are available to look at, tweak, and learn from. Our hope is that the linked data API that provides access to lists of schools, departments and railway stations can make the linked data learning curve a little less steep.

Standardising an RDF API

Jeni — Sat, 04 Dec 2010 08:05:43 +0000

I got a little bit of pushback on my previous blog post for suggesting that W3C should standardise an API for RDF. (I’m talking here about a programming-interface-kind-of-API to enable developers to extract information out of an RDF document rather than a website-API to enable them to access RDF data in the first place.)

I just wanted to talk about a couple of actual real-life scenarios that make me want a standard RDF API:

TimBL wants an RDFa parser for Tabulator. There are a few RDFa parsers in Javascript; he chooses to use rdfQuery’s. Tabulator works on top of its own datastore, which has its own interface for inserting data. rdfQuery’s RDFa parser works on top of its own datastore, which has a different interface for inserting data. To use rdfQuery, TimBL has to either rewrite some of its internal code to call the methods that insert data into Tabulator’s datastore, or rewrite some of Tabulator’s internal code to call the methods that query rdfQuery’s datastore. The lack of a standard API for RDF has made it harder for TimBL to reuse my code.
I’m working on Puelia, which needs to both parse and generate RDF in various ways and uses Moriarty to do so. I am editing the code to create triples in an in-memory RDF graph. I want to add a triple with a literal value. I have no idea how to do so, because I haven’t used Moriarty before, so I have to hunt through its documentation to find the add_literal_triple() function. The lack of a standard API for RDF has made it harder for me to use the library. If I ever wanted to switch to using some other PHP RDF library, such as EasyRDF or Graphite, for whatever reason, I would have to rewrite substantial parts of Puelia to use the functions provided by that library. The lack of a standard API for RDF has made Puelia less modular and adaptable.

For all that the W3C XML DOM seems to be universally reviled as an API for querying and creating XML, it and SAX mean that people can write XSLT and XProc processors (etc) without writing their own XML parser. They mean that whatever programming language I find myself writing code in, I know that I’ll be able to use getElementsByTagName() to get hold of elements with a particular name. They mean that XML parsers have a reason to improve over time, because applications can easily switch to better parsers when they come along. DOM and SAX provide a foundation, a level of standardisation and pluggability, that improves the XML landscape as a whole.

Of course sometimes components need tighter integration in order to achieve performance benefits; that’s a modularity/performance judgement on the part of the developer of the application. And of course there are better object model APIs for XML than the W3C XML DOM around. But better APIs are almost always programming-language or library specific; they are better simply because cross-platform APIs like DOM and SAX cannot take full advantage of the idioms of a particular programming language or style.

Now regarding the W3C’s involvement in creating such a standard, the argument seems to be “W3C created the horror that is the XML DOM and therefore every API specification that comes out of the W3C will be horrendous”.

I think sometimes that W3C is seen as a kind of monolithic organisation that exists over there, with secret committees whose work takes place out of public eyes until they deign to let us mere mortals read the results of their machinations. And who then fend off all comments and criticism in order to protect their lovingly crafted (but completely impractical) specifications.

What this overlooks is that the standards organisation merely provides the framework and administrative support within which groups who are interested in creating a standard can come together. The existing RDFa Working Group’s meetings are documented and discussion takes place in public and is open to all. I’m sure this will continue in the RDF Core Working Group when it is set up.

It will happen anyway. There is already work going on with the W3C to create a standard RDFa API, out of which, so I am told, will arise a Working Draft of an RDF API. From the looks of the most recent Working Draft I will be able to add a literal triple to a DataStore using something like

$store->add(
  $store->createTriple(
    $store->createBlankNode('puelia'),
    $store->createIRI('http://www.w3.org/2000/01/rdf-schema#label'),
    $store->createPlainLiteral('Puelia', 'en')
  )
);

(compared to

$graph->add_literal_triple('_:puelia', 'http://www.w3.org/2000/01/rdf-schema#label', 'Puelia', 'en')

in Moriarty). So OK, it needs a bit of work. But these are early days, and from the looks of the editor’s draft it’s likely to change quite rapidly.

W3C’s standardisation is what we make it; wherever it is done, it is a self-fulfilling prophecy that an API will not be suited to its purpose if the people who would benefit from implementing and using that API don’t get involved in its design. And to be clear, I am talking to myself more than anyone.

Priorities for RDF

Jeni — Sun, 28 Nov 2010 21:44:52 +0000

A couple of weeks ago I did a talk at the TPAC Plenary Day about why RDF hasn’t had the uptake that it might and what could be done about it.

I felt quite uncomfortable about doing this for many reasons. The predominant one is that I’m well aware that the world is made by the people who turn up. It is far far easier to snipe from the sidelines than it is to put in the effort to attend telcons and face-to-face meetings, to engage on mailing lists, to write specifications and implementations and tutorials.

On the other hand, what I hope is that the perspective of someone who is outside that process, someone who tries to understand and interpret and use the results of that process, might be valuable. And so I aimed to provide that honestly.

In that spirit, I’m going to put my stake in the ground and say that there are three areas where I think W3C should be concentrating its efforts:

standardising (something like) TriG — Turtle plus named graphs
standardising an API for the RDF data model
standardising a path language for RDF that can be used by that API and others for easy access

and that it should specifically not put its efforts into standardising another syntax for RDF based on JSON.

Format Standardisation

The first point is that I think we need to decide on a single recommended format for RDF.

Fundamentally, unlike XML or JSON, RDF is defined first and foremost as a model rather than as a syntax. That means it can be expressed in a number of syntaxes, the most common of which are RDF/XML, Turtle and N-Triples though of course there’s also RDFa, RDF/JSON, JSON-LD and N3 and if you start factoring in named graphs you can add TriX, TriG and N-Quads to the list.

Except for a few corner cases, it would be perfectly possible to express the same RDF model in any of these syntaxes. Why is this so bad? Surely having choice is a good thing, because publishers can choose an option that fits with their workflows? And aren’t all these formats generated automatically anyway, such that the same data can be provided in many ways with no overhead?

Well, no, there are actually two ways in which having multiple syntaxes makes adoption harder.

First, publishers aren’t always generating data automatically; in a number of cases (which I think and hope will grow) RDF data is being generated just like CSV files are, as static documents which are simply published in the same way as other static documents. In these cases, publishers either have to do the research and make a decision about which format to use, or produce the data in multiple formats. This is a particular challenge when people aren’t convinced they want to generate RDF anyway.

Second, toolsets have to handle producing or consuming multiple formats. That means more code, more testing and more maintenance on both the production and consumption sides of the equation, all of which raise the implementation burden.

Of course it’s natural that during the initial stages of the use of a technology that we should see a variety of patterns of use: there need to be innovations and experiments so that we can find what works and what doesn’t. But as that technology matures, we need to start bedding down some basics. There need to be agreed foundations that, even if imperfect, are solid enough for the majority of us to build upon. And we need to exercise some self-restraint to concentrate on doing that building rather than revisiting those decisions.

We have a number of years of experience now about what formats are easy to understand, to pass around, to create and to process. It is time, I think, to pick one, to get it standardised, to deprecate others and to provide a much cleaner and clearer picture to publishers and consumers.

Of the formats that we have, the one that fits best with the RDF data model and is simplest for humans to understand is Turtle. But it needs to support named graphs, so that it’s possible to share the RDF datasets that are exposed within a SPARQL endpoint, which is why I say W3C should standardise something like TriG.

RDF APIs

The second point is to work on standardising the APIs that are available for developers who work with RDF. Why standardise APIs? Because it would make accessing RDF easier and more predictable for web developers, who often work across multiple languages and platforms. Developers don’t really care about syntax — although having something readable is useful for debugging — they care about the way in which they get to interact with in-memory structures that hold the data.

RDF needs an API that exposes its internal model (of literals and resources and triples and graphs and datasets) in a way that isn’t too onerous for people to use. There are lots and lots of RDF APIs about, within the various parsers that are available for different platforms; the only one that’s approaching a standard is the one embedded within the RDFa API specification. I would like to see that disentangled from RDFa and for it, or something like it, to gain traction amongst the writers of RDF libraries such as the Redland RDF libraries, RDFLib, Moriarty, Reddy, rdfQuery and so on and on.

But having an API for RDF’s data model is not enough.

I think there is a lot that we can learn from XML’s experience here. James Clark’s recent blog post about XML and the web describes what it’s like for developers working with XML compared to JSON:

The fundamental problem is the mismatch between programming language data structures and the XML element/attribute data model of elements. This leaves the developer with three choices, all unappetising:

live with an inconvenient element/attribute representation of the data;

descend into XML Schema hell in the company of your favourite data binding tool;

write reams of code to convert the XML into a convenient data structure.

By contrast with JSON, especially with a dynamic programming language, you can get a reasonable in-memory representation just by calling a library function.

So JSON is popular because accessing information within the JSON is really easy. And that’s for two reasons:

it’s parsed with a single simple function call in a common library
the result of parsing is simple to navigate; typically you can do so using native methods such as dot-notation paths

The first of these is a simple matter of winning hearts and minds. The second is the important one: it’s easy to use because the underlying JSON model fits neatly onto the object-oriented programming paradigm that most developers use.

XML isn’t so popular among web developers because its underlying model doesn’t fit well into most programming languages: it has attributes and mixed content and a whole bunch of other things that don’t map straight-forwardly onto objects-with-properties. Navigating through XML (or HTML) structures using a DOM is tedious and automatic binding mostly doesn’t work.

What about RDF? On the face of it, RDF is a good fit with object-oriented models; they both follow a basic entity-attribute-value approach. However, there are (at least) three things in RDF that do not fit with the object-oriented model:

Properties in RDF are identified through URIs rather than simple names (ie those containing just letters and numbers and underscores). Some programming languages, such as Javascript, let you have properties that aren’t simple names, but you then have to access them through the relatively clunky [] notation rather than dot-notation paths. Properties are first-class objects in RDF with things like labels and ranges and inverses; fitting with standard programming languages here means using reflection and having the ability to annotate fields, and everything gets a bit mind-bending.
Values in RDF often have datatypes or languages associated with them, and the set of datatypes that you can use is completely extensible (and of course datatypes are first-class objects, with their own properties, too). This wouldn’t be so bad except that making every value an object means comparisons with basic strings or numbers won’t generally work.
In RDF, a property can have more than one value (an unordered bag of values, if you like) or it can have a value that is a rdf:List (an ordered sequence of values); it can even have many values which are rdf:Lists. On the other hand, the object-oriented model generally supports values that are arrays (and of course you can have arrays within arrays), which are always ordered. So there is always a choice to be made when mapping from an object-oriented model to RDF, about whether the values should be at the same level or be rdf:Lists.

In other words, just as with XML, there is no straight-forward mapping from RDF to an object structure that developers can immediately use. That doesn’t stop us trying, of course:

There’s the approach that we use within the linked data API and elsewhere, which is to make it easy for data publishers to create simple JSON versions of the RDF they publish. A website-specific configuration file determines what the mapping looks like. URIs of properties get turned into readable names (and provide the map back to the original property URI, so that it’s possible to get more information about the property). Datatypes and languages are ignored by default (but mapped onto structured values if so configured). And the distinction between properties-with-multiple-values and properties-whose-value-is-a-List is ignored. We purposefully lose some of RDF’s expressivity and power in order to gain usability. You can see the result in action at http://education.data.gov.uk/doc/school and http://education.data.gov.uk/doc/department for example.

There’s the approach that Nathan has taken within js3 which is to create libraries that, with a bit of work on the client side, give a way of mapping RDF into an object-oriented structure which is easy to manipulate (or vice versa, create RDF from OO structures). It’s the same basic principle as helping publishers to generate JSON, but the interpretation and mapping is done by the client rather than the publisher. The work that Nathan’s done to manage this in Javascript is very impressive; I don’t know whether the same approach can be mapped to other languages.

But as James intimated, data binding is hairy and scary. Mappings between different data models are always imperfect, lossy, sensitive to what seem like small changes and therefore hard to maintain. I remember nodding along as Mike Kay talked about this at the XML Summer School in relation to the use of XML: the horrors of working with systems in which there three way maps between relational and object-oriented and XML structures, and the relief that comes with working with an XML-only architecture. I suspect this same observation is one of the drivers behind the growth of JSON databases.

On the other hand, you know, perhaps RDF is close enough to the object-oriented model that it won’t be so bad. Perhaps we could find a way to standardise on a method of configuring applications that do the mapping, such as defining short names for properties, describing how to handle objects with datatypes and languages and so on. We have a body of experience that can be learnt from, including the ones above, and perhaps it can be tied into the RDB-to-RDF work too. The biggest challenge, I suspect, will be to create something round-trippable.

Path Languages

The other option that James didn’t mention but that I touched on in my TPAC talk is to learn from how working with HTML and XML has been made easier in libraries such as jQuery or hpricot. These libraries still allow the HTML and XML to be accessed through a DOM, rather than mapping HTML or XML into object structures, but make the lives of developers simpler by supporting querying of the HTML/XML using path languages that are designed to be used to query those kinds of structures. For HTML, that’s CSS; for XML that’s XPath. (It’s the same approach as is used for strings: we use regular expressions for many operations rather than working with them at the character level.) Path languages work over the native model; all that’s offered in the library are functions that take strings (holding the path language) and return objects or values as appropriate.

I don’t know exactly what it looks like, and it might already be out there (the world moves fast and I know I’m not aware of everything), but what I think we need is a path language for navigating around RDF, probably based on SPARQL property paths or the FRESNEL selector language and an API (or APIs) that uses it. For example, something that lets developers use code like:

graph.find("*[foaf:nick = 'web3r']/foaf:name")

to pick values out of an in-memory graph. In my opinion, something like this would be much more likely to bring benefits than a data binding approach.

Why Not RDF in JSON?

What I’ve tried to explain above is firstly that we already have too many syntaxes for RDF, and secondly that the main barrier to developers using RDF is the way in which they are forced to interact with that RDF once they have hold of it, not the syntax itself. The syntax that we use for RDF really doesn’t matter, because developers interact with the in-memory dataset, not directly on the syntax.

Nathan’s recent post on Opening Linked Data, which is worth reading in its entirety, captures the essence of the issue:

You can’t shoe horn RDF in to JSON, no matter how hard you try - well, you can, but you loose all the benefits of JSON in the first place, because the data is RDF, triples and not objects, rdf nodes and not simple values

In other words, using JSON as the basis for an RDF syntax doesn’t actually win you anything in terms of the ease of processing of that RDF. In fact, I’ll go further and say it has exactly the same bad qualities as RDF/XML.

One of the bad things about RDF/XML is that it misleads people into thinking they can use normal XML tooling to process RDF, but XML tooling exposes the XML tree, not the RDF graph that they need. It’s good enough in some circumstances, of course, but it’s not working with RDF as RDF. Similarly, just because you’re using XML tools doesn’t mean RDF/XML is easy to generate; you’re a lot safer to generate correct RDF/XML from an in-memory graph, in the same way as generating XML using string manipulation is harder work than it first appears.

In exactly the same way, I think that a JSON-based syntax for RDF will mislead developers into thinking that they can interpret and generate that JSON like they can normal JSON, and interact with it at that level, when this simply isn’t the case.

The only advantage that I can see for a JSON-based RDF syntax is equivalent to the only advantage of RDF/XML: it is easier to store for people who use JSON databases, just as RDF/XML is easier to store for people who use XML databases. I am not sure that benefit is worth the cost of an additional RDF syntax; isn’t RDF better stored in a triplestore?

Summary

So to reiterate, as far as I’m concerned, W3C and the RDF community should be concentrating on a syntax for RDF that doesn’t come saddled with those kinds of assumptions, which I think is Turtle + graphs; something like TriG. They should be concentrating on developing a standard API for RDF access that has a chance of adoption among the developers of RDF libraries, and on working out what parts of SPARQL and FRESNEL could be used to create a path language that could be reused in several contexts, including within such an API. And these should be done in preference to a RDF syntax in JSON which doesn’t solve the core problems, and in fact just adds another syntax to the mix.

Translating Existing Models to RDF

Jeni — Sat, 13 Mar 2010 20:35:46 +0000

As we encourage linked data adoption within the UK public sector, something we run into again and again is that (unsurprisingly) particular domain areas have pre-existing standard ways of thinking about the data that they care about. There are existing models, often with multiple serialisations, such as in XML and a text-based form, that are supported by existing tool chains.

In contrast, if there is existing RDF in that domain area, it’s usually been designed by people who are more interested in the RDF than in the domain area, and is thus generally more focused on the goals of the typical casual data re-user rather than the professionals in the area.

To give an example, the international statistics community uses SDMX for representing and exchanging statistics (and a lot more besides; it’s a huge standard). SDMX includes a well-thought through model for statistical datasets and the observations within them, as well as standard concepts for things like gender, age, unit multipliers and so on. By comparison, SCOVO, the main RDF model for representing statistics, barely scratches the surface in comparison.

This isn’t the only example: the INSPIRE Directive defines how geographic information must be made available. GEMINI defines the kind of geospatial metadata that that community cares about. The Open Provenance Model is the result of many contributors from multiple fields, and again has a number of serialisations.

You could view this as a challenge: experts in their domains already have models and serialisations for the data that they care about; how can we persuade them to adopt an RDF model and serialisations instead?

But that’s totally the wrong question. Linked data doesn’t, can’t and won’t replace existing ways of handling data. But it has got some interesting features that can bring great benefit to people who want to publish their data, namely:

web-scale addresses — being able to name and refer to things like individual observations in a statistical hypercube, a particular road junction, or the particular process that led to something being created
annotation — the ability to record metadata about everything that you can name, which is everything!
distributed publication — enabling multiple publishers to control the publication of their data without having to upload it to a central location
links — the joining of information to other information, providing more context, supporting more queries and reducing the requirement for duplication

The question is really about how to enable people to reap these benefits; the answer, because HTTP-based addressing and typed linkage is usually hard to introduce into existing formats, is usually to publish data using an RDF-based model alongside existing formats. This might be done by generating an RDF-based format (such as RDF/XML or Turtle) as an alternative to the standard XML or HTML, accessible via content negotiation, or by providing a GRDDL transformation that maps an XML format into RDF/XML.

Either way, the underlying model needs to be mapped into RDF. We’re furthest down this road with statistical data. I wanted to explore here what it might look like for the Open Provenance Model, building on lessons learned from the statistical domain.

Open Provenance Model

The Open Provenance Model talks about three main nodes:

artifacts, which are the things that are produced or used by processes
processes, which are actions that are performed using or producing artifacts
agents, which are the people or systems that perform actions

and five kinds of edges that can be defined between them:

process A used artifact B
artifact A was generated by process B
process A was controlled by agent B
process A was triggered by process B
artifact A was derived from artifact B

Then things start getting more complicated. OPM indicates that each artifact and agent plays a different role when it is used by, generated by or controls a process. What’s more, each artifact and agent might be involved in the process at different times (though timing information is optional within OPM). And a given provenance graph may contain several accounts of how artifacts, processes and agents fit together.

Existing Mapping to RDF

The OWL ontology for OPM for OPM is a very literal mapping of OPM into RDF. Each of the types of nodes is a separate class, and each of the types of edges is a separate class. Thus, it introduces a lot of n-ary relationships. Take a really simple example of an XML file being transformed into HTML using XSLT. With the OPM ontology, the RDF would look something like:

_:transformation a opm:Process .
<doc.html> a opm:Artifact .
<doc.xml> a opm:Artifact .
<doc.xsl> a opm:Artifact .
_:processor a opm:Agent .
_:Jeni a opm:Agent .

_:stylesheetLink a opm:Used ;
  opm:effect _:transformation ;
  opm:cause <doc.xml> ;
  opm:role eg:xsltSource .

_:sourceLink a opm:Used ;
  opm:effect _:transformation ;
  opm:cause <doc.xsl> ;
  opm:role eg:xsltStylesheet .

_:resultLink a opm:WasGeneratedBy ;
  opm:effect <doc.html> ;
  opm:cause _:transformation ;
  opm:role eg:xsltResult .

_:processorLink a opm:WasControlledBy ;
  opm:effect _:transformation ;
  opm:cause _:processor ;
  opm:role xslt:processor .

_:userLink a opm:WasControlledBy ;
  opm:effect _:transformation ;
  opm:cause _:Jeni ;
  opm:role xslt:user .

_:derivation a opm:WasDerivedFrom ;
  opm:effect <doc.html> ;
  opm:cause <doc.xml> .

xslt:source a opm:Role ;
  opm:value "source" .

xslt:stylesheet a opm:Role ;
  opm:value "stylesheet" .

xslt:result a opm:Role ;
  opm:value "result" .

xslt:processor a opm:Role ;
  opm:value "processor" .

xslt:user a opm:Role ;
  opm:value "user" .

To give you an idea of what this mapping means, if I wanted to work out who created doc.html, I would have to do a query like:

SELECT ?who
WHERE {
  ?generatedBy 
    opm:cause <doc.html> ;
    opm:role xslt:result ;
    opm:effect ?transformation .
  ?controlledBy
    opm:effect ?transformation ;
    opm:role xslt:user ;
    opm:cause ?who .
}

Some Observations

There are two things that I want to pull out about the RDF mapping described above.

it’s incredibly literal; every entity type within the model is mapped onto an RDF class, including the edges, the roles and the accounts (which I didn’t show above)
it doesn’t reuse any existing vocabularies, even when they might help (such as for the ‘value’ of a role, which is really a label)

It reminds me of the mapping of object-oriented or relational data models into each other or into XML, which often result in a god awful mess and people swearing that technology X is goddamned ugly.

The fact is that elegant uses of each modelling paradigm — ones that are easy to understand and efficient to query — always take advantage of the unique features of that paradigm. For example, good XML vocabularies take advantage of the distinctions between attributes and elements, of nesting and hierarchies, and of the ability to hold mixed content.

It’s the same with RDF. There are four features of RDF that I think good vocabularies will take suitable advantage of:

existing vocabularies
inheritance
shortcuts and reasoning
named graphs

Reusing existing vocabularies takes advantage of the ease of bringing together diverse domains within RDF, and it makes data more reusable. For example, an OPM mapping that encourages the reuse of FOAF for people and organisations saves time and effort for the developers of the OPM RDF vocabulary, that they would otherwise have spent modelling the details of agents; and it means that any agents that are described within the description of a piece of provenance are automatically available as agents in the wider FOAF cloud. The same goes for using DOAP to describe software.

By reusing vocabularies, the data isn’t isolated any more, locked within a single context designed for a single use. This is a huge benefit of the linked data approach and it makes sense to leverage it.

Using inheritance means creating general purpose classes and properties and encouraging other people to use rdfs:subClassOf or rdfs:subPropertyOf to specialise them according to their own requirements. Within OPM, the different roles that artifacts and agents might play in a process is a natural fit with either sub-properties or sub-classes, depending on how the edges in the model are represented. For example, rather than

_:stylesheetLink a opm:Used ;
  opm:effect _:transformation ;
  opm:cause <doc.xsl> ;
  opm:role eg:xsltStylesheet .

xslt:stylesheet a opm:Role ;
  opm:value "stylesheet" .

you could generate data that looked like:

_:stylesheetLink a xslt:Stylesheet ;
  opm:effect _:transformation ;
  opm:cause <doc.xsl> .

where xslt:Stylesheet is defined as a subclass of opm:Used.

Inheritance is a basic form of reasoning. In the case of the subclass relationship outlined above, the reasoning is that anything that is a xslt:Stylesheet is also a opm:Used, and thus:

_:stylesheetLink a xslt:Stylesheet .

implies

_:stylesheetLink a xslt:Used .

Taking the scenario where you’re doing native linked data publishing — storing data in a triplestore and then publishing it out from there — you have two choices:

you can store just the basic data, and let the application retrieving it carry out whatever reasoning is necessary to derive the information they need; this limits the size of the triplestore, but can place a large burden on people using it — either they have to be very familiar with the exact choices made in modelling the basic data, or they have to construct complex SPARQL queries that take account of the fact that the data might be modelled in many different ways
you can store not only the basic data but also anything that can be derived from it; this increases the number of triples you have to store, but means that people can query it without having to perform any reasoning themselves

The latter is obviously the more user-friendly approach. (And a triplestore could make it easy by understanding and applying schemas, ontologies and rules as data is loaded in.)

To take a more complex example, provenance could be modelled in a much more direct way, such as:

<doc.html> a opm:Artifact ;
  opm:derivedFrom <doc.xml> ;
  opm:generatedBy [
    xslt:source <doc.xml> ;
    xslt:stylesheet <doc.xsl> ;
    xslt:processor _:processor ;
    xslt:user _:Jeni ;
  ] .

where xslt:source and xslt:stylesheet are sub-properties of a property called opm:used, and xslt:processor and xslt:user are sub-properties of opm:controlledBy. This removes the n-ary properties, which (given the use of inheritance to represent roles) are only actually needed if the model needs to capture the timing of the involvement of particular artifacts or agents within a process, and makes the provenance information much easier to query than before:

SELECT ?who
WHERE {
  <doc.html> opm:generatedBy ?transformation .
  ?transformation xslt:user ?who .
}

But what if we also want to support the more complex, n-ary-relation-based models? We would need to assert, somehow, a rule that said that the presence of a opm:controlledBy relationship from a process to an agent was equivalent to having a opm:WasControlledBy instance with a opm:cause pointing to the agent and an opm:effect pointing to the process. Combine this with xslt:user being sub-property of opm:controlledBy and you have the statement:

_:transformation xslt:user _:Jeni .

implying:

_:transformation opm:controlledBy _:Jeni .

which in turn implies:

[] a opm:WasControlledBy ;
  opm:effect _:transformation ;
  opm:cause _:Jeni .

The same reasoning could be applied in the opposite direction, of course. Part of the definition of the use of OPM in RDF could be that the presence of a opm:WasControlledBy with a opm:cause pointing to an agent and opm:effect pointing to a process implies a opm:controlledBy link between the opm:effect and the opm:cause. Whichever was used in the initial modelling of the data, the same query could be used to query the data (accepting some loss of precision along the way, but if you’re not interesting in timing information then why should you suffer the cost of querying through n-ary relations?).

The final thing that I mentioned above that mappings from existing models to RDF should take advantage of is named graphs. In OPM, the obvious way that named graphs could play a role is in providing support for the different accounts of provenance. Separate named graphs could be used to represent separate accounts, referencing the same artifacts, agents and processes where appropriate. Individually, the graphs can remain simple; together, you have the full power of OPM.

Conclusions

Modelling is a complex design activity, and you’re best off avoiding doing it if you can. That means reusing conceptual models that have been built up for a domain as much as possible and reusing existing vocabularies wherever you can. But you can’t and shouldn’t try to avoid doing design when mapping from a conceptual model to a particular modelling paradigm such as a relational, object-oriented, XML or RDF model.

If you’re mapping to RDF, remember to take advantage of what it’s good at such as web-scale addressing and extensibility, and always bear in mind how easy or difficult your data will be to query. There is no point publishing linked data if it is unusable.

Creating Linked Data - Part V: Finishing Touches

Jeni — Sat, 05 Dec 2009 08:50:28 +0000

This is the fifth part in this series about creating linked data. I’ve talked previously about analysis and modelling, defining URIs, defining concept schemes and defining a vocabulary. In this instalment I’ll talk about the finishing touches that can make linked data easier to browse, query, locate and trust.

Note that we don’t have to do any of these things; they’re not part of the core data. We shouldn’t beat ourselves up if we don’t have time to do it right now, because we can always add them later, and it might be that you just don’t agree that they should be done. But many of them don’t take a lot of time and can enhance the user’s experience of the data.

Labels

Every resource should have a label, even blank nodes. Adding labels makes it easier for people to generate HTML views from the data. Sometimes we have resources that have an obvious label (like the name of a local authority); at other times, the label needs to be constructed based on the other information that’s available about the resource.

I talked in the last instalment about skos:prefLabel (preferred label), skos:altLabel (alternative label) and rdfs:label. Technically, skos:prefLabel and skos:altLabel are sub properties of rdfs:label, which means that if a resource has a skos:prefLabel it also has a rdfs:label with that value. However, drawing that conclusion requires either built-in knowledge of SKOS or the ability to both automatically get hold of the SKOS ontology and reason with it, which is feasible (this is one of the advantages of RDF, after all), but adds an extra hurdle for people wanting to use your data.

So it’s best to give everything a rdfs:label, even if they already have a skos:prefLabel or skos:altLabel. It’s also good to try to imagine that label in the context of having no other information about the thing that it’s labelling, such as in the title of a page. For example, if you’re looking at the observation http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00/type/bicycle in the context of a traffic count, it may seem sensible to label it just “bicycle” (as I did in the first iteration of turning this traffic count data into RDF). But without that context, it makes no sense. Better to label it “Bicycles - 8 Oct 2001 17:00-18:00 - East - Salterton Road, EAST OF DINAN WAY, EXMOUTH” and provide an even more descriptive rdfs:comment like “Number of bicycles counted travelling East at Salterton Road, EAST OF DINAN WAY, EXMOUTH on 8 October 2001 between 17:00 and 18:00.”.

Datasets

There are two kinds of datasets that are applicable to this particular …err… set of data … and that we should describe within the RDF. They are:

datasets that are sets of statistical data items (such as the observations in the traffic count data); these are best described using SCOVO
datasets that are general descriptions of particular sets of linked data (such as roads or local authorities); these are best described using voiD

Both kinds of datasets can be identified for UK government data using URIs in the form:

http://{sector}.data.gov.uk/set/{dataset}/

SCOVO Datasets

Every scovo:Item should be part of a scovo:Dataset, associated through a scovo:dataset (and a reverse scovo:datasetOf). A scovo:Dataset is pretty simple: all you really need to do is give it an identifier and, of course, a label. In this case, something like:

http://transport.data.gov.uk/set/traffic-count/2001-2008/

This is an identifier that the various scovo:Items should use to indicate where the data comes from:

<http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00/type/bicycle>
  a scovo:Item ;
  scovo:dataset <http://transport.data.gov.uk/set/traffic-count/2001-2008/> ;
  traffic:count <http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00> ;
  traffic:vehicleType <http://transport.data.gov.uk/def/vehicle/bicycle> ;
  rdf:value 2 .

It’s also an identifier that we can attach some metadata to. Obviously it needs a label, but we can also attach other metadata, such as the provenance of the dataset.

<http://transport.data.gov.uk/set/traffic-count/2001-2008/>
  a scovo:Dataset ;
  a prv:DataItem ;
  rdfs:label "Traffic counts between 2001 and 2008"@en ;
  prv:createdBy [
    a prv:DataCreation ;
    prv:performedAt ... ;
    prv:performedBy ... ;
    prv:usedData ... ;
    prv:usedGuideline ... ;
  ] .

VoiD Datasets

VoiD is designed to be used to describe sets of linked data, their contents, their provenance and their relationships with each other. There are many ways of dividing up the data that we’ve been looking at into datasets. We can start with a simple example: the dataset containing linked data about countries:

<http://statistics.data.gov.uk/set/country/>
  a void:Dataset ;
  rdfs:label "Countries"@en ;
  foaf:homepage <http://statistics.data.gov.uk/set/country> ;
  dct:subject <http://dbpedia.org/resource/Country> ;
  cc:license [
    a cc:License ;
    rdfs:label "data.gov.uk Licence"@en ;
    foaf:homepage <http://data.hmg.gov.uk/terms-privacy> ;
    cc:permits cc:DerivativeWorks, cc:Distribution, cc:Reproduction ;
    cc:requires cc:Attribution ;
  ] ;
  void:exampleResource <http://statistics.data.gov.uk/id/country?name=England> ;
  void:sparqlEndpoint <http://services.data.gov.uk/statistics/sparql> ;
  void:uriRegexPattern "http://statistics.data.gov.uk/id/country?name=.+"^^xs:string ;
  void:vocabulary <http://statistics.data.gov.uk/def/administrative-geography/> ;
  void:vocabulary <http://www.w3.org/2000/01/rdf-schema#> .

This provides a link to a home page for the dataset, which should contain information about the dataset itself. (Accessing the URI for the dataset should also redirect users to this home page.) I’ve used the same URI as the dataset URI but without the slash at the end. (This is probably too subtle a difference between URIs; we don’t currently have official guidance for URIs for documents-about-datasets or documents-about-definitions.)

The void:exampleResource property can be used to point to resources that can act as starting points for exploring the data, and the void:sparqlEndpoint property points at a SPARQL endpoint that can be used for deeper querying. The void:uriRegexPattern property provides a regular expression for the URIs that are used to identify the resources that the dataset is about. void:vocabulary points to the vocabularies that the dataset uses.

Various Dublin Core properties can be used to provide metadata about the dataset, such as its subject matter. The Creative Commons schema provides a way of indicating the licence that the dataset is made available under, which is essential information to enable reuse. (I’ve derived some RDF about the licence here from the one described on the data.hmg.gov.uk pages; there should be an official version some time soon.)

The data that we can actually produce from this traffic count dataset is actually a subset of the dataset of all countries, and we can indicate this through a void:subset relationship:

<http://statistics.data.gov.uk/set/country/>
  ...
  void:subset [
    a void:Dataset ;
    a prv:DataItem ;
    rdfs:label "Country data from the DfT traffic count dataset 2001-2008"@en ;
    prv:createdBy [
      a prv:DataCreation ;
      prv:performedAt ... ;
      prv:performedBy ... ;
      prv:usedData ... ;
      prv:usedGuideline ... ;
    ] ;
  ] .

The other kind of subset that we should describe are link sets. Link sets are datasets that contain links between datasets. The country dataset doesn’t (currently) contain any links to other datasets, but the count dataset does:

<http://transport.data.gov.uk/set/traffic-count>
  a void:Dataset ;
  rdfs:label "Traffic Counts"@en ;
  foaf:homepage <http://transport.data.gov.uk/set/traffic-count> ;
  dct:subject <http://dbpedia.org/resource/Traffic> ;
  dct:subject <http://dbpedia.org/resource/Counting> ;
  cc:license [
    a cc:License ;
    rdfs:label "data.gov.uk Licence"@en ;
    foaf:homepage <http://data.hmg.gov.uk/terms-privacy> ;
    cc:permits cc:DerivativeWorks, cc:Distribution, cc:Reproduction ;
    cc:requires cc:Attribution ;
  ] ;
  void:exampleResource <http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00> ;
  void:uriRegexPattern <http://transport.data.gov.uk/id/traffic-count-point/[0-9]+/direction/[NSEW]/hour/[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:00:00> ;
  void:sparqlEndpoint <http://services.data.gov.uk/transport/sparql> ;
  void:vocabulary <http://transport.data.gov.uk/def/traffic/> ;
  void:subset [
    a void:Dataset ;
    rdfs:label "Traffic Counts from the DfT traffic count dataset 2001-2008"@en ;
    prv:createdBy ...
  ] ;
  void:subset [
    a void:Linkset ;
    rdfs:label "Traffic count / count point links"@en ;
    rdfs:comment "Links from a traffic count to the count point at which the count was taken."@en ;
    void:subjectsTarget <http://transport.data.gov.uk/set/traffic-count/> ;
    void:linkPredicate <http://transport.data.gov.uk/def/count> ;
    void:objectsTarget <http://transport.data.gov.uk/set/traffic-count-point/> ;
  ] ;
  void:subset [
    a void:Linkset ;
    rdfs:label "Traffic count / cardinal direction"@en ;
    rdfs:comment "Links from a traffic count to the direction in which the traffic was going."@en ;
    void:subjectsTarget <http://transport.data.gov.uk/set/traffic-count/> ;
    void:linkPredicate <http://transport.data.gov.uk/def/direction> ;
    void:objectsTarget <http://dbpedia.org/void/Dataset> ;
  ] ;
  void:subset [
    a void:Linkset ;
    rdfs:label "Traffic count / hour"@en ;
    rdfs:comment "Links from a traffic count to the hour when the traffic was being monitored."@en ;
    void:subjectsTarget <http://transport.data.gov.uk/set/traffic-count/> ;
    void:linkPredicate <http://transport.data.gov.uk/def/direction> ;
    void:objectsTarget [
      a void:Dataset ;
      rdfs:label "URIs for places and times" ;
      foaf:homepage "http://placetime.com/" ;
    ] ;
  ] .

scovo:Datasets are often subsets of void:Datasets. In the case of the traffic count data, the observations described by the scovo:Dataset above are a subset of the void:Dataset that is the set of all such observations (including ones from other years).

Derivable Data

The discussion about rdfs:label above touched on another set of information that should be included within the RDF data we produce: data that is automatically derivable from the data we provide. There are three main reasons for including derivable data within what we publish:

Given the current adoption of RDF-aware technologies, the consumers of our data are pretty unlikely to be able to (or to want to) use schemas, ontologies or rule sets to help them to reason over the data and draw conclusions. The consumers of this data might include semantic search engines and people scraping the data into their own triplestores, but they’re far more likely to be developers who really don’t care about RDF at all. It would be a shame to publish the data and then have no one use it.
Computing derivable data once saves overall effort. We calculate it once, centrally, and it means that the people using the data don’t have to spend processing time doing it themselves. (There’s a classic time/space trade-off here, of course; the down side of including data that isn’t strictly necessary is that the documents will end up larger.)
If we provide information that people are likely to need within the document that they get when they request a given resource, they’re less likely to need to resort to (harder to construct and more intensive to process) SPARQL queries to get what they need.

The overriding principle that we can use to help us decide what to include is to consider what we would like to see if we visited a page about the particular thing.

How we manage to provide the derived data depends on how we publish the data. I’m not talking here about how to do the publishing, but rather about what the consumers of the data should expect to see eventually. So, for example, if we publish the data as static files then we’re going to have to include all this data in those files. If we generate the RDF dynamically, we just have to make sure that the generated RDF includes the derived data; we might be able to set up rules in a triplestore, or a transformation of the data that it naturally produces, to include the derivable data.

Superclasses and Super-properties

One set of derived data is that inferred from the superclasses and super-properties that are defined with the RDF vocabularies we use in our data. Basically, if a resource has a type that is a subclass of another type, then the resource should have that superclass as a type as well. Similarly, if a triple includes a property that has a super-property, then there ought also to be a triple that links the subject and object of the original triple with the super-property as well.

To understand when it’s important to include this kind of derived data, we need to be aware of the kind of applications that will use the data. Some applications will be targeting just this dataset about traffic counts, and will be written to use whatever properties and classes that we’ve made available. Other applications will be targeted at specific vocabularies at a more general-purpose level. There might be applications that can be used to visualise SKOS hierarchies as a tree, for example, or applications that can plot any geo:lat/geo:long coordinates on a map, or any OWL-Time intervals and instants on a timeline. Still other applications, such as viewers like Tabulator, will be used with any old RDF. We need to provide enough information to make the data easily usable by these more generic applications.

As an example, in the last instalment we introduced classes for traffic:VehicleType and traffic:RoadCategory which were subclasses of skos:Concept. If we want generic SKOS visualisers to be able to display the vehicle type and road category concept schemes, we should try to make it easy for them to work out which things are concepts, by indicating that they are concepts as well. Bearing in mind what I’ve said above about labels, that means that the original RDF:

<motorway> a traffic:RoadCategory ;
  skos:prefLabel "Motorway"@en ;
  skos:broader <major> ;
  skos:scopeNote "Major roads often used for long distance travel. They are usually three or more lanes in each direction and generally have the maximum speed limit of 70mph."@en ;
  skos:inScheme <> .

should include a reference to skos:Concept and a rdfs:label:

<motorway> a traffic:RoadCategory ;
  a skos:Concept ;
  rdfs:label "Motorway"@en ;
  skos:prefLabel "Motorway"@en ;
  skos:broader <major> ;
  skos:scopeNote "Major roads often used for long distance travel. They are usually three or more lanes in each direction and generally have the maximum speed limit of 70mph."@en ;
  skos:inScheme <> .

Note that I haven’t included the results of all the reasoning that we could anticipate. The property skos:scopeNote is a sub-property of skos:note, for example, but I haven’t included a skos:note explicitly because any SKOS-aware processor should have that kind of knowledge built in. The rule of thumb is that if the result of the reasoning involves a resource from another vocabulary, then we should include it.

Derivable Values

There are other kinds of derivable data in this data set. In particular, there are eastings and northings, but not latitudes and longitudes. When there’s useful derivable data, especially when it’s not trivial to derive, it makes sense to make that available explicitly, otherwise everyone else will have to go through the effort of deriving it themselves.

We’ve already done this with the information about the hours of the traffic counts, by pulling out the year and hour of the count rather than having them tucked away within a xs:dateTime literal. The same should be true of the eastings and northings. For small numbers of values, you can use the Ordnance Survey’s online converter; for larger numbers of values you can download the (Windows only and very dated) software or try one of the various converters you can find with a Google search.

Latitudes and longitudes for points should, of course, be expressed using the geo:lat and geo:long properties from the http://www.w3.org/2003/01/geo/wgs84_pos# vocabulary.

Inverses

Statements in RDF link two things. For example, you can view the statement:

<http://transport.data.gov.uk/id/traffic-count-point/13>
  traffic:road <http://transport.data.gov.uk/id/road/B3178> .

as saying that traffic count point 13 is on the B3178 and that the B3178 has a count point on it that is traffic count point 13.

So it’s always possible, when creating a query about or a representation of the road to include the ‘backward links’ — the statements in which the road features as an object as well as those in which it features as a subject. This has caused some people to argue that relationships should only be defined in one direction.

Personally, I don’t agree, for two reasons.

Although it’s possible to create queries and representations that include backward links, it often doesn’t happen like that. It’s different with different triplestores, but result of a the DESCRIBE SPARQL query commonly only includes triples in which the thing being described in the subject, not the object. Also, when constructing queries, it seem more natural to always “travel forward” through the graph. For example:
```
SELECT ?count
WHERE {
  ?point
    area:localAuthority <http://statistics.data.gov.uk/id/local-authority/43UC> ;
    traffic:count ?count .
}
```
rather than:
```
SELECT ?count
WHERE {
  ?point
    area:localAuthority <http://statistics.data.gov.uk/id/local-authority/43UC> .
  ?count
    traffic:countPoint ?point .
}
```
So although it introduces redundancy, I think that including inverse relationships in RDF aids usability and navigability.
Sometimes both directions of a relationship contain meaningful information. For example, it’s not enough to include a gen:mother relationship from a person to their mother because the implied reverse relationship is simply that the person is a child of their mother — you need to include a gen:son or gen:daughter relationship as well to tell which type of child.

So in this dataset, I’m going to include inverse relationships where appropriate:

from countries to regions
from regions to local authority districts
from roads to count points
from count points to counts
from counts to observations

Shortcuts

Another thing that can aid the navigability of a set of RDF data is to provide “shortcuts”. For example, at the moment we have links that say which country a region belongs to and which region a local authority district belongs to, but we don’t have a link that says which country a local authority district belongs to. These kind of links can make it easier to navigate through (and to query) a dataset, so they can be worth adding so long as they don’t clutter up the data too much.

Just think of what you’d like to know about a particular thing when you visit its page. If you’re looking at transport in a local authority district, it would be useful to know what region and country it belongs to and about what roads and traffic count points it contains. But it would be too much to have a list of all the counts and observations that have been taken on those count points.

For this dataset, I’m going to add shortcuts from:

countries to local authority districts (and vice versa)
count points to regions and countries
roads to local authority districts (and vice versa)
roads to regions and countries
roads to road categories and road names
roads to counts (and vice versa)
observations to count points, roads, directions and count hours

These are all judgement calls — there are no hard and fast rules — and as you can see I’m not adding inverses everywhere here because to do so would lead to unnecessarily large sets of RDF in some cases.

That’s the end of this instalment. I had been intending to make this the final one, but there are a couple of things still left to talk about: the publication of RDF, and the supplementary documents that we need to provide (including RDF about those supplementary documents). I’ve also had a request to talk about OWL ontologies, so I’ll probably do that, and there are things to say about how to manage things changing over time. So this may end up being an eight-part series!

To keep us up to date, with all the extra derived information added, the RDF looks as follows:

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#"> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix time: <http://www.w3.org/2006/time> .
@prefix scovo: <http://purl.org/NET/scovo#> .
@prefix area: <http://statistics.data.gov.uk/def/administrative-geography/> .
@prefix admingeo: <http://data.ordnancesurvey.co.uk/ontology/admingeo/> .
@prefix space: <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/> .
@prefix traffic: <http://transport.data.gov.uk/def/traffic/> .

<http://statistics.data.gov.uk/id/country?name=England>
  a area:Country ;
  rdfs:label "England"@en ;
  area:region <http://statistics.data.gov.uk/id/government-office-region/K> ;
  area:district <http://statistics.data.gov.uk/id/local-authority-district/18> .

<http://statistics.data.gov.uk/id/government-office-region/K>
  a admingeo:GovernmentOfficeRegion ;
  rdfs:label "South West"@en ;
  skos:notation "K"^^area:StandardCode ;
  area:country <http://statistics.data.gov.uk/id/country?name=England> ;
  area:district <http://statistics.data.gov.uk/id/local-authority-district/18> .

<http://statistics.data.gov.uk/id/local-authority-district/18>
  a area:LocalAuthorityDistrict ;
  rdfs:label "Devon"@en ;
  skos:notation "18"^^area:StandardCode ;
  skos:notation "1115"^^traffic:LAcode ;
  area:localAuthority <http://statistics.data.gov.uk/id/local-authority/18> ;
  area:country <http://statistics.data.gov.uk/id/country?name=England> ;
  area:region <http://statistics.data.gov.uk/id/government-office-region/K> ;
  traffic:countPoint <http://transport.data.gov.uk/id/traffic-count-point/13> ;
  traffic:road <http://transport.data.gov.uk/id/road/B3178> .

<http://transport.data.gov.uk/id/local-authority-district/1115>
  owl:sameAs <http://statistics.data.gov.uk/id/local-authority-district/18> .

<http://statistics.data.gov.uk/id/local-authority/18>
  a area:LocalAuthority ;
  rdfs:label "Devon County Council"@en ;
  skos:notation "18"^^area:StandardCode ;
  skos:notation "1115"^^traffic:LAcode ;
  area:coverage <http://statistics.data.gov.uk/id/local-authority-district/18> .

<http://transport.data.gov.uk/id/local-authority/1116>
  owl:sameAs <http://statistics.data.gov.uk/id/local-authority/18> .

<http://transport.data.gov.uk/id/road/B3178>
  a traffic:Road ;
  rdfs:label "B3178" ;
  rdfs:label "Salterton Road"@en ;
  skos:prefLabel "B3178" ;
  skos:altLabel "Salterton Road"@en ;
  skos:notation "B3178"^^traffic:RoadNumber ;
  area:country <http://statistics.data.gov.uk/id/country?name=England> ;
  area:region <http://statistics.data.gov.uk/id/government-office-region/K> ;
  area:localAuthority <http://statistics.data.gov.uk/id/local-authority/18> ;
  area:district <http://statistics.data.gov.uk/id/local-authority-district/18> ;
  traffic:roadCategory 
    <http://transport.data.gov.uk/def/road-category/b> ,
    <http://transport.data.gov.uk/def/road-category/urban> ;
  traffic:count <http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00> .

<http://transport.data.gov.uk/id/traffic-count-point/13>
  a traffic:CountPoint ;
  rdfs:label "Salterton Road, EAST OF DINAN WAY, EXMOUTH"@en ;
  rdfs:comment "Salterton Road, EAST OF DINAN WAY, EXMOUTH"@en ;
  skos:notation "13"^^traffic:CountPointNumber ;
  traffic:road <http://transport.data.gov.uk/id/road/B3178> ;
  traffic:roadName "Salterton Road"@en ;
  traffic:roadCategory 
    <http://transport.data.gov.uk/def/road-category/b> ,
    <http://transport.data.gov.uk/def/road-category/urban> ;
  space:easting 302600 ;
  space:northing 81984 ;
  geo:lat 50.6294 ;
  geo:long -3.3784 ;
  area:country <http://statistics.data.gov.uk/id/country?name=England> ;
  area:region <http://statistics.data.gov.uk/id/government-office-region/K> ;
  area:localAuthority <http://statistics.data.gov.uk/id/local-authority/18> ;
  area:district <http://statistics.data.gov.uk/id/local-authority-district/18> ;
  traffic:count <http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00> .

<http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00>
  a traffic:Count ;
  rdfs:label "8 Oct 2001 17:00-18:00 - East - Salterton Road, EAST OF DINAN WAY, EXMOUTH"@en ;
  traffic:countPoint <http://transport.data.gov.uk/id/traffic-count-point/13> ;
  traffic:road <http://transport.data.gov.uk/id/road/B3178> ;
  traffic:direction <http://dbpedia.org/resource/East> ;
  traffic:countHour <http://placetime.com/interval/gregorian/2001-10-08T17:00:00Z/PT1H> ;
  traffic:observation <http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00/type/bicycle> .

<http://dbpedia.org/resource/East>
  rdfs:label "East"@en .

<http://placetime.com/interval/gregorian/2001-10-08T17:00:00Z/PT1H>
  a traffic:CountHour ;
  rdfs:label "8 Oct 2001, 17:00-18:00"@en ;
  time:hasBeginning <http://placetime.com/instant/gregorian/2001-10-08T17:00:00Z> ;
  time:hasEnd <http://placetime.com/instant/gregorian/2001-10-08T18:00:00Z> ;
  time:hasDurationDescription _:OneHour ;
  time:intervalDuring <http://dbpedia.org/resource/2001> .

_:OneHour a time:DurationDescription ;
  rdfs:label "one hour"@en ;
  time:years 0 ;
  time:months 0 ;
  time:days 0 ;
  time:hours 1 ;
  time:minutes 0 ;
  time:seconds 0 .

<http://placetime.com/instant/gregorian/2001-10-08T17:00:00Z>
  a time:Instant ;
  rdfs:label "8 Oct 2001, 17:00"@en ;
  time:inXSDDateTime "2001-10-08T17:00:00Z"^^xsd:dateTime ;
  time:inDateTime [
    a time:DateTimeDescription ;
    time:unitType time:unitHour ;
    time:year "2001"^^xsd:gYear ;
    time:month "--10"^^xsd:gMonth ;
    time:day "---08"^^xsd:gDay ;
    time:hour 17 ;
  ] .

<http://placetime.com/instant/gregorian/2001-10-08T18:00:00Z>
  a time:Instant ;
  rdfs:label "8 Oct 2001, 18:00"@en ;
  time:inXSDDateTime "2001-10-08T18:00:00Z"^^xsd:dateTime ;
  time:inDateTime [
    a time:DateTimeDescription ;
    time:unitType time:unitHour ;
    time:year "2001"^^xsd:gYear ;
    time:month "--10"^^xsd:gMonth ;
    time:day "---08"^^xsd:gDay ;
    time:hour 18 ;
  ] .

<http://dbpedia.org/resource/2001>
  a time:Interval ;
  rdfs:label "2001" ;
  rdf:value "2001"^^xsd:gYear ;
  time:intervalEquals <http://placetime.com/interval/gregorian/2001-01-01T00:00:00Z/P1Y> .

<http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00/type/bicycle>
  a scovo:Item ;
  rdfs:label "8 Oct 2001 17:00-18:00 - East - Salterton Road, EAST OF DINAN WAY, EXMOUTH"@en ;
  traffic:count <http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00> ;
  traffic:countPoint <http://transport.data.gov.uk/id/traffic-count-point/13> ;
  traffic:direction <http://dbpedia.org/resource/East> ;
  traffic:countHour <http://placetime.com/interval/gregorian/2001-10-08T17:00:00Z/PT1H> ;
  traffic:vehicleType <http://transport.data.gov.uk/def/vehicle/bicycle> ;
  rdf:value 2 .

Creating Linked Data - Part IV: Developing RDF Schemas

Jeni — Thu, 26 Nov 2009 10:35:32 +0000

This is the fourth instalment in a series about turning an existing dataset into some linked data. I’ve previously talked about analysis and modelling, defining URIs and defining concept schemes. In this instalment, we’ll look at developing a schema in which we define the classes, properties and datatypes that we want to use in the RDF that describes the things in our dataset.

We’ll start by writing out some RDF for our record, using Turtle here for readability, and use unprefixed names to indicate classes, properties and datatypes, just so we can see what we need. Then we’ll see how those requirements match up to existing vocabularies and ontologies that we can reuse. Anything that’s left over we’re going to have to put in our own vocabulary. We’ll call this

http://transport.data.gov.uk/def/traffic/

All the classes, properties and datatypes that we define will eventually use that namespace.

Let’s focus on this record; I find it easiest to use an actual example rather than talk in abstract:

"England","South West","K",1115.00,"18","Devon County Council",
13,"B3178",,"B Urban","Salterton Road",
"Salterton Road, EAST OF DINAN WAY, EXMOUTH",302600,81984,
8/10/2001 00:00:00,"E",17,2,2,400,5,41,0,2,0,0,0,0,2,450

We’ll put this into RDF bit by bit.

Areas

First, let’s look at the areas and local authorities. The kind of RDF that we want to have looks like:

<http://statistics.data.gov.uk/id/country?name=England>
  a :Country ;
  :name "England"@en .

<http://statistics.data.gov.uk/id/government-office-region/K>
  a :GovernmentOfficeRegion ;
  :name "South West"@en ;
  :code "K"^^:ONScode ;
  :containedBy <http://statistics.data.gov.uk/id/country?name=England> .

<http://statistics.data.gov.uk/id/local-authority-district/18>
  a :LocalAuthorityDistrict ;
  :code "18"^^:ONScode ;
  :code "1115"^^:DfTLAcode ;
  :localAuthority <http://statistics.data.gov.uk/id/local-authority/18> ;
  :containedBy <http://statistics.data.gov.uk/id/country?name=England> ;
  :containedBy <http://statistics.data.gov.uk/id/government-office-region/K> .

<http://transport.data.gov.uk/id/local-authority-district/1115>
  :sameAs <http://statistics.data.gov.uk/id/local-authority-district/18> .

<http://statistics.data.gov.uk/id/local-authority/18>
  a :LocalAuthority ;
  :name "Devon County Council"@en ;
  :code "18"^^:ONSLAcode ;
  :code "1115"^^:DfTLAcode ;
  :localAuthorityDistrict <http://statistics.data.gov.uk/id/local-authority-district/18> .

<http://transport.data.gov.uk/id/local-authority/1116>
  :sameAs <http://statistics.data.gov.uk/id/local-authority/18> .

To work out what we need to put in our schema, we should first look at what existing vocabularies there are that could help. These areas are already defined elsewhere, so we can just use the same vocabulary for countries, regions, local authority districts and local authorities as is used there. The vocabularies that are useful here are:

http://statistics.data.gov.uk/def/administrative-geography/ which defines classes and properties related to administrative areas and local authorities (as described by the Office of National Statistics)
http://data.ordnancesurvey.co.uk/ontology/admingeo/ which also defines classes and properties related to administrative areas (as described by the Ordnance Survey)
http://data.ordnancesurvey.co.uk/ontology/spatialrelations/, also developed by John Goodwin at the Ordnance Survey, which defines spatial relationships between areas

There are other commonly used vocabularies that it’s helpful to know about:

RDFS is designed for representing RDF schemas, but it has a few general-purpose properties that are good to know, namely rdfs:label (the label for a thing) and rdfs:comment (a comment or description about the thing).
SKOS is designed for representing concept schemes, but again it has a few properties that can be used with any set of linked data, in particular skos:prefLabel (the preferred label for a thing), skos:altLabel (an alternative label for a thing) and skos:notation (a code for the thing).
OWL is designed for representing ontologies, but it has one very important property that you should know about — owl:sameAs — which is used to link two things that are the same thing.
XML Schema datatypes can be used within RDF, which is useful for things like dates, times, integers and so on.
For our purposes here, OWL-Time is going to prove useful, as it has a bunch of properties that are used to represent instants and durations.

If we look through the RDF above, the only thing that isn’t covered by these vocabularies is the DfTLAcode datatype. If we use the http://transport.data.gov.uk/def/traffic/ namespace, there’s not really any need to indicate that this is a transport-related code, so we can just call it LAcode. Let’s define that datatype:

<http://transport.data.gov.uk/def/traffic/LAcode>
  a rdfs:Datatype ;
  rdfs:label "Local Authority Code"@en .

That’s it. Now here’s the Turtle for the areas with the relevant namespaces added, and property names changed where appropriate:

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#"> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix area: <http://statistics.data.gov.uk/def/administrative-geography/> .
@prefix space: <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/> .
@prefix admingeo: <http://data.ordnancesurvey.co.uk/ontology/admingeo/> .
@prefix traffic: <http://transport.data.gov.uk/def/traffic/> .

<http://statistics.data.gov.uk/id/country?name=England>
  a area:Country ;
  rdfs:label "England"@en .

<http://statistics.data.gov.uk/id/government-office-region/K>
  a admingeo:GovernmentOfficeRegion ;
  rdfs:label "South West"@en ;
  skos:notation "K"^^area:StandardCode ;
  area:country <http://statistics.data.gov.uk/id/country?name=England> .

<http://statistics.data.gov.uk/id/local-authority-district/18>
  a area:LocalAuthorityDistrict ;
  skos:notation "18"^^area:StandardCode ;
  skos:notation "1115"^^traffic:LAcode ;
  area:localAuthority <http://statistics.data.gov.uk/id/local-authority/18> ;
  area:country <http://statistics.data.gov.uk/id/country?name=England> ;
  area:region <http://statistics.data.gov.uk/id/government-office-region/K> .

<http://transport.data.gov.uk/id/local-authority-district/1115>
  owl:sameAs <http://statistics.data.gov.uk/id/local-authority-district/18> .

<http://statistics.data.gov.uk/id/local-authority/18>
  a area:LocalAuthority ;
  rdfs:label "Devon County Council"@en ;
  skos:notation "18"^^area:StandardCode ;
  skos:notation "1115"^^traffic:LAcode ;
  area:coverage <http://statistics.data.gov.uk/id/local-authority-district/18> .

<http://transport.data.gov.uk/id/local-authority/1116>
  owl:sameAs <http://statistics.data.gov.uk/id/local-authority/18> .

Roads

Here’s the kind of RDF we want to create for roads:

<http://transport.data.gov.uk/id/road/B3178>
  a :Road ;
  :code "B3178"^^:RoadNumber .

Obviously, we need a class for roads:

<http://transport.data.gov.uk/def/traffic/Road>
  a rdfs:Class ;
  rdfs:label "Road"@en .

Wherever there’s a code, I like to reuse skos:notation. But it’s important to define a datatype for the values used with that notation because (as we saw with local authorities above) there may be several different coding schemes that apply to the same Thing, and we need to be able to distinguish between them in case they clash. So:

<http://transport.data.gov.uk/def/traffic/RoadNumber>
  a rdfs:Datatype ;
  rdfs:label "Road Number"@en .

That’s all we have to define for roads; now the RDF can look like:

@prefix traffic: <http://transport.data.gov.uk/def/traffic/> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .

<http://transport.data.gov.uk/id/road/B3178>
  a traffic:Road ;
  skos:notation "B3178"^^traffic:RoadNumber .

Count Points

On to count points. Here’s the sketch of the RDF we want to create:

<http://transport.data.gov.uk/id/traffic-count-point/13>
  a :TrafficCountPoint ;
  :description "Salterton Road, EAST OF DINAN WAY, EXMOUTH"@en ;
  :code "13"^^:CountPointNumber ;
  :road <http://transport.data.gov.uk/id/road/B3178> ;
  :roadName "Salterton Road"@en ;
  :roadCategory 
    <http://transport.data.gov.uk/def/road-category/b> ,
    <http://transport.data.gov.uk/def/road-category/urban> ;
  :easting 302600 ;
  :northing 81984 ;
  :localAuthority <http://statistics.data.gov.uk/id/local-authority/18> ;
  :localAuthorityDistrict <http://statistics.data.gov.uk/id/local-authority-district/18> .

Of these, the description could be done with rdfs:comment. The code can be held by a skos:notation (provided we define a datatype for the count point number):

<http://transport.data.gov.uk/def/traffic/CountPointNumber>
  a rdfs:Datatype ;
  rdfs:label "Traffic Count Point Number"@en .

Properties for easting and northing are actually defined by the OS’s spatial relations ontology (although unfortunately neither the ontology nor the property is currently resolvable; the only way you’d know this is through looking at their use in the conversion of the edubase data). Links to local authorities and local authority districts can be done using the ONS-based administrative geography ontology, which again is currently only guessable at by looking at the online data.

That leaves us with a traffic:CountPoint class (no point calling it TrafficCountPoint if the namespace provides sufficient disambiguation):

<http://transport.data.gov.uk/def/traffic/CountPoint>
  a rdfs:Class ;
  rdfs:label "Traffic Count Point"@en .

A road property to point to a road:

<http://transport.data.gov.uk/def/traffic/road>
  a rdf:Property ;
  rdfs:label "road"@en ;
  rdfs:range <http://transport.data.gov.uk/def/traffic/Road> .

Note that properties are by convention named with a lowercase first letter, whereas classes are named with an uppercase first letter. It’s a good idea to follow that convention. Note also that I’ve defined a rdfs:range for this property, which means that anything that’s the object in a RDF statement that involves this property must be a traffic:Road.

We need a road name property to give the name of the road at the count point.

<http://transport.data.gov.uk/def/traffic/road>
  a rdf:Property ;
  rdfs:label "road name"@en ;
  rdfs:range <http://transport.data.gov.uk/def/traffic/Road> .

We also need a road category property to point to the categor(ies) of the road at the count point:

<http://transport.data.gov.uk/def/traffic/roadCategory>
  a rdf:Property ;
  rdfs:label "road category"@en .

You’ll remember that we defined different road categories using SKOS, such that each road category is a skos:Concept. But to give a range to the traffic:roadCategory property, we need to create a class for all the things that are categories of road. These are all skos:Concepts, and we can indicate that through an rdfs:subClassOf property:

<http://transport.data.gov.uk/def/traffic/RoadCategory>
  a rdfs:Class ;
  rdfs:subClassOf skos:Concept ;
  rdfs:label "Road Category"@en .

use this as the range of the traffic:roadCategory property:

<http://transport.data.gov.uk/def/traffic/roadCategory>
  a rdf:Property ;
  rdfs:label "road category"@en ;
  rdfs:range <http://transport.data.gov.uk/def/traffic/RoadCategory> .

and amend the concept scheme we created to include references to this new class, for example:

<motorway> a traffic:RoadCategory ;
  skos:prefLabel "Motorway"@en ;
  skos:broader <major> ;
  skos:scopeNote "Major roads often used for long distance travel. They are usually three or more lanes in each direction and generally have the maximum speed limit of 70mph."@en ;
  skos:inScheme <> .

So here is the RDF with the relevant properties properly defined:

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix area: <http://statistics.data.gov.uk/def/administrative-geography/> .
@prefix space: <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/> .
@prefix traffic: <http://transport.data.gov.uk/def/traffic/> .

<http://transport.data.gov.uk/id/traffic-count-point/13>
  a traffic:CountPoint ;
  rdfs:comment "Salterton Road, EAST OF DINAN WAY, EXMOUTH"@en ;
  skos:notation "13"^^traffic:CountPointNumber ;
  traffic:road <http://transport.data.gov.uk/id/road/B3178> ;
  traffic:roadName "Salterton Road"@en ;
  traffic:roadCategory 
    <http://transport.data.gov.uk/def/road-category/b> ,
    <http://transport.data.gov.uk/def/road-category/urban> ;
  space:easting 302600 ;
  space:northing 81984 ;
  area:localAuthority <http://statistics.data.gov.uk/id/local-authority/18> ;
  area:district <http://statistics.data.gov.uk/id/local-authority-district/18> .

Traffic Counts

On to traffic counts. The un-namespaced RDF should look like:

<http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00>
  a :TrafficCount ;
  :countPoint <http://transport.data.gov.uk/id/traffic-count-point/13> ;
  :direction <http://dbpedia.org/resource/East> ;
  :hour <http://placetime.com/interval/gregorian/2001-10-08T17:00:00Z/PT1H> .

So for that we need a class for traffic counts:

<http://transport.data.gov.uk/def/traffic/Count>
  a rdfs:Class ;
  rdfs:label "Traffic Count"@en .

a property that can link to the traffic count to the count point where the count is taken:

<http://transport.data.gov.uk/def/traffic/countPoint>
  a rdf:Property ;
  rdfs:label "traffic count point"@en ;
  rdfs:range <http://transport.data.gov.uk/def/traffic/CountPoint> .

a property to link to the the direction the traffic is flowing in (we can’t put a range on this one because the DBPedia resources we’re using don’t have a common type):

<http://transport.data.gov.uk/def/traffic/direction>
  a rdf:Property ;
  rdfs:label "traffic direction"@en .

and finally a property to link to the hour during which the measurement was taken. This last one is a very common thing to need to do, so we’d imagine that there might be an existing property defined somewhere that we could use. SDMX, which includes a standard for representing statistical information in XML, defines a REF_PERIOD field which would seem to suit our purposes, but we don’t yet have a proper mapping of SDMX into RDF (I’ve had an initial cut, but it needs some input from statisticians).

So for now, we’ll use a specific property in our own namespace; we can always indicate that it’s a sub-property of a future SDMX property at a later date. I’m going to call it countHour and give it a domain of traffic:Count to indicate that the property has a pretty specific use for providing the count for an hour. We could just give its range as a generic time:Interval, but the kind of hours that are traffic count hours are kinda special intervals: they’re obviously an hour long, but are also restricted to start and end on the hour, cover an hour between 7am and 7pm, and don’t occur in winter. So it feels like we should have a special kind of interval for that purpose:

<http://transport.data.gov.uk/def/traffic/countHour>
  a rdf:Property ;
  rdfs:label "hour of count"@en ;
  rdfs:domain <http://transport.data.gov.uk/def/traffic/Count> ;
  rdfs:range <http://transport.data.gov.uk/def/traffic/CountHour> .

<http://transport.data.gov.uk/def/traffic/CountHour>
  a rdfs:Class ;
  rdfs:subClassOf time:Interval ;
  rdfs:label "Count Hour"@en .

All those properties were in the traffic namespace, so here’s the RDF with it added:

@prefix traffic: <http://transport.data.gov.uk/def/traffic/> .

<http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00>
  a traffic:Count ;
  traffic:countPoint <http://transport.data.gov.uk/id/traffic-count-point/13> ;
  traffic:direction <http://dbpedia.org/resource/East> ;
  traffic:countHour <http://placetime.com/interval/gregorian/2001-10-08T17:00:00Z/PT1H> .

Cardinal Directions

As I discussed in the last instalment, we’re not actually going to mint URIs for cardinal directions, but that doesn’t mean we can’t make statements about them in the RDF we generate. As I’ll discuss in more depth in the next instalment, it’s always good to provide a label at the very least:

<http://dbpedia.org/resource/East>
  rdfs:label "East"@en .

Intervals and Instants

Let’s look now at the RDF we want to generate about the hour during which the count was taken. As I’ve said above, these hours are a special kind of interval, and we’ve already created a class for them. I also discussed earlier that the things about this interval that are really useful for the purposes of querying are the year during which the count was taken and the hour at which it was taken, so we should pull out at least those pieces of information. Time-based data can be represented in RDF using the OWL-Time ontology.

Unfortunately, expressing time very specifically gets. This is what the statements we want to make look like using OWL-Time:

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix time: <http://www.w3.org/2006/time> .

<http://placetime.com/interval/gregorian/2001-10-08T17:00:00Z/PT1H>
  a traffic:CountHour ;
  rdfs:label "8 Oct 2001, 17:00-18:00"@en ;
  time:hasBeginning <http://placetime.com/instant/gregorian/2001-10-08T17:00:00Z> ;
  time:hasEnd <http://placetime.com/instant/gregorian/2001-10-08T18:00:00Z> ;
  time:hasDurationDescription _:OneHour ;
  time:intervalDuring <http://dbpedia.org/resource/2001> .

_:OneHour a time:DurationDescription ;
  rdfs:label "one hour"@en ;
  time:years 0 ;
  time:months 0 ;
  time:days 0 ;
  time:hours 1 ;
  time:minutes 0 ;
  time:seconds 0 .

<http://placetime.com/instant/gregorian/2001-10-08T17:00:00Z>
  a time:Instant ;
  rdfs:label "8 Oct 2001, 17:00"@en ;
  time:inXSDDateTime "2001-10-08T17:00:00Z"^^xsd:dateTime ;
  time:inDateTime [
    a time:DateTimeDescription ;
    time:unitType time:unitHour ;
    time:year "2001"^^xsd:gYear ;
    time:month "--10"^^xsd:gMonth ;
    time:day "---08"^^xsd:gDay ;
    time:hour 17 ;
  ] .

<http://placetime.com/interval/gregorian/2001-10-08T18:00:00Z>
  a time:Instant ;
  rdfs:label "8 Oct 2001, 18:00"@en ;
  time:inXSDDateTime "2001-10-08T18:00:00Z"^^xsd:dateTime ;
  time:inDateTime [
    a time:DateTimeDescription ;
    time:unitType time:unitHour ;
    time:year "2001"^^xsd:gYear ;
    time:month "--10"^^xsd:gMonth ;
    time:day "---08"^^xsd:gDay ;
    time:hour 18 ;
  ] .

<http://dbpedia.org/resource/2001>
  a time:Interval ;
  rdfs:label "2001" ;
  rdf:value "2001"^^xsd:gYear ;
  time:intervalEquals <http://placetime.com/interval/gregorian/2001-01-01T00:00:00Z/P1Y> .

Observations

Finally we’re on to the observations themselves. The un-namespaced RDF looks like:

<http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00/type/bicycle>
  a :Observation ;
  :count <http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00> ;
  :vehicleType <http://transport.data.gov.uk/def/vehicle/bicycle> ;
  :value 2 .

The SCOVO vocabulary exists to represent statistical information like this. In SCOVO, observations are called scovo:Items, the value of the statistical measure itself (the count in this case) should be held in the rdf:value property, and any other properties should be subtypes of scovo:dimension, which has a domain of scovo:Dimension.

To fit in with SCOVO, then, we need to have the pointer to the count that this observation belongs to as a property that is a sub-property of scovo:dimension:

<http://transport.data.gov.uk/def/traffic/count>
  a rdf:Property ;
  rdfs:subPropertyOf scovo:dimension ;
  rdfs:label "count"@en ;
  rdfs:range <http://transport.data.gov.uk/def/traffic/Count> .

We might be tempted to indicate that the type of thing pointed to by the traffic:count property is a subclass of scovo:Dimension, but this is unnecessary and probably untrue: there might exist some traffic counts that aren’t dimensions, and the ones that are will be linked to by the traffic:count property can be inferred to be dimensions.

Similarly, the property that provides the pointer to the vehicle type should be a sub-property of scovo:dimension and we need a class for those various vehicle types in order to restrict the range of that property:

<http://transport.data.gov.uk/def/vehicleType>
  a rdf:Property ;
  rdfs:subPropertyOf scovo:dimension ;
  rdfs:label "vehicle type"@en ;
  rdfs:range <http://transport.data.gov.uk/def/VehicleType> .

<http://transport.data.gov.uk/def/VehicleType>
  a rdfs:Class ;
  rdfs:subClassOf skos:Concept ;
  rdfs:label "Vehicle Type"@en .

Of course all the concepts that we created for the vehicle types need to be designated as instances of this new traffic:VehicleType class:

<bicycle> a traffic:VehicleType ;
  ... .

So, the RDF with the proper namespaces is:

@prefix scovo: <http://purl.org/NET/scovo#> .
@prefix traffic: <http://transport.data.gov.uk/def/traffic/> .

<http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00/type/bicycle>
  a scovo:Item ;
  traffic:count <http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00> ;
  traffic:vehicleType <http://transport.data.gov.uk/def/vehicle/bicycle> ;
  rdf:value 2 .

That concludes our initial walkthrough of the data to create a vocabulary. I’ve duplicated the schema and the example data below so that it’s all in one place. But it’s not quite done. In the next instalment, I’ll look at adding some finishing touches that make the RDF easier to use.

Schema

This is the full schema. It contains just six classes, seven properties and three datatypes at the moment, so it’s pretty small as vocabularies go. We’ve been able to reuse a lot of classes, properties and datatypes that have already been defined elsewhere in the RDF itself, so this vocabulary is pretty focused on just what we need to describe traffic counts.

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix scovo: <http://purl.org/NET/scovo#> .
@prefix time: <http://www.w3.org/2006/time> .

# Classes #

<http://transport.data.gov.uk/def/traffic/Road>
  a rdfs:Class ;
  rdfs:label "Road"@en .

<http://transport.data.gov.uk/def/traffic/CountPoint>
  a rdfs:Class ;
  rdfs:label "Traffic Count Point"@en .

<http://transport.data.gov.uk/def/traffic/Count>
  a rdfs:Class ;
  rdfs:label "Traffic Count"@en .

<http://transport.data.gov.uk/def/traffic/RoadCategory>
  a rdfs:Class ;
  rdfs:subClassOf skos:Concept ;
  rdfs:label "Road Category"@en .    

<http://transport.data.gov.uk/def/traffic/CountHour>
  a rdfs:Class ;
  rdfs:subClassOf time:Interval ;
  rdfs:label "Count Hour"@en .

<http://transport.data.gov.uk/def/VehicleType>
  a rdfs:Class ;
  rdfs:subClassOf skos:Concept ;
  rdfs:label "Vehicle Type"@en .

# Properties #

<http://transport.data.gov.uk/def/traffic/road>
  a rdf:Property ;
  rdfs:label "road name"@en ;
  rdfs:range <http://transport.data.gov.uk/def/traffic/Road> .

<http://transport.data.gov.uk/def/traffic/countPoint>
  a rdf:Property ;
  rdfs:label "traffic count point"@en ;
  rdfs:range <http://transport.data.gov.uk/def/traffic/CountPoint> .

<http://transport.data.gov.uk/def/traffic/count>
  a rdf:Property ;
  rdfs:subPropertyOf scovo:dimension ;
  rdfs:label "count"@en ;
  rdfs:range <http://transport.data.gov.uk/def/traffic/Count> .

<http://transport.data.gov.uk/def/traffic/roadCategory>
  a rdf:Property ;
  rdfs:label "road category"@en ;
  rdfs:range <http://transport.data.gov.uk/def/traffic/RoadCategory> .

<http://transport.data.gov.uk/def/traffic/direction>
  a rdf:Property ;
  rdfs:label "traffic direction"@en .

<http://transport.data.gov.uk/def/traffic/countHour>
  a rdf:Property ;
  rdfs:label "hour of count"@en ;
  rdfs:domain <http://transport.data.gov.uk/def/traffic/Count> ;
  rdfs:range <http://transport.data.gov.uk/def/traffic/CountHour> .

<http://transport.data.gov.uk/def/vehicleType>
  a rdf:Property ;
  rdfs:subPropertyOf scovo:dimension ;
  rdfs:label "vehicle type"@en ;
  rdfs:range <http://transport.data.gov.uk/def/VehicleType> .

# Datatypes #

<http://transport.data.gov.uk/def/traffic/LAcode>
  a rdfs:Datatype ;
  rdfs:label "Local Authority Code"@en .

<http://transport.data.gov.uk/def/traffic/RoadNumber>
  a rdfs:Datatype ;
  rdfs:label "Road Number"@en .

<http://transport.data.gov.uk/def/traffic/CountPointNumber>
  a rdfs:Datatype ;
  rdfs:label "Traffic Count Point Number"@en .

RDF Data

Here’s a sample set of data. It looks like rather a lot to simply describe the number of bicycles at a particular point on a road (and it doesn’t even include the SKOS concept schemes that we did last time), but (a) it all provides valuable context for that measurement and (b) most of it will be reused by a lot of other measurements.

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#"> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix time: <http://www.w3.org/2006/time> .
@prefix scovo: <http://purl.org/NET/scovo#> .
@prefix area: <http://statistics.data.gov.uk/def/administrative-geography/> .
@prefix admingeo: <http://data.ordnancesurvey.co.uk/ontology/admingeo/> .
@prefix space: <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/> .
@prefix traffic: <http://transport.data.gov.uk/def/traffic/> .

<http://statistics.data.gov.uk/id/country?name=England>
  a area:Country ;
  rdfs:label "England"@en .

<http://statistics.data.gov.uk/id/government-office-region/K>
  a admingeo:GovernmentOfficeRegion ;
  rdfs:label "South West"@en ;
  skos:notation "K"^^area:StandardCode ;
  area:country <http://statistics.data.gov.uk/id/country?name=England> .

<http://statistics.data.gov.uk/id/local-authority-district/18>
  a area:LocalAuthorityDistrict ;
  skos:notation "18"^^area:StandardCode ;
  skos:notation "1115"^^traffic:LAcode ;
  area:localAuthority <http://statistics.data.gov.uk/id/local-authority/18> ;
  area:country <http://statistics.data.gov.uk/id/country?name=England> ;
  area:region <http://statistics.data.gov.uk/id/government-office-region/K> .

<http://transport.data.gov.uk/id/local-authority-district/1115>
  owl:sameAs <http://statistics.data.gov.uk/id/local-authority-district/18> .

<http://statistics.data.gov.uk/id/local-authority/18>
  a area:LocalAuthority ;
  rdfs:label "Devon County Council"@en ;
  skos:notation "18"^^area:StandardCode ;
  skos:notation "1115"^^traffic:LAcode ;
  area:coverage <http://statistics.data.gov.uk/id/local-authority-district/18> .

<http://transport.data.gov.uk/id/local-authority/1116>
  owl:sameAs <http://statistics.data.gov.uk/id/local-authority/18> .

<http://transport.data.gov.uk/id/road/B3178>
  a traffic:Road ;
  skos:notation "B3178"^^traffic:RoadNumber .

<http://transport.data.gov.uk/id/traffic-count-point/13>
  a traffic:CountPoint ;
  rdfs:comment "Salterton Road, EAST OF DINAN WAY, EXMOUTH"@en ;
  skos:notation "13"^^traffic:CountPointNumber ;
  traffic:road <http://transport.data.gov.uk/id/road/B3178> ;
  traffic:roadName "Salterton Road"@en ;
  traffic:roadCategory 
    <http://transport.data.gov.uk/def/road-category/b> ,
    <http://transport.data.gov.uk/def/road-category/urban> ;
  space:easting 302600 ;
  space:northing 81984 ;
  area:localAuthority <http://statistics.data.gov.uk/id/local-authority/18> ;
  area:district <http://statistics.data.gov.uk/id/local-authority-district/18> .

<http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00>
  a traffic:Count ;
  traffic:countPoint <http://transport.data.gov.uk/id/traffic-count-point/13> ;
  traffic:direction <http://dbpedia.org/resource/East> ;
  traffic:countHour <http://placetime.com/interval/gregorian/2001-10-08T17:00:00Z/PT1H> .

<http://dbpedia.org/resource/East>
  rdfs:label "East"@en .

<http://placetime.com/interval/gregorian/2001-10-08T17:00:00Z/PT1H>
  a traffic:CountHour ;
  rdfs:label "8 Oct 2001, 17:00-18:00"@en ;
  time:hasBeginning <http://placetime.com/instant/gregorian/2001-10-08T17:00:00Z> ;
  time:hasEnd <http://placetime.com/instant/gregorian/2001-10-08T18:00:00Z> ;
  time:hasDurationDescription _:OneHour ;
  time:intervalDuring <http://dbpedia.org/resource/2001> .

_:OneHour a time:DurationDescription ;
  rdfs:label "one hour"@en ;
  time:years 0 ;
  time:months 0 ;
  time:days 0 ;
  time:hours 1 ;
  time:minutes 0 ;
  time:seconds 0 .

<http://placetime.com/instant/gregorian/2001-10-08T17:00:00Z>
  a time:Instant ;
  rdfs:label "8 Oct 2001, 17:00"@en ;
  time:inXSDDateTime "2001-10-08T17:00:00Z"^^xsd:dateTime ;
  time:inDateTime [
    a time:DateTimeDescription ;
    time:unitType time:unitHour ;
    time:year "2001"^^xsd:gYear ;
    time:month "--10"^^xsd:gMonth ;
    time:day "---08"^^xsd:gDay ;
    time:hour 17 ;
  ] .

<http://placetime.com/instant/gregorian/2001-10-08T18:00:00Z>
  a time:Instant ;
  rdfs:label "8 Oct 2001, 18:00"@en ;
  time:inXSDDateTime "2001-10-08T18:00:00Z"^^xsd:dateTime ;
  time:inDateTime [
    a time:DateTimeDescription ;
    time:unitType time:unitHour ;
    time:year "2001"^^xsd:gYear ;
    time:month "--10"^^xsd:gMonth ;
    time:day "---08"^^xsd:gDay ;
    time:hour 18 ;
  ] .

<http://dbpedia.org/resource/2001>
  a time:Interval ;
  rdfs:label "2001" ;
  rdf:value "2001"^^xsd:gYear ;
  time:intervalEquals <http://placetime.com/interval/gregorian/2001-01-01T00:00:00Z/P1Y> .

<http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00/type/bicycle>
  a scovo:Item ;
  traffic:count <http://transport.data.gov.uk/id/traffic-count-point/13/direction/E/hour/2001-10-08T17:00:00> ;
  traffic:vehicleType <http://transport.data.gov.uk/def/vehicle/bicycle> ;
  rdf:value 2 .

Creating Linked Data - Part III: Defining Concept Schemes

Jeni — Sun, 22 Nov 2009 21:04:41 +0000

This is the third instalment in a series that I’m writing about turning data into linked data. I’m using traffic count data as the example, since that’s a dataset that I’m currently working on. In the last two instalments, I talked about analysing and modelling the data and about designing URIs for the things in that model.

Within the model, there are three sets of things that are concepts:

road categories
vehicle types
cardinal directions

As I discussed last time, cardinal directions have URIs defined within DBPedia which are good enough for our purposes. The categorisation of roads and vehicles, on the other hand, is something specific to UK transport data, so they are up to us to define.

There’s a really useful RDF vocabulary called SKOS which is designed precisely for defining the kind of concept schemes that we want to use here. SKOS provides classes for concepts, concept schemes and collections (groupings of concepts within a scheme), and properties for linking them and providing labels, codes, definitions and so forth. Many of the SKOS properties can be used outside concept schemes — for example skos:prefLabel can be used anywhere you want to indicate the preferred label for a thing — so it’s good to get to know them.

Vehicle Types

Before we dive into RDF, let’s take some time to understand the classification that we need to model. We’re modelling vehicle types because counts are made of each different type of vehicle passing a traffic count point over a particular hour. Within the CSV data, the relevant column headings are:

Pedal cycles
Two wheeled motor vehicles
Cars and taxis
Buses and coaches
Light vans
HGVr2
HGVr3
HGVr4+
HGVa3/4
HGVa5
HGVa6
All HGV
All motor vehicles

These classifications are detailed in the Department for Transport documentation of the dataset. It’s clear that it’s not a flat classification, but can be arranged into a hierarchy as follows:

+- Pedal cycles
+- All motor vehicles
   +- Two wheeled motor vehicles
   +- Cars and taxis
   +- Buses and coaches
   +- Light vans
   +- All HGV
      +- Rigid HGV
      |  +- HGVr2
      |  +- HGVr3
      |  +- HGVr4+
      +- Articulated HGV
         +- HGVa3/4
         +- HGVa5
         +- HGVa6

So all we have to do is define that in SKOS. We’ve already decided that the URIs will look like:

http://transport.data.gov.uk/def/vehicle-category/{type}

so for URI-hackability reasons we’ll call the concept scheme:

http://transport.data.gov.uk/def/vehicle-category/

It’s probably easiest to just show what the concept scheme looks like. This is in Turtle.

@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@base <http://transport.data.gov.uk/def/vehicle-category/> .

<> a skos:ConceptScheme ;
  skos:prefLabel "Vehicle Types"@en ;
  skos:hasTopConcept <bicycle> ;
  skos:hasTopConcept <motor-vehicle> .
...
<motor-vehicle> a skos:Concept ;
  skos:prefLabel "Motor Vehicle"@en ;
  skos:topConceptOf <> ;
  skos:narrower <motorbike> ;
  skos:narrower <car> ;
  skos:narrower <bus> ;
  skos:narrower <van> ;
  skos:narrower <HGV> .
...
<HGV> a skos:Concept ;
  skos:prefLabel "Heavy Goods Vehicle"@en ;
  skos:altLabel "HGV"@en ;
  skos:definition "Goods vehicles over 3,500 kgs gross vehicle weight."@en ;
  skos:scopeNote "Includes tractors (without trailers), road rollers, box vans and similar large vans. A two axle motor tractive unit without trailer is also included."@en ;
  skos:broader <motor-vehicle> ;
  skos:narrower <HGVr> ;
  skos:narrower <HGVa> ;
  skos:inScheme <> .
...

The properties shown here are:

skos:prefLabel - the preferred label for something; there can only be one in any given language
skos:altLabel - an alternative label for the thing; there can be any number
skos:definition - provides a definition of the term
skos:scopeNote - provides information about the scope of the term (eg what’s included or excluded)
skos:broader/skos:narrower - link together concepts into a hierarchy
skos:hasTopConcept/skos:topConceptOf - links together the concept schemes and the concepts at the top of the concept hierarchy defined within the scheme
skos:inScheme - points from a concept the concept scheme it’s defined in; it’s necessary to use either this or skos:topConceptOf on every skos:Concept otherwise it’s not clear which concept scheme they belong to

Note that in the RDF I’ve assigned every string a language (English). That’s good practice when values are textual; a Welsh translation could be provided for each one as well, for example.

Road Categories

Road categories are also described within the documentation for this dataset. The hierarchy is shown in the documentation as:

+- Major Roads
|  +- Motorways
|  |  +- Trunk
|  |  +- Principal
|  +- A Roads
|     +- Trunk
|     |  +- Urban
|     |  +- Rural
|     +- Principal
|        +- Urban
|        +- Rural
+- Minor Roads
   +- B Roads
   |  +- Urban
   |  +- Rural
   +- C Roads
   |  +- Urban
   |  +- Rural
   +- Unclassified Roads
      +- Urban
      +- Rural

But this is actually the result of three sets of overlapping concepts:

roads by classification (major/minor, motorway/A/B/C/unclassified)
roads by locale (urban/rural)
major roads by maintenance responsibility (trunk/principal)

These kinds of subdivisions of concepts can be managed in SKOS through skos:Collections, which group together concepts without being broader than those concepts. Here’s a snippet from the concept scheme that shows how this works.

@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@base <http://transport.data.gov.uk/def/road-category/> .

<> a skos:ConceptScheme ;
  skos:prefLabel "Road Categories"@en ;
  skos:hasTopConcept <major> ;
  skos:hasTopConcept <minor> ;
  skos:hasTopConcept <urban> ;
  skos:hasTopConcept <rural> .

<classification> a skos:Collection ;
  skos:prefLabel "Road by Classification"@en ;
  skos:member <major> ;
  skos:member <minor> .

<maintenance> a skos:Collection ;
  skos:prefLabel "Major Road by Maintenance Responsibility"@en ;
  skos:member <principal> ;
  skos:member <trunk> .

<major> a skos:Concept ;
  skos:prefLabel "Major Road"@en ;
  skos:altLabel "Major"@en ;
  skos:scopeNote "Include motorways and A roads. These roads usually have high traffic flows and are often the main arteries to major destinations."@en ;
  skos:narrower <motorway> ;
  skos:narrower <a> ;
  skos:narrower <principal> ;
  skos:narrower <trunk> ;
  skos:topConceptOf <> .

<motorway> a skos:Concept ;
  skos:prefLabel "Motorway"@en ;
  skos:broader <major> ;
  skos:scopeNote "Major roads often used for long distance travel. They are usually three or more lanes in each direction and generally have the maximum speed limit of 70mph."@en ;
  skos:inScheme <> .
...
<trunk> a skos:Concept ;
  a skos:Concept ;
  skos:prefLabel "Trunk Road"@en ;
  skos:altLabel "Trunk"@en ;
  skos:scopeNote "Most motorways and many of the long distance rural A roads are trunk roads."@en ;
  skos:note "The responsibility for the maintenance of trunk roads lies with the Secretary of State and they are managed by the Highways Agency in England, the National Assembly of Wales in Wales and the Scottish Executive in Scotland (National Through Routes)."@en ;
  skos:broader <major> ;
  skos:inScheme <> .
...

In a hierarchy, these multiple overlapping concepts can be shown as:

+- <Road by Classification>
|  +- Major Road
|  |  +- <Major Road by Classification>
|  |  |  +- Motorway
|  |  |  +- A Road
|  |  +- <Major Road by Maintenance Responsibility>
|  |     +- Principal Road
|  |     +- Trunk Road
|  +- Minor Road
|     +- B Road
|     +- C Road
|     +- Unclassified Road
+- <Road by Locale>
   +- Urban Road
   +- Rural Road

That’s our concept schemes done. Next it will be time to turn to defining a vocabulary for the particular things that we want to describe from this dataset.