rdfQuery

On Resolvability

Jeni — Fri, 28 Aug 2009 22:02:16 +0000

In my last post about RDFa and HTML I talked about how one of the gulfs that separates the HTML5 and Semantic Web communities is the attitude to the resolvability of property (and class) URIs.

I’m currently experimenting with introducing the ability to automatically locate information about properties and other resources that are referenced within triples to rdfQuery, so now is a good time, as far as I’m concerned, to look more closely at what the ability to resolve properties gives you and how to avoid problems if the property URI is (temporarily or permanently) unresolvable or resolvable to something new.

I’m going to attempt to answer:

How do or might applications use property and class URIs?
How can data and ontology publishers assist them in doing so?
What should frameworks (such as rdfQuery) do to help application developers?

Application Developers

We can divide applications using online data into three general categories:

data-specific applications are constructed around particular data sets that are known to the developer of the application; the visualisations that I’ve been doing are examples of data-specific applications
vocabulary-specific applications are constructed around particular vocabularies, wherever the data might be found that uses them; Google’s Social Graph API and Yahoo! SearchMonkey are examples
generic applications are constructed to navigate through any RDF that they find; Tabulator is one example

Most mashups are data-specific applications. When you, as a developer, create a data-specific application, the thing that you need to know most of all is what information the dataset contains. Part of that is working out the meaning of a particular property (or class). What the data publisher needs to do is make sure that the data they publish is documented.

There are three ways of locating the documentation about a particular property or class:

looking through the general documentation the data publisher has provided
resolving the URI of the class or property
searching

For a developer, it’s very useful to find out about a property by bunging its URI into a browser and hitting return. Want to know what http://xmlns.com/foaf/0.1/name means? Look up that URI. By comparison, if you want to know what a vevent is, your best bet is a search engine. In the results I get from Google, the microformat definition of vevent is currently second on the list. (The Microdata definition of vevent doesn’t even feature.) Even if a property isn’t available at its URI, its URI gives a more unique identifier to search for than an short term: you’re more likely to find relevant information if you search for http://xmlns.com/foaf/0.1/name than if you search for name.

But there’s no requirement for data-specific applications to use computer-readable information about properties or classes. If you know the data that’s available in a dataset, you can find out the semantics of the properties and classes it contains and hard-code those within your application. Most applications that reuse data are currently of this type, and it tends to be the only kind that non-Semantic Web people think about.

Vocabulary-specific and generic applications will have some vocabularies built in but may also operate with unknown vocabularies. For example, an application that cares about FOAF profiles is almost certainly going to want to hard-code information about FOAF rather than download its schema every time it’s used.

There are three reasons for building-in knowledge about particular vocabularies:

some information about a vocabulary simply can’t be represented in a schema or ontology; if you want special handling for particular properties, you’re going to want to hard-code it
downloading, parsing and interpreting a schema that you know you’re going to need every time you run the application is really inefficient
relying on the network to provide information about a vocabulary you know you’re going to need makes your application fragile, especially if you do not have control over the publication of the schema yourself

It’s worth noting that applications increasingly do rely on the availability of networked resources in order to operate — that’s what cloud computing is all about — but the resources are usually ones that the application developers have some kind of control over.

It helps to use URIs for properties and classes for well-known vocabularies only in as much as it means that property and class names from different vocabularies won’t clash, so you don’t have to worry about your application confusing http://xmlns.com/foaf/0.1/title with http://purl.org/dc/terms/title.

On the other hand, if data uses an unknown vocabulary, vocabulary-specific and generic applications would like to get hold of extra information. This falls into three categories:

human-readable information includes things that help with the display of data, such as human-readable labels for properties and classes; the expected datatype of the values of a property might also fall into this category
mapping information helps applications map unknown properties and classes onto known ones; for example, if http://people.example.org/ontology/fullName is defined as a sub-property of http://xmlns.com/foaf/0.1/name then the application can use or display the value of http://people.example.org/ontology/fullName in exactly the same way as the value of http://xmlns.com/foaf/0.1/name
reasoning information helps applications draw further conclusions about the resources for which there’s information based on what they already know; for example, if http://people.example.org/ontology/fullName has a domain of http://xmlns.com/foaf/0.1/Person then anything that has the property http://people.example.org/ontology/fullName must be a http://xmlns.com/foaf/0.1/Person

These are in descending order of priority: many applications will want to interact with the user in some way, in which case human-readable information is vital. Applications that have built-in knowledge about one or more vocabularies are likely to have special handling for those vocabularies, so being able to map unknown properties and classes into those known vocabularies will enhance the behaviour of the application, although it adds a bit of complexity in the implementation to do so. Further reasoning has the potential to increase the value of sparse data but again increases the complexity of implementation.

Using URIs for classes and properties provides a mechanism for applications to get hold of this extra information about unknown vocabularies. They might try four tactics, in order of priority:

look at the data they already know; the information they need about the unknown properties and classes may be included in the files they’ve already accessed (including those containing data)
look in an application-specific (possibly cloud-hosted) cache of vocabularies that the application has already downloaded
resolve the URI of the class or property by performing an HTTP GET (and add it to the application-specific cache)
look in a general-purpose cache, such as the Internet Archive or an ontology repository such as Swoogle

Robust applications will not break if they don’t manage to locate the definition of a property or class. They can certainly continue to parse any data that they come across. To create a human-readable label, they might use the part of the URI after the last # or /. It’s no loss (to the application) if they cannot perform other reasoning: they might display the data in some default way or simply ignore it.

It’s worth noting, because of the fear of DDoS attacks that some people have, that the majority of applications won’t need to actually GET property or class URIs, either because they are data-specific applications or because they only work with vocabularies that are hard-coded into them. Applications that are good web citizens will avoid DDoS attacks on popular vocabularies by hard-coding knowledge about those vocabularies and/or maintaining a cache, either locally or in the cloud, of vocabularies that have already been resolved.

Publishers

With what I’ve said above in mind, what can publishers do to help applications to understand the data that they provide?

If a publisher is only concerned about data-specific, point-to-point mashups, all they have to provide is the data itself. It will help developers if there is some documentation of the dataset and the properties and classes used within it. But data publishers who only want their data to be discoverable by people can rely on human intelligence for locating information, and for them using URIs for properties and classes may seem like overkill.

But in a linked data world, publishers should really support their data being discovered automatically via the links from other data. Here we’re talking about making life easier for vocabulary-specific and generic applications to use the data that you provide.

The vocabularies that you use within your data fall into three general categories:

well-known vocabularies are vocabularies that are commonly enough used that vocabulary-specific and generic applications are likely to have them built-in; these vocabularies tend to be useful across domains, such as FOAF, which is useful whenever you want to express information about people or organisations
local vocabularies are vocabularies that are specific to the dataset that you are publishing; you have as much control over their publication as you do over the publication of the data itself
reused vocabularies are vocabularies that you are using that are owned by other people but that do not have the take-up of well-known vocabularies; these are typically domain-specific; one example is Metalex, which is a vocabulary about legislation

As a data publisher, the first thing you can do is to use well-known vocabularies in your data wherever possible, even if you also use local or reused vocabularies to express the same properties or classes.

For example, say you have some data describing a cricket team and use http://cricket.example.org/ontology#name for the name of a member of a team, and that you mean it to be a sub-property of http://xmlns.com/foaf/0.1/name (which is itself a sub-property of http://www.w3.org/2000/01/rdf-schema#label). If you just publish the http://cricket.example.org/ontology#name property then the only way that a generic application can know that http://cricket.example.org/ontology#name can be used as a label for a resource (which is a person) is by attempting to resolve http://cricket.example.org/ontology and reasoning based on what it finds. On the other hand, if you also provide http://xmlns.com/foaf/0.1/name and http://www.w3.org/2000/01/rdf-schema#label properties, applications are no longer dependent on the network, nor on having the ability to reason, to use that information.

You could also provide mappings onto any reused vocabularies that you specialise, but this is less worthwhile given that vocabulary-specific and generic applications are unlikely to understand them either.

The second thing you can do is to include information about the properties that you use within the data that you publish. This isn’t important for well-known vocabularies (because they’re… uh… well-known) and it’s only useful for local vocabularies if you’re not publishing those vocabularies, because if someone can access your data, odds are they’re able to access your local vocabulary’s property URIs as well. But it is useful for reused vocabularies, where you can’t guarantee access, in just the same way as it’s useful to provide basic labelling information about any resources you reference.

If you’re publishing your data embeddded within a web page, as well as marking up the data, you can mark up the labels that you use for those values, which more than likely appear as headings in a table or something similar.

If you are publishing a schema or ontology that describes your properties and types, there are also things that you can do to help applications. The most important thing is to assist caches in their caching of the ontology, which will reduce the number of times that it needs to be accessed directly and help you avoid DDoS attacks: see Mark Nottingham’s Caching Tutorial. You can also reduce the number of hits on your server by using hash URIs for your property and class names and use standard load-balancing techniques to manage the traffic.

If you’re referring to reused vocabularies within your own, you can also embed information about the relevant properties and classes from those vocabularies within your own ontology. This can save applications an extra hop, and lessens the risk of the reused vocabulary disappearing (perhaps forever).

If you want to help people who might reuse your ontology, you can make the process of copying it easier by publishing it as a single file, or broken up into segments that are likely to be reused individually. At a non-technical level, it’s also a good idea to provide a announcement mailing list or a feed so that people who reuse your vocabulary can be kept up to date with any changes you make to it.

Framework Developers

Bearing all this in mind, what should I (and other framework developers) do to support the reusers of data? I think I need to make it easy for application developers to:

load in known ontologies from known locations
hard-code relevant semantics in the script
create catalogs that map known property or class names onto known locations of documents that contain details about them
use caching proxies when accessing unknown vocabularies
access vocabularies directly at the relevant URI
fallback on archives when the URI cannot be found

In other words, I need to make it easy for people to use a range of strategies for getting hold of information about a property or class, aside from simply trying to access it at its URI. I think that means that it’s better to provide a lightweight solution, giving developers the opportunity to be in control of which URIs get resolved rather than automatically downloading extra information from the URI that’s actually used for the property or class. It also means I need to provide hooks in the code that they can use to trigger that resolution.

It would also be useful, of course, for developers to be able to use information about properties and classes easily, in particular to reason with it. That kind of support is something I’ve been working on for rdfQuery. It’s not quite ready yet.

Conclusions

My (somewhat contentious) view is that we place too much emphasis on the resolvability of property and class names, and that this can put people off the idea of the Semantic Web. You can do useful things with data without resolving properties or classes. And for a large number of useful applications, being able to actually reason over the data you get at the end of a property URI would have a high implementation cost without providing a great deal of functional benefit.

Further, for data publishers, the requirement to enable the resolution of every property and class URI you use within your data just adds to the publishing burden, especially if you’re made to feel it has to resolve to some kind of grand OWL ontology.

There’s a concept in psychology of the zone of proximal development. The idea is that if someone is operating at a particular level then as a teacher you should help them to achieve something slightly above that level, rather than trying to get them to do everything straight away.

The same is true here. We need to help publishers make the small steps that they can make, one at a time, to gradually get them to full Semantic Web goodness:

publish a dataset in some kind of open format (CSV, XML etc) so that people can get hold of it
publish the data with distinct URIs for distinct resources so that people can reference them
publish the data in a machine-readable format so that people can easily reuse it
publish the data in a way that can be interpreted as RDF, with URIs for properties and types, to avoid conflicts with other vocabularies and so that the data can be “understood” even when discovered automatically
put some human-readable documentation at the end of the property/type URIs, so that developers can easily discover what your data’s about
embed machine-readable labels and descriptions for your properties/types within your data, so that applications can display it
embed rdfs:subPropertyOf/rdfs:subClassOf mappings from your properties/types to well-known properties/types within your data, so that it can be displayed in custom ways
put the machine-readable information about the properties/types at the end of the property/type URIs, so that you can update your vocabulary easily and so that other people can reuse it
add other RDFS and OWL statements about the properties/types, so that reasoners can add value to your data

The biggest leap, the one that requires the most persuasion and the most justification, is probably from simply publishing the data in a machine-readable format to using the RDF model with URIs for properties and types. But if you remove the cost of having to provide anything at the end of the URI and factor in the potential benefits you may reap in the future (as you step further up that ladder), the question becomes less “why?” and more “why not?”.

More Crime

Jeni — Sun, 23 Aug 2009 20:26:49 +0000

I wrote previously about a visualisation using Home Office data to navigate around categories of offences. The second interesting set of data from the Home Office that I found, tucked away in a small link on a page about Crime Reduction Toolkits was a spreadsheet of recorded crime statistics between 1898 and the present day. Each column is a different category of offence (I won’t say class because they don’t map onto the Classes from the spreadsheet of notifiable offences).

This time I wanted to try out the jQuery sparklines plug-in to illustrate how crime notifications have changed over time. The resulting page is available at http://www.jenitennison.com/visualisation/crime.html; here’s a screenshot for Bigamy:

(I chose Bigamy because there are some interesting humps in the data, roughly aligned with the two World Wars, which demonstrate the value of looking at timelines.)

I got this working by:

cleaning up and processing the spreadsheet into RDF/XML
putting the RDF/XML on my server in http://www.jenitennison.com/data/scheme/crime/, with more or less the same .htaccess as I used previously
creating a web page that included jQuery sparklines and rdfQuery as libraries and populates the page with details

You can see the source code for crime.html if you just go and look at it, but the relevant piece for populating the sparkline is:

$.get(selected, null, function (rdfXml) {
  var values = {}, sparkline = [],
    ...
    counts = $.rdf()
      .load(rdfXml)
      .prefix('rdf', 'http://www.w3.org/1999/02/22-rdf-syntax-ns#')
      .prefix('crime', 'http://www.jenitennison.com/data/ontology/crime#')
      .where('<' + selected + '> crime:count ?count')
      .where('?count crime:startYear ?year')
      .where('?count rdf:value ?value')
      .each(function () {
        ...
        values[this.year.value] = this.value.value;
      });
  ...
  for (v in values) {
    sparkline.push([v, values[v]]);
  }
  $('#sparkline')
    .sparkline(sparkline, { 
      chartRangeMin: 0,
      lineColor: '#999',
      fillColor: '#EEE',
      spotColor: 'blue',
      minSpotColor: 'green',
      maxSpotColor: 'red'
    });
}, 'xml');

(There is a reason for first creating an object representation of the values and then generating the array-of-arrays that the sparkline needs, but it’s in the elided code so you’ll have to look at the original source to see.)

jQuery sparklines are ridiculously easy to create. I’m looking forward to using more of the great variety of visualisations that they support.

Now, the big problem with the data is that the ways in which crimes are classified and notified has changed over time. If you look at the original spreadsheet you’ll see a bunch of notes that describe three kinds of changes:

Changes in the codes used for particular offences: for example Arson changed from classification 51 to 56 in 1934. I’m not using the Home Office codes at the moment anyway, but this is going to be something to be wary of when I start doing so.
Changes when a code is split into separate codes: for example “Other wounding” (8) split into “Other wounding” (8A), “Possession of weapons” (8B) and “Harassment” (8C) in 1998. In this case, offences stop being recorded under the original category and start being recorded under the new ones.
Changes in whether a crime is notifiable: for example “Cruelty to or neglect of children” (11) became notifiable in 1998 (according to the notes; according to the data, at least some instances were notified up to 1952, but then there was a gap).
Changes in how crimes are recorded: for example “The introduction of the Sexual Offences Act 2003 in May 2004 resulted in substantial changes to the sexual offences. This means that sexual offences data for 2004/05 are not comparable with those for previous years.”

There are undoubtedly other effects that aren’t listed in the notes, either in the legislation that covers a particular crime, or other environmental factors such as being at war (as shown above).

What I’d really like to do is to indicate these events in the sparkline and in the data table. Unfortunately it involves the translation of the loose notes from the spreadsheet into handcrafted RDF/XML, which is a little tedious. It’s also frustrating that there’s no good means of identification for the categories of offences. I’ve ended up arbitrarily naming them ‘A’ to ‘FZ’ which is somewhat unsatisfactory.

It’s worth noting that although I have a closed data set I’m explicitly using the Linked Data paradigm to go from a list of categories of crimes to retrieving information about a particular category (because the identifier for a category is a URI). If I weren’t using RDF, and wanted to split up the data in the way that I have for manageability, I’d have to document that particular properties contain pointers to information held at other locations. (Kris Zyp has attempted to formalise this in a kind of schema for JSON, but I have no idea how much support for this there is.)

I’ve also used SKOS to describe the categories, which is nice because all I have to tell you is that http://www.jenitennison.com/data/scheme/crime/ is a SKOS Concept Scheme and if you know SKOS you’ll know how to locate the top concepts in that scheme, nice human readable labels for them, and so on.

But if you want to reuse the counts of offences you will still have to actually look at the data to find the name of the property that I’ve used to go from a category to a count, and for the years and values themselves. These semantics are local to this particular application and the only way you can know them is by being told, just as it would be if I were using JSON.

So using RDF has bought us some things — a level of understanding about reaching data and a common vocabulary for organising concept schemes — but certainly not everything. It should be no surprise to anyone that it is not a magic bullet.

Offence Hierarchy Visualisation with rdfQuery and JIT

Jeni — Sun, 16 Aug 2009 20:00:50 +0000

The Home Office recently opened up some of its data, mostly in the form of PDF reports and Excel spreadsheets. Right after, I went on holiday and offline (!) for a week, so I set myself the task of putting together some visualisations of the data using two client-side visualisation libraries that I liked the look of:

jQuery sparklines which I think look simply gorgeous and which follow the jQuery tradition of being incredibly easy to put on a page
the JavaScript InfoVis Toolkit (JIT) which can be used to create some very attractive and interactive visualisations for hierarchical information

As a quick summary, I ended up with solutions that use an HTML page with rdfQuery code that pulls in static RDF/XML files and performs queries on them to create the particular formats that the two client-side libraries require.

The first one I’m going to talk about is a visualisation of types of offences using JIT. There’s a screenshot below to give you a flavour, but you’d be better off actually visiting the page because it’s interactive: mousing over and clicking on the labels enables you to navigate around the hierarchy.

The data for this visualisation comes from a spreadsheet of notifiable offences, available amongst a bunch of interesting information about counting rules for recording crime. The columns are:

Home Office code, which is split into a major and minor part (eg 6/4)
Mode (‘I’ for indictable, ‘E’ for triable-either-way or ‘S’ for summary)
Max sentence (eg life, 15, 3m, fine)
Class (eg “Violence Against The Person”)
Subclass (eg “Endangering Railway Passengers”)
Offence (eg “Destroying, damaging etc. a Channel Tunnel train or the Tunnel system or committing acts of violence likely to endanger safety of operation”)
Act(s) (eg “Channel Tunnel Act 1987 Sec 1(7)”)

There are a couple of things that I find particularly interesting about this data.

First, it includes references to legislation! Since my day job at TSO is currently all about publishing legislation as linked data, I find this really exciting! I haven’t done anything with those links yet, but I aim to.

Second, you would have thought that the Home Office code would be tied to a particular Subclass or Offence, but it’s not. The same Subclass can have multiple codes, but two Offences can have the same Home Office code. There doesn’t seem to be a natural way of identifying the Offences, except through their (often long) descriptive name. The terminology for the Offence often comes straight out of a piece of legislation, but sometimes it’s simply common law.

On the other hand, the offence Classes have reasonably short labels like “Burglary” and “Drug Offences” which can be turned into URIs like:

http://www.jenitennison.com/data/scheme/offence/drug-offences

The fact that Class-Subclass-Offence defines a hierarchy of concepts led me to think that SKOS would be a good ontology to use to model it. The Classes and Subclasses can be plain old skos:Concepts but the Offences need to have their own type so that extra information, such as the maximum sentence that applies to the offence, can be associated with it.

So if you look at http://www.jenitennison.com/data/scheme/offence/drug-offences you’ll see RDF/XML that includes the triples:

<http://www.jenitennison.com/data/scheme/offence/drug-offences>
  a skos:Concept ;
  skos:topConceptOf <http://www.jenitennison.com/data/scheme/offence/> ;
  skos:prefLabel "Drug Offences"@en ;
  skos:narrower
    [ 
      a skos:Concept ;
      skos:inScheme <http://www.jenitennison.com/data/scheme/offence/> ;
      skos:broader <http://www.jenitennison.com/data/scheme/offence/drug-offences> ;
      skos:prefLabel "Trafficking in controlled drugs"@en ;
      skos:narrower
        [
          a crime:Offence ;
          skos:inScheme <http://www.jenitennison.com/data/scheme/offence/> ;
          skos:prefLabel "Manufacturing a scheduled substance"@en ;
          crime:maxSentence "P14Y"^^xsd:yearMonthDuration
        ],
        [
          a crime:Offence ;
          skos:inScheme <http://www.jenitennison.com/data/scheme/offence/> ;
          skos:prefLabel "Supplying a scheduled substance to another person"@en ;
          crime:maxSentence "P14Y"^^xsd:yearMonthDuration
        ],
        ...
    ],
    [
      a skos:Concept ;
      skos:inScheme <http://www.jenitennison.com/data/scheme/offence/> ;
      skos:broader <http://www.jenitennison.com/data/scheme/offence/drug-offences> ;
      skos:prefLabel "Possession of controlled drugs"@en ;
      skos:narrower
        ...
    ],
    ...

You’ll notice that I’ve used blank nodes in the above rather than constructing an identifier for each Subclass or Offence. This makes things simpler because it means I can easily publish the dataset as a few flat files. An alternative would have been to use hash URIs, I suppose, but anyway this is the way I went. The (big) disadvantage is that it means the individual offences themselves aren’t referenceable. So I might work on that, especially if I migrate the data over to data.gov.uk rather than just using it to try out a visualisation.

The URI for the concept scheme is http://www.jenitennison.com/data/scheme/offence/. The slash on the end is entirely the result of trying to make Apache serve the static files correctly. As it is, I have one RDF/XML file for each Class of offence, plus an index.rdf within the same (offence) directory, with the .htaccess file:

AddType application/rdf+xml .rdf
DirectoryIndex index.rdf
RewriteEngine On
RewriteRule ^([^\.]+)$ $1.rdf [L]

The concept scheme file itself contains a list of the top concepts in the scheme (the Classes) and their labels. This serves as a useful entry point to the data.

So to the code. To create the visualisation, I needed to construct a Javascript structure that adhered to the JIT Input JSON Structure. Basically, each “node” within the visualisation needed to have an id, a name and a number of children. This structure needed to be constructed from the RDF/XML for a particular offence Class, ie that held within a particular RDF/XML document. The RDF/XML document can be accessed using the standard $.get() jQuery method. This passes the DOM for the document into the callback function passed as the third argument, which can then invoke rdfQuery’s $.rdf.load() method to load the triples encoded in the RDF/XML into an rdfQuery object that can then operate over those triples.

Here’s the relevant part of the code, in which view is the URI for the particular offence class and ht is a JIT HyperTree instance:

$.get(view, null, function (rdfXml) {
  var rdf, offences = {};
  rdf = $.rdf()
    .load(rdfXml)
    .prefix('skos', 'http://www.w3.org/2004/02/skos/core#')
    .prefix('crime', 'http://www.jenitennison.com/data/ontology/crime#');
  rdf
    .where('<' + view + '> skos:prefLabel ?label')
    .each(function () {
      offences.id = view;
      offences.name = this.label.value;
      offences.data = {
        '$color': '#0CC',
        'type': 'class'
      };
      offences.children = rdf
        .where('<' + view + '> skos:narrower ?subclass')
        .where('?subclass skos:prefLabel ?label')
        .map(function () {
          return {
            id: this.subclass.id,
            name: this.label.value,
            data: {
              'type': 'subclass'
            },
            children: rdf
              .where(this.subclass + ' skos:narrower ?offence')
              .where('?offence skos:prefLabel ?label')
              .where('?offence crime:maxSentence ?sentence')
              .map(function () {
                var sentence;
                if (this.sentence.datatype.toString() === 'http://www.w3.org/2001/XMLSchema#token') {
                  sentence = this.sentence.value;
                } else if (this.sentence.value > 12) {
                  sentence = this.sentence.value / 12 + ' years';
                } else {
                  sentence = this.sentence.value + ' months';
                }
                return {
                  id: this.offence.id,
                  name: this.label.value,
                  data: {
                    'type': 'offence',
                    'sentence': sentence
                  },
                  children: []
                };
              })
              .get()
          };
        })
        .get();
    });
  ht.loadJSON(offences);
  ht.refresh();
}, 'xml');

You can look at the rest of the code simply by viewing source on offences.html if you want to. It’s mostly the same as the HyperTree animation example but with a bit of refactoring particularly to add some jQuery goodness.

Some random thoughts having done this:

rdfQuery is really good to use, even if I do say so myself. It provides a very flexible way of creating data structures based on RDF accessed from elsewhere, particularly because you have the full power of Javascript at your fingertips.
JIT itself is OK to work with, though it doesn’t have the ease of use that it could have. The visualisation’s reasonably attractive, but my attempts to do clever things with the size of nodes to reflect the severity of the sentence proved fruitless.
The HyperTree visualisation works far better for smallish hierarchies (eg for Criminal Damage) than for large ones (eg Violence Against The Person or, if you have the patience, Other Offences).
The Offence hierarchy itself is a bit of a mess. There are 738 ‘Other Offences’ compared with 453 offences categorised within the other Classes, some of which contain only a handful of Offences. If this visualisation shows anything, it’s how disorganised the offences are. Even more so if you take into account some of the other data that’s been made available which I’ll post about another time and shows a completely different classification. I wonder if there’s data or other visualisations that would help identify where it could be rationalised.
The data is also out of date. I was surprised to see that it said that under the Piracy Act 1837 Section 2, Piracy with violence (one of the many ‘Other Offences’) still attracted a death penalty. But looking at the relevant Section on the Statue Law Database it appears that the death penalty was replaced with life imprisonment by Section 36 of the Crime and Disorder Act 1998. Getting better links into the legislation itself might help identify similar problems with the offence data.
I don’t know how easy it would have been to create this visualisation if I hadn’t been hosting the data myself. Danny Ayers put together a helpful post recently in which he listed the various ways of getting around the restrictions in doing cross-domain Ajax, which I’ll no doubt draw on if and when I need to do that.

rdfQuery Dazzle, Oxford, 11-12 July

Jeni — Sat, 13 Jun 2009 19:14:28 +0000

If you’re anywhere near Oxford on the weekend of the 11-12th July, and are interested in parsing, querying and manipulating RDF(a) in a browser, come along to the rdfQuery Dazzle (hack days). The official page lists some of the things we might work on:

Applications: widgets, adding to existing RDFa sites
Core development: documentation, packaging, microformats, named graphs, and ontologies
Interfaces: Talis change markup, N3 and Turtle, and SPARQL

It’s free to attend, you can come for either or both days, and refreshments, entertainment and wifi will be provided, so register now!

rdfQuery: Progressive Enhancement with RDFa

Jeni — Sat, 15 Nov 2008 16:46:02 +0000

Earlier this week I presented at SWIG-UK about rdfQuery. rdfQuery is a set of plugins that I’ve developed for jQuery in order to support RDFa parsing, querying and generation. There are a bunch of other Javascript libraries for RDFa around, such as Mark Birbeck’s Ubiquity RDFa and Ben Adida’s RDFa library. What I’ve really tried to do with rdfQuery is tie it in with the “Write Less, Do More” philosophy of jQuery and provide a neat, elegant API. At least that’s the aim!

So what does it do? Well, I’ve just added the demo that I used on Tuesday into the repository, so if you grab hold of that you can take a look. Here’s a screenshot of the demo.

The demo shows the overall concept of rdfQuery, namely that semantic markup can be useful not only to the crawlers that extract data from your pages to pump into massive triplestores, but also for you as a developer. In this case, which is a simple genealogy-type application, I want to have the people and places that are relevant to this particular extract highlighted within the text. I also want them listed on the left, with their details summarised.

So the demo illustrates three things that rdfQuery does to help:

gleaning triples from a section of the page (not just the whole page); in this case the triples are marked up with RDFa
querying the data to construct objects that represent the results of those queries, then doing things with those results
automatically adding RDFa to elements within the page, to update the data that it holds

It also demonstrates rough versions of a couple of things that rdfQuery could and should do that I aim to work on soon:

reasoning based on the data that’s found in the page
using ontologies to decide how to handle the data you find

By the by, it shows a simple, natural-language interface for making statements based on some text, but don’t let that distract you. It’s just regular expression processing.

There are four parts of this page. The main part contains some text about Charles Darwin. If you look at the source, you’ll see that it’s been marked up with some RDFa like this:

<span about="#CharlesRobertDarwin" typeof="foaf:Person" 
      property="rdfs:label" datatype="">
  <span property="foaf:firstName">Charles</span> Robert 
  <span property="foaf:surname">Darwin</span>
</span> was <span about="#CharlesRobertDarwin" rel="biografr:hasBirthPlace">born in 
<span about="#Shrewsbury" typeof="vcard:Address" 
      property="rdfs:label" datatype="">
  <span property="vcard:locality">Shrewsbury</span>, 
  <span property="vcard:region">Shropshire</span>, 
  <span property="vcard:country">England</span>
</span></span> on <span about="#CharlesRobertDarwin" property="biografr:bornOn" 
  content="1809-02-12" datatype="xsd:date">12 February 1809</span> at his 
  family home, the Mount. ...

This source is also shown in the textarea at the bottom of the page. Obviously this wouldn’t be visible in a real application, but it enables you to see what’s going on in the demo. Changing it won’t do anything; you can imagine that string being POSTed back to the server. I’ve also added some CSS that gives a border to any elements with a typeof attribute. Elements that have a property change their colour when you mouse over them.

So the page contains some RDFa. But what it doesn’t contain (in the source) is any information in the menu on the left. This gets populated based on the RDFa. If you click on Charles Robert Darwin, you’ll see the data that’s been gleaned about him, including (at the bottom), the derived fact that Robert Darwin was Charles Darwin’s father. Anything in black is information pulled from the RDFa; anything in orange is derived.

Next, try typing “Susannah Darwin was a person” into the text input and hit return. You should get Susannah Darwin added to the list of people. More importantly, if you look at the new source of the page, you’ll see that the phrase “Susannah Darwin” has been marked up with some RDFa to indicate that she was, indeed, a person and that “Susannah Darwin” can be used as a label for her.

You can try typing a few more facts into the box if you like. I suggest:

“Charles Robert Darwin was also known as Darwin”
“Susannah Darwin was Darwin’s mother”
“Susannah Darwin’s surname was Darwin”
“Josiah Wedgwood was a person”
“Susannah Darwin’s father was Josiah Wedgwood”

So this is all very cool and all, but the point was to show the code. So here we go.

Gleaning and Querying

Let’s look at how the lists are populated:

457: populateLists = function () {
458:   var rdf = $('#content').rdf();
459:   people.empty();
460:   places.empty();
461:   rdf
462:     .prefix('rdfs', ns.rdfs)
463:     .prefix('foaf', ns.foaf)
464:     .where('?person a foaf:Person')
465:     .where('?person rdfs:label ?label')
466:     .each(function () {
467:       addIndividual(people, this.person, this.label.value);
468:     })
469:     .reset()
470:     .where('?place a vcard:Address')
471:     .where('?place rdfs:label ?label')
472:     .each(function () {
473:       addIndividual(places, this.place, this.label.value);
474:     });
475: },

Line 458 parses the RDFa within the element with the id content (this is a <div> that wraps around the paragraphs about Charles Darwin). The rdf variable holds an rdfQuery object, which is very similar to a jQuery object except that it queries over RDF triples rather than DOM nodes. The rdfQuery object holds a pointer to a databank, which holds the triples that have been collected.

Lines 459 and 460 empty out the existing lists of people and places if there are any. The people and places variables are set earlier in the script and are jQuery objects.

Now the fun begins. I first set some prefixes on the rdfQuery object so that I can use those prefixes in CURIEs within the queries. In fact, these prefixes will have been set up by default anyway, because they’re declared in the HTML page, but it doesn’t hurt.

Lines 464 and 465 locate triples in the databank based on simple SPARQL-based queries. The first where() call creates a new rdfQuery object that, like a jQuery object, looks a bit like an array. The array contains objects, one for each triple that matches the pattern ?person a foaf:Person. Each of the objects has a person property containing the resource that represents the person. So the rdfQuery that results from this where() call looks like:

{ length: 2,
  0: { person: $.rdf.resource('<#CharlesRobertDarwin>') },
  1: { person: $.rdf.resource('<#RobertDarwin>') },
  ... bunch of other properties and methods that aren't important here ... }

The second where() call creates another new rdfQuery object based on combining the previous query results with the any triples that match the pattern ?person rdfs:label ?label. This holds objects with person and label properties, one for each person and their label. So the result of this looks like:

{ length: 2,
  0: { person: $.rdf.resource('<#CharlesRobertDarwin>'),
       label:  $.rdf.literal('"Charles Robert Darwin"') },
  1: { person: $.rdf.resource('<#RobertDarwin>'),
       label:  $.rdf.literal('"Robert Darwin"') },
  ... bunch of other properties and methods that aren't important here ... }

The each() on lines 466-468 then iterates over the objects that it’s constructed and calls the addIndividuals() function (which is particular for this demo), passing in the list in the HTML, the person resource and the value of the label literal.

Line 469 uses the reset() method to go back to the original rdfQuery object. If I didn’t do this, any further queries would simply add to the objects that I already have, or remove them if nothing matched.

Lines 470-474 do the same thing for the places that are marked up within the text.

There are various other places within markup.js that glean and query RDF. For example, the addDescription() function, which populates the list items on the left with data about particualr people and places. That function demonstrates the use of the about() method, which gives you all the triples about a particular subject, and shows how to use arguments with the each() method when you want to use the index of the query result or the triples that were used to create the query result.

Updating RDFa

So how easy is it to update the RDFa on the web page? Well, if you know what you want to add, then it’s dead easy. Here’s the code that does it on lines 452 and 532:

span.rdfa(triple);

The variable span here is a jQuery object. The triple itself is a $.rdf.triple. I’d tell you more about them but I think I’ve gone on long enough.

Final Words

rdfQuery is a Google Code project, released under an MIT license. If you’re interested in contributing, send me an email and I’ll add you as a member, or an owner if you’re really keen. If you’re interested, I’ve set up a discussion group. You can post any questions there, although of course if you find bugs, do add an issue.