jQuery

More Crime

Jeni — Sun, 23 Aug 2009 21:26:49 +0100

I wrote previously about a visualisation using Home Office data to navigate around categories of offences. The second interesting set of data from the Home Office that I found, tucked away in a small link on a page about Crime Reduction Toolkits was a spreadsheet of recorded crime statistics between 1898 and the present day. Each column is a different category of offence (I won’t say class because they don’t map onto the Classes from the spreadsheet of notifiable offences).

This time I wanted to try out the jQuery sparklines plug-in to illustrate how crime notifications have changed over time. The resulting page is available at http://www.jenitennison.com/visualisation/crime.html; here’s a screenshot for Bigamy:

(I chose Bigamy because there are some interesting humps in the data, roughly aligned with the two World Wars, which demonstrate the value of looking at timelines.)

I got this working by:

cleaning up and processing the spreadsheet into RDF/XML
putting the RDF/XML on my server in http://www.jenitennison.com/data/scheme/crime/, with more or less the same .htaccess as I used previously
creating a web page that included jQuery sparklines and rdfQuery as libraries and populates the page with details

You can see the source code for crime.html if you just go and look at it, but the relevant piece for populating the sparkline is:

$.get(selected, null, function (rdfXml) {
  var values = {}, sparkline = [],
    ...
    counts = $.rdf()
      .load(rdfXml)
      .prefix('rdf', 'http://www.w3.org/1999/02/22-rdf-syntax-ns#')
      .prefix('crime', 'http://www.jenitennison.com/data/ontology/crime#')
      .where('<' + selected + '> crime:count ?count')
      .where('?count crime:startYear ?year')
      .where('?count rdf:value ?value')
      .each(function () {
        ...
        values[this.year.value] = this.value.value;
      });
  ...
  for (v in values) {
    sparkline.push([v, values[v]]);
  }
  $('#sparkline')
    .sparkline(sparkline, { 
      chartRangeMin: 0,
      lineColor: '#999',
      fillColor: '#EEE',
      spotColor: 'blue',
      minSpotColor: 'green',
      maxSpotColor: 'red'
    });
}, 'xml');

(There is a reason for first creating an object representation of the values and then generating the array-of-arrays that the sparkline needs, but it’s in the elided code so you’ll have to look at the original source to see.)

jQuery sparklines are ridiculously easy to create. I’m looking forward to using more of the great variety of visualisations that they support.

Now, the big problem with the data is that the ways in which crimes are classified and notified has changed over time. If you look at the original spreadsheet you’ll see a bunch of notes that describe three kinds of changes:

Changes in the codes used for particular offences: for example Arson changed from classification 51 to 56 in 1934. I’m not using the Home Office codes at the moment anyway, but this is going to be something to be wary of when I start doing so.
Changes when a code is split into separate codes: for example “Other wounding” (8) split into “Other wounding” (8A), “Possession of weapons” (8B) and “Harassment” (8C) in 1998. In this case, offences stop being recorded under the original category and start being recorded under the new ones.
Changes in whether a crime is notifiable: for example “Cruelty to or neglect of children” (11) became notifiable in 1998 (according to the notes; according to the data, at least some instances were notified up to 1952, but then there was a gap).
Changes in how crimes are recorded: for example “The introduction of the Sexual Offences Act 2003 in May 2004 resulted in substantial changes to the sexual offences. This means that sexual offences data for 2004/05 are not comparable with those for previous years.”

There are undoubtedly other effects that aren’t listed in the notes, either in the legislation that covers a particular crime, or other environmental factors such as being at war (as shown above).

What I’d really like to do is to indicate these events in the sparkline and in the data table. Unfortunately it involves the translation of the loose notes from the spreadsheet into handcrafted RDF/XML, which is a little tedious. It’s also frustrating that there’s no good means of identification for the categories of offences. I’ve ended up arbitrarily naming them ‘A’ to ‘FZ’ which is somewhat unsatisfactory.

It’s worth noting that although I have a closed data set I’m explicitly using the Linked Data paradigm to go from a list of categories of crimes to retrieving information about a particular category (because the identifier for a category is a URI). If I weren’t using RDF, and wanted to split up the data in the way that I have for manageability, I’d have to document that particular properties contain pointers to information held at other locations. (Kris Zyp has attempted to formalise this in a kind of schema for JSON, but I have no idea how much support for this there is.)

I’ve also used SKOS to describe the categories, which is nice because all I have to tell you is that http://www.jenitennison.com/data/scheme/crime/ is a SKOS Concept Scheme and if you know SKOS you’ll know how to locate the top concepts in that scheme, nice human readable labels for them, and so on.

But if you want to reuse the counts of offences you will still have to actually look at the data to find the name of the property that I’ve used to go from a category to a count, and for the years and values themselves. These semantics are local to this particular application and the only way you can know them is by being told, just as it would be if I were using JSON.

So using RDF has bought us some things — a level of understanding about reaching data and a common vocabulary for organising concept schemes — but certainly not everything. It should be no surprise to anyone that it is not a magic bullet.

rdfQuery: Progressive Enhancement with RDFa

Jeni — Sat, 15 Nov 2008 16:46:02 +0000

Earlier this week I presented at SWIG-UK about rdfQuery. rdfQuery is a set of plugins that I’ve developed for jQuery in order to support RDFa parsing, querying and generation. There are a bunch of other Javascript libraries for RDFa around, such as Mark Birbeck’s Ubiquity RDFa and Ben Adida’s RDFa library. What I’ve really tried to do with rdfQuery is tie it in with the “Write Less, Do More” philosophy of jQuery and provide a neat, elegant API. At least that’s the aim!

So what does it do? Well, I’ve just added the demo that I used on Tuesday into the repository, so if you grab hold of that you can take a look. Here’s a screenshot of the demo.

The demo shows the overall concept of rdfQuery, namely that semantic markup can be useful not only to the crawlers that extract data from your pages to pump into massive triplestores, but also for you as a developer. In this case, which is a simple genealogy-type application, I want to have the people and places that are relevant to this particular extract highlighted within the text. I also want them listed on the left, with their details summarised.

So the demo illustrates three things that rdfQuery does to help:

gleaning triples from a section of the page (not just the whole page); in this case the triples are marked up with RDFa
querying the data to construct objects that represent the results of those queries, then doing things with those results
automatically adding RDFa to elements within the page, to update the data that it holds

It also demonstrates rough versions of a couple of things that rdfQuery could and should do that I aim to work on soon:

reasoning based on the data that’s found in the page
using ontologies to decide how to handle the data you find

By the by, it shows a simple, natural-language interface for making statements based on some text, but don’t let that distract you. It’s just regular expression processing.

There are four parts of this page. The main part contains some text about Charles Darwin. If you look at the source, you’ll see that it’s been marked up with some RDFa like this:

<span about="#CharlesRobertDarwin" typeof="foaf:Person" 
      property="rdfs:label" datatype="">
  <span property="foaf:firstName">Charles</span> Robert 
  <span property="foaf:surname">Darwin</span>
</span> was <span about="#CharlesRobertDarwin" rel="biografr:hasBirthPlace">born in 
<span about="#Shrewsbury" typeof="vcard:Address" 
      property="rdfs:label" datatype="">
  <span property="vcard:locality">Shrewsbury</span>, 
  <span property="vcard:region">Shropshire</span>, 
  <span property="vcard:country">England</span>
</span></span> on <span about="#CharlesRobertDarwin" property="biografr:bornOn" 
  content="1809-02-12" datatype="xsd:date">12 February 1809</span> at his 
  family home, the Mount. ...

This source is also shown in the textarea at the bottom of the page. Obviously this wouldn’t be visible in a real application, but it enables you to see what’s going on in the demo. Changing it won’t do anything; you can imagine that string being POSTed back to the server. I’ve also added some CSS that gives a border to any elements with a typeof attribute. Elements that have a property change their colour when you mouse over them.

So the page contains some RDFa. But what it doesn’t contain (in the source) is any information in the menu on the left. This gets populated based on the RDFa. If you click on Charles Robert Darwin, you’ll see the data that’s been gleaned about him, including (at the bottom), the derived fact that Robert Darwin was Charles Darwin’s father. Anything in black is information pulled from the RDFa; anything in orange is derived.

Next, try typing “Susannah Darwin was a person” into the text input and hit return. You should get Susannah Darwin added to the list of people. More importantly, if you look at the new source of the page, you’ll see that the phrase “Susannah Darwin” has been marked up with some RDFa to indicate that she was, indeed, a person and that “Susannah Darwin” can be used as a label for her.

You can try typing a few more facts into the box if you like. I suggest:

“Charles Robert Darwin was also known as Darwin”
“Susannah Darwin was Darwin’s mother”
“Susannah Darwin’s surname was Darwin”
“Josiah Wedgwood was a person”
“Susannah Darwin’s father was Josiah Wedgwood”

So this is all very cool and all, but the point was to show the code. So here we go.

Gleaning and Querying

Let’s look at how the lists are populated:

457: populateLists = function () {
458:   var rdf = $('#content').rdf();
459:   people.empty();
460:   places.empty();
461:   rdf
462:     .prefix('rdfs', ns.rdfs)
463:     .prefix('foaf', ns.foaf)
464:     .where('?person a foaf:Person')
465:     .where('?person rdfs:label ?label')
466:     .each(function () {
467:       addIndividual(people, this.person, this.label.value);
468:     })
469:     .reset()
470:     .where('?place a vcard:Address')
471:     .where('?place rdfs:label ?label')
472:     .each(function () {
473:       addIndividual(places, this.place, this.label.value);
474:     });
475: },

Line 458 parses the RDFa within the element with the id content (this is a <div> that wraps around the paragraphs about Charles Darwin). The rdf variable holds an rdfQuery object, which is very similar to a jQuery object except that it queries over RDF triples rather than DOM nodes. The rdfQuery object holds a pointer to a databank, which holds the triples that have been collected.

Lines 459 and 460 empty out the existing lists of people and places if there are any. The people and places variables are set earlier in the script and are jQuery objects.

Now the fun begins. I first set some prefixes on the rdfQuery object so that I can use those prefixes in CURIEs within the queries. In fact, these prefixes will have been set up by default anyway, because they’re declared in the HTML page, but it doesn’t hurt.

Lines 464 and 465 locate triples in the databank based on simple SPARQL-based queries. The first where() call creates a new rdfQuery object that, like a jQuery object, looks a bit like an array. The array contains objects, one for each triple that matches the pattern ?person a foaf:Person. Each of the objects has a person property containing the resource that represents the person. So the rdfQuery that results from this where() call looks like:

{ length: 2,
  0: { person: $.rdf.resource('<#CharlesRobertDarwin>') },
  1: { person: $.rdf.resource('<#RobertDarwin>') },
  ... bunch of other properties and methods that aren't important here ... }

The second where() call creates another new rdfQuery object based on combining the previous query results with the any triples that match the pattern ?person rdfs:label ?label. This holds objects with person and label properties, one for each person and their label. So the result of this looks like:

{ length: 2,
  0: { person: $.rdf.resource('<#CharlesRobertDarwin>'),
       label:  $.rdf.literal('"Charles Robert Darwin"') },
  1: { person: $.rdf.resource('<#RobertDarwin>'),
       label:  $.rdf.literal('"Robert Darwin"') },
  ... bunch of other properties and methods that aren't important here ... }

The each() on lines 466-468 then iterates over the objects that it’s constructed and calls the addIndividuals() function (which is particular for this demo), passing in the list in the HTML, the person resource and the value of the label literal.

Line 469 uses the reset() method to go back to the original rdfQuery object. If I didn’t do this, any further queries would simply add to the objects that I already have, or remove them if nothing matched.

Lines 470-474 do the same thing for the places that are marked up within the text.

There are various other places within markup.js that glean and query RDF. For example, the addDescription() function, which populates the list items on the left with data about particualr people and places. That function demonstrates the use of the about() method, which gives you all the triples about a particular subject, and shows how to use arguments with the each() method when you want to use the index of the query result or the triples that were used to create the query result.

Updating RDFa

So how easy is it to update the RDFa on the web page? Well, if you know what you want to add, then it’s dead easy. Here’s the code that does it on lines 452 and 532:

span.rdfa(triple);

The variable span here is a jQuery object. The triple itself is a $.rdf.triple. I’d tell you more about them but I think I’ve gone on long enough.

Final Words

rdfQuery is a Google Code project, released under an MIT license. If you’re interested in contributing, send me an email and I’ll add you as a member, or an owner if you’re really keen. If you’re interested, I’ve set up a discussion group. You can post any questions there, although of course if you find bugs, do add an issue.