visualisation

SPARQL & Visualisation Frustrations: Aggregation and Projection

Jeni — Sat, 12 Sep 2009 22:42:45 +0000

Today, I’m going to moan about the lack of features in SPARQL that are necessary to do many kinds of data analysis and visualisation. Going from raw data, held in RDF, to data like

the average traffic flow along the M5
the total amount claimed by each MP
the number of corporate insolvency notices published each day

cannot be done with SPARQL on its own. These calculations involve aggregation, grouping and projection which are planned for SPARQL vNext, but not here yet (at least, not in any standard way or in every triplestore).

Here’s the pretty graph to illustrate today’s rant:

The graph shows the number of notices of certain types placed in the London Gazette each day. The notices it summarises are those related to companies being liquidated, indicated by:

winding-up petitions indicating compulsory liquidation
members resolutions for winding-up indicating members’ voluntary liquidation
creditors resolutions for winding-up indicating creditors’ voluntary liquidation
appointment of administrators indicating companies going into administration

The graph is a version of:

with each data point averaged over 20 days. (The raw data spikes every Wednesday, presumably due to notices building up over the weekend and taking two days to appear in the Gazette.) It shows how the number of creditors’ voluntary liquidations (indicating companies that go insolvent and are unable to pay their creditors) doubled from around 30/day in May 2008 to around 60/day in the Spring of this year, but seems to be falling again (as far as we can tell; the data is not up-to-date).

This data is brought to you by the RDFa embedded by TSO in the notices on the London Gazette website and the scraping of said data into the datagovuk datastore held on the Talis platform, for both of which we have OPSI to thank.

The visualisation is brought to you by a touch of experimental “AJAR” in rdfQuery and the graphing power of Flot. Here are the lengths I have to go to to get the pretty graph:

First, I use rdfQuery to request a list of London Gazette issues since 1st May 2008. The SPARQL for the request is:

PREFIX corp-insolvency: <http://www.gazettes-online.co.uk/ontology/corp-insolvency#>
PREFIX g: <http://www.gazettes-online.co.uk/ontology#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>

CONSTRUCT {
  ?issue a g:Issue .
  ?issue g:hasPublicationDate ?date .
}
WHERE {
  ?issue a g:Issue .
  ?issue g:hasPublicationDate ?date .
  FILTER ( ?date > "2008-05-01"^^xsd:date ) .
}

This is a CONSTRUCT request because the resulting RDF/XML can be loaded into rdfQuery for querying. I could do a SELECT query and request JSON as the output format, but I’m doing a kind of end-to-end RDF thing here. So I use rdfQuery to make the request, load the result into an rdfQuery object, query it, and iterate over the results.

For each of the returned issues (all 293 of them), I make a separate request for all the relevant notices within that issue. The SPARQL looks like this:

PREFIX corp-insolvency: <http://www.gazettes-online.co.uk/ontology/corp-insolvency#>
PREFIX g: <http://www.gazettes-online.co.uk/ontology#>

CONSTRUCT {
  ?notice a ?type
}
WHERE {
  ?notice g:isInIssue $issue .
  { ?notice a corp-insolvency:MembersResolutionsForWindingUpNotice } UNION
  { ?notice a corp-insolvency:CreditorsResolutionsForWindingUpNotice } UNION
  { ?notice a corp-insolvency:AppointmentOfAdministratorNotice } UNION
  { ?notice a corp-insolvency:PetitionsToWindUpCompaniesNotice } .
  ?notice a ?type .
}

Once I’ve got the RDF for those notices, I can use rdfQuery to select just those of a particular type, then count how many there are and use the result to plot the graph.

Creating the graph involves 294 requests to the Talis store via the proxy that I’m using to get around the cross-site scripting issues, each of which takes (in my experience) between 200ms and 4s. So it’s pretty server-intensive for both the Talis servers and my proxy server (which is why I’m not actually going to make the page available generally). It’s also slow.

What I want to do is to be able to make four SPARQL requests that return RDF that summarise the number of notices of each of the different types on each date (or in each issue). I want to write SPARQL queries that look something like:

PREFIX corp-insolvency: <http://www.gazettes-online.co.uk/ontology/corp-insolvency#>
PREFIX g: <http://www.gazettes-online.co.uk/ontology#>

CONSTRUCT {
  ?issue a g:Issue .
  ?issue g:hasPublicationDate ?date .
  ?issue corp-insolvency:membersResolutionsForWindingUpNotices COUNT(?notice) .
}
WHERE {
  ?issue a g:Issue .
  ?issue g:hasPublicationDate ?date .
  ?notice g:isInIssue ?issue .
  ?notice a corp-insolvency:MembersResolutionsForWindingUpNotice .
}
GROUP BY ?issue

Four requests would be so much better than 294.

The thing of it is that this kind of facility is available as standard in SQL, the Google Visualisation API’s simple query language, or in the “reduce” part of map/reduce. If we’re to think of triplestores as a serious alternative to either relational or non-relational databases, and SPARQL as a serious alternative to either SQL or NoSQL, then it really must support these operations. And Real Soon.

In the meantime, I think the lesson for the publishers of linked data is to provide aggregated values for the obvious kinds of aggregations that people might want to do over your data. In the London Gazette data, that would be the counts of the various kinds of notices it contains. In the traffic flow data it would be the average, minimum and maximum traffic flow over each of the measured days, at each hour over the known dates and overall for each point.

On a more philosophical note, it strikes me that the concept of aggregation contradicts the Open World assumption. I can only know that the number of members’ winding-up order notices was exactly 30 if I know that I know of all the members’ winding-up order notices that exist. Pragmatically, in many cases this is going to be just fine, because we know that the datasets that we’re using are complete (our World is Closed), but it does slightly concern me that it’s impossible to do much useful data analysis without contradicting one of the fundamental tenets of the Semantic Web.

More Crime

Jeni — Sun, 23 Aug 2009 20:26:49 +0000

I wrote previously about a visualisation using Home Office data to navigate around categories of offences. The second interesting set of data from the Home Office that I found, tucked away in a small link on a page about Crime Reduction Toolkits was a spreadsheet of recorded crime statistics between 1898 and the present day. Each column is a different category of offence (I won’t say class because they don’t map onto the Classes from the spreadsheet of notifiable offences).

This time I wanted to try out the jQuery sparklines plug-in to illustrate how crime notifications have changed over time. The resulting page is available at http://www.jenitennison.com/visualisation/crime.html; here’s a screenshot for Bigamy:

(I chose Bigamy because there are some interesting humps in the data, roughly aligned with the two World Wars, which demonstrate the value of looking at timelines.)

I got this working by:

cleaning up and processing the spreadsheet into RDF/XML
putting the RDF/XML on my server in http://www.jenitennison.com/data/scheme/crime/, with more or less the same .htaccess as I used previously
creating a web page that included jQuery sparklines and rdfQuery as libraries and populates the page with details

You can see the source code for crime.html if you just go and look at it, but the relevant piece for populating the sparkline is:

$.get(selected, null, function (rdfXml) {
  var values = {}, sparkline = [],
    ...
    counts = $.rdf()
      .load(rdfXml)
      .prefix('rdf', 'http://www.w3.org/1999/02/22-rdf-syntax-ns#')
      .prefix('crime', 'http://www.jenitennison.com/data/ontology/crime#')
      .where('<' + selected + '> crime:count ?count')
      .where('?count crime:startYear ?year')
      .where('?count rdf:value ?value')
      .each(function () {
        ...
        values[this.year.value] = this.value.value;
      });
  ...
  for (v in values) {
    sparkline.push([v, values[v]]);
  }
  $('#sparkline')
    .sparkline(sparkline, { 
      chartRangeMin: 0,
      lineColor: '#999',
      fillColor: '#EEE',
      spotColor: 'blue',
      minSpotColor: 'green',
      maxSpotColor: 'red'
    });
}, 'xml');

(There is a reason for first creating an object representation of the values and then generating the array-of-arrays that the sparkline needs, but it’s in the elided code so you’ll have to look at the original source to see.)

jQuery sparklines are ridiculously easy to create. I’m looking forward to using more of the great variety of visualisations that they support.

Now, the big problem with the data is that the ways in which crimes are classified and notified has changed over time. If you look at the original spreadsheet you’ll see a bunch of notes that describe three kinds of changes:

Changes in the codes used for particular offences: for example Arson changed from classification 51 to 56 in 1934. I’m not using the Home Office codes at the moment anyway, but this is going to be something to be wary of when I start doing so.
Changes when a code is split into separate codes: for example “Other wounding” (8) split into “Other wounding” (8A), “Possession of weapons” (8B) and “Harassment” (8C) in 1998. In this case, offences stop being recorded under the original category and start being recorded under the new ones.
Changes in whether a crime is notifiable: for example “Cruelty to or neglect of children” (11) became notifiable in 1998 (according to the notes; according to the data, at least some instances were notified up to 1952, but then there was a gap).
Changes in how crimes are recorded: for example “The introduction of the Sexual Offences Act 2003 in May 2004 resulted in substantial changes to the sexual offences. This means that sexual offences data for 2004/05 are not comparable with those for previous years.”

There are undoubtedly other effects that aren’t listed in the notes, either in the legislation that covers a particular crime, or other environmental factors such as being at war (as shown above).

What I’d really like to do is to indicate these events in the sparkline and in the data table. Unfortunately it involves the translation of the loose notes from the spreadsheet into handcrafted RDF/XML, which is a little tedious. It’s also frustrating that there’s no good means of identification for the categories of offences. I’ve ended up arbitrarily naming them ‘A’ to ‘FZ’ which is somewhat unsatisfactory.

It’s worth noting that although I have a closed data set I’m explicitly using the Linked Data paradigm to go from a list of categories of crimes to retrieving information about a particular category (because the identifier for a category is a URI). If I weren’t using RDF, and wanted to split up the data in the way that I have for manageability, I’d have to document that particular properties contain pointers to information held at other locations. (Kris Zyp has attempted to formalise this in a kind of schema for JSON, but I have no idea how much support for this there is.)

I’ve also used SKOS to describe the categories, which is nice because all I have to tell you is that http://www.jenitennison.com/data/scheme/crime/ is a SKOS Concept Scheme and if you know SKOS you’ll know how to locate the top concepts in that scheme, nice human readable labels for them, and so on.

But if you want to reuse the counts of offences you will still have to actually look at the data to find the name of the property that I’ve used to go from a category to a count, and for the years and values themselves. These semantics are local to this particular application and the only way you can know them is by being told, just as it would be if I were using JSON.

So using RDF has bought us some things — a level of understanding about reaching data and a common vocabulary for organising concept schemes — but certainly not everything. It should be no surprise to anyone that it is not a magic bullet.

Offence Hierarchy Visualisation with rdfQuery and JIT

Jeni — Sun, 16 Aug 2009 20:00:50 +0000

The Home Office recently opened up some of its data, mostly in the form of PDF reports and Excel spreadsheets. Right after, I went on holiday and offline (!) for a week, so I set myself the task of putting together some visualisations of the data using two client-side visualisation libraries that I liked the look of:

jQuery sparklines which I think look simply gorgeous and which follow the jQuery tradition of being incredibly easy to put on a page
the JavaScript InfoVis Toolkit (JIT) which can be used to create some very attractive and interactive visualisations for hierarchical information

As a quick summary, I ended up with solutions that use an HTML page with rdfQuery code that pulls in static RDF/XML files and performs queries on them to create the particular formats that the two client-side libraries require.

The first one I’m going to talk about is a visualisation of types of offences using JIT. There’s a screenshot below to give you a flavour, but you’d be better off actually visiting the page because it’s interactive: mousing over and clicking on the labels enables you to navigate around the hierarchy.

The data for this visualisation comes from a spreadsheet of notifiable offences, available amongst a bunch of interesting information about counting rules for recording crime. The columns are:

Home Office code, which is split into a major and minor part (eg 6/4)
Mode (‘I’ for indictable, ‘E’ for triable-either-way or ‘S’ for summary)
Max sentence (eg life, 15, 3m, fine)
Class (eg “Violence Against The Person”)
Subclass (eg “Endangering Railway Passengers”)
Offence (eg “Destroying, damaging etc. a Channel Tunnel train or the Tunnel system or committing acts of violence likely to endanger safety of operation”)
Act(s) (eg “Channel Tunnel Act 1987 Sec 1(7)”)

There are a couple of things that I find particularly interesting about this data.

First, it includes references to legislation! Since my day job at TSO is currently all about publishing legislation as linked data, I find this really exciting! I haven’t done anything with those links yet, but I aim to.

Second, you would have thought that the Home Office code would be tied to a particular Subclass or Offence, but it’s not. The same Subclass can have multiple codes, but two Offences can have the same Home Office code. There doesn’t seem to be a natural way of identifying the Offences, except through their (often long) descriptive name. The terminology for the Offence often comes straight out of a piece of legislation, but sometimes it’s simply common law.

On the other hand, the offence Classes have reasonably short labels like “Burglary” and “Drug Offences” which can be turned into URIs like:

http://www.jenitennison.com/data/scheme/offence/drug-offences

The fact that Class-Subclass-Offence defines a hierarchy of concepts led me to think that SKOS would be a good ontology to use to model it. The Classes and Subclasses can be plain old skos:Concepts but the Offences need to have their own type so that extra information, such as the maximum sentence that applies to the offence, can be associated with it.

So if you look at http://www.jenitennison.com/data/scheme/offence/drug-offences you’ll see RDF/XML that includes the triples:

<http://www.jenitennison.com/data/scheme/offence/drug-offences>
  a skos:Concept ;
  skos:topConceptOf <http://www.jenitennison.com/data/scheme/offence/> ;
  skos:prefLabel "Drug Offences"@en ;
  skos:narrower
    [ 
      a skos:Concept ;
      skos:inScheme <http://www.jenitennison.com/data/scheme/offence/> ;
      skos:broader <http://www.jenitennison.com/data/scheme/offence/drug-offences> ;
      skos:prefLabel "Trafficking in controlled drugs"@en ;
      skos:narrower
        [
          a crime:Offence ;
          skos:inScheme <http://www.jenitennison.com/data/scheme/offence/> ;
          skos:prefLabel "Manufacturing a scheduled substance"@en ;
          crime:maxSentence "P14Y"^^xsd:yearMonthDuration
        ],
        [
          a crime:Offence ;
          skos:inScheme <http://www.jenitennison.com/data/scheme/offence/> ;
          skos:prefLabel "Supplying a scheduled substance to another person"@en ;
          crime:maxSentence "P14Y"^^xsd:yearMonthDuration
        ],
        ...
    ],
    [
      a skos:Concept ;
      skos:inScheme <http://www.jenitennison.com/data/scheme/offence/> ;
      skos:broader <http://www.jenitennison.com/data/scheme/offence/drug-offences> ;
      skos:prefLabel "Possession of controlled drugs"@en ;
      skos:narrower
        ...
    ],
    ...

You’ll notice that I’ve used blank nodes in the above rather than constructing an identifier for each Subclass or Offence. This makes things simpler because it means I can easily publish the dataset as a few flat files. An alternative would have been to use hash URIs, I suppose, but anyway this is the way I went. The (big) disadvantage is that it means the individual offences themselves aren’t referenceable. So I might work on that, especially if I migrate the data over to data.gov.uk rather than just using it to try out a visualisation.

The URI for the concept scheme is http://www.jenitennison.com/data/scheme/offence/. The slash on the end is entirely the result of trying to make Apache serve the static files correctly. As it is, I have one RDF/XML file for each Class of offence, plus an index.rdf within the same (offence) directory, with the .htaccess file:

AddType application/rdf+xml .rdf
DirectoryIndex index.rdf
RewriteEngine On
RewriteRule ^([^\.]+)$ $1.rdf [L]

The concept scheme file itself contains a list of the top concepts in the scheme (the Classes) and their labels. This serves as a useful entry point to the data.

So to the code. To create the visualisation, I needed to construct a Javascript structure that adhered to the JIT Input JSON Structure. Basically, each “node” within the visualisation needed to have an id, a name and a number of children. This structure needed to be constructed from the RDF/XML for a particular offence Class, ie that held within a particular RDF/XML document. The RDF/XML document can be accessed using the standard $.get() jQuery method. This passes the DOM for the document into the callback function passed as the third argument, which can then invoke rdfQuery’s $.rdf.load() method to load the triples encoded in the RDF/XML into an rdfQuery object that can then operate over those triples.

Here’s the relevant part of the code, in which view is the URI for the particular offence class and ht is a JIT HyperTree instance:

$.get(view, null, function (rdfXml) {
  var rdf, offences = {};
  rdf = $.rdf()
    .load(rdfXml)
    .prefix('skos', 'http://www.w3.org/2004/02/skos/core#')
    .prefix('crime', 'http://www.jenitennison.com/data/ontology/crime#');
  rdf
    .where('<' + view + '> skos:prefLabel ?label')
    .each(function () {
      offences.id = view;
      offences.name = this.label.value;
      offences.data = {
        '$color': '#0CC',
        'type': 'class'
      };
      offences.children = rdf
        .where('<' + view + '> skos:narrower ?subclass')
        .where('?subclass skos:prefLabel ?label')
        .map(function () {
          return {
            id: this.subclass.id,
            name: this.label.value,
            data: {
              'type': 'subclass'
            },
            children: rdf
              .where(this.subclass + ' skos:narrower ?offence')
              .where('?offence skos:prefLabel ?label')
              .where('?offence crime:maxSentence ?sentence')
              .map(function () {
                var sentence;
                if (this.sentence.datatype.toString() === 'http://www.w3.org/2001/XMLSchema#token') {
                  sentence = this.sentence.value;
                } else if (this.sentence.value > 12) {
                  sentence = this.sentence.value / 12 + ' years';
                } else {
                  sentence = this.sentence.value + ' months';
                }
                return {
                  id: this.offence.id,
                  name: this.label.value,
                  data: {
                    'type': 'offence',
                    'sentence': sentence
                  },
                  children: []
                };
              })
              .get()
          };
        })
        .get();
    });
  ht.loadJSON(offences);
  ht.refresh();
}, 'xml');

You can look at the rest of the code simply by viewing source on offences.html if you want to. It’s mostly the same as the HyperTree animation example but with a bit of refactoring particularly to add some jQuery goodness.

Some random thoughts having done this:

rdfQuery is really good to use, even if I do say so myself. It provides a very flexible way of creating data structures based on RDF accessed from elsewhere, particularly because you have the full power of Javascript at your fingertips.
JIT itself is OK to work with, though it doesn’t have the ease of use that it could have. The visualisation’s reasonably attractive, but my attempts to do clever things with the size of nodes to reflect the severity of the sentence proved fruitless.
The HyperTree visualisation works far better for smallish hierarchies (eg for Criminal Damage) than for large ones (eg Violence Against The Person or, if you have the patience, Other Offences).
The Offence hierarchy itself is a bit of a mess. There are 738 ‘Other Offences’ compared with 453 offences categorised within the other Classes, some of which contain only a handful of Offences. If this visualisation shows anything, it’s how disorganised the offences are. Even more so if you take into account some of the other data that’s been made available which I’ll post about another time and shows a completely different classification. I wonder if there’s data or other visualisations that would help identify where it could be rationalised.
The data is also out of date. I was surprised to see that it said that under the Piracy Act 1837 Section 2, Piracy with violence (one of the many ‘Other Offences’) still attracted a death penalty. But looking at the relevant Section on the Statue Law Database it appears that the death penalty was replaced with life imprisonment by Section 36 of the Crime and Disorder Act 1998. Getting better links into the legislation itself might help identify similar problems with the offence data.
I don’t know how easy it would have been to create this visualisation if I hadn’t been hosting the data myself. Danny Ayers put together a helpful post recently in which he listed the various ways of getting around the restrictions in doing cross-domain Ajax, which I’ll no doubt draw on if and when I need to do that.

SPARQL & Visualisation Frustrations: Linked Data

Jeni — Mon, 03 Aug 2009 20:36:34 +0000

I’ll start with the problem. To create the graphs I showed in my last post, I wanted to split MPs into groups based on their party affiliation. Ideally, I wanted the Google Visualisation query to look like:

select mp, additionalCosts, totalTravel, totalBasic 
where party = 'Conservative' 
order by totalClaim desc 
limit 25

because this is reasonably easy to understand and for a developer to create without having to know any magic URIs.

The party affiliation for an MP is given in the RDF supplied within the Talis store as a pointer to one of the resources:

http://dbpedia.org/resource/Labour_Party_(UK)
http://dbpedia.org/resource/Conservative_Party_(UK)
http://dbpedia.org/resource/Liberal_Democrats

Now, if you visit http://dbpedia.org/resource/Conservative_Party_(UK) then you’ll see precious few properties and none of them give you access to the string ‘Conservative’. If you look at http://dbpedia.org/resource/Liberal_Democrats, you’ll see plenty of properties, one of which is dbpprop:partyName. But trying to query on dbpprop:partyName within the Talis data store gives me nothing, because that information hasn’t been imported into the particular store that this SPARQL query is running on.

What I did in utils.php was extend the parsing of the tq parameter, which is supposed to be in the Google Visualisation query language, to understand <URI> as a reference to a resource. In other words, you can create a query like:

select mp, additionalCosts, totalTravel, totalBasic 
where rParty = <http://dbpedia.org/resource/Conservative_Party_(UK)> 
order by totalClaim desc 
limit 25

and this will be mapped to a SPARQL query that looks like:

SELECT ?mp ?additionalCosts ?totalTravel ?totalBasic 
WHERE {
  ...
  FILTER (?rParty = <http://dbpedia.org/resource/Conservative_Party_(UK)>)
}
ORDER BY desc(?totalClaim)
LIMIT 25

I don’t like having done this, because I don’t want Data Sources that happen to be SPARQL queries to look any different from other Data Sources. Introducing a new syntax for URI literals isn’t really on.

The superficial fix is to always provide basic labelling information for the resources referenced within a triplestore. In this case, Leigh actually did include an rdfs:label property for each of the party URIs within the Guardian store, so it was possible to use the query I wanted to use after all (though it took some experimentation to find this out).

But underlying this is a bigger issue. Much is made of linked data — that you can find out more about a particular thing by resolving the link to that thing — but the best illustrations of the power and benefits of the semantic web tend to revolve around analysis and visualisations of moderately large amounts of data using SPARQL. And SPARQL (as yet) only runs on individual triplestores, which do not contain the entire semantic web. Every SPARQL query is limited by what has been loaded into the particular triplestore that is queried.

Now, one of the “time-permitting” requirements for SPARQL 1.1 is Federated Queries:

Federated query is the ability to take a query and provide solutions based on information from many different sources. It is a hard problem in its most general form and is the subject of continuing (and continuous) research. A building block is the ability to have one query be able to issue a query on another SPARQL endpoint during query execution.

Time-permitting, the SPARQL Working Group will define the syntax and semantics for handling a basic class of federated queries in which the SPARQL endpoints to use in executing portions of the query are explicitly given by the query author.

That’s certainly “a building block”, but it can’t be the only method. For many data publishers, it’s going to be far far simpler to publish their data as linked data in RDF/XML than it is to provide a SPARQL endpoint for that data. We can ask organisations like Talis to crawl our data and provide a SPARQL endpoint for it, and hope that the SPARQL Working Group have time to address federated search, but really we need tools that make it easy to aggregate, analyse and visualise linked data directly rather than through a triplestore silo.

So how about it?

Map Visualisation of MPs Travel Expenses

Jeni — Fri, 31 Jul 2009 22:33:13 +0000

During Guardian Hack Day 2, Leigh ported the Guardian’s MP’s Expenses data into Talis. Most wonderfully, this gives a SPARQL endpoint that can be used to query the data. I thought I’d try to use the same approach as I blogged about recently, using a SPARQL query as a Data Source for a Google Visualisation of the MP’s expenses data.

To cut to the chase, here’s a screenshot of the result (follow the link for the more interactive version):

I created this visualisation with the same general approach as I explained last time.

First, I’ve been working on the visualisation utils.php, which is a reasonably simple PHP script that exposes a SPARQL endpoint as a Google Visualisation Data Source. Requests to a Data Source use a special query language to indicate the information that should be included, how it should be sorted, how many rows of data there should be, and so on.

Previously, utils.php only understood the select portion of the tq parameter which contains this query; I’ve expanded it to understand (somewhat limited versions of) the select, where, order by, limit and offset parts of the query, which of course have equivalents in SPARQL. Since these parts of the Google Visualisation query language are pretty close to SPARQL, this is actually just a bunch of string munging, which isn’t particularly interesting, so just grab hold of it if you want to use it.

Second, I created a PHP script (mp-travel.php) specifically for the MPs expenses data that pulls out the parts that I’m interested in and exposes them as variables which can be used in the query language. This is what the file looks like:

<?php
  include "utils.php";
  proxy('?rMP a <http://guardian.dataincubator.org/ns/MemberOfParliament> .
         ?rMP <http://xmlns.com/foaf/0.1/name> ?mp .
         ?rMP <http://guardian.dataincubator.org/ns/mp-expenses/majority> ?majority .
         ?rMP <http://dbpedia.org/property/constituency> ?rConstituency .
         ?rConstituency rdfs:label ?constituency .
         ?rConstituency <http://www.w3.org/2003/01/geo/wgs84_pos#lat> ?lat .
         ?rConstituency <http://www.w3.org/2003/01/geo/wgs84_pos#long> ?long .
         ?rMP <http://guardian.dataincubator.org/ns/mp-expenses/total-travel> ?totalTravel .',
        'desc(?totalTravel)', 
        'guardian');
?>

The second argument to the proxy() function is the default ordering (desc(?totalTravel)) and the third is the name of the Talis data store that’s being used (guardian).

The first argument is a query which determines the variables that are exposed by the Data Source. This Data Source exposes the variables:

mp: the name of the MP
majority: the majority that they have in their constituency
constituency: the name of the constituency
lat, long: the latitude and longitude of the constituency (presumably the centre of it)
totalTravel: the total amount claimed for travel by the MP
rMP: the URI used to identify the MP
rConstituency: the URI used to identify the constituency

Third, I created an HTML document that used the Google Visualisation API to create the map visualisation that I’ve shown above. The really important lines are:

var query = new google.visualization.Query('http://www.jenitennison.com/visualisation/data/mp-travel');
query.setQuery('select lat, long, totalTravel, mp order by majority limit 100');

The first line shows the URL for the Data Source, which is essentially a pointer to the mp-travel.php script. The second line shows the query that’s sent to the Data Source: “select lat, long, totalTravel, mp order by majority limit 100”.

Put together, when you load http://www.jenitennison.com/visualisation/mp-travel.html, you create a Google Visualisation GeoMap which uses as its data the result of the SPARQL query

SELECT ?lat ?long ?totalTravel ?mp
WHERE {
  ?rMP a <http://guardian.dataincubator.org/ns/MemberOfParliament> .
  ?rMP <http://xmlns.com/foaf/0.1/name> ?mp .
  ?rMP <http://guardian.dataincubator.org/ns/mp-expenses/majority> ?majority .
  ?rMP <http://dbpedia.org/property/constituency> ?rConstituency .
  ?rConstituency rdfs:label ?constituency .
  ?rConstituency <http://www.w3.org/2003/01/geo/wgs84_pos#lat> ?lat .
  ?rConstituency <http://www.w3.org/2003/01/geo/wgs84_pos#long> ?long .
  ?rMP <http://guardian.dataincubator.org/ns/mp-expenses/total-travel> ?totalTravel .
}
ORDER By ?majority
LIMIT 100

on the SPARQL endpoint at http://api.talis.com/stores/guardian/services/sparql.

Here’s hoping you can reuse the Data Source or the code that was used to make it. Let me know if you do!

Creating Google Visualisations of Linked Data

Jeni — Thu, 23 Jul 2009 21:37:33 +0000

Update: For the people who couldn’t read the post because the graph didn’t have 0 as its x-axis minimum, here is the version of the graph that does. I haven’t removed the other version, since doing so would make the comments confusing and I think it’s interesting to compare the two.

There are idealists who immediately see the publication of Open Data as a Good Thing, and leap up and down (metaphorically or physically) shouting “Raw Data Now”. There are also a whole bunch of people who need to “see the shiny”. They need to understand why publishing Open Data is a Good Thing, and most particularly what the benefit is going to be to them.

This is understandable. Publishers bear the cost of the development of URI schemes, XML formats, RDF ontologies, and other infrastructure for serving data, and the ongoing maintenance cost of domain resolution, bandwidth usage and user support. Even publishers with a public-service remit (who may not need to see monetary payback) need to be convinced that there will be some kind of return on the investment.

One result of making data available is that it enables you and others to easily construct nice visualisations over the data, and maybe spot useful patterns within it. This is particularly useful for public sector information because it can provide feedback on how effective a particular policy has been or where more resources need to be spent.

So I thought it would be worthwhile trying to explore how to create visualisations of some data, starting with the London Borough data that I’ve published using Talis.

Nowadays, there are a bunch of visualisation libraries around. My first experiment is going to be using the Google Visualisation API, which offers a range of different, reasonably pretty, visualisations that work cross-browser using either SVG or VML. In this post, I’m going to use the barchart visualisation, to create a chart of male and female life expectancy in the London Boroughs. It looks like this:

One of the interesting things about the Google Visualisation API is that you can provide the data for every visualisation using a Data Source. A Data Source is any web page that, in the face of particular requests, returns JSON in a particular format. The format basically encodes a table with named (and typed) columns and a number of rows containing cells which have values for each of those columns.

For the London Borough data, the JSON needs to look something like:

google.visualization.Query.setResponse({
  version:0.6,
  status:'ok',
  reqId:0,
  table:{
    cols:[
      {id:'label', type:'string'},
      {id:'maleLE', type:'number'},
      {id:'femaleLE', type:'number'}
    ],
    rows:[
      {c:[
        {v:'Barking & Dagenham'},
        {v:76.3},
        {v:80.3}]},
      {c:[
        {v:'Barnet'},
        {v:79.5},
        {v:83.6}]},
      {c:[
        {v:'Bexley'},
        {v:78.7},
        {v:82.7}]},
      ...
    ]}
})

Now we already know, from my last couple of blog posts, how to query the Talis platform through its SPARQL endpoint, and how to use Talis’ transformation service to invoke an XSLT transformation over the XML that we get back from that. The same principle applies:

create a SPARQL query that will return the information we want
create an XSLT transformation that will tidy the result into the format we want it
set up a PHP page to access the relevant URI and pass through the results
create an .htaccess file so that invoking the PHP can be done without revealing it’s PHP

First, the SPARQL query. As I said, a Data Source is essentially a table of named columns. While we could do a DESCRIBE or CREATE query, a SELECT query is a lot closer match to the tabular layout that the Data Source needs because it’s also essentially a table of named columns (the variables you select). For the data that we want in the table, an appropriate SPARQL query would be:

SELECT ?label ?maleLE ?femaleLE
WHERE {
  ?borough a <http://www.jenitennison.com/ontology/data#LondonBorough> .
  ?borough <http://www.w3.org/2000/01/rdf-schema#label> ?label .
  ?borough <http://www.jenitennison.com/ontology/data#maleLifeExpectancy> ?maleLE .
  ?borough <http://www.jenitennison.com/ontology/data#femaleLifeExpectancy> ?femaleLE .
}
ORDER BY ?label

Plugging this query into the Talis SPARQL endpoint (have a go at http://api.talis.com/stores/rdfquery-dev1/services/sparql if you like) gives a response in the SPARQL Query Results Format which looks like:

<sparql xmlns="http://www.w3.org/2005/sparql-results#">
  <head>
    <variable name="label"/>
    <variable name="maleLE"/>
    <variable name="femaleLE"/>
  </head>
  <results>
    <result>
      <binding name="label">
        <literal>Barking &amp; Dagenham</literal>
      </binding>
      <binding name="maleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">76.3</literal>
      </binding>
      <binding name="femaleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">80.3</literal>
      </binding>
    </result>
    <result>
      <binding name="label">
        <literal>Barnet</literal>
      </binding>
      <binding name="maleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">79.5</literal>
      </binding>
      <binding name="femaleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">83.6</literal>
      </binding>
    </result>
    <result>
      <binding name="label">
        <literal>Bexley</literal>
      </binding>
      <binding name="maleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">78.7</literal>
      </binding>
      <binding name="femaleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">82.7</literal>
      </binding>
    </result>
    ...
  </results>
</sparql>

The second step — transforming this SPARQL result into JSON — just take a little bit of XSLT (1.0, remember, because that’s all the Talis’ Transformation Service can manage). My aim in this post is to show that anyone, even if they don’t have write access to a Talis data store, can create these visualisations, so I’ve just put the XSLT on my site at http://www.jenitennison.com/visualisation/data/SRXtoGoogleVisData.xsl. I’m not going to duplicate it here; it’s generic enough for reuse should you want to.

The third step is to create some PHP that handles a query from the Google Visualisation. The requests will include two parameters:

tqx defines details about how the data should be returned, such as its format
tq defines a query in the Google Visualisation API query language that identifies precisely the data that should be returned

I want this PHP to be reusable, so I’ve created a utils.php that looks like this:

<?php
  $store = 'rdfquery-dev1';

  function proxy($filter, $order) {
    global $store;
    $tqx = $_GET['tqx'];
    $tq = $_GET['tq'];

    // Parse tq parameter
    if ($tq) {
      $select = stristr($tq, 'select ');
      $select = substr($select, 7);
      $select = explode(',', $select);
      foreach ($select as $var) {
        $var = trim($var);
        $vars[] = "?$var";
      }
      $vars = implode(' ', $vars);
    } else {
      $vars = '*';
    }
    $sparql = "SELECT $vars WHERE { $filter } ORDER BY $order";

    $params = array('query' => $sparql, 'output' => 'xml');
    $query = http_build_query($params);
    $rdfURL = "http://api.talis.com/stores/$store/services/sparql?$query";

    // URL for the transformation
    $params = array('xml-uri' => $rdfURL, 
      'xsl-uri' => "http://www.jenitennison.com/visualisation/data/SRXtoGoogleVisData.xsl",
      'tqx' => $tqx);
    $query = http_build_query($params);
    $txURL = "http://api.talis.com/tx?$query";

    $resource = fopen($txURL, 'rb');
    header('Content-Type: application/json', true);
    fpassthru($resource);
    return;
  }
?>

Things to note here are about the processing of the tq and tqx parameters that are sent by the Google Visualisation to control what and how the data appears. The tqx parameter gets passed through into the stylesheet as a parameter, and parsed there. The tq parameter is used to construct the SPARQL query itself, specifically which variables will get included within the result. The rest of the SPARQL query — the filter and the ordering — are set in the code which calls the proxy() function, which is in london-borough.php within the same directory, and looks like:

<?php
  include "utils.php";
  proxy('?borough a <http://www.jenitennison.com/ontology/data#LondonBorough> .
         ?borough <http://www.w3.org/2000/01/rdf-schema#label> ?label .
         ?borough <http://www.jenitennison.com/ontology/data#maleLifeExpectancy> ?maleLE .
         ?borough <http://www.jenitennison.com/ontology/data#femaleLifeExpectancy> ?femaleLE .', 
        '?label');
?>

This code defines the names of the variables that can be used within the tq parameter, and therefore selected and displayed within the graph. So for example, if I request:

http://www.jenitennison.com/visualisation/data/london-borough?tq=select+label,+maleLE

this translates into the SPARQL query:

SELECT ?label ?maleLE
WHERE {
  ?borough a <http://www.jenitennison.com/ontology/data#LondonBorough> .
  ?borough <http://www.w3.org/2000/01/rdf-schema#label> ?label .
  ?borough <http://www.jenitennison.com/ontology/data#maleLifeExpectancy> ?maleLE .
  ?borough <http://www.jenitennison.com/ontology/data#femaleLifeExpectancy> ?femaleLE .
}
ORDER BY ?label

At the moment, the PHP just uses the select portion of the tq parameter to determine which data to display. It would be possible to map other aspects of the Google Visualisation query language onto SPARQL, but this will do for now.

The final step is to amend the .htaccess to do a basic rewrite to prevent people from having to put .php at the end of the URI because I don’t like URIs that indicate the technology they’re using. In this case it looks like:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d

  RewriteRule ^(.+) $1.php [L,QSA]
</IfModule>

as this gives me flexibility later on to add more PHP files that can do similar things.

So now we have a Data Source that can provide the label, male life expectancy and female life expectancy for the London Boroughs. Using it requires a copy and tweak of an example from Google:

<html>
  <head>
    <script type="text/javascript" src="http://www.google.com/jsapi"></script>
    <script type="text/javascript">
      google.load("visualization", "1", {packages:["barchart"]});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        // Replace the data source URL on next line with your data source URL.
        var query = new google.visualization.Query('http://www.jenitennison.com/visualisation/data/london-borough');
        query.setQuery('select label, maleLE, femaleLE');
        // Send the query with a callback function.
        query.send(handleQueryResponse);
      };
      function handleQueryResponse(response) {
        if (response.isError()) {
          alert('Error in query: ' + response.getMessage() + ' ' + response.getDetailedMessage());
          return;
        }

        var data = response.getDataTable();
        data.setColumnLabel(1, 'Male Life Expectancy');
        data.setColumnLabel(2, 'Female Life Expectancy');
        var chart = new google.visualization.BarChart(document.getElementById('chart_div'));
        chart.draw(data, {
          width: 600, 
          height: 450, 
          is3D: false, 
          title: 'Life Expectancy in London Boroughs',
          axisFontSize: 10,
          colors: ['#4162B5', '#CF413A']});
      };
    </script>
  </head>

  <body>
    <div id="chart_div"></div>
  </body>
</html>

The drawChart() function is where the URL to the Data Source gets set. It’s actually done in two parts:

var query = new google.visualization.Query('http://www.jenitennison.com/visualisation/data/london-borough');

sets the base URI to be used, and:

query.setQuery('select label, maleLE, femaleLE');

sets the value of the tq parameter without you having to worry about escaping the special characters it may contain.

After getting the data, I set the column labels in the code itself: they aren’t provided in the Data Source because it’s easier and more generic this way. Plus I set a bunch of other display options within the HTML page, so it seems like the right place for it.

The result of all this is a graph that you can see at http://www.jenitennison.com/visualisation/london-borough-life-expectancy.html.

If you want to do something similar, feel free to grab hold of utils.php. You can either reuse my hosted copy of SRXtoGoogleVisData.xsl or move it onto your own site. Then all you have to do is:

adjust the graph variable in utils.php, and the location of SRXtoGoogleVisData.xsl if you need to
create another PHP file similar to london-borough.php that defines a filter and an ordering over a set of data
tweak your .htaccess if you want to
create an HTML page that references your Data Source to create a Google Visualisation

My current plan is to continue to refine utils.php and SRXtoGoogleVisData.xsl to make it easy to create SPARQL-backed visualisations. More soon.