Talis

SPARQL & Visualisation Frustrations: Linked Data

Jeni — Mon, 03 Aug 2009 20:36:34 +0000

I’ll start with the problem. To create the graphs I showed in my last post, I wanted to split MPs into groups based on their party affiliation. Ideally, I wanted the Google Visualisation query to look like:

select mp, additionalCosts, totalTravel, totalBasic 
where party = 'Conservative' 
order by totalClaim desc 
limit 25

because this is reasonably easy to understand and for a developer to create without having to know any magic URIs.

The party affiliation for an MP is given in the RDF supplied within the Talis store as a pointer to one of the resources:

http://dbpedia.org/resource/Labour_Party_(UK)
http://dbpedia.org/resource/Conservative_Party_(UK)
http://dbpedia.org/resource/Liberal_Democrats

Now, if you visit http://dbpedia.org/resource/Conservative_Party_(UK) then you’ll see precious few properties and none of them give you access to the string ‘Conservative’. If you look at http://dbpedia.org/resource/Liberal_Democrats, you’ll see plenty of properties, one of which is dbpprop:partyName. But trying to query on dbpprop:partyName within the Talis data store gives me nothing, because that information hasn’t been imported into the particular store that this SPARQL query is running on.

What I did in utils.php was extend the parsing of the tq parameter, which is supposed to be in the Google Visualisation query language, to understand <URI> as a reference to a resource. In other words, you can create a query like:

select mp, additionalCosts, totalTravel, totalBasic 
where rParty = <http://dbpedia.org/resource/Conservative_Party_(UK)> 
order by totalClaim desc 
limit 25

and this will be mapped to a SPARQL query that looks like:

SELECT ?mp ?additionalCosts ?totalTravel ?totalBasic 
WHERE {
  ...
  FILTER (?rParty = <http://dbpedia.org/resource/Conservative_Party_(UK)>)
}
ORDER BY desc(?totalClaim)
LIMIT 25

I don’t like having done this, because I don’t want Data Sources that happen to be SPARQL queries to look any different from other Data Sources. Introducing a new syntax for URI literals isn’t really on.

The superficial fix is to always provide basic labelling information for the resources referenced within a triplestore. In this case, Leigh actually did include an rdfs:label property for each of the party URIs within the Guardian store, so it was possible to use the query I wanted to use after all (though it took some experimentation to find this out).

But underlying this is a bigger issue. Much is made of linked data — that you can find out more about a particular thing by resolving the link to that thing — but the best illustrations of the power and benefits of the semantic web tend to revolve around analysis and visualisations of moderately large amounts of data using SPARQL. And SPARQL (as yet) only runs on individual triplestores, which do not contain the entire semantic web. Every SPARQL query is limited by what has been loaded into the particular triplestore that is queried.

Now, one of the “time-permitting” requirements for SPARQL 1.1 is Federated Queries:

Federated query is the ability to take a query and provide solutions based on information from many different sources. It is a hard problem in its most general form and is the subject of continuing (and continuous) research. A building block is the ability to have one query be able to issue a query on another SPARQL endpoint during query execution.

Time-permitting, the SPARQL Working Group will define the syntax and semantics for handling a basic class of federated queries in which the SPARQL endpoints to use in executing portions of the query are explicitly given by the query author.

That’s certainly “a building block”, but it can’t be the only method. For many data publishers, it’s going to be far far simpler to publish their data as linked data in RDF/XML than it is to provide a SPARQL endpoint for that data. We can ask organisations like Talis to crawl our data and provide a SPARQL endpoint for it, and hope that the SPARQL Working Group have time to address federated search, but really we need tools that make it easy to aggregate, analyse and visualise linked data directly rather than through a triplestore silo.

So how about it?

Map Visualisation of MPs Travel Expenses

Jeni — Fri, 31 Jul 2009 22:33:13 +0000

During Guardian Hack Day 2, Leigh ported the Guardian’s MP’s Expenses data into Talis. Most wonderfully, this gives a SPARQL endpoint that can be used to query the data. I thought I’d try to use the same approach as I blogged about recently, using a SPARQL query as a Data Source for a Google Visualisation of the MP’s expenses data.

To cut to the chase, here’s a screenshot of the result (follow the link for the more interactive version):

I created this visualisation with the same general approach as I explained last time.

First, I’ve been working on the visualisation utils.php, which is a reasonably simple PHP script that exposes a SPARQL endpoint as a Google Visualisation Data Source. Requests to a Data Source use a special query language to indicate the information that should be included, how it should be sorted, how many rows of data there should be, and so on.

Previously, utils.php only understood the select portion of the tq parameter which contains this query; I’ve expanded it to understand (somewhat limited versions of) the select, where, order by, limit and offset parts of the query, which of course have equivalents in SPARQL. Since these parts of the Google Visualisation query language are pretty close to SPARQL, this is actually just a bunch of string munging, which isn’t particularly interesting, so just grab hold of it if you want to use it.

Second, I created a PHP script (mp-travel.php) specifically for the MPs expenses data that pulls out the parts that I’m interested in and exposes them as variables which can be used in the query language. This is what the file looks like:

<?php
  include "utils.php";
  proxy('?rMP a <http://guardian.dataincubator.org/ns/MemberOfParliament> .
         ?rMP <http://xmlns.com/foaf/0.1/name> ?mp .
         ?rMP <http://guardian.dataincubator.org/ns/mp-expenses/majority> ?majority .
         ?rMP <http://dbpedia.org/property/constituency> ?rConstituency .
         ?rConstituency rdfs:label ?constituency .
         ?rConstituency <http://www.w3.org/2003/01/geo/wgs84_pos#lat> ?lat .
         ?rConstituency <http://www.w3.org/2003/01/geo/wgs84_pos#long> ?long .
         ?rMP <http://guardian.dataincubator.org/ns/mp-expenses/total-travel> ?totalTravel .',
        'desc(?totalTravel)', 
        'guardian');
?>

The second argument to the proxy() function is the default ordering (desc(?totalTravel)) and the third is the name of the Talis data store that’s being used (guardian).

The first argument is a query which determines the variables that are exposed by the Data Source. This Data Source exposes the variables:

mp: the name of the MP
majority: the majority that they have in their constituency
constituency: the name of the constituency
lat, long: the latitude and longitude of the constituency (presumably the centre of it)
totalTravel: the total amount claimed for travel by the MP
rMP: the URI used to identify the MP
rConstituency: the URI used to identify the constituency

Third, I created an HTML document that used the Google Visualisation API to create the map visualisation that I’ve shown above. The really important lines are:

var query = new google.visualization.Query('http://www.jenitennison.com/visualisation/data/mp-travel');
query.setQuery('select lat, long, totalTravel, mp order by majority limit 100');

The first line shows the URL for the Data Source, which is essentially a pointer to the mp-travel.php script. The second line shows the query that’s sent to the Data Source: “select lat, long, totalTravel, mp order by majority limit 100”.

Put together, when you load http://www.jenitennison.com/visualisation/mp-travel.html, you create a Google Visualisation GeoMap which uses as its data the result of the SPARQL query

SELECT ?lat ?long ?totalTravel ?mp
WHERE {
  ?rMP a <http://guardian.dataincubator.org/ns/MemberOfParliament> .
  ?rMP <http://xmlns.com/foaf/0.1/name> ?mp .
  ?rMP <http://guardian.dataincubator.org/ns/mp-expenses/majority> ?majority .
  ?rMP <http://dbpedia.org/property/constituency> ?rConstituency .
  ?rConstituency rdfs:label ?constituency .
  ?rConstituency <http://www.w3.org/2003/01/geo/wgs84_pos#lat> ?lat .
  ?rConstituency <http://www.w3.org/2003/01/geo/wgs84_pos#long> ?long .
  ?rMP <http://guardian.dataincubator.org/ns/mp-expenses/total-travel> ?totalTravel .
}
ORDER By ?majority
LIMIT 100

on the SPARQL endpoint at http://api.talis.com/stores/guardian/services/sparql.

Here’s hoping you can reuse the Data Source or the code that was used to make it. Let me know if you do!

The Real Deal: data.gov.uk

Jeni — Sun, 26 Jul 2009 15:38:54 +0000

I’m sure that you’ve noticed that my recent posts have been somewhat obsessed with publishing and using public sector information. It’s because I’ve somehow been sucked into the work going on within the UK government, with Tim Berners-Lee and Nigel Shadbolt advising, to publish its data as linked data.

My recent blog posts about publishing data using Talis have actually been a front for much more complex work that I’ve been doing with a different data set.

As an early demonstration of how existing government data sets might be turned into linked data, a few weeks ago I was given a CSV file containing road traffic counts; the raw data that lies behind the traffic flow information available on the Department for Transport website. The data is really interesting and ripe for visualisations and analysis. For each hour of particular days each year, at particular points on many roads within the UK, the Department for Transport measures the number of bicycles, motorbikes, cars, vans, buses and HGVs of various types that roll past in each direction. The data contains information about:

the count of each of the various classes of traffic that pass the point in a particular direction on a particular hour of a particular day
the points at which these measurements were taken
the roads on which the points are situated
the areas in which the points are situated
the local authority that is in charge of these areas
the region that the area is in
the country that the region is in

The challenge was to turn the 386Mb CSV file into linked data. The result is up and available for you to look at; a good starting point is http://geo.data.gov.uk/0/country. Just follow the links from there.

With a few false starts and mis-steps, this is the process that I went through:

Tidied the CSV file so that it could be processed using awk. That meant replacing the commas that were delimiters with |s. It also meant removing a couple of weird ^M characters that had snuck into the file.
Examined the data and came up with an informal ontology and prototype URI scheme.
Created a bunch of awk scripts to extract different data from the files and create RDF/XML from it.
Ran the scripts to create RDF/XML.
Uploaded the data into a Talis store.
Created appropriate PHP for the data and put it into a proxy server.

Some of this has been covered by my recent posts, so I’m just going to talk about a few of these steps in a bit more detail.

First, the URIs. Frankly, they’re an experiment to see how it plays. The templates are:

countries: http://geo.data.gov.uk/0/id/country/{name}, eg http://geo.data.gov.uk/0/id/country/england
regions: http://geo.data.gov.uk/0/id/region/{name}, eg http://geo.data.gov.uk/0/id/region/north-west
areas: http://geo.data.gov.uk/0/id/area/{ONS code}, eg http://geo.data.gov.uk/0/id/area/00KA
local authorities: http://local-government.data.gov.uk/0/id/local-authority/{ONS code for area}, eg http://local-government.data.gov.uk/0/id/local-authority/00KA
roads: http://transport.data.gov.uk/0/id/road/{name} or http://transport.data.gov.uk/0/id/road/U-{random number}, eg http://transport.data.gov.uk/0/id/road/M5
traffic count points: http://transport.data.gov.uk/0/id/traffic-count-point/{number}, eg http://transport.data.gov.uk/0/id/traffic-count-point/36195
traffic counts: http://transport.data.gov.uk/0/id/traffic-count/{point number}/{direction}/{date}/{hour}/{traffic type}, eg http://transport.data.gov.uk/0/id/traffic-count/4/N/2008-06-05/08:00:00/HGVr2

The subdomains are one way of subdividing the vast set of public sector information into vague categories that might be handled by different departments, without using the (highly changeable) department names in the URI. The /0 portion of each URI is a version number: these URIs are experimental and liable to be unsupported in the future so they’re marked with a version 0. The /id portion of each URI indicates that these are URIs for non-information resources; the response is a 303 See Other redirect to the same URIs but without the /id.

After the /id, the URIs follow a common pattern of naming a class of resource, followed by an appropriate identifier for that resource. The identifiers themselves are designed to be unique, unlikely to change, and human readable.

The ontologies, well, actually they don’t exist as yet except in my head. It’s been more important to make the data available than to provide ontologies for it. Triplestores and SPARQL queries work without ontologies; indeed you have to go out of your way to find applications that actually reason with them. Like schemas for XML documents, they’re not absolutely essential, but useful for documentation purposes and potentially useful for applications.

There are, though, a couple of SKOS schemes for categorising roads and vehicle types. These are available via:

http://transport.data.gov.uk/0/category/road
http://transport.data.gov.uk/0/category/vehicle

They were informed by the British Roads FAQ and the data definitions from the Department for Transport. I heartily recommend a read; it’s scintillating stuff!

Anyway, with this size of file, and the kind of processing that needed to be done with it, the simple XSLT that I talked about previously for extracting data out of CSV files just wasn’t going to cut it. Awk, on the other hand, is designed for this kind of processing. Most of the RDF/XML could be generated by collecting unique values from the file. For example, to generate the RDF/XML for the regions I used:

BEGIN { 
  FS = "|";
  print "<rdf:RDF xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"";
  print "  xmlns:rdfs=\"http://www.w3.org/2000/01/rdf-schema#\"";
  print "  xmlns:g=\"http://geo.data.gov.uk/0/ontology/geo#\">";
}
FNR > 1 {
  countries[$2] = substr($1, 2, length($1) - 2);
  regions[$2] = substr($2, 2, length($2) - 2);
  codes[$2] = substr($3, 2, length($3) - 2);
}
END { 
  for (region in regions) {
    country = countries[region];
    name = regions[region];
    code = codes[region];
    path = tolower(name);
    gsub(" ", "-", path);
    print "<g:Region rdf:about=\"http://geo.data.gov.uk/0/id/region/" path "\">";
    print "  <rdfs:label>" name "</rdfs:label>";
    print "  <g:isInCountry>";
    print "    <g:Country rdf:about=\"http://geo.data.gov.uk/0/id/country/" tolower(country) "\">";
    print "      <g:hasRegion rdf:resource=\"http://geo.data.gov.uk/0/id/region/" path "\" />";
    print "    </g:Country>";
    print "  </g:isInCountry>";
    if (code != "") {
      print "  <g:ONScode rdf:datatype=\"http://www.w3.org/2001/XMLSchema#NCName\">" code "</g:ONScode>";
    }
    print "</g:Region>";
  }
  print "</rdf:RDF>"; 
}

This generated RDF/XML that looks like:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:g="http://geo.data.gov.uk/0/ontology/geo#">
<g:Region rdf:about="http://geo.data.gov.uk/0/id/region/london">
  <rdfs:label>London</rdfs:label>
  <g:isInCountry>
    <g:Country rdf:about="http://geo.data.gov.uk/0/id/country/england">
      <g:hasRegion rdf:resource="http://geo.data.gov.uk/0/id/region/london" />
    </g:Country>
  </g:isInCountry>
  <g:ONScode rdf:datatype="http://www.w3.org/2001/XMLSchema#NCName">H</g:ONScode>
</g:Region>
<g:Region rdf:about="http://geo.data.gov.uk/0/id/region/yorkshire-and-the-humber">
  <rdfs:label>Yorkshire and The Humber</rdfs:label>
  <g:isInCountry>
    <g:Country rdf:about="http://geo.data.gov.uk/0/id/country/england">
      <g:hasRegion rdf:resource="http://geo.data.gov.uk/0/id/region/yorkshire-and-the-humber" />
    </g:Country>
  </g:isInCountry>
  <g:ONScode rdf:datatype="http://www.w3.org/2001/XMLSchema#NCName">D</g:ONScode>
</g:Region>
...
</rdf:RDF>

In other cases, I needed to split up the RDF/XML that was generated into several files. Uploads to Talis of more than about 2Mb cause the upload to fail. The traffic count point RDF/XML needed to be split into 13 separate files. The traffic counts themselves… well, I haven’t managed to do it all yet but to give you an idea, the 2008 data alone generated 1800 RDF/XML files, each about 1.6Mb in size and each taking about a minute to upload. What’s there now is all the 2008 data, and the overall motor vehicle counts from all the years. More will be added gradually.

The awk script that generates the count data in separate files is:

BEGIN { 
  FS = "|";
  fileCount = 0;
  countCount = 99999;
  curlFile = "traffic-counts.curl.sh";
}
FNR > 1 && $15 ~ /\/2008 / {
  countCount += 1;
  if (countCount > 200) {
    if (fileCount != 0) {
      print "</rdf:RDF>" > fileName; 
      close(fileName);
    }
    countCount = 0;
    fileCount += 1;
    fileName = "traffic-counts/traffic-counts." fileCount ".rdf";
    print "creating", fileName;
    print "echo loading", fileName > curlFile;
    print "curl -H \"Content-type: application/rdf+xml\" -o progress.txt --digest -u username:password --data-binary @" fileName " http://api.talis.com/stores/transport/meta" > curlFile;

    print "<?xml version=\"1.0\" encoding=\"ASCII\"?>" > fileName;
    print "<rdf:RDF xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"" > fileName;
    print "  xmlns:rdfs=\"http://www.w3.org/2000/01/rdf-schema#\"" > fileName;
    print "  xmlns:xsd=\"http://www.w3.org/2001/XMLSchema#\"" > fileName;
    print "  xmlns:t=\"http://transport.data.gov.uk/0/ontology/traffic#\"" > fileName;
    print "  xml:base=\"http://transport.data.gov.uk/0/id/traffic-count/\">" > fileName;
  }

  cp = $7;
  date = $15;
  direction = substr($16, 2, length($16) - 2);
  split(date, dateFields, " ");
  date = dateFields[1];
  split(date, dateFields, "/");
  date = sprintf("%04d-%02d-%02d", dateFields[3], dateFields[2], dateFields[1]);
  hour = sprintf("%02d:00:00", $17);
  base = "http://transport.data.gov.uk/0/id/traffic-count/" cp "/" direction "/" date "/" hour;

  cycles = $18;
  motorbikes = $19;
  ...

  print "<t:Count rdf:about=\"" base "/cycle\">" > fileName;
  print "  <t:point>" > fileName;
  print "    <t:CountPoint rdf:about=\"http://transport.data.gov.uk/0/id/traffic-count-point/" cp "\">" > fileName;
  print "      <t:count rdf:resource=\"" base "/cycle\" />" > fileName;
  print "    </t:CountPoint>" > fileName;
  print "  </t:point>" > fileName;
  print "  <t:hour rdf:datatype=\"http://www.w3.org/2001/XMLSchema#dateTime\">" date "T" hour "</t:hour>" > fileName;
  print "  <t:direction>" direction "</t:direction>" > fileName;
  print "  <t:category rdf:resource=\"http://transport.data.gov.uk/0/category/bicycle\" />" > fileName;
  print "  <rdf:value  rdf:datatype=\"http://www.w3.org/2001/XMLSchema#integer\">" cycles "</rdf:value>" > fileName;
  print "</t:Count>" > fileName;
  print "<t:Count rdf:about=\"" base "/motorbike\">" > fileName;
  print "  <t:point>" > fileName;
  print "    <t:CountPoint rdf:about=\"http://transport.data.gov.uk/0/id/traffic-count-point/" cp "\">" > fileName;
  print "      <t:count rdf:resource=\"" base "/motorbike\" />" > fileName;
  print "    </t:CountPoint>" > fileName;
  print "  </t:point>" > fileName;
  print "  <t:hour rdf:datatype=\"http://www.w3.org/2001/XMLSchema#dateTime\">" date "T" hour "</t:hour>" > fileName;
  print "  <t:direction>" direction "</t:direction>" > fileName;
  print "  <t:category rdf:resource=\"http://transport.data.gov.uk/0/category/motorbike\" />" > fileName;
  print "  <rdf:value  rdf:datatype=\"http://www.w3.org/2001/XMLSchema#integer\">" motorbikes "</rdf:value>" > fileName;
  print "</t:Count>" > fileName;
  ...
}
END {
  print "</rdf:RDF>" > fileName; 
  close(fileName);
}

This also generates a shall script that includes the curl instructions to upload the files.

The original data contained easing/northing information about each point when generally latitude/longitude is easier for mapping. So I extracted the easting/northings, used the free (Windows only) software available via the Ordnance Survey to turn these into latitude/longitude — there is a web service to do the same, but you can only do 200 coordinates at a time — converted those into decimals, then RDF, and uploaded them.

The PHP scripts that serve the data as linked data are exactly what I’ve shown before. I amended the .htaccess file to redirect to an appropriate PHP script like this:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d

  RewriteRule ^id/(.+)$  id.php [L]

  RewriteCond %{REQUEST_URI} !\.php
  RewriteRule ^([^/]+)(/.+)? $1.php$2 [L,QSA]
</IfModule>

and created PHP scripts for each of the types of data being published. For example, region.php is:

<?php
  include "utils.php";
  proxy('http://geo.data.gov.uk/0/ontology/geo#Region', 50);
?>

And there we have it. Linked traffic count data on the web.

(And because this is all published through Talis, there’s also a SPARQL endpoint that you could use to run queries and create visualisations. Knock yourself out.)

Please take a look and comment on what we’ve done. What’s your opinion of the URI scheme? Is it useful to be able to access the data as linked data? Which other formats would you like to see?

Creating Google Visualisations of Linked Data

Jeni — Thu, 23 Jul 2009 21:37:33 +0000

Update: For the people who couldn’t read the post because the graph didn’t have 0 as its x-axis minimum, here is the version of the graph that does. I haven’t removed the other version, since doing so would make the comments confusing and I think it’s interesting to compare the two.

There are idealists who immediately see the publication of Open Data as a Good Thing, and leap up and down (metaphorically or physically) shouting “Raw Data Now”. There are also a whole bunch of people who need to “see the shiny”. They need to understand why publishing Open Data is a Good Thing, and most particularly what the benefit is going to be to them.

This is understandable. Publishers bear the cost of the development of URI schemes, XML formats, RDF ontologies, and other infrastructure for serving data, and the ongoing maintenance cost of domain resolution, bandwidth usage and user support. Even publishers with a public-service remit (who may not need to see monetary payback) need to be convinced that there will be some kind of return on the investment.

One result of making data available is that it enables you and others to easily construct nice visualisations over the data, and maybe spot useful patterns within it. This is particularly useful for public sector information because it can provide feedback on how effective a particular policy has been or where more resources need to be spent.

So I thought it would be worthwhile trying to explore how to create visualisations of some data, starting with the London Borough data that I’ve published using Talis.

Nowadays, there are a bunch of visualisation libraries around. My first experiment is going to be using the Google Visualisation API, which offers a range of different, reasonably pretty, visualisations that work cross-browser using either SVG or VML. In this post, I’m going to use the barchart visualisation, to create a chart of male and female life expectancy in the London Boroughs. It looks like this:

One of the interesting things about the Google Visualisation API is that you can provide the data for every visualisation using a Data Source. A Data Source is any web page that, in the face of particular requests, returns JSON in a particular format. The format basically encodes a table with named (and typed) columns and a number of rows containing cells which have values for each of those columns.

For the London Borough data, the JSON needs to look something like:

google.visualization.Query.setResponse({
  version:0.6,
  status:'ok',
  reqId:0,
  table:{
    cols:[
      {id:'label', type:'string'},
      {id:'maleLE', type:'number'},
      {id:'femaleLE', type:'number'}
    ],
    rows:[
      {c:[
        {v:'Barking & Dagenham'},
        {v:76.3},
        {v:80.3}]},
      {c:[
        {v:'Barnet'},
        {v:79.5},
        {v:83.6}]},
      {c:[
        {v:'Bexley'},
        {v:78.7},
        {v:82.7}]},
      ...
    ]}
})

Now we already know, from my last couple of blog posts, how to query the Talis platform through its SPARQL endpoint, and how to use Talis’ transformation service to invoke an XSLT transformation over the XML that we get back from that. The same principle applies:

create a SPARQL query that will return the information we want
create an XSLT transformation that will tidy the result into the format we want it
set up a PHP page to access the relevant URI and pass through the results
create an .htaccess file so that invoking the PHP can be done without revealing it’s PHP

First, the SPARQL query. As I said, a Data Source is essentially a table of named columns. While we could do a DESCRIBE or CREATE query, a SELECT query is a lot closer match to the tabular layout that the Data Source needs because it’s also essentially a table of named columns (the variables you select). For the data that we want in the table, an appropriate SPARQL query would be:

SELECT ?label ?maleLE ?femaleLE
WHERE {
  ?borough a <http://www.jenitennison.com/ontology/data#LondonBorough> .
  ?borough <http://www.w3.org/2000/01/rdf-schema#label> ?label .
  ?borough <http://www.jenitennison.com/ontology/data#maleLifeExpectancy> ?maleLE .
  ?borough <http://www.jenitennison.com/ontology/data#femaleLifeExpectancy> ?femaleLE .
}
ORDER BY ?label

Plugging this query into the Talis SPARQL endpoint (have a go at http://api.talis.com/stores/rdfquery-dev1/services/sparql if you like) gives a response in the SPARQL Query Results Format which looks like:

<sparql xmlns="http://www.w3.org/2005/sparql-results#">
  <head>
    <variable name="label"/>
    <variable name="maleLE"/>
    <variable name="femaleLE"/>
  </head>
  <results>
    <result>
      <binding name="label">
        <literal>Barking &amp; Dagenham</literal>
      </binding>
      <binding name="maleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">76.3</literal>
      </binding>
      <binding name="femaleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">80.3</literal>
      </binding>
    </result>
    <result>
      <binding name="label">
        <literal>Barnet</literal>
      </binding>
      <binding name="maleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">79.5</literal>
      </binding>
      <binding name="femaleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">83.6</literal>
      </binding>
    </result>
    <result>
      <binding name="label">
        <literal>Bexley</literal>
      </binding>
      <binding name="maleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">78.7</literal>
      </binding>
      <binding name="femaleLE">
        <literal datatype="http://www.w3.org/2001/XMLSchema#decimal">82.7</literal>
      </binding>
    </result>
    ...
  </results>
</sparql>

The second step — transforming this SPARQL result into JSON — just take a little bit of XSLT (1.0, remember, because that’s all the Talis’ Transformation Service can manage). My aim in this post is to show that anyone, even if they don’t have write access to a Talis data store, can create these visualisations, so I’ve just put the XSLT on my site at http://www.jenitennison.com/visualisation/data/SRXtoGoogleVisData.xsl. I’m not going to duplicate it here; it’s generic enough for reuse should you want to.

The third step is to create some PHP that handles a query from the Google Visualisation. The requests will include two parameters:

tqx defines details about how the data should be returned, such as its format
tq defines a query in the Google Visualisation API query language that identifies precisely the data that should be returned

I want this PHP to be reusable, so I’ve created a utils.php that looks like this:

<?php
  $store = 'rdfquery-dev1';

  function proxy($filter, $order) {
    global $store;
    $tqx = $_GET['tqx'];
    $tq = $_GET['tq'];

    // Parse tq parameter
    if ($tq) {
      $select = stristr($tq, 'select ');
      $select = substr($select, 7);
      $select = explode(',', $select);
      foreach ($select as $var) {
        $var = trim($var);
        $vars[] = "?$var";
      }
      $vars = implode(' ', $vars);
    } else {
      $vars = '*';
    }
    $sparql = "SELECT $vars WHERE { $filter } ORDER BY $order";

    $params = array('query' => $sparql, 'output' => 'xml');
    $query = http_build_query($params);
    $rdfURL = "http://api.talis.com/stores/$store/services/sparql?$query";

    // URL for the transformation
    $params = array('xml-uri' => $rdfURL, 
      'xsl-uri' => "http://www.jenitennison.com/visualisation/data/SRXtoGoogleVisData.xsl",
      'tqx' => $tqx);
    $query = http_build_query($params);
    $txURL = "http://api.talis.com/tx?$query";

    $resource = fopen($txURL, 'rb');
    header('Content-Type: application/json', true);
    fpassthru($resource);
    return;
  }
?>

Things to note here are about the processing of the tq and tqx parameters that are sent by the Google Visualisation to control what and how the data appears. The tqx parameter gets passed through into the stylesheet as a parameter, and parsed there. The tq parameter is used to construct the SPARQL query itself, specifically which variables will get included within the result. The rest of the SPARQL query — the filter and the ordering — are set in the code which calls the proxy() function, which is in london-borough.php within the same directory, and looks like:

<?php
  include "utils.php";
  proxy('?borough a <http://www.jenitennison.com/ontology/data#LondonBorough> .
         ?borough <http://www.w3.org/2000/01/rdf-schema#label> ?label .
         ?borough <http://www.jenitennison.com/ontology/data#maleLifeExpectancy> ?maleLE .
         ?borough <http://www.jenitennison.com/ontology/data#femaleLifeExpectancy> ?femaleLE .', 
        '?label');
?>

This code defines the names of the variables that can be used within the tq parameter, and therefore selected and displayed within the graph. So for example, if I request:

http://www.jenitennison.com/visualisation/data/london-borough?tq=select+label,+maleLE

this translates into the SPARQL query:

SELECT ?label ?maleLE
WHERE {
  ?borough a <http://www.jenitennison.com/ontology/data#LondonBorough> .
  ?borough <http://www.w3.org/2000/01/rdf-schema#label> ?label .
  ?borough <http://www.jenitennison.com/ontology/data#maleLifeExpectancy> ?maleLE .
  ?borough <http://www.jenitennison.com/ontology/data#femaleLifeExpectancy> ?femaleLE .
}
ORDER BY ?label

At the moment, the PHP just uses the select portion of the tq parameter to determine which data to display. It would be possible to map other aspects of the Google Visualisation query language onto SPARQL, but this will do for now.

The final step is to amend the .htaccess to do a basic rewrite to prevent people from having to put .php at the end of the URI because I don’t like URIs that indicate the technology they’re using. In this case it looks like:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d

  RewriteRule ^(.+) $1.php [L,QSA]
</IfModule>

as this gives me flexibility later on to add more PHP files that can do similar things.

So now we have a Data Source that can provide the label, male life expectancy and female life expectancy for the London Boroughs. Using it requires a copy and tweak of an example from Google:

<html>
  <head>
    <script type="text/javascript" src="http://www.google.com/jsapi"></script>
    <script type="text/javascript">
      google.load("visualization", "1", {packages:["barchart"]});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        // Replace the data source URL on next line with your data source URL.
        var query = new google.visualization.Query('http://www.jenitennison.com/visualisation/data/london-borough');
        query.setQuery('select label, maleLE, femaleLE');
        // Send the query with a callback function.
        query.send(handleQueryResponse);
      };
      function handleQueryResponse(response) {
        if (response.isError()) {
          alert('Error in query: ' + response.getMessage() + ' ' + response.getDetailedMessage());
          return;
        }

        var data = response.getDataTable();
        data.setColumnLabel(1, 'Male Life Expectancy');
        data.setColumnLabel(2, 'Female Life Expectancy');
        var chart = new google.visualization.BarChart(document.getElementById('chart_div'));
        chart.draw(data, {
          width: 600, 
          height: 450, 
          is3D: false, 
          title: 'Life Expectancy in London Boroughs',
          axisFontSize: 10,
          colors: ['#4162B5', '#CF413A']});
      };
    </script>
  </head>

  <body>
    <div id="chart_div"></div>
  </body>
</html>

The drawChart() function is where the URL to the Data Source gets set. It’s actually done in two parts:

var query = new google.visualization.Query('http://www.jenitennison.com/visualisation/data/london-borough');

sets the base URI to be used, and:

query.setQuery('select label, maleLE, femaleLE');

sets the value of the tq parameter without you having to worry about escaping the special characters it may contain.

After getting the data, I set the column labels in the code itself: they aren’t provided in the Data Source because it’s easier and more generic this way. Plus I set a bunch of other display options within the HTML page, so it seems like the right place for it.

The result of all this is a graph that you can see at http://www.jenitennison.com/visualisation/london-borough-life-expectancy.html.

If you want to do something similar, feel free to grab hold of utils.php. You can either reuse my hosted copy of SRXtoGoogleVisData.xsl or move it onto your own site. Then all you have to do is:

adjust the graph variable in utils.php, and the location of SRXtoGoogleVisData.xsl if you need to
create another PHP file similar to london-borough.php that defines a filter and an ordering over a set of data
tweak your .htaccess if you want to
create an HTML page that references your Data Source to create a Google Visualisation

My current plan is to continue to refine utils.php and SRXtoGoogleVisData.xsl to make it easy to create SPARQL-backed visualisations. More soon.

Publishing Linked Data on the Talis Platform, Part 3

Jeni — Tue, 21 Jul 2009 21:39:21 +0000

This is the third in a series of posts about using the Talis Platform as a back end for serving linked data. In the first part, I showed how to add data to a store. In the second post, I showed how to use some PHP scripts to publish the data as Linked Data, at the URLs you use as your identifiers.

In this post, I’m going to begin the process of exposing the data in a way that makes it easy to locate and reuse. One of the biggest lessons I learned after the initial publication of the London Gazette data as RDFa is that the publication of data and metadata about individual items is not enough. To make the data usable, you have to make it discoverable. To make it discoverable there must be an entry point from which you can locate the data. One kind of easy entry point is a list.

In the case of the data about London Boroughs that I’ve been using, there aren’t currently any links to the data, so there is no way to discover it aside from me telling you the URI template (http://www.jenitennison.com/data/id/london-borough/{name}, where name is hyphenated and in lowercase) and you knowing the name of a London Borough that you want to look up. Discovery via a URI template that I told you relies on out-of-band information, and contradicts the RESTful tenet of “hypertext as the engine of application state”.

Instead, I need to offer an entry point from which you can follow links (or fill in forms) to discover information about the various London Boroughs. Since I’m dealing with a small set of information here, I’m going to do this in the straight-forward way of having http://www.jenitennison.com/data/london-borough contain a brief description of each of the known London Boroughs, including (obviously) a link to the URI for the London Borough, from which you can get more information.

I don’t want to return all the information about each of the London Boroughs within the list, so I’m going to use a CONSTRUCT SPARQL query to create triples that include just the type and label (if it has one) of the borough. Here’s the initial query:

CONSTRUCT { 
  ?thing a <http://www.jenitennison.com/ontology/data#LondonBorough> . 
  ?thing <http://www.w3.org/2000/01/rdf-schema#label> ?label . } 
WHERE { 
  ?thing a <http://www.jenitennison.com/ontology/data#LondonBorough> . 
  OPTIONAL { ?thing <http://www.w3.org/2000/01/rdf-schema#label> ?label . }}

Now I want to make this request a bit more generic. The type that I’m looking for here is http://www.jenitennison.com/ontology/data#LondonBorough but if I (or you!) wanted to use it for resources of different types, you’d want it to reference a different class. In addition, while the list of London Boroughs is reasonably small, lists of individuals of other types might be much larger, in which case you’d want a facility to page through them.

So in the PHP code I’m going to have three variables: $type is the URI of the class of things that should be listed, $limit is the number of those things that should appear on each page, and $start is the first item in the page. In addition, to ensure a consistent and (hopefully) meaningful order, I’m going to order the results based on their URI. So this is what the query looks like:

CONSTRUCT { 
  ?thing a <$type> . 
  ?thing <http://www.w3.org/2000/01/rdf-schema#label> ?label . } 
WHERE { 
  ?thing a <$type> . 
  OPTIONAL { ?thing <http://www.w3.org/2000/01/rdf-schema#label> ?label . }} 
ORDER BY ?thing
LIMIT $limit
OFFSET $start

I’m also going to rejig the PHP that I’ve been using to use this query when it receives the request http://www.jenitennison.com/data/london-borough. First, utils.php is now going to hold a proxy() function that performs requests based on the URI of the request or the arguments passed to it. Aside from the body of the SPARQL query, the code is pretty similar to the ASK query (it’s getting close to the point where I should at least refactor, or maybe start using Moriarty though I’d like to keep this lightweight to provide the minimum overhead for other people wanting to use this mechanism for publishing their data). Here’s the function:

function proxy($type, $limit = 10) {
  global $store;
  $docUri = $_SERVER['REQUEST_URI'];

  // URL for the RDF
  if ($_SERVER['PATH_INFO']) {
    // Request for a specific thing
    $dir = dirname($_SERVER['SCRIPT_NAME']);
    $path = substr($docUri, strlen($dir));
    $idUri = "$dir/id$path";
    if (exists($idUri)) {
      $domain = $_SERVER['HTTP_HOST'];
      $id = "http://$domain$idUri";
      $params = array('about' => $id, 'output' => 'rdf');
      $query = http_build_query($params);
      $rdfURL = "http://api.talis.com/stores/$store/meta?$query";
    } else {
      error();
      return;
    }
  } else {
    // Request for a list of $limit individuals of type $type
    $start = (int)$_GET['start'];
    $sparql = "CONSTRUCT { ?thing a <$type> . ?thing <http://www.w3.org/2000/01/rdf-schema#label> ?label . } WHERE { ?thing a <$type> . OPTIONAL { ?thing <http://www.w3.org/2000/01/rdf-schema#label> ?label . }} ORDER BY ?thing LIMIT $limit OFFSET $start";
    $params = array('query' => $sparql, 'output' => 'rdf');
    $query = http_build_query($params);
    $rdfURL = "http://api.talis.com/stores/$store/services/sparql?$query";
  }

  // URL for the transformation
  $params = array('xml-uri' => $rdfURL, 
    'xsl-uri' => "http://api.talis.com/stores/$store/items/tidyRDF.xsl");
  $query = http_build_query($params);
  $txURL = "http://api.talis.com/tx?$query";

  $resource = fopen($txURL, 'rb');
  header("Content-Type: application/rdf+xml", true);
  header("Content-Location: $docUri.rdf", true);
  fpassthru($resource);
  return;
}

Now I need to make sure that requests to http://www.jenitennison.com/data/london-borough calls this function with the correct type and limit. To do this I create a london-borough.php that looks like:

<?php
  include "utils.php";
  proxy('http://www.jenitennison.com/ontology/data#LondonBorough', 50);
?>

and adjust .htaccess to redirect to this PHP script:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d

  RewriteRule ^id/(.+)$ id.php [L]
  RewriteRule ^london-borough(/.+)? london-borough.php$1 [L,QSA]
</IfModule>

Note that in the RewriteRule for URIs starting with london-borough, I’ve included QSA in the options, which means that query string parameters (such as the start) parameter will enable paging through the results. For example, if I were only reporting 10 London Boroughs at a time, I could use

http://www.jenitennison.com/data/london-borough

to get the first ten London Boroughs and

http://www.jenitennison.com/data/london-borough?start=10

to get the next ten.

When you request http://www.jenitennison.com/data/london-borough what you get back is the neatened RDF for the London Boroughs, which looks like:

<rdf:RDF xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
         xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <LondonBorough xmlns="http://www.jenitennison.com/ontology/data#"
                  rdf:about="http://www.jenitennison.com/data/id/london-borough/barking-and-dagenham">
    <rdfs:label>Barking &amp; Dagenham</rdfs:label>
  </LondonBorough>
  <LondonBorough xmlns="http://www.jenitennison.com/ontology/data#"
                  rdf:about="http://www.jenitennison.com/data/id/london-borough/barnet">
    <rdfs:label>Barnet</rdfs:label>
  </LondonBorough>
  ...
</rdf:RDF>

Now this is OK, but I think the best way of serving lists of things is through Atom or RSS, with RSS 1.0 fitting better with the RDF world because it is RDF. Both formats provide mechanisms to give metadata about the list, including links to the next set of information, to enable pagination through the list. So what I’d like to do is provide a mechanism for serving back different formats for this information. And not only for the lists, but for the data about the London Boroughs themselves: Talis supports serving data in Turtle and RDF/JSON as well as RDF/XML, so providing those formats should be cheap. This is something to come back to later.

Publishing Linked Data on the Talis Platform, Part 2

Jeni — Fri, 17 Jul 2009 20:49:40 +0000

In my last post, I showed how to add data to a Talis store. In this post, I’m going to show how you can use the Talis Platform as a back end for a Linked Data view on the RDF you added to it.

As you’ll see, the great thing about this method is that it only takes a couple of PHP files and an .htaccess file on a server. Assuming that you’ve got a web server that supports PHP, it’s an approach you can use without installing anything. The code I’ve written is pretty generic and should be widely applicable; feel free to reuse and adapt it.

One of the principles of Linked Data is that if you make a GET request to a URI that’s used as an identifier within an RDF triple, you’ll get back some useful information about that resource. I’ve created URIs like http://www.jenitennison.com/data/id/london-borough/barnet and added triples to Talis about those resources, but I haven’t yet put anything in place such that actually requesting http://www.jenitennison.com/data/id/london-borough/barnet will provide a useful response. So how do I do that?

Well, it’s easy enough with a bit of PHP to do the forwarding. (By the way, this is the first bit of PHP I’ve ever done, so feel free to point out all the glaring problems with it; I’d love to learn.)

Now, the URI http://www.jenitennison.com/data/id/london-borough/barnet is a URI that I’ve made up for a London Borough, and obviously when you request that URI you’re not actually going to get the London Borough delivered to you through your computer screen. Instead, based on Cool URIs for the Semantic Web, I want to either respond with a 303 See Other redirection to a document resource describing the borough, or a 404 Not Found to say that it doesn’t exist.

Note that I don’t just want to blindly respond with a 303 See Other. If someone requests http://www.jenitennison.com/data/id/london-borough/rubbish I want to tell them that the London Borough of ‘Rubbish’ doesn’t exist. If I redirected them to a document URI which then 404’ed, it would mean the London Borough of ‘Rubbish’ exists, but we have no information about it. So I can’t use a simple URL rewrite; I have to check for its presence first.

Existence Tests

The first task, then, is to test whether the resource exists. To do that, I can execute an ASK request on the SPARQL endpoint that Talis provides for the store. The ASK request simply looks like:

ASK { <http://www.jenitennison.com/data/id/london-borough/barnet> ?p ?v . }

which asks if there are any triples at all that involve that URI. I request the JSON response using the output=json parameter. The JSON looks like:

{"head":{},"boolean":true}

if the store holds any triples about the borough and:

{"head":{},"boolean":false}

if it doesn’t. The URI for the request looks like:

http://api.talis.com/stores/rdfquery-dev1/services/sparql?query=ASK+%7B+%3Chttp%3A%2F%2Fwww.jenitennison.com%2Fdata%2Fid%2Flondon-borough%2Fbarnet%3E+%3Fp+%3Fv+.+%7D

which looks pretty horrendous when you write it out but is easy enough to construct with PHP. Here’s the exists() function which does the test based on the server host name used in the request and a path that’s passed in.

$store = 'rdfquery-dev1';

function exists($idUri) {
  global $store;
  $host = $_SERVER['HTTP_HOST'];
  $id = "http://$host$idUri";
  $sparql = "ASK { <$id> ?p ?v . }";
  $params = array('query' => $sparql, 'output' => 'json');
  $query = http_build_query($params);
  $request = "http://api.talis.com/stores/$store/services/sparql?$query";
  $resource = file_get_contents($request, 'rb');
  $result = strstr(strstr($resource, "\"boolean\":"), ":");
  return !strstr($result, "false");
}

Handling Identifier URIs

With that function in utils.php, it’s pretty easy to create a id.php that does the redirection that I need to do. For my purposes, I’m using /id/ in all the URIs that identify abstract resources, and removing it for the document URIs that describe them. So the URI for the abstract resource http://www.jenitennison.com/data/id/london-borough/barnet will redirect to the document resource http://www.jenitennison.com/data/london-borough/barnet. Here’s id.php:

<?php
  include "utils.php";
  $idUri = $_SERVER['REQUEST_URI'];
  if (exists($idUri)) {
    $docUri = str_replace('/id/', '/', $idUri);
    header("Location: $docUri", true, 303);
  } else {
    error(404);
  }
?>

The error() function is also in utils.php and looks like:

  function error() {
    header("HTTP/1.1 404 Not Found");
    echo <<<EOF
<html>
  <head>
    <title>404 Not Found</title>
  </head>
  <body>
    <h1>404 Not Found</h1>
    <p>No such resource</p>
  </body>
</html>
EOF;
  }

I have id.php which will check for the presence of triples about the requested resource, and respond with either a 404 Not Found or a 303 See Other. Now I need to invoke id.php whenever someone requests an identifier URI like http://www.jenitennison.com/data/id/london-borough/barnet. To do this, I put id.php in the /data directory within my webserver’s documents and added a .htaccess file that looks like:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d

  RewriteRule ^id/([^/]+)/(.+)$  id.php [L]
</IfModule>

This says that any requests that aren’t for existing files or directories and that start with id should be redirected to id.php. Since id.php picks up on the original request URI, I don’t need to pass anything extra into it by way of query parameters and what have you.

To make this .htaccess file work, you have to have mod_rewrite enabled and have AllowOverride include FileInfo (in http.conf) . My ISP allows this, but the Apache installation on my Mac doesn’t, and Apache generally doesn’t out of the box, so you may need to do a bit of fiddling with configuration files.

Now, requesting http://www.jenitennison.com/data/id/london-borough/barnet redirects me with a 303 See Other to http://www.jenitennison.com/data/london-borough/barnet, while requesting http://www.jenitennison.com/data/id/london-borough/rubbish gives me a 404 Not Found response.

Handling Document URIs

The next stage is supporting the document URIs like http://www.jenitennison.com/data/london-borough/barnet. For them, I need to actually get the data about the resource out of the Talis Platform. Fortunately, there’s a really easy way of doing that using a simple request on the metabox like:

http://api.talis.com/stores/rdfquery-dev1/meta?about=http%3A%2F%2Fwww.jenitennison.com%2Fdata%2Fid%2Flondon-borough%2Fbarnet&output=rdf

In other words, you pass the URI of the resource that you’re interested in as the value of the about parameter to the metabox store URI of http://api.talis.com/stores/{store}/meta?about={resource}&output=rdf. This gives you back some RDF/XML. For the particular request above, the RDF/XML looks like:

<rdf:RDF
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:j.0="http://www.jenitennison.com/ontology/data#"
    xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" > 
  <rdf:Description rdf:about="http://www.jenitennison.com/data/id/london-borough/barnet">
    <rdfs:label>Barnet</rdfs:label>
    <rdf:type rdf:resource="http://www.jenitennison.com/ontology/data#LondonBorough"/>
    <j.0:maleLifeExpectancy rdf:datatype="http://www.w3.org/2001/XMLSchema#decimal">79.5</j.0:maleLifeExpectancy>
    <j.0:femaleLifeExpectancy rdf:datatype="http://www.w3.org/2001/XMLSchema#decimal">83.6</j.0:femaleLifeExpectancy>
  </rdf:Description>
</rdf:RDF>

Now I don’t know about you, but this RDF/XML really makes me cringe. It’s very obviously RDF, and it has a horrible j.0 prefix that no one would ever actually write if they were creating it in an editor. Readability matters, even for data that’s aimed at computers. If I’m going to use RDF/XML, I’d really like it to be sensible XML as well as being RDF (and Leigh Dodds has given some good guidelines about how to do it).

But of course since it’s XML it’s amendable to a spot of transformation. So it’s not hard to transform the RDF/XML above into:

<LondonBorough xmlns="http://www.jenitennison.com/ontology/data#"
               xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
               xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
               rdf:about="http://www.jenitennison.com/data/id/london-borough/sutton">
   <rdfs:label>Sutton</rdfs:label>
   <femaleLifeExpectancy rdf:datatype="http://www.w3.org/2001/XMLSchema#decimal">82.6</femaleLifeExpectancy>
   <maleLifeExpectancy rdf:datatype="http://www.w3.org/2001/XMLSchema#decimal">78.7</maleLifeExpectancy>
</LondonBorough>

which is a little more acceptable. Talis offers a transformation service at:

http://api.talis.com/tx

It only supports XSLT 1.0. (There’s also the W3C XSLT 2.0 Service based on Saxon, but I get the impression they don’t like people to use it in anger.)

Anyway, each Talis store contains a contentbox as well as a metabox. The metabox holds the RDF/XML, and the contentbox can hold anything you like. I can put the XSLT stylesheet (tidyRDF.xsl) into my store’s contentbox using the command:

curl -X PUT -H "Content-type: application/xslt+xml" --digest -u username:password --data-binary @tidyRDF.xsl 
  http://api.talis.com/stores/rdfquery-dev1/items/tidyRDF.xsl

which then makes it accessible at:

http://api.talis.com/stores/rdfquery-dev1/items/tidyRDF.xsl

(I could also use my own server of course, but if Talis are offering free hosting, why not?…)

And that means that I can get the RDF/XML associated with http://www.jenitennison.com/data/london-borough/barnet and transform it into some decent XML using a horrendous double-escaped URI that I’m not going to replicate here. The proxy.php script does this all nicely behind the scenes:

<?php
  include "utils.php";
  $docUri = $_SERVER['REQUEST_URI'];
  $dir = dirname($_SERVER['SCRIPT_NAME']);
  $path = substr($docUri, strlen($dir));
  $idUri = "$dir/id$path";
  if (exists($idUri)) {
    $domain = $_SERVER['HTTP_HOST'];

    // URL for the RDF
    $id = "http://$domain$idUri";
    $params = array('about' => $id, 'output' => 'rdf');
    $query = http_build_query($params);
    $rdfURL = "http://api.talis.com/stores/$store/meta?$query";

    // URL for the transformation
    $params = array('xml-uri' => $rdfURL, 
      'xsl-uri' => "http://api.talis.com/stores/$store/items/tidyRDF.xsl");
    $query = http_build_query($params);
    $txURL = "http://api.talis.com/tx?$query";

    $resource = fopen($txURL, 'rb');
    header("Content-Type: application/rdf+xml");
    header("Content-Location: $docUri.rdf");
    fpassthru($resource);
    return;
  } else {
    error(404);
  }
?>

With proxy.php in the /data directory on my server, I need a slight tweak to the .htaccess to make sure that all non-id requests go to it:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d

  RewriteRule ^id/(.+)$  id.php [L]

  RewriteCond %{REQUEST_URI} !\.php
  RewriteRule ^(.+)$ proxy.php [L]
</IfModule>

And Bob, as they say, is your uncle.

Requests to identifier URIs redirect to document URIs. Requests to document URIs return relevant RDF/XML for the resource. Have a look at http://www.jenitennison.com/data/id/london-borough/barnet for example.

Updated: fixed the link in the final paragraph so it actually pointed to the right location. Duh.

Publishing Linked Data on the Talis Platform

Jeni — Wed, 15 Jul 2009 19:53:10 +0000

I was at OpenTech a couple of weekends ago, and heard a lot of great talks. I particularly enjoyed the one by Simon Willison in which he talked about the Guardian Data Blog. Essentially, the data collected by the journalists at the Guardian, that form the basis of their pretty visualisations and so forth, gets published in Google Spreadsheets.

Looking through the data blog today, I saw that the Greater London Authority have similarly released their data using Google Spreadsheets.

Now Google Spreadsheets are just fine — they’re easy for end-users to use and it’s not hard for data nerds to extract data from them. They have real advantages for publishing because they are quick and easy to set up.

But take a look through the page listing the tables of data and you can see that many of them are about the same areas. The Guardian Data Blog have actually created a new spreadsheet that pulls together that information. Even with the aggregated data, in Google Spreadsheets there’s no way to address the data held in each table about Sutton (say).

Now, a few months ago, Talis announced the Talis Connected Commons, which enables anyone to publish public domain data using the Talis Platform for free. It turns out that it’s really easy to publish addressable data using the Talis Platform as a host.

The first stage is to get hold of the data and convert it into some RDF/XML that can be loaded into Talis. You can get hold of a CSV version of a Google Spreadsheet by exporting it as CSV. Converting the CSV into RDF/XML can be done in any number of ways. Given that this is a small dataset and it has commas in some of the fields I’ve used some simple reusable XSLT rather than awk. Here’s csv.xsl for parsing the CSV:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
      xmlns:xs="http://www.w3.org/2001/XMLSchema"
      xmlns:csv="http://www.jenitennison.com/xslt/csv"
      xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
      xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
      exclude-result-prefixes="xs csv"
      version="2.0">

<xsl:param name="filename" as="xs:string" required="yes" />

<xsl:variable name="csv" as="xs:string" select="unparsed-text($filename)" />
<xsl:variable name="lines" as="xs:string+" select="tokenize($csv, '\n')[normalize-space(.) != '']" />
<xsl:variable name="fields" as="xs:string+" select="csv:values($lines[1])" />
<xsl:variable name="data" as="xs:string+" select="$lines[position() > 1]" />

<xsl:template match="/" name="main">
  <rdf:RDF>
    <xsl:for-each select="$lines[position() > 1]">
      <xsl:call-template name="csv:line">
        <xsl:with-param name="values" select="csv:values(.)" />
      </xsl:call-template>
    </xsl:for-each>
  </rdf:RDF>
</xsl:template>

<xsl:template name="csv:line">
  <xsl:param name="values" as="xs:string+" required="yes" />
</xsl:template>

<xsl:function name="csv:values" as="xs:string+">
  <xsl:param name="line" as="xs:string" />
  <xsl:analyze-string select="$line" regex="(&quot;([^&quot;]+)&quot;|([^,]+))?,">
    <xsl:matching-substring>
      <xsl:choose>
        <xsl:when test="regex-group(2) != ''">
          <xsl:sequence select="regex-group(2)" />
        </xsl:when>
        <xsl:otherwise>
          <xsl:sequence select="regex-group(3)" />
        </xsl:otherwise>
      </xsl:choose>
    </xsl:matching-substring>
    <xsl:non-matching-substring>
      <xsl:variable name="value" as="xs:string" select="normalize-space(.)" />
      <xsl:choose>
        <xsl:when test="starts-with($value, '&quot;') and ends-with($value, '&quot;')">
          <xsl:sequence select="substring($value, 2, string-length($value) - 2)" />
        </xsl:when>
        <xsl:otherwise>
          <xsl:sequence select="$value" />
        </xsl:otherwise>
      </xsl:choose>
    </xsl:non-matching-substring>
  </xsl:analyze-string>
</xsl:function>

<xsl:function name="csv:field" as="xs:string">
  <xsl:param name="values" as="xs:string+" />
  <xsl:param name="field" as="xs:string" />
  <xsl:sequence select="$values[index-of($fields, $field)]" />
</xsl:function>

</xsl:stylesheet>

and here’s the stylesheet that I’ve used to create some basic RDF data about the boroughs:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
      xmlns:xs="http://www.w3.org/2001/XMLSchema"
      xmlns:csv="http://www.jenitennison.com/xslt/csv"
      exclude-result-prefixes="xs csv"
      xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
      xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
      xmlns:data="http://www.jenitennison.com/ontology/data#"
      version="2.0">

<xsl:import href="../csv.xsl" />

<xsl:param name="filename" select="resolve-uri('boroughs.txt', static-base-uri())" />

<xsl:template match="/" name="main">
  <rdf:RDF>
    <xsl:for-each select="$data">
      <xsl:variable name="values" as="xs:string+" select="csv:values(.)" />
      <xsl:variable name="borough" as="xs:string" select="csv:field($values, 'BOROUGH')" />
      <xsl:variable name="maleLifeExpectancy" as="xs:string" 
        select="csv:field($values, '2005-07 Life expectancy at birth (years), Males')" />
      <xsl:variable name="femaleLifeExpectancy" as="xs:string" 
        select="csv:field($values, 'Life expectancy at birth (years) Females')" />
      <data:LondonBorough rdf:about="http://www.jenitennison.com/data/id/london-borough/{translate(lower-case(replace($borough, '&amp;', 'and')), ' ', '-')}">
        <rdfs:label><xsl:value-of select="$borough" /></rdfs:label>
        <xsl:if test="$maleLifeExpectancy != ''">
          <data:maleLifeExpectancy rdf:datatype="http://www.w3.org/2001/XMLSchema#integer">
            <xsl:value-of select="$maleLifeExpectancy" />
          </data:maleLifeExpectancy>
        </xsl:if>
        <xsl:if test="$femaleLifeExpectancy != ''">
          <data:femaleLifeExpectancy rdf:datatype="http://www.w3.org/2001/XMLSchema#integer">
            <xsl:value-of select="$femaleLifeExpectancy" />
          </data:femaleLifeExpectancy>
        </xsl:if>
      </data:LondonBorough>
    </xsl:for-each>
  </rdf:RDF>
</xsl:template>

</xsl:stylesheet>

The result is some RDF/XML for each borough which looks like:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:data="http://www.jenitennison.com/ontology/data#">
  <data:LondonBorough rdf:about="http://www.jenitennison.com/data/id/london-borough/city-of-london">
    <rdfs:label>City of London</rdfs:label>
  </data:LondonBorough>
  <data:LondonBorough
    rdf:about="http://www.jenitennison.com/data/id/london-borough/barking-and-dagenham">
    <rdfs:label>Barking &amp; Dagenham</rdfs:label>
    <data:maleLifeExpectancy rdf:datatype="http://www.w3.org/2001/XMLSchema#integer">76.3</data:maleLifeExpectancy>
    <data:femaleLifeExpectancy rdf:datatype="http://www.w3.org/2001/XMLSchema#integer>80.3</data:femaleLifeExpectancy>
  </data:LondonBorough>
  <data:LondonBorough rdf:about="http://www.jenitennison.com/data/id/london-borough/barnet">
    <rdfs:label>Barnet</rdfs:label>
    <data:maleLifeExpectancy rdf:datatype="http://www.w3.org/2001/XMLSchema#integer">79.5</data:maleLifeExpectancy>
    <data:femaleLifeExpectancy rdf:datatype="http://www.w3.org/2001/XMLSchema#integer">83.6</data:femaleLifeExpectancy>
  </data:LondonBorough>
  ...
</rdf:RDF>

To load this into a Talis store, you have to first have one set up, which I do (primarily for experimenting with rdfQuery). You can get one by filling in the form on the Talis website. Loading data into Talis just means a POST request like this:

> curl -H "Content-type: application/rdf+xml" --digest -u username:password 
  --data-binary @LondonBoroughs/boroughs.rdf http://api.talis.com/stores/rdfquery-dev1/meta

Once that’s done, you can check whether the data has been successfully loaded or not by visiting the URI of the store in an ordinary browser, in this case:

http://api.talis.com/stores/rdfquery-dev1/meta

You can enter the URI of a resource in the form, for example http://www.jenitennison.com/data/id/london-borough/barnet and indicate the format for the response. Requesting Turtle gets you something like:

<http://www.jenitennison.com/data/id/london-borough/barnet>
  <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>
    <http://www.jenitennison.com/ontology/data#LondonBorough> ;
  <http://www.w3.org/2000/01/rdf-schema#label>
    "Barnet" ;
  <http://www.jenitennison.com/ontology/data#femaleLifeExpectancy>
    "83.6"^^<http://www.w3.org/2001/XMLSchema#integer> ;
  <http://www.jenitennison.com/ontology/data#maleLifeExpectancy>
    "79.5"^^<http://www.w3.org/2001/XMLSchema#integer> .

All is well and good, but this isn’t really linked data. For it to be linked data, this RDF needs to be accessible at the URI http://www.jenitennison.com/data/id/london-borough/barnet. So how do we do that? Tune in next time to find out.