psi

Three challenges for alpha.gov.uk

Jeni — Fri, 20 May 2011 19:45:40 +0000

The new alpha.gov.uk website was launched recently, as a prototype for the “single Government website” described in Martha Lane Fox’s report Directgov 2010 and Beyond: Revolution Not Evolution. Apparently the real deal could go live “in about a year”.

The site is lovely, a far cry from the standard government fare. But this isn’t exactly surprising: it’s been developed using modern technologies by a top team with a set of design rules far removed from those usually applied to government websites, a budget that’s not exactly tight and using an Agile methodology. These factors mark it out from the majority (though not all) government websites. And this is part of the point, to illustrate the gap between what we have and what a revolution could bring.

There are three challenges where I am and have been particularly interested to see the alpha.gov.uk approach. These are in balancing:

simplicity and complexity
centralisation and distribution
end-user and data re-user

It is not currently clear to me whether alpha.gov.uk has decided an approach on any of these — whether the way the site works currently is the way that they have decided it should work — or whether these are areas that are still up in the air at the moment. I’m hoping it’s the latter.

Disclaimer: What I write here is largely coloured by having worked on legislation.gov.uk for the last few years, both in terms of the difficulties that we face and the question at the back of my mind of how legislation.gov.uk fits in this brave new world.

Simplicity vs Complexity

alpha.gov.uk does a great job at providing the simple message that is needed by the majority of people. The first challenge, I think, lies in also addressing the complex situations that are faced by the minority. For example, alpha.gov.uk provides information about child car seats but what if my child is disabled? What if I want to check whether it’s OK to go without in a taxi? Similarly, there’s a tool for calculating holiday entitlement, but what if I work part time and my days include Mondays; how do bank holidays fit in?

Does a single Government website need to address all these complexities? I can see arguments both ways. On the one hand, handling minority cases can mean making things unnecessarily complex for everyone — this is the trap that DirectGov falls into too much of the time. On the other hand, if people can’t find this information from the single Government website, what other authoritative source is there? Can we even assume that non-authoritative third parties will fill the gaps? Will people trust the information if it isn’t from a government source?

My feeling is that, unlike conventional websites, a single Government website has an additional responsibility to provide complex content, and that this includes helping people find information as well as perform tasks. For example, if I’m writing a holiday travel guide and want to include a page describing what to do when you’ve lost your passport, including contact details for common countries, I want a description and a list, not a tool.

One of the alpha.gov.uk design rules was “optimise for the common case”, and the emphasis on clear content and straightforward tools is admirable. But optimisation for a common situation shouldn’t mean completely ignoring the uncommon one. The challenge — which I think is primarily a design challenge and handled pretty nicely in the Guides — is to make a website that contains complex information appear simple.

Centralisation vs Distribution

In her report, Martha Lane-Fox laid out a number of advantages to a single government website, including providing consistency in user experience, ensuring that there is no “wrong part” of the government web estate from which to start your search for government information, and saving money by building on shared platforms. These imply a high level of centralisation.

Against that is the fact that there is no single owner of government content. Departments have the best understanding of their policies. HMRC collects taxes. DVLA licenses drivers. The Identity and Passport Service issues passports. The greater the distance between the website and the people who understand the area, the more likely the content and tools are to be simply wrong or out of date and ultimately worthless. To completely centralise all government websites runs counter to the way the web works and the advantages that it gives in distributing control over content. If Google is your front page, then what does it matter how many domains the content is distributed across?

So this is what I see as the second challenge for a single government website. It is a question of governance and of technology. How does a single Government website enable the people that own information to keep control of it while providing a consistent user experience?

While there are very likely large swathes of government content that could be perfectly well served by a Wordpress or Drupal installation with some custom templates, other content really couldn’t fit into that kind of content management system (yes, I’m thinking of legislation), and need custom solutions which are best owned and updated by those that understand the content. What does the “single Government website” mean for these sites? Do they continue running their own sites with a common set of styles and links? Do they only run a service with an API that is skinned centrally?

In my opinion, alpha.gov.uk should be trying out the different ways in which different content could be brought together from distributed sources, learning lessons from experience at the BBC and beyond, to find approaches that work within government.

End User vs Data Reuser

The third challenge is particularly important for those of us who care about freeing government data. We don’t want information locked within government’s web pages, but available for us to re-present, mash together and build into our own products. The balancing act here is between the end-user who really doesn’t care at all about the data underlying the pages they see, and the citizen developer who wants to reuse that information.

There is some evidence of the idea of providing access to underlying data in some of the alpha.gov.uk Guides, such as the An employee’s guide to Redundancy, which include a “Syndication API” box with (not yet working) links to JSON, XML and Atom versions of the content. But there’s no provision of a feed on the search results page, no RDFa or microdata markup, or separate data versions, on pages that contain data such as What is the minimum wage?, and the tools such as that used to Calculate holiday pay are provided as client-side scripts rather than server-side APIs which could be called from other applications.

As we’ve found with legislation.gov.uk, building websites based on data and API access to that data is very possible, but doing so can change your approach to both site design and technology use. I’d love to see alpha.gov.uk being built around making reuse easy rather than retrofitting it as a secondary concern.

Other Challenges

There are undoubtedly other challenging balancing acts for alpha.gov.uk.

Steph talks about lack of community; I view this as something to balance against the need to provide authoritative set of information, which is hard to do when you let just anyone participate in a site.

There’s also the balance between the need for small-g government to provide a persistent and consistent website while big-g Government wants to create short-lived special-case sites to tout particular policies, such as Red Tape Challenge or Your Freedom. How do these sites fit within the single Government website?

Navigating a way through these challenges will require making hard decisions, reaching difficult compromises and finding imaginative third ways. I for one am very interested to see the directions alpha.gov.uk chooses to take.

Government Should Do its Own Data Homework

Jeni — Sun, 26 Sep 2010 21:41:30 +0000

I’ve been reflecting a little since OpenTech on the relationship between the developer community and government.

Let me set out my perspective first. My goal is to help ensure that the public sector publishes reusable data in the long term.

To do that, data publication needs to be sustainable. It needs to be embedded within the day-to-day activity of the public sector, something that seems as natural as the generation of PDF reports seems today. It also needs to be useful. It needs to be easy for anyone to understand and reuse the data, with minimal effort. It cannot be the case, long term, that you need to be an expert hacker to reuse government data.

To get there, we need to work towards a virtuous cycle in which the public sector is rewarded for publishing useful data well. The reward may come from financial savings, from increasing data quality, from better delivery of its remit, or simply from kudos. It doesn’t matter how, but there needs to be some reward, or it just won’t happen.

Over the last few years, government has had to be persuaded that it’s a good idea to release their data at all. The message from the developer community has been “give us your data and we’ll show you what we can do with it!” Through hack days and various similar activities, developers have excited, wowed and dazzled officials and politicians, opening their eyes to what could be done. Through sustained argument and political pressure, developers have set out the economic and moral case that releasing data not only could, but should happen.

They have been incredibly successful. We have data.gov.uk, open data from Ordnance Survey, strong commitments to open data within the Coalition Agreement, and the Public Sector Transparency Board who are now applying that pressure, with authority, at the heart of government.

My perception is that the argument that government should open up its data has basically been won. The questions within the public sector are now about how, not whether. And as a result, in this changed environment, I’m growing slightly uneasy about the core developer message of “give us your data and we’ll show you what we can do with it!”

There are two things about that message that concern me. First, it implies government is doing it all wrong. Second, it implies that government doesn’t need to do any better, because the developer community can take up all the slack and fill in all the gaps. It’s like getting fed up with a child struggling with their homework, and saying “oh, just give it here and I’ll do it!” It’s a narrative that simultaneously undermines the best efforts of those within government and removes from them the motivation and opportunity to learn to do better.

Of course there is a tricky balance here. We don’t want to let up pressure on the government to release important information. We don’t want government to feel that they have to get their data perfect before releasing it. And we can’t always wait for government, which can be slow-moving as an organisation, to provide everything we need right now.

However, there are certain things that only the owners of data — those within the public sector — can do. People who own data understand it so much better than third parties: what codes mean, what values are used to indicate missing data, what gets included and what gets left out, which columns aren’t really used any more, which interpretations are safe and which are meaningless. Data owners can be trusted in a way that no one outside could be; when data publication becomes a sustainable part of their activity, they are much better placed to provide a steady, reliable, flow of data than a third-party API that could disappear or get out of date whenever the volunteer behind it moves on to something new.

People in government must be given the responsibility to publish their data well. And there are three core ways in which I think developers could help them.

First, while there are many more technically savvy people within government than is sometimes made out, the average civil servant lacks both know-how and tooling. I think developers could help a huge amount here. What about hack days where developers sit side by side with civil servants to help them clean and publish their data? What about engaging with the owners of a particular data set to help them to publish it in a way that was reusable and sustainable? What about writing services, accessible through the locked-down IT systems that civil servants have to use, that enabled them to convert their data into multiple formats, and to link up the ways they refer to things with the way other people do?

Second, while government needs to be responsible for publishing its data, it can’t be responsible for building everything that end-users need based on that data. Developers have the facility to create applications that bring together data from diverse parts of the public sector, and combine it with data from outside. This has always been a feature of hack days, of course; all I’m arguing for is a focus on applications that the public sector shouldn’t be doing itself.

Third, we need to build the virtuous cycle that I talked about above. Government needs to hear about what works for developers, as well as what doesn’t. What data releases have been helpful and why? Who are the stars? Who should be rewarded and emulated? We need ways of feeding back in a constructive way to public sector workers who are trying their best with the resources they have — often extensive subject-matter expertise but little time, locked-down technology and contracting finances.

The vitality and engagement of the developer community has played a massively important role in the open government data initiative within the UK, and I’m sure it will continue to do so. We are incredibly lucky, here, to have a collection of talented and motivated developers who volunteer their time to work with government data. My hope is simply that the relationship between government and developers can grow into one that is more encouraging and supportive, that understands the constraints and concerns of those within government, and that provides practical help to overcome them.

On Standards

Jeni — Sun, 19 Sep 2010 15:54:51 +0000

I’m beginning to think that ‘to recommend’ is an irregular verb like those that appeared every so often in Yes, Minister:

Bernard: It’s one of those irregular verbs, isn’t it: I have an independent mind; you are an eccentric; he is round the twist.

Something like: I recommend, you tell people what to do, he engages in premature standardisation.

Our own recommendations are much more reasonable than those made by other people. We understand the requirements, whereas they haven’t talked to anyone. We are issuing them as guidance and are open to feedback, whereas they are ramming them down people’s throats.

Of course without guidance, recommendations and standards of some description, it becomes near impossible to do anything useful. Take a look at the wide variety of information released by different councils to meet their commitment to publish spending data. Many use different formats but even amongst those that use Excel or CSV, the column names are different. Look closer and you see that they actually report at different levels of granularity as well. Some report each transaction, some each invoice item, some the ways these items are assigned to different cost centres. Some stick to the £500 limit, some report everything. Some include VAT in the amounts they quote, some don’t. Some provide the dates of each transaction, some just the period that it occurred in. If you are clever and committed, you can find some wood in the trees but it’s hard work.

This variety is not due to pigheadedness or stupidity on the part of the councils. It’s down to the very different technical and political constraints and approaches, and the fact that there was little guidance at all, up until this week, about what was expected of them.

The point I’m making is that people in different circumstances will naturally do things differently; common practice does not appear overnight by magic.

Should councils have held off publishing their data until there was some kind of guidance in place? Absolutely 100% No! It is far better to have the data in some form than to not have it at all, and it’s only by making real data available that they and we get to start informed discussions about what kind of guidance is necessary.

Should they be working towards publishing something better? Hell Yeah! Data is not really open if the people who consume it have to put in hours or days of effort to understand it, map it, merge it, to be able to do something useful with it.

What that ‘something better’ looks like, I really don’t know. My prediction is that councils will converge gradually, over time, into a handful of different approaches (rather than the basketful that we have now). Some will converge by choosing to use particular publishers for their data. Others will converge because they want to take advantage of particular tools for analysing or visualising the data that they produce, which will require certain formats. Still others will converge through an interest in “doing what’s right”, based on guidance from groups and organisations that they trust.

From chaos will come order, eventually. But this is a process that is led by politics — negotiation, persuasion, socialisation and cultural change — not by technology. It’s only to be expected that there will be differences in approaches along the way, because we need to try, to learn, and we need for there to be choice, to evolve.

legislation.gov.uk: Credit Where it's Due

Jeni — Sat, 14 Aug 2010 11:18:38 +0000

I’m aware I’ve been quiet for the past few months. This isn’t because nothing interesting has been going on — rather the opposite. It’s been difficult to get a chance to sit down and write about the work I’ve been doing, when actually doing the work has been taking up so much time.

Most of my time has been spent on the new legislation.gov.uk website and its underlying API. There’s so much to say about this project that I hardly know where to start, so I’ll just try to do an overview and we can take it from there. Let me know what you’re interested in.

legislation.gov.uk is a government website built on the principles of transparency and open data, including ideas laid out in the Power of Information Taskforce Report. We have a lovely user interface which helps end-users find and understand legislation, but it’s layered over the top of an API that anyone is free to use to construct their own websites based on the same data.

In fact, we built the API first, and it’s been around (though not in a particularly stable state) for about a year. However, it turned out that building the user interface really helped in two ways. First, it helped the legislation experts who were looking at the documents to spot errors in a way that they unsurprisingly struggled to do when presented with raw XML. Second, it helped to identify things that the API needed to do to support a useful website, such as always providing links to the table of contents for an item of legislation or providing a search based on modification date.

Now, if you’ve been reading Sean McGrath’s blog you’ll know that as far as content goes, legislation is about as tough as you can get. For a start, Acts and Statutory Instruments are semi-structured documents, not tabular data. It’s not a simple matter of storing and extracting rows in a database: we need to be able to address portions of an item of legislation such as “Local Government Act 1988 (c. 9, SIF 81:2), Sch. 3 para. 13(1)(b)(2)” (this an actual citation! I am not making this up!).

The content itself is complex. For legislation.gov.uk, the main challenge is not to do with faithfully reconstructing page and line breaks (fortunately!) but how to represent complex, annotated, changes to legislation over time, and then how to present them. Much of this had already been done (in terms of technology) within the Statute Law and OPSI websites, although the data comes from a variety of sources over time, each with its own set of peculiarities to be navigated. The larger challenge here was to provide a mechanism of navigating through the content that made clear the distinctions between the various versions of legislation that people can look at and warning them about their status without overwhelming them with information.

We also have a lot of documents, some of which are very large. There are nearly 60,000 items of legislation on the site. The largest and most complex of them has hundreds of sections and about a hundred distinct versions. When you consider all the versions of all the possible fragments of all the items of legislation, you’re talking about 6.5 million distinct documents, each of which is available in HTML, XML, PDF and for which there is some RDF metadata.

On top of this, the content is constantly changing. New legislation is published every working day, first as PDFs, then as HTML (and XML), and then various associated documents the most important of which are Explanatory Notes, again first in PDF and then in HTML/XML form. Old legislation changes too; the legislation.gov.uk editorial team is constantly working through a backlog of changes to existing legislation brought about by new legislation. Simply hooking up the site to keep up to date with these changes has been an enormous challenge.

The content also changes because we intend to add features to the site over time. The site has already seen bug fixes and tweaks to address problems that we’ve encountered post-launch, and there are a number of new features in the pipeline to bring the site up to the level of completeness where it can fully replace the existing OPSI and Statute Law websites.

Then we needed something that was reasonably fast and robust in the face of moderately heavy traffic. Providing fast access to ever changing content, especially when the changes themselves are unpredictable, is an ongoing challenge.

All of this has only been possible by having an excellent team of experts and developers. One of the things that made this project quite different from the majority of government projects of this size was that it was much closer to Agile than Prince2: clients and providers working closely in the same team, chatting on daily calls, working side-by-side. From the developer perspective, it gave us direct access to the people who both had the expertise about the content and knew what they wanted. From the customer side, I hope and believe that it gave them as close involvement in the development of the site as they could want and a far deeper level of understanding about exactly how it works (and therefore what is easy and what is hard, and where compromises are best made) than they would have had otherwise.

So here are some credits. First, from TSO, where I work:

Carey Farrell ran the project, keeping track of the many and various bits and pieces that needed doing and finding the people to get them done; he has been the project’s backbone
Paul Appleby may have moved on to better things part way through, but made his mark early on in its design and architecture, and much much earlier in the design of the XML schema, and in many of the stylesheets that underlie the HTML and PDF views of this data
Lee Goodby put together the system infrastructure, arranging more machines and memory and disk space to satisfy our endless demands
Chunyu Cong performed many a thankless data wrangling task without complaint
Griff Chamberlain has worked doggedly on this project (with occasional pauses for beer) since he came aboard, among many many other things working on the generation of PDF (via XSL-FO) from the XML source and dealing with the difficulties of usable next/previous navigation
Tony Graham made the publication of Tables of Effects his own, as well as constantly improving our build processes
Gavin Mannings achieved quite remarkable things with a combination of HTML, CSS and Javascript. If you don’t believe me, take a look at the source underlying the histograms on the browse pages or the timelines for legislation content
Faiz Muhammad quickly got to grips with a whole set of complex and unfamiliar content and technologies, to create the UI from the API data
Paul Harvey furnished us with data, warned us of bear pits, and remained astonishingly uncomplaining of the changes we were putting him through
Marc Sturman brought all his expertise to bear in managing the publication of legislation from the SLD editorial system into the new website and pulled our fat from the fire both on deployment and in the creation of the larger PDFs
Vinod Sathyamoorthy worked on all aspects of the infrastructure: scaling out the environment, testing it, configuring it and so on, to make it into a site that more than a few people could access at a time
Rob Bullen brought a little more order to a kind of controlled chaos, in the way a good project manager should
Terry Blake had the clout and the clear-sighted vision to get things done, as well as (and perhaps secretly enjoying) getting his hands dirty on occasion

From Bunnyfoot:

Mark Pierce designed the look and feel of the site, having to get to grips with the complexities of legislative content as well as treading the fine line between making the site look modern yet authoritative, appealing whilst not detracting from the content
Rebecca Gill provided clear eyes, analysis and insight to help us understand how to improve the site for our users

And from The National Archives:

Clare Allison has devoted her life to ensuring that the content on the site is as accurate and meaningful as it can be, working with the astonishing complexities of the legislation content with an amazing depth of knowledge and expertise
Claire Lait has poured her soul into providing a meaningful and useful experience for the end users of the site with insight, intelligence and unparalleled openness and enthusiasm
Catherine Tabone has dealt with the traumas of the ups and downs of deployment with fortitude and good humour

And finally, none of this would have happened without John Sheridan having the ambition and the vision for how legislation should be published on the web, creating the environment that enabled this project to be done, setting a positive tone and providing support, encouragement and a gently guiding hand throughout the process.

This isn’t everyone who has been involved in the project: there are system administrators and testers and beta users and a whole cloud of other support particularly from MarkLogic, Orbeon and Akamai. But these are the people who let it consume their lives for at least a while. Every one of them was vitally important to the project, bringing their own expertise and skills and personality. I admire them all hugely. No project of this size is completely plain sailing, and I am convinced that we would be in a very different position today if the project hadn’t been built on mutual respect and trust. I’ve sketched some of the challenges that we faced. If it all looks easy, it’s only because this group of people did their jobs incredibly well. This is my public thanks to them for all their work.

Expressing Statistics with RDF

Jeni — Fri, 23 Oct 2009 22:07:50 +0000

Update: If you’re interested in expressing statistics in RDF, I’d encourage you to join the publishing statistical data group and take a look at the documentation for ‘SDMX-RDF’ described there.

One of the things that we’ve been discussing over on the UK Government Data Developers mailing list is how best to represent the vast quantities of statistical data that the government produces, in RDF. This is what we’ve come up with.

We’ll use SCOVO as our main vocabulary.
Dimensions (the things a statistic are about) should be instances of specialised classes such as ‘Hospital’ or ‘School’; these will often be SKOS concepts. We will try to reuse these as much as possible across datasets (see below).
We will create subproperties of scv:dimension that have appropriate names and different subclasses of scv:Dimensions as ranges. We will try to reuse these as much as possible across datasets (see below).
The scv:Items we use (representing individual statistics) should not be blank nodes (because giving them URIs allows us to attach other information to them); they will each have a scv:dataset property that points to the scv:Dataset they belong to (which will probably also be a void:Dataset).
Every scv:Item will also be the object of at least one triple that involves one of its dimensions; this will usually be the real-world thing that the statistic is associated with (eg the school or hospital).
Most statistics are provided for a particular time period; for these, we will define relationships from OWL-Time to placetime.com resources, but will also use appropriately datatyped literals where possible to make querying easier.

Here’s an example of what this looks like:

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix scv: <http://purl.org/NET/scovo#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix void: <http://rdfs.org/ns/void#> .
@prefix time: <http://www.w3.org/2006/time#> .
@prefix sdmx: <http://proxy.data.gov.uk/sdmx.org/def/sdmx/> .
@prefix pop: <http://statistics.data.gov.uk/def/population/> .
@prefix year: <http://statistics.data.gov.uk/def/census-year/> .

# The statistics themselves

<http://statistics.data.gov.uk/id/local-authority-district/00HE>
  rdfs:label "Cornwall" ;
  pop:totalPopulation <http://statistics.data.gov.uk/id/local-authority-district/00HE/population/total/year/2001> ;
  pop:ruralPopulation <http://statistics.data.gov.uk/id/local-authority-district/00HE/population/rural/year/2001> ;
  ... .

<http://statistics.data.gov.uk/id/local-authority-district/00HE/population/total/year/2001>
  a scv:Item ;
  rdf:value "499399"^^xsd:integer ;
  scv:dataset <http://statistics.data.gov.uk/doc/local-authority-district/*/population> ;
  sdmx:refArea <http://statistics.data.gov.uk/id/local-authority-district/00HE> ;
  pop:populationType pop:total ;
  sdmx:timePeriod <http://statistics.data.gov.uk/def/census-year/2001> .

<http://statistics.data.gov.uk/id/local-authority-district/00HE/population/rural/year/2001>
  a scv:Item ;
  rdf:value "127904"^^xsd:integer ;
  scv:dataset <http://statistics.data.gov.uk/doc/local-authority-district/*/population> ;
  sdmx:refArea <http://statistics.data.gov.uk/id/local-authority-district/00HE> ;
  pop:populationType pop:rural ;
  sdmx:timePeriod <http://statistics.data.gov.uk/def/census-year/2001> .

...

# Datasets

<http://statistics.data.gov.uk/doc/local-authority-district/*/population/*/year/2001>
  a scv:Dataset ;
  a void:Dataset ;
  dct:title "Populations of Local Authority Districts" ;
  ... .

# Common definitions for the dataset

pop:totalPopulation a rdf:Property ;
  rdfs:label "total population" ;
  rdfs:range scv:Item .
pop:ruralPopulation a rdf:Property ;
  rdfs:label "rural population" ;
  rdfs:range scv:Item .
...

pop:populationType rdfs:subPropertyOf scv:dimension ;
  rdfs:label "population type" ;
  rdfs:domain scv:Item ;
  rdfs:range pop:Population .

pop:Population a rdfs:Class ;
  rdfs:subClassOf skos:Concept ;
  rdfs:subClassOf scv:Dimension ;
  rdfs:label "population type" .

pop:populationScheme a skos:ConceptScheme ;
  skos:prefLabel "Population Types" ;
  pop:hasTopConcept pop:total .

pop:total a pop:Population ;
  skos:prefLabel "total population" ;
  skos:topConceptOf pop:populationScheme ;
  skos:narrower pop:rural ;
  ... .

pop:rural a pop:Population ;
  skos:prefLabel "rural population" ;
  skos:inScheme pop:populationScheme ;
  skos:broader pop:total ;
  ... .

year:Year a rdfs:Class ;
  rdfs:subClassOf time:Interval ;
  rdfs:subClassOf scv:Dimension .

<http://statistics.data.gov.uk/def/census-year/2001>
  rdfs:label "mid-2001" ;
  time:intervalDuring <http://www.placetime.com/interval/gregorian/2001-01-01T00:00:00Z/P1Y> .

<http://www.placetime.com/interval/gregorian/2001-01-01T00:00:00Z/P1Y>
  rdfs:label "2001" ;
  rdf:value "2001"^^xsd:gYear .

One source of sub-properties of scv:dimension (and subtypes of scv:Dimension) is SDMX (Statistical Data and Metadata eXchange). This provides standard ways of indicating things like the area and time that a statistic applies to. I’ve made an initial mapping into some RDFS properties and SKOS schemes as an indication of the kind of thing that would work here, but expect it to change.

We’re currently working on providing identifiers for the areas that statistics are likely to be about (such as local authority districts, MSOAs or wards). They are of the form:

http://statistics.data.gov.uk/id/{area-type}/{ONS-area-code}

and they tie into the newly released OS data. I hope we’ll have them available as Linked Data soon.

One issue that hasn’t been resolved is how to handle the huge amount of repetition that is inherent in this method of representing statistical data. For example, in the data above, all the scv:DataItems in the scv:Dataset http://statistics.data.gov.uk/doc/local-authority-district/*/population/*/year/2001 are from 2001. Rather than indicating the year of each individual scv:DataItem, it would be nice if we could have a property on the dataset that indicated that all the items in that dataset had the same value for a particular dimension. If this were called scv:itemDimension, for example, then we could do:

<http://statistics.data.gov.uk/doc/local-authority-district/*/population/*/year/2001>
  a scv:Dataset ;
  a void:Dataset ;
  dct:title "Populations of Local Authority Districts" ;
  sdmx:itemTimePeriod <http://statistics.data.gov.uk/def/census-year/2001> ;
  ... .

sdmx:itemTimePeriod rdfs:subPropertyOf scv:itemDimension ;
  rdfs:label "time period of items in the dataset" ;
  rdfs:domain scv:Dataset .

and the individual scv:Items would not have to have any sdmx:timePeriod properties explicitly. Perhaps this is something that the people beind SCOVO might consider, or we might create the property ourselves.

As far as I know, this pattern for representing statistics has yet to be used “in anger”, but I hope that we’ll have some illustrations soon which will help us assess whether it’s viable. Any comments and suggestions would, of course, be very welcome!

hmg.gov.uk/data and What We Can Do

Jeni — Sat, 03 Oct 2009 14:00:24 +0000

This week, the Cabinet Office went live with a preview version of hmg.gov.uk/data, available only to those who subscribe to the UK Government Data Developers Google Group. Harry Metcalfe has written a great review, or of course you can check it out yourselves.

Already, though, there are discussions starting on the mailing list about how the data is being made available, and I’m worried that these might distract us from getting things done.

There are a range of ways in which data can be made available on the web:

embedded in PDFs and Word documents (ie proprietary formats that make it nearly impossible to get hold of the data itself)
through a Web Services / RPC style interface whereby a custom API is available for each data set
as downloadable (possibly compressed) files in a machine-readable format such as CSV or XML
through a RESTful API whereby there is a URI for each resource and GETting that URI provides information about the resource, and there are also URIs for lists of resources
through a search interface, such as a SQL or SPARQL interface onto a database or triplestore, which may enable aggregations over multiple resources

All of us Open Data advocates agree that we need to encourage government away from the first two methods of making data “available” and towards the last three. But there’s a vast array of opinion within the developer community about which of the latter three are most useful, and precisely which technologies to use in each of those categories.

What’s the real answer? “All of them!”

We need the raw data because it enables us to double-check all the other interfaces which are provided to it. We need RESTful APIs. We need them to serve RDF and XML and JSON and CSV and all the other formats that people ask for. We need the data to be made available in SQL databases and NoSQL databases and triplestores; we need access to SQL queries and SPARQL queries and map/reduce processing. All of that, all of them, and more.

This is not a zero-sum game. Just because someone makes edubase available on the Talis platform through a SPARQL interface does not prevent someone else making it available on Amazon S3. The more methods of access there are, the more widely available and therefore useful the data is. The more things we try, the more lessons we learn, the better we get.

One thing is certain, though: the government cannot do all of this itself. They simply don’t have the resources or expertise. If we think something’s important, we can help by doing it. And we can help them, and each other, by sharing both the results of our work (so that others can build on it) and how we got them (so that others can follow the same patterns for other datasets). That, as far as I’m concerned, is what hmg.gov.uk/data is for.

Whatever our technology preferences, we can help each other by sharing our results whenever we:

clean up the data that the government have supplied
analyse the data (which is often in de-normalised formats)
find areas of commonality between different data sets
transform the data into other formats
build other types of APIs on top of the ones others have constructed

The hmg.gov.uk/data has a certain bias towards Linked Data, it’s true, and this should come as no surprise given its advisors. But whichever side of that particular argument we’re on, we’re shooting ourselves in the feet if we assert that this is an exclusive choice.

The Real Deal: data.gov.uk

Jeni — Sun, 26 Jul 2009 15:38:54 +0000

I’m sure that you’ve noticed that my recent posts have been somewhat obsessed with publishing and using public sector information. It’s because I’ve somehow been sucked into the work going on within the UK government, with Tim Berners-Lee and Nigel Shadbolt advising, to publish its data as linked data.

My recent blog posts about publishing data using Talis have actually been a front for much more complex work that I’ve been doing with a different data set.

As an early demonstration of how existing government data sets might be turned into linked data, a few weeks ago I was given a CSV file containing road traffic counts; the raw data that lies behind the traffic flow information available on the Department for Transport website. The data is really interesting and ripe for visualisations and analysis. For each hour of particular days each year, at particular points on many roads within the UK, the Department for Transport measures the number of bicycles, motorbikes, cars, vans, buses and HGVs of various types that roll past in each direction. The data contains information about:

the count of each of the various classes of traffic that pass the point in a particular direction on a particular hour of a particular day
the points at which these measurements were taken
the roads on which the points are situated
the areas in which the points are situated
the local authority that is in charge of these areas
the region that the area is in
the country that the region is in

The challenge was to turn the 386Mb CSV file into linked data. The result is up and available for you to look at; a good starting point is http://geo.data.gov.uk/0/country. Just follow the links from there.

With a few false starts and mis-steps, this is the process that I went through:

Tidied the CSV file so that it could be processed using awk. That meant replacing the commas that were delimiters with |s. It also meant removing a couple of weird ^M characters that had snuck into the file.
Examined the data and came up with an informal ontology and prototype URI scheme.
Created a bunch of awk scripts to extract different data from the files and create RDF/XML from it.
Ran the scripts to create RDF/XML.
Uploaded the data into a Talis store.
Created appropriate PHP for the data and put it into a proxy server.

Some of this has been covered by my recent posts, so I’m just going to talk about a few of these steps in a bit more detail.

First, the URIs. Frankly, they’re an experiment to see how it plays. The templates are:

countries: http://geo.data.gov.uk/0/id/country/{name}, eg http://geo.data.gov.uk/0/id/country/england
regions: http://geo.data.gov.uk/0/id/region/{name}, eg http://geo.data.gov.uk/0/id/region/north-west
areas: http://geo.data.gov.uk/0/id/area/{ONS code}, eg http://geo.data.gov.uk/0/id/area/00KA
local authorities: http://local-government.data.gov.uk/0/id/local-authority/{ONS code for area}, eg http://local-government.data.gov.uk/0/id/local-authority/00KA
roads: http://transport.data.gov.uk/0/id/road/{name} or http://transport.data.gov.uk/0/id/road/U-{random number}, eg http://transport.data.gov.uk/0/id/road/M5
traffic count points: http://transport.data.gov.uk/0/id/traffic-count-point/{number}, eg http://transport.data.gov.uk/0/id/traffic-count-point/36195
traffic counts: http://transport.data.gov.uk/0/id/traffic-count/{point number}/{direction}/{date}/{hour}/{traffic type}, eg http://transport.data.gov.uk/0/id/traffic-count/4/N/2008-06-05/08:00:00/HGVr2

The subdomains are one way of subdividing the vast set of public sector information into vague categories that might be handled by different departments, without using the (highly changeable) department names in the URI. The /0 portion of each URI is a version number: these URIs are experimental and liable to be unsupported in the future so they’re marked with a version 0. The /id portion of each URI indicates that these are URIs for non-information resources; the response is a 303 See Other redirect to the same URIs but without the /id.

After the /id, the URIs follow a common pattern of naming a class of resource, followed by an appropriate identifier for that resource. The identifiers themselves are designed to be unique, unlikely to change, and human readable.

The ontologies, well, actually they don’t exist as yet except in my head. It’s been more important to make the data available than to provide ontologies for it. Triplestores and SPARQL queries work without ontologies; indeed you have to go out of your way to find applications that actually reason with them. Like schemas for XML documents, they’re not absolutely essential, but useful for documentation purposes and potentially useful for applications.

There are, though, a couple of SKOS schemes for categorising roads and vehicle types. These are available via:

http://transport.data.gov.uk/0/category/road
http://transport.data.gov.uk/0/category/vehicle

They were informed by the British Roads FAQ and the data definitions from the Department for Transport. I heartily recommend a read; it’s scintillating stuff!

Anyway, with this size of file, and the kind of processing that needed to be done with it, the simple XSLT that I talked about previously for extracting data out of CSV files just wasn’t going to cut it. Awk, on the other hand, is designed for this kind of processing. Most of the RDF/XML could be generated by collecting unique values from the file. For example, to generate the RDF/XML for the regions I used:

BEGIN { 
  FS = "|";
  print "<rdf:RDF xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"";
  print "  xmlns:rdfs=\"http://www.w3.org/2000/01/rdf-schema#\"";
  print "  xmlns:g=\"http://geo.data.gov.uk/0/ontology/geo#\">";
}
FNR > 1 {
  countries[$2] = substr($1, 2, length($1) - 2);
  regions[$2] = substr($2, 2, length($2) - 2);
  codes[$2] = substr($3, 2, length($3) - 2);
}
END { 
  for (region in regions) {
    country = countries[region];
    name = regions[region];
    code = codes[region];
    path = tolower(name);
    gsub(" ", "-", path);
    print "<g:Region rdf:about=\"http://geo.data.gov.uk/0/id/region/" path "\">";
    print "  <rdfs:label>" name "</rdfs:label>";
    print "  <g:isInCountry>";
    print "    <g:Country rdf:about=\"http://geo.data.gov.uk/0/id/country/" tolower(country) "\">";
    print "      <g:hasRegion rdf:resource=\"http://geo.data.gov.uk/0/id/region/" path "\" />";
    print "    </g:Country>";
    print "  </g:isInCountry>";
    if (code != "") {
      print "  <g:ONScode rdf:datatype=\"http://www.w3.org/2001/XMLSchema#NCName\">" code "</g:ONScode>";
    }
    print "</g:Region>";
  }
  print "</rdf:RDF>"; 
}

This generated RDF/XML that looks like:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:g="http://geo.data.gov.uk/0/ontology/geo#">
<g:Region rdf:about="http://geo.data.gov.uk/0/id/region/london">
  <rdfs:label>London</rdfs:label>
  <g:isInCountry>
    <g:Country rdf:about="http://geo.data.gov.uk/0/id/country/england">
      <g:hasRegion rdf:resource="http://geo.data.gov.uk/0/id/region/london" />
    </g:Country>
  </g:isInCountry>
  <g:ONScode rdf:datatype="http://www.w3.org/2001/XMLSchema#NCName">H</g:ONScode>
</g:Region>
<g:Region rdf:about="http://geo.data.gov.uk/0/id/region/yorkshire-and-the-humber">
  <rdfs:label>Yorkshire and The Humber</rdfs:label>
  <g:isInCountry>
    <g:Country rdf:about="http://geo.data.gov.uk/0/id/country/england">
      <g:hasRegion rdf:resource="http://geo.data.gov.uk/0/id/region/yorkshire-and-the-humber" />
    </g:Country>
  </g:isInCountry>
  <g:ONScode rdf:datatype="http://www.w3.org/2001/XMLSchema#NCName">D</g:ONScode>
</g:Region>
...
</rdf:RDF>

In other cases, I needed to split up the RDF/XML that was generated into several files. Uploads to Talis of more than about 2Mb cause the upload to fail. The traffic count point RDF/XML needed to be split into 13 separate files. The traffic counts themselves… well, I haven’t managed to do it all yet but to give you an idea, the 2008 data alone generated 1800 RDF/XML files, each about 1.6Mb in size and each taking about a minute to upload. What’s there now is all the 2008 data, and the overall motor vehicle counts from all the years. More will be added gradually.

The awk script that generates the count data in separate files is:

BEGIN { 
  FS = "|";
  fileCount = 0;
  countCount = 99999;
  curlFile = "traffic-counts.curl.sh";
}
FNR > 1 && $15 ~ /\/2008 / {
  countCount += 1;
  if (countCount > 200) {
    if (fileCount != 0) {
      print "</rdf:RDF>" > fileName; 
      close(fileName);
    }
    countCount = 0;
    fileCount += 1;
    fileName = "traffic-counts/traffic-counts." fileCount ".rdf";
    print "creating", fileName;
    print "echo loading", fileName > curlFile;
    print "curl -H \"Content-type: application/rdf+xml\" -o progress.txt --digest -u username:password --data-binary @" fileName " http://api.talis.com/stores/transport/meta" > curlFile;

    print "<?xml version=\"1.0\" encoding=\"ASCII\"?>" > fileName;
    print "<rdf:RDF xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"" > fileName;
    print "  xmlns:rdfs=\"http://www.w3.org/2000/01/rdf-schema#\"" > fileName;
    print "  xmlns:xsd=\"http://www.w3.org/2001/XMLSchema#\"" > fileName;
    print "  xmlns:t=\"http://transport.data.gov.uk/0/ontology/traffic#\"" > fileName;
    print "  xml:base=\"http://transport.data.gov.uk/0/id/traffic-count/\">" > fileName;
  }

  cp = $7;
  date = $15;
  direction = substr($16, 2, length($16) - 2);
  split(date, dateFields, " ");
  date = dateFields[1];
  split(date, dateFields, "/");
  date = sprintf("%04d-%02d-%02d", dateFields[3], dateFields[2], dateFields[1]);
  hour = sprintf("%02d:00:00", $17);
  base = "http://transport.data.gov.uk/0/id/traffic-count/" cp "/" direction "/" date "/" hour;

  cycles = $18;
  motorbikes = $19;
  ...

  print "<t:Count rdf:about=\"" base "/cycle\">" > fileName;
  print "  <t:point>" > fileName;
  print "    <t:CountPoint rdf:about=\"http://transport.data.gov.uk/0/id/traffic-count-point/" cp "\">" > fileName;
  print "      <t:count rdf:resource=\"" base "/cycle\" />" > fileName;
  print "    </t:CountPoint>" > fileName;
  print "  </t:point>" > fileName;
  print "  <t:hour rdf:datatype=\"http://www.w3.org/2001/XMLSchema#dateTime\">" date "T" hour "</t:hour>" > fileName;
  print "  <t:direction>" direction "</t:direction>" > fileName;
  print "  <t:category rdf:resource=\"http://transport.data.gov.uk/0/category/bicycle\" />" > fileName;
  print "  <rdf:value  rdf:datatype=\"http://www.w3.org/2001/XMLSchema#integer\">" cycles "</rdf:value>" > fileName;
  print "</t:Count>" > fileName;
  print "<t:Count rdf:about=\"" base "/motorbike\">" > fileName;
  print "  <t:point>" > fileName;
  print "    <t:CountPoint rdf:about=\"http://transport.data.gov.uk/0/id/traffic-count-point/" cp "\">" > fileName;
  print "      <t:count rdf:resource=\"" base "/motorbike\" />" > fileName;
  print "    </t:CountPoint>" > fileName;
  print "  </t:point>" > fileName;
  print "  <t:hour rdf:datatype=\"http://www.w3.org/2001/XMLSchema#dateTime\">" date "T" hour "</t:hour>" > fileName;
  print "  <t:direction>" direction "</t:direction>" > fileName;
  print "  <t:category rdf:resource=\"http://transport.data.gov.uk/0/category/motorbike\" />" > fileName;
  print "  <rdf:value  rdf:datatype=\"http://www.w3.org/2001/XMLSchema#integer\">" motorbikes "</rdf:value>" > fileName;
  print "</t:Count>" > fileName;
  ...
}
END {
  print "</rdf:RDF>" > fileName; 
  close(fileName);
}

This also generates a shall script that includes the curl instructions to upload the files.

The original data contained easing/northing information about each point when generally latitude/longitude is easier for mapping. So I extracted the easting/northings, used the free (Windows only) software available via the Ordnance Survey to turn these into latitude/longitude — there is a web service to do the same, but you can only do 200 coordinates at a time — converted those into decimals, then RDF, and uploaded them.

The PHP scripts that serve the data as linked data are exactly what I’ve shown before. I amended the .htaccess file to redirect to an appropriate PHP script like this:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d

  RewriteRule ^id/(.+)$  id.php [L]

  RewriteCond %{REQUEST_URI} !\.php
  RewriteRule ^([^/]+)(/.+)? $1.php$2 [L,QSA]
</IfModule>

and created PHP scripts for each of the types of data being published. For example, region.php is:

<?php
  include "utils.php";
  proxy('http://geo.data.gov.uk/0/ontology/geo#Region', 50);
?>

And there we have it. Linked traffic count data on the web.

(And because this is all published through Talis, there’s also a SPARQL endpoint that you could use to run queries and create visualisations. Knock yourself out.)

Please take a look and comment on what we’ve done. What’s your opinion of the URI scheme? Is it useful to be able to access the data as linked data? Which other formats would you like to see?