Jeni's Musings

I’ve been reflecting a little since OpenTech on the relationship between the developer community and government.

Let me set out my perspective first. My goal is to help ensure that the public sector publishes reusable data in the long term.

To do that, data publication needs to be sustainable. It needs to be embedded within the day-to-day activity of the public sector, something that seems as natural as the generation of PDF reports seems today. It also needs to be useful. It needs to be easy for anyone to understand and reuse the data, with minimal effort. It cannot be the case, long term, that you need to be an expert hacker to reuse government data.

To get there, we need to work towards a virtuous cycle in which the public sector is rewarded for publishing useful data well. The reward may come from financial savings, from increasing data quality, from better delivery of its remit, or simply from kudos. It doesn’t matter how, but there needs to be some reward, or it just won’t happen.

Over the last few years, government has had to be persuaded that it’s a good idea to release their data at all. The message from the developer community has been “give us your data and we’ll show you what we can do with it!” Through hack days and various similar activities, developers have excited, wowed and dazzled officials and politicians, opening their eyes to what could be done. Through sustained argument and political pressure, developers have set out the economic and moral case that releasing data not only could, but should happen.

They have been incredibly successful. We have data.gov.uk, open data from Ordnance Survey, strong commitments to open data within the Coalition Agreement, and the Public Sector Transparency Board who are now applying that pressure, with authority, at the heart of government.

My perception is that the argument that government should open up its data has basically been won. The questions within the public sector are now about how, not whether. And as a result, in this changed environment, I’m growing slightly uneasy about the core developer message of “give us your data and we’ll show you what we can do with it!”

There are two things about that message that concern me. First, it implies government is doing it all wrong. Second, it implies that government doesn’t need to do any better, because the developer community can take up all the slack and fill in all the gaps. It’s like getting fed up with a child struggling with their homework, and saying “oh, just give it here and I’ll do it!” It’s a narrative that simultaneously undermines the best efforts of those within government and removes from them the motivation and opportunity to learn to do better.

Of course there is a tricky balance here. We don’t want to let up pressure on the government to release important information. We don’t want government to feel that they have to get their data perfect before releasing it. And we can’t always wait for government, which can be slow-moving as an organisation, to provide everything we need right now.

However, there are certain things that only the owners of data — those within the public sector — can do. People who own data understand it so much better than third parties: what codes mean, what values are used to indicate missing data, what gets included and what gets left out, which columns aren’t really used any more, which interpretations are safe and which are meaningless. Data owners can be trusted in a way that no one outside could be; when data publication becomes a sustainable part of their activity, they are much better placed to provide a steady, reliable, flow of data than a third-party API that could disappear or get out of date whenever the volunteer behind it moves on to something new.

People in government must be given the responsibility to publish their data well. And there are three core ways in which I think developers could help them.

First, while there are many more technically savvy people within government than is sometimes made out, the average civil servant lacks both know-how and tooling. I think developers could help a huge amount here. What about hack days where developers sit side by side with civil servants to help them clean and publish their data? What about engaging with the owners of a particular data set to help them to publish it in a way that was reusable and sustainable? What about writing services, accessible through the locked-down IT systems that civil servants have to use, that enabled them to convert their data into multiple formats, and to link up the ways they refer to things with the way other people do?

Second, while government needs to be responsible for publishing its data, it can’t be responsible for building everything that end-users need based on that data. Developers have the facility to create applications that bring together data from diverse parts of the public sector, and combine it with data from outside. This has always been a feature of hack days, of course; all I’m arguing for is a focus on applications that the public sector shouldn’t be doing itself.

Third, we need to build the virtuous cycle that I talked about above. Government needs to hear about what works for developers, as well as what doesn’t. What data releases have been helpful and why? Who are the stars? Who should be rewarded and emulated? We need ways of feeding back in a constructive way to public sector workers who are trying their best with the resources they have — often extensive subject-matter expertise but little time, locked-down technology and contracting finances.

The vitality and engagement of the developer community has played a massively important role in the open government data initiative within the UK, and I’m sure it will continue to do so. We are incredibly lucky, here, to have a collection of talented and motivated developers who volunteer their time to work with government data. My hope is simply that the relationship between government and developers can grow into one that is more encouraging and supportive, that understands the constraints and concerns of those within government, and that provides practical help to overcome them.