Jeni's Musings

A Fair Web

2013-02-03T23:38:00+00:00

The new membership of the W3C’s Technical Architecture Group (TAG), and some of the recent discussions on the TAG list about polyglot markup, have made me think about what the TAG should stand for and the role the TAG should play.

Fundamentally, the web is for everyone, whatever gender, whatever race, whatever sexual orientation, whatever visual or mental ability and so on. The web community should fight to keep the web open to all. And it should try to be a community that is open to all.

With same-sex marriages shortly being voted on by the UK Parliament, I have been struck, reading the recent threads about polyglot, how similar the arguments against polyglot seem to those used against homosexuality:

claiming that there is no use for polyglot, in the face of those who say they have use for it, is similar to denying that homosexuality exists, in the face of people saying that they are homosexual
stating that you can see no use for polyglot and therefore no one else should use it is similar to saying that since you are heterosexual, everyone else must be too
claiming that creating a Recommendation that describes polyglot will make people use it is similar to saying that talking about homosexuality will make people gay
saying that you don’t want to implement polyglot in a validator or editor is similar to being a priest who declines to marry gay people

By this comparison, those who argue that polyglot must be the only output anyone generates also has an analogy: someone arguing that all churches must marry only gay people.

I want the web community to be a fair and good society. To me the question about whether there should be a polyglot Recommendation is just the latest example of a need to ensure that our community is equitable.

The web is for everyone, whatever technology stack they use. The reason we have standards is to enable people to make their own choices about what they do in the privacy of their own servers. We don’t have to use the same libraries, so long as we implement the same standards. It doesn’t matter if you programme with Ruby or PHP or C# or Scala or XQuery, you can build a web application because we have the standards of HTTP, HTML, CSS, Javascript and so on.

Just as in wider society, we need to find compromises that balance the needs and desires of different constituencies. We need to balance the rights that everyone has to code as they wish against the rights that everyone has to have a web that works. We need to make sure that the quiet voices are heard, and support the equal rights of those who tread the less worn paths.

When there are conflicts between technologies, developers necessarily think of them in terms of which they would use, given their experience, expertise, environment and so on. I think the TAG needs to judge technologies in a different way. We have to consider the extent to which standardising their use disrupts the fabric of the web and prevents others from operating as they wish to. And, because we want a web that is fair and open and free, if there is no or minimal risk to the fabric of the web, and it does not overly constrain how others act, I believe we should err on the side of supporting diversity.

Making and expressing these judgements is just one of the things that I hope the newly formed TAG will manage to do better.

Open Data Business Models

2012-08-20T21:31:30+01:00

As you may have seen, I have been appointed to the Cabinet Office’s Open Data User Group which had its first meeting in July. The minutes and slides are available.

The purpose of the group is to act as an “intelligent customer” to the government on the release of open data. This is a bit of a misnomer, as the word “customer” implies that the group will in some way buy data that should be made open, which it’s unlikely to do. Perhaps “intelligent consumer” would be more appropriate: our task is to advise the government about which data should be opened up, and (if the commitment has already been made to open it) which should be opened first or how access to it could be improved.

One of the tasks that we face, particularly for datasets that are currently being sold by government (mostly from the Public Data Group: Met Office, Ordnance Survey, Land Registry and Companies House), is making a strong economic argument for opening up data. To do that, it’s useful to understand two things:

the ways in which open data can be used in the wider economy, to aid innovation, growth and thereby lift the country out of the economic doldrums
the business models that are being used by open data publishers to support open data releases, to illustrate the benefits that they can bring to publishers themselves

I find business cases for data publishers much more compelling than examples of how open data can be used. For a start, I don’t think it’s possible to predict how open data will be used or what that will mean in terms of economic or societal impact: the wide world into which it’s released is just too complex to know. Sure, there’s the development of “Apps”, but there are many more hidden uses for open data within businesses, which data publishers and those external to the process are unlikely to ever be aware of. There are also impacts on vendors of support products like analysis and visualisation software, and on society as a whole when you have better informed people and organisations.

As well as wider impact being hard to measure, I don’t think anyone is likely to publish open data well if they don’t have some motivation that is a lot nearer to home than “helping grow the economy”. To be useful for reusers, open data needs to be structured, supported, timely, accurate and so on. To be useful for business, data releases need to be reliable and sustained over time: data can go out of date very rapidly and single releases are rarely interesting. So open data publishers need to have business models that enable their data-publishing activities to be self-sustaining and preferably improving.

Below I’ve described several such models, many of which are based on the business models that currently exist around open source. I’d be really interested to hear of any other business models that you know of for open data, and in particular to hear about examples where any of these models have been employed successfully.

Cost Avoidance

One argument I’ve heard made about government open data is that releasing it can help organisations avoid the costs of Freedom of Information requests. This probably applies only to data that is likely to be requested (perhaps it is published annually and was requested last year) or has a very low publishing cost. Organisations that have a high FOI spend with lots of successful requests may find that they can lower that FOI spend by proactively releasing data (and making it easy to find).

There are other, less direct, costs that organisations might avoid by releasing data. Most obviously in the public sector, there is political cost in not toeing the Open Data line.

Sponsorship

The reverse of cost avoidance is finding sponsors for open data publication. If there are people who strongly believe that a particular dataset should be open and available to all, they may be prepared to sponsor its publication (which isn’t the same as licensing it; the consequence is that the data is open for all, not just for those who pay). As I understand it, some data that has been opened up by government, such as that from Ordnance Survey, has essentially been opened up through the sponsorship of the Treasury: they have paid for the data to be made open, for their own reasons (to do with belief that it will enable the economy to grow).

How could you persuade others to sponsor opening up data? If it’s something that they would otherwise license, perhaps they are in a better place to face any disruption that will come from the data being freely available than their competitors. Perhaps, if it’s the type of dataset that is hard to close up again after it has been made open, they might gamble that it would lower their long term costs. Perhaps they sell analysis or visualisation products that they know those who use the data will find useful, and so getting the data available widely will aid their business.

Freemium

The freemium model has been used with some success for web-based services; it might also work for open data. Under this model, an organisation would publish open data in a basic form — perhaps with some limitations on formats and throttling of API calls — and offer advanced access to those who are willing to pay.

There are many ways in which open data can be made more useful than static publication of spreadsheets or a basic API; under a freemium model some of these enhancements would only be offered to those who pay for it:

availability of different machine-readable formats
unconstrained numbers of API calls
more sophisticated querying
access to data dumps rather than through an API (or vice versa)
provision of feeds of changes to the data
enhancement of the data with additional information
early access to data
provision of data on DVDs or hard disks rather than over the net

Dual Licensing

Data publishers could provide data under an open license for certain purposes, and under a closed license for others. This technique has worked for some open source products. The ‘certain purposes’ might not be simply ‘non-commercial’: publishers could still encourage start-up use of the data by charging based on the size or revenue of the organisation. Or the license could state that the data can be used in products but cannot be used in further “added value” data feeds without being licensed (this is roughly equivalent to dual-licensing with a share-alike license). This is the model used by OpenCorporates.

Support and Services

Offering support and services is a business model which seems to work well for companies built around open source. In the open data world, data publishers could offer paid packages with:

guarantees on data availability
prioritisation on bug fixes (both in data and its provision) for paying customers
timely help for customers using the data
services around data visualisation, analysis and mashing with other data

These kinds of services still tend to be coupled with licenses in the data world, whereas in open source they have been successfully disentangled.

Charging for Changes

In some cases, individuals or organisations are obliged to provide information to public bodies (and they have a statutory duty to collect it), so that it is available within government and more generally in society. These public bodies can (and sometimes do) charge the providers of that information “administration costs”. Examples of this are Companies House information, the Gazettes, Land Registrations, VAT Registrations and so on.

In these cases, those who supply the information to the register are bound to by law, so it would be possible to charge them whatever it took to support providing the data as open data. Indeed, supplying the data as open data is likely to increase its usage (both within government and more widely), and therefore the political pressure to retain the registry and thereby maintain its longevity.

Increasing Quality through Participation

The model that we are using at legislation.gov.uk is based on increasing the quality of the data that we have to publish — bringing the statute book up to date — by enlisting the help of other parties who would benefit from having an up-to-date open statute book. Because otherwise this information is very costly to get hold of, there are any number of potential contributors, including publishers, lawyers, academics, and government itself.

This model doesn’t entirely cover the costs of opening up data: contributors aren’t generally paying money to be involved, but donating effort to maintaining the published data. Thus this business model doesn’t completely cover costs, but it’s a very useful one for organisations that have an obligation to publish information but lack the resources to do it well.

Supporting Primary Business

The final business model that I have seen being used is where releasing open data naturally supports the primary business goal of the organisation. The best example of this is around the Barclays Cycle Hire in London (or Boris Bikes as we call them for some reason), where releasing open data about the bikes drives the development of Apps that make it easier for potential customers to use the scheme, thus bringing in revenue to the core business.

Another example is the recent release of data about Manchester City football players which, they hope, will lead people to create better ways of measuring player performance, which they will then be able to take advantage of. (And if it means that they are being talked about in the blogosphere, so much the better.)

I’d also place under this category situations where the organisation that publishes the data ends up improving its own use of its data by using the third-party tools that are created because the data is open and available. The kind of thing that I’m thinking about here is how MPs (reportedly) use TheyWorkForYou.com. There’s great opportunity, I think, for the public sector to create a market place for tools that enable it to work more efficiently, by opening up its data.

In cases where organisations are releasing data to support their primary business, they may even find it worthwhile backing up such releases with hackdays and competitions and so on in order to drive the initial creation of some products.

Discussion

I have listed here business models for open data that I have either seen being used or think could viably be used by organisations, particularly public sector organisations. There may be others business models, or examples, that I don’t know about, and as I said at the start, I’d really value your suggestions for more.

One thing that I wanted to touch on was about motivations for data publishers. Although, as I’ve said, I think it’s going to be very hard to measure or predict the impact of particular datasets in terms of how it is used in the wider world, it’s fairly obvious that high quality data, supplied in a timely and consistent fashion, is going to be easier to use and more accurate than low quality data, supplied as and when, using different formats and coding schemes within each release. In other words, it seems likely that data that is published well will lead to greater usage and thus the better economic outcomes on which much of the open data argument is based.

The different business models above provide different incentives for data publishers. In fact, only the last two include any incentive to publish data well: when publishing data to support your primary business, the whole point is to make it easily reusable; when you are supporting data publication by eliciting contributions from others, they are more likely to contribute if the released data is useful for them.

In other cases, the incentive for the data publisher is towards doing the least amount of work so that they can retain as much money as possible, or sometimes (as in the Freemium or Support/Service models) to make the data hard to use unless you are a paying customer. Of course that doesn’t mean that organisations using these models will deliberately restrict the utility of the data that they publish — public sector employees tend to be more motivated towards doing the right thing than making a profit — but they will have a lot less incentive to invest in making the data easy to use than those employing either of the latter two models.

Another aspect of the business models above that’s worth thinking about is about knowing who is using your data. Many data releases have been done in a “fire and forget” mode, using either cost avoidance or sponsorship as a model, which has the advantage of being a cheap method of releasing data. But having a rudimentary idea of who is using data and what they are using it for helps you to understand where there are gaps in your provision, be it in formats, coding schemes, timeliness and so on. Many of the other business models on the list above (as well as the ‘selling licenses’ business model of closed data) put you in direct contact with at least some of those using the data, which helps you improve provision in the right direction. In particular, getting people to participate to help improve data quality is good open data engagement and something that I think most data releases should aim for.

Which brings me back to ODUG. The best indications that we have about what data will prove useful to people are the data that people are currently using and the data that they tell us they want. If you’re a potential or actual reuser of public sector information, ODUG will shortly be providing routes to tell us about what you’d like to have access to, but in the meantime do get in touch with me if you have any requests.

RDF Chimera

2012-06-30T21:42:44+01:00

During my keynote at XML Prague (the video might make more sense than the slides on their own; there are notes on the slides but Slideshare doesn’t do well with Keynote), I talked about how the advantages of using chimeras created from two formats with different underlying models are seldom outweighed by the disadvantages. RDF/XML gets knocked so frequently it’s not even much fun to do it any more, but I’ve applied the same arguments to JSON-LD in the past. My argument was that RDF, XML, JSON and HTML should each be used individually for their strengths rather than trying to find a middle ground that rarely satisfies anyone.

Leigh Dodds’ post on principled use of RDF/XML makes the point that RDF/XML can be useful when it is used in a regular, principled way. And in fact, I am using RDF/XML extensively in my work on Expert Participation for legislation.gov.uk, though slightly differently from how Leigh describes. What I want to explore in this post is when and how it makes sense to use RDF/XML and how that might translate into usage of JSON versions of RDF. The key point I want to make is that RDF chimera are roads, not destinations, and when you’re choosing a road you have to think about the destination you’re aiming for.

My Destination

Within legislation.gov.uk, our primary content is stored as XML (because it’s documents) and our primary toolchain is therefore based on XML: we use XSLT, XSL-FO and Orbeon pipelines (for historic reasons; today I’d try to find something similar that supported XProc) to create the various views of legislation we have on the site.

We store RDF content in a triplestore, and query it with SPARQL, because these are technologies designed for easy and efficient storage and querying of RDF, but when it comes to integrating that data into HTML pages, it makes sense to reuse the same core pipelines and processing tooling as we have been using thus far. If we’d been using Ruby on Rails we’d have wanted to use Ruby, if Django then Python, if Node.js then Javascript — the point is that once we’ve extracted the data we needed, we needed to get it into a format that we could easily process with the tooling we are using.

Using SPARQL Results

One of the nice things about SPARQL is that you can use it over HTTP and get the results of a standard SELECT SPARQL query in XML, JSON or CSV/TSV. Making HTTP requests is easy across programming languages, and every environment is going to be able to handle at least one of the three possible output formats. So for us, requesting an XML result to a SELECT query fits right into our processing pipeline. Well, to be honest, writing paths like

sparql:result[sparql:binding[@name = 'process']/sparql:uri = $update]/sparql:binding[@name = 'count']/sparql:literal

is a little irritating, but if I cared enough it would be easy to transform the SPARQL results XML format into something I could more directly query through XPath.

What I’ve found, however, is that it is usually a lot more flexible and often more efficient to use a CONSTRUCT or DESCRIBE SPARQL query to extract a subgraph from the store to process. At that point, you need a format that can represent an RDF graph, and you need your processing chain to be able to process an RDF graph as an RDF graph. I’ll come on to other languages in a second, but sticking with XML: if your processing toolchain is XML-based, you need it in XML.

There are actually two choices here: RDF/XML or TriX. TriX is fine, but it suffers similarly to the SPARQL XML results format when you try to use it with XPath: you have to do things like

$task[trix:uri[2] = $task:assignedTo]/trix:uri[3]

where $task is a sequence of trix:triple elements and $task:assignedTo is the URI of a property that I’m interested in, to work out who is assigned to a particular task in the system I’m building. Of course that gnarliness could be hidden behind a function call, but it means using functions everywhere. Using RDF/XML on the other hand can be somewhat easier and use natural XML paths but only if it is normalised.

Normalised RDF/XML for XML/XSLT Processing

The biggest problem with RDF/XML for use with XML tooling is that as a format it is too damned flexible. There are multiple ways of representing everything: resource types can be indicated by the name of the resource element or through an explicit rdf:type child element; literal property values can be held as attributes or elements; resource property values can be referenced or nested. All these different options for representing the same information makes the format a complete sod to work with.

This is the thrust of Leigh’s post. The question is then: how do you normalise RDF/XML into a regular format that you can process? In Leigh’s post, he advises “Use all of the [RDF/XML] shortcuts”, which he describes as basically:

using a single root resource element rather than a flat structure with a rdf:RDF wrapper
using the element name of a resource element to indicate the type of the element
using a single resource element to represent each resource

(I note these aren’t all the RDF/XML shortcuts: you can use attributes for literal properties for example. It looks like Leigh instead chooses the regularity of having all properties be represented as elements in the XML.)

This normalisation routine gives you XML that is very close to “natural” XML: it will only have a few rdf:about and rdf:resource attributes here and there to give away that it can be processed as RDF/XML. However, if you are starting with RDF, it only gives you a nice, non-repetitive, regular structure if:

it’s easy to identify a root resource within which everything is nested within a given graph
resources only have one type
you don’t get resources used in more than one place in the graph

For the legislation.gov.uk Expert Participation work, I needed to have a normalisation routine that would work across graphs which included resources with multiple types and that were highly repetitive. I also needed a normalisation routine that I could use without it having to guess what the appropriate root node would be, or me having to feed in that decision or artificially create a root node for each graph (which would have just been extra work). The normalisation that I’ve found that works for me is instead:

always use a rdf:RDF wrapper
all resource elements are directly within the rdf:RDF wrapper, including blank nodes
all resource elements are represented by a rdf:Description element
there’s one resource element per resource
all properties are represented by elements

There are disadvantages to this algorithm: if you want to find all the leg:Legislation items in the RDF, for example, then you need to do something like //rdf:Description[rdf:type/@rdf:resource = $leg:Legislation] with an appropriately set $leg:Legislation variable, whereas under Leigh’s scheme you would do //leg:Legislation. The thing is that in the data that I’m dealing with, leg:Legislation resources might also be leg:UnitedKingdomPublicGeneralActs or leg:CommencementOrders: I can’t know at normalisation time which of the various types of a given resource is the one that I’ll want to easily be able to query over, and the cost of changing my mind later on would be quite high.

Similarly, not having nesting means that I can’t write simple paths like $task/task:assignedTo/sioc:User/sioc:name which I would have been able to do under Leigh’s suggested normalisation. What I do instead is define a couple of keys that index the descriptions by their rdf:about or rdf:nodeID attributes

and a function that makes it easy to traverse through properties:

The normalisation I describe coupled with this bit of utility coding means that I can just use a simple path to get to the value of a (literal) property like this: $task/task:assignedAt and the rdf:get() function when I want to navigate to another resource: $task/rdf:get(., xs:QName('task:assignedTo'))/sioc:name. In XSLT 3.0 I think I’d be able to define some funky functional functions and write things like $task($task:assignedTo)($sioc:name), but we’re not quite there yet.

Another Destination

What I wanted to bring out is that for all its failings RDF/XML is a useful format, but to process RDF/XML using XML tooling, you need to normalise it into a regular structure that retains the semantics of the underlying RDF graph. I thought it was interesting that Leigh suggested a different normalisation algorithm from the one I use: working through it, it’s clear there are different advantages and disadvantages of the different algorithms, both in terms of how easy they are to apply to given data and how easy the results are to process.

But the other thing I wonder is whether Leigh’s destination was different from mine. The goal of producing a natural-looking XML format, and one that you would want to validate using RELAX-NG, implies that Leigh was going along the RDF/XML road in another direction: basically scattering some RDF attributes around an XML format so that it could be interpreted by RDF tools. I’m sure that this was the reason RDF/XML was made so flexible to begin with: if you can take some naturalistic XML and just add attributes to turn it into RDF then (the argument goes) you get the best of both worlds.

But this is what I’m really sceptical about. XML formats tend not to fit into the “striped syntax” of resource/property/resource/property required by RDF/XML: they miss out resource elements or they group them in wrappers, or they put references in attributes. To get an XML format to be compliant RDF/XML you really have to design it that way from the start, and this goes to the root of my objection to chimera: when you do that, you stop using XML in the way it’s naturally used, and you lose (some of) its advantages. It’s like putting yourself in a straitjacket. It seems a lot wiser to me to let XML be XML, and run it through a transformation to create an RDF format (TriX or RDF/XML are both easy targets here) when you want to extract the data within it as RDF.

JSON and RDF

And so I turn again to JSON and RDF, because the same arguments apply. If my destination were using RDF data in a non-XML-based toolchain, I think that I would have a very similar experience to the one described above. Getting the results of a SELECT SPARQL query as JSON would be great, but eventually I’d need to handle an RDF graph as an RDF graph and I’d want to be able to manipulate it in my programming language of choice using the idioms of that programming language.

There are RDF processing libraries for most programming languages, of course, that make it easy to load RDF graphs into a data structure that you can then query and process “naturally”. If you’re using one of these libraries, then the format of the data that you get back from a SPARQL query doesn’t really matter, so long as it can be loaded into that data structure through the library.

Does a JSON format for RDF help here? Well, if there isn’t a library for a given programming language, or you don’t like the API that any of them give you, then a JSON format for RDF is a format that you will be able to load (because every programming language supports JSON) and manipulate.

If you’re in this world, you need a regular structure that enables you do that processing in a regular way. You know the format that you’re dealing with, so putting URIs into variables isn’t an issue (there’s no need for shorthands in the syntax itself). You want a flat and predictable structure which you can query into easily to follow links to information about other resources in the graph. You can of course use JSON-LD in this way. It might look something like:

{
  "@graph": [{
    "@id": "http://www.legislation.gov.uk/id/task/research/effects/uksi/2011/1901",
    "http://www.legislation.gov.uk/def/legislation/assignedTo": [{
      "@value": "http://www.legislation.gov.uk/id/user/tso.co.uk/jeni.tennison"
    }]
  }, {
    "@id": "http://www.legislation.gov.uk/id/user/tso.co.uk/jeni.tennison",
    "http://rdfs.org/sioc/ns#name": [{
      "@value": "Jeni Tennison"
    }]
  }]
}

This isn’t great for navigation, at least in Javascript — you really need to build a hash table to get quickly from the object ids to the details of the objects themselves — but once you have that you can do things like:

objects[task[leg.assignedTo][0].@value][sioc.name][0].@value

which isn’t too bad. It gets nastier when you start to have multiple values for properties and want to navigate through them, but people who use these kinds of object structures are used to that.

What about going the other direction? One of the JSON-LD raisons d’être is to provide a quick and easy annotation route for adding RDF semantics on top of existing JSON formats. Just as with RDF/XML used in this way, I’m really not convinced that the majority of existing JSON formats are going to be easily coercible into JSON that can be processed into sensible RDF through the JSON-LD processing rules, nor that JSON designed from scratch to be JSON-LD compatible will have the advantages of “natural” JSON. I can see JSON-LD as an easy-to-generate target format for people wanting to extract RDF from JSON, though given how people generate HTML and XML they might just stick with string manipulation and generate N-triples or N-quads.

Thoughts?

What do you think? Are there advantages in RDF chimeras like RDF/XML and JSON-LD as destinations that I’m just not seeing? Are there other ways of normalising them that make them easy to process as XML or JSON?

Using "Punning" to Answer httpRange-14

2012-05-11T21:11:43+01:00

As part of the TAG’s work on httpRange-14, Jonathan Rees has assessed how a variety of use cases could be met by various proposals put before the TAG. The results of the assessment are a matrix which shows that “punning” is the most promising method, unique in not failing on either ease of use (use case J) or HTTP consistency (use case M).

In normal use, “punning” is about making jokes based around a word that has two meanings. In this context, “punning” is about using the same URI to mean two (or more) different things. It’s most commonly used as a term of art in OWL but normal people don’t need to worry particularly about that use. Here I’ll explore what that might actually mean as an approach to the httpRange-14 issue.

Note: The material here is a summary of what I think is the best way forward following various discussions within and outside the TAG, in particular with Jonathan, Henry Thompson and TimBL. Not all these people agree with or endorse the approach described here, but neither do all the ideas in this post originate from me.

Background

Five things recently make me more convinced than ever that the TAG must either provide some direction to the community, and soon, or get out of the way.

The proposed Linked Data Platform Working Group charter and the Submission that is the main input to the group specifically brings together linked data and REST, and the only mention of 303 redirections so far is to do with paging.
A recent thread on the W3C public-vocabs mailing list, raised the question of whether to embed schema.org markup about the page itself within a given page, or only about the thing that the page is about. I wonder how many pages are being described as schema:WebPage as well as things like schema:Organisation, and how people choose which class to use.
The initial version of Dan’s proposal for handling external enumerations within schema.org talked about minting new URIs in the ext.schema.org domain specifically to proxy existing URIs so that they can be guaranteed to provide the right (303) HTTP response. I can see the reasoning (persuading people to use 303 redirections is difficult) but it would be frustrating if the end result were a centralisation of the URI space.
Talking with my colleague John Sheridan about updating the UK government’s guidance on Designing URI Sets for the UK Public Sector, I really don’t know what to advise. Should the guidance continue to be to use 303 redirections, when I know from experience that these can be impractically slow? Should it change to recommend using hash URIs to identify things?
The very first message on the Technical Architecture Community Group was in part about how to identify people with URIs.

Of course the httpRange-14 issue has been running for so long now that I’d estimate that currently 80% of the discussion about it is meta-discussion about whether there it needs to be discussed, how much it should be discussed, how to raise the quality of the discussion, how anyone who discusses it is a time-wasting idiot and should just shut up and so on. It’s a terrible destructive cycle: the more this goes on, the higher the proportion of time being spent on the meta-discussion, and the longer any discussion takes.

But I believe that we can get to a point where we don’t have to discuss it any more (except to reminisce about what a waste of time it was), and I believe that the only way to get to that point is for the TAG to push through and provide a practical way forward.

Terminology

Let’s start off with some terminology. The basic scenario is a three-way interaction between three agents:

a supplier who manages the information that is accessible at a URI; it’s worth noting that the supplier for a particular URI might change over time and what exactly is provided at a URI is controlled by multiple parties, as there may be many service providers involved in routing the resolution of a URI, others in constructing what’s shown on the page served from the origin web server, and still others who transform that content en route to the consumer
a third-party publisher who publishes some information about a URI; unlike the supplier, they generally have no control and incomplete knowledge about the information available at a particular URI, its stability over time or consistency across representations
a consumer, typically an application of some kind, who has discovered the information published by the supplier or third-party publisher and wants to do something with it

It’s also useful to include terms for three things that are passed around or referenced during the interaction, which are defined with the URI specification and HTTPbis. (The current HTTP specification, RFC 2616 is also of interest, of course, but HTTPbis is a better reflection of current practical use of HTTP, and is close to complete, at which point it will replace RFC 2616.)

a URI which is a string of characters matching the syntax in the URI specification that identifies a resource; we only care about http: URIs here, although similar considerations may apply to URIs that use other schemes
a resource which is identified by the URI; the debate over whether HTTP constrains the nature of the resource is at the heart of some discussions around httpRange-14; here, as in the URI specification and HTTPbis, a resource could be anything
representations which are media-typed sequences of bytes (often characters) encoded within the response to an HTTP GET request on a given URI; per HTTPbis, the response to a GET request contains a representation of the current state of the resource identified by the URI

Content and Meaning

Now I will introduce a few new terms that aren’t used in the URI specification of HTTPbis but which are useful for discussion.

The content located by a http: URI is whatever core information a consumer interacts with through the HTTP interface provided by the server for that URI. This is the information that is common across all the representations that are returned through a GET on a given URI (through content-negotiated variants). We can say that the http: URI locates the content of the resource that it identifies, because you can get hold of the content of a resource by performing an HTTP GET request.
The sense referred to by a http: URI is a social construct that arises from the properties associated with the URI by publishers and the way that these invoke action in consumers. We can say that a http: URI refers to a sense. While you can GET content from a URI, determining its sense can only be achieved by examining the way in which the URI is used within data published on the web. The sense referred to by a URI might vary in different contexts, but equally a single sense may emerge in the use of the URI. One sense referred to by a URI may be its content, and for some types of information that may be the only sense referred to by the URI by anyone.

Here is a diagram that shows how these different terms hang together.

For example, take the URI http://www.amazon.com/gp/product/B004TRXX7C. The content located by this URI is the core information in the web pages we GET from the URI. The sense referred to by the URI could the same as the content, or it could be the novel Moby Dick, or the particular Kindle edition of the book. We can’t tell from any interaction at the level of the HTTP protocol what the sense of the resource is: that information has to come from the application level.

Like the meaning of a word, the sense that a URI refers to is a social understanding which emerges from use of the URI across the web, and a given URI may be used to refer to different senses in different sources of information or over time. Consumers interpret the information that uses a URI and is made available to them on the web in order to draw conclusions and perform a task. Different consumers will have different levels of trust in the particular interpretation of the URI that a given publisher provides; in particular, the information published by the supplier of the URI might be given a higher weight than that from third-party publishers. Tools like sig.ma illustrate how information can be combined from multiple sources with different weights, by associating metadata about the location of the data with the data itself; unpicking commonalities between groups of sources may help to work out the different senses referred to by these different sources.

The content located by a URI is more concrete, and is important because certain classes of application may infer something about what they can do with the content found at a given URI based on information published about the URI. The canonical example of this is a consumer that searches the web for public-domain pages, based on information published about the licensing of those pages, and displays a portion of one each day within a feed. This application can’t work properly if it doesn’t know what actual content is public domain. In the Amazon example above, if there is a statement saying http://www.amazon.com/gp/product/B004TRXX7C is public domain (referring to the novel “Moby Dick”, which is one possible sense referred by the URI, one on which the copyright has expired), a consumer that assumes the URI is being used to locate the content at that URI will assume that the representation retrieved through a GET on that URI is public domain. The consumer might pick out the first major paragraph from the HTML page for display, but that first paragraph is actually an editorial review that is marked with a separate copyright which therefore shouldn’t be displayed in a feed of public-domain content.

Of course interpreting assertions made by publishers about particular content can be just as complex as interpreting statements about a particular sense of a URI, especially when those assertions come from a third party. The content located by a URI may change over time, and potentially in dramatic ways if the domain name of the URI changes hands, so any statements that a consumer discovers about content needs to be assessed by a consumer in that context. That said, while the validity or truthfulness of a given piece of information about content may be variable, the bytes that are located through a URI by a consumer are tangible and discoverable in a way that the sense referred to by a URI can never be.

The core disagreements around httpRange-14 arise from whether you view the content located by a hash-less http: URI which provides a successful (2XX) response to be the only valid sense referred to by that URI, or whether you think they could be different things, and if you think they can be different things then which of those you think the URI identifies when it is used.

Current State

The URI and HTTP specifications talk only about URIs identifying resources and being able to make requests using URIs to get a representation of the resource, they do not talk about senses or content.

The httpRange-14 decision is based on a design where if you can successfully GET a representation using a URI (ie you receive a 200 OK response) then the sense referred to by the URI is the content located by that URI. This is pictured in the diagram below.

Sometimes, a server simply doesn’t store the content for something that it wishes to provide information about (for example, the Amazon website doesn’t store the content of Moby Dick), and sometimes the sense that a publisher wishes to confer on a URI is such that no content can be transmitted over the wire, such as a Person. In these cases, under this design, the server cannot give a 200 OK response because it does not have the content for the URI. There are then two patterns a publisher can use to assign a http: URI to something in these cases.

One pattern is to use a 303 See Other redirection to a URI whose content (which is the only sense of the URI in this design, remember) describes the original URI. This is pictured below.

The second pattern is to use a hash URI. This gives a very similar pattern, as you can see from the diagram below; the only difference is that instead of following a 303 See Other redirection to get from one URI to the other, you can use URI parsing: you chop off the fragment part of the URI and perform a GET on the resulting URI to get a description. Often this leads to several hash URIs being described by the same content, as illustrated here:

However, hash URIs are also used to identify fragments within pages, which are bits of content. For hash URIs that identify fragments of a page, the picture looks more like this:

A consumer can’t tell just from looking at a hash URI whether it identifies a fragment of content or is being used to refer to something described by the content located by its base URI: the consumer has to make a request and understand the fragment identifier as it applies to the media type of the representation that it gets back. Also, if there’s any content negotiation going on, even if the fragment identifier doesn’t make sense for the media type, or doesn’t locate a fragment within the representation that the consumer retrieves, it might still locate a fragment of content within a different representation of the resource than that the consumer has retrieved.

In any case, common usage of hash-less http: URIs differs from this model of content and sense being one and the same for all URIs that give a successful response. Often URIs are used in data formats such as JSON, XML, RDF or HTML where the content and the sense of the URI are different things. For example, on the Flickr the content located through the URI http://www.flickr.com/photos/45701084@N08/7051652969/ is a landing page that provides a bunch of information about an image, but the data within the page includes a statement about its license:


  cc:license  ;
  .

in which that same URI refers to the photograph itself: the sense referred to by the URI. The only way that you can tell this is by being a human: reading the page and the context in which the text describing the license is used.

This mismatch between the design specified by RFC 2616 and the httpRange-14 decision, and practice on the web today results in arguments back and fro with people saying, in essence, that the resource a URI identifies is the sense conferred by the URI’s supplier, or that a URI should always be taken as identifying the content of the resource, and then discussions about how to signal to an application that in particular cases the supplier really does mean the URI to identify some content or really does mean the URI to identify a particular sense, and if so which sense is being referred to.

Punning

“Punning” approaches attempt to cut through these disagreements by saying that the context in which the URI is used determines whether it is locating content or referring to a sense.

If we look at some Open Graph Protocol (OGP) statements on http://www.imdb.com/title/tt1334573/, we see:

Some of these properties — the url, title, type and image — are about the Moby Dick TV Series — the sense referred to by the URI http://www.imdb.com/title/tt1334573/. Others — the site name and Facebook application id — are about the content located by the URI. The properties that are provided by this data are all related to the same URI, but they aren’t all properties of the same thing. In natural language we might say:

a URL of the thing described by the page is http://www.imdb.com/title/tt1334573/
a title of the thing described by the page is “Moby Dick (TV Series 2010)”
a type of the thing described by the page is “video.tv_show”
an image of the thing described by the page is the content at http://i.media-imdb.com/images/SFc0774313bf9ccbfe22050c8bb4029e41/imdb-share-logo.gif
the site of the page is IMDb
the Facebook application to use with the page has the identifier 115109575169727

The property itself determines whether it applies to the content located by the URI (the page) or a sense referred to by the URI (in this case, the thing the page describes). Here’s a diagram that shows the distinction:

Defining URI Usage

The way in which a URI is interpreted — as referring to a sense or locating content — is dependent on where it is used. In XML, for example, an xmlns:* attribute contains a URI; when this is a hash-less http: URI, this refers to an XML namespace (a sense of that URI): it doesn’t matter what content you GET from dereferencing the URI, or even if it can be dereferenced at all. On the other hand, the href attribute on a xi:include element defined by XInclude is used to locate some content to be included within the referring XML.

It is really up to the format in which data is encoded to determine how the URI should be interpreted: as locating some content or referring to a sense. As with interpreting any information with which it’s presented, an application that needs to work out which is meant might use:

built-in knowledge (eg an application might know that the og:title property is always about the sense referred to by the subject URI, based on documentation about the property [this is essentially the same as if the information were embedded within a schema, but without the implication that every application must download and interpret a schema every time it happens across a property])
information encoded within a schema (eg a schema might classify the og:title property as a PropertyWhoseSubjectIsTheSubstanceOfAResource)
a default for the format of the data (eg given OGP uses RDFa, RDF could specify that by default URIs refer to a sense, and therefore barring other information to the contrary, properties cannot be assumed to be about the page itself)
a default for the web (eg we might say that barring overriding information, all hash-less http: URIs are assumed to locate content, as this is consistent with the current definition of HTTP in RFC 2616)

Thus an implication of this approach is that the people who define languages and vocabularies must specify what aspect of a resource a URI used in a particular way identifies. There are four possibilities for a given URI:

the URI is being used to locate some content
the URI is being used to refer to a sense
the URI is being used to identify either content or sense but it’s not specified which
the URI is being used to both locate content and refer to a sense (ie a property applies equally to both)

Equality

Now let’s consider what happens when there is more information available about something, but it uses a different URI. The page https://en.wikipedia.org/wiki/Moby_Dick_(miniseries) is about the same TV series as http://www.imdb.com/title/tt1334573/. Imagine that this similarly made available the information that it held using OGP. It might contain:

The two pages describe the same thing: the sense referred to by the two URIs is the same. However, the content of the two pages is different. If you simply smushed the properties together, ignoring the fact that some properties apply to the content and others the sense of the resource, you’d get some data that wasn’t quite right:

{
  url: [
    'http://www.imdb.com/title/tt1334573/',
    'https://en.wikipedia.org/wiki/Moby_Dick_(miniseries)'
  ],
  title: [
    'Moby Dick (TV Series 2010)',
    'Moby Dick (miniseries)'
  ],
  type: [
    'video.tv_show'
  ],
  image: [
    'http://i.media-imdb.com/images/SFc0774313bf9ccbfe22050c8bb4029e41/imdb-share-logo.gif'
  ],
  site_name: [
    'IMDb',
    'Wikipedia'
  ],
  app_id: [
    '115109575169727'
  ]
}

Having two URLs, two titles and so on is fine, but having two site names doesn’t make sense: the og:site_name property is related to the content located by the URI, and the content is different for the two URIs. This is illustrated below.

Conversely, imagine a situation in which there is a single document on a web server that is served up from the two URIs

http://example.org/gender/male
http://example.org/gender/female

In this case, the content located by the two URIs is exactly the same, but the sense referred to by the two URIs is different: one refers to the gender ‘male’ and the other to the gender ‘female’, as illustrated here.

So there are three types of equality that we have to be concerned with:

equality between the desired sense and content of a single URI, as described earlier
equality between the senses referred to by different URIs
equality between the content located by different URIs

In general equality between the resources identified between two URIs is a controversial thing to assert, because different contexts may refer to different senses of a particular URI, some of which may be equal with a sense referred to by another URI and some not. Like any statement made about URIs, the source of statements about equality must be considered.

Formats that wish to make assertions about equality between resources should provide ways of saying that the sense referred to by two URIs is the same, without implying that the content located at those two URIs are the same, and vice versa, and to assert that the sense and content of a URI are equal. What these properties are — how exactly these kinds of equality are asserted in a given format — is up to the format, but it’s important that the properties are kept distinct to enable people to articulate the full range of equality relationships between resources.

Implications for Linked Data

I have tried to keep the description above neutral in terms of technology choice, because I believe that the issue of how to interpret URIs within data is common across all languages that use URIs. However, as I’ve discussed previously, linked data is particularly affected by these issues both because URIs form a central part of the way it works as a data format and because culturally the community tries very hard to adhere to “good web architectural practice” in the hope that this will confer long-term benefits.

For that reason, I’ll look at what I think the impacts are on linked data practice of using the “punning” approach that I’ve described above.

RDF

The definition of RDF is currently in flux, as RDF 1.1 is developed, so now is a good time to consider its use of URIs.

RDF itself is not particularly concerned with what URIs identify: it is simply a model that can be used to associate properties between “resources”, where in the RDF context this term means anything that can be the subject or object of an RDF statement, including literals. (RDF’s use of the term “resource” is not the same as that used in the URI specification or HTTPbis.) The only real limitation in RDF 1.0 Concepts, is that a hash URI identifies something described by the RDF/XML representation retrieved when the URI is resolved. In the current Editor’s Draft of RDF 1.1, the same section is less specific about what such a fragment might denote. So if the language and concepts above were adopted, RDF 1.1 should be more careful in its use of terminology, and attempt to be consistent with the URI and HTTP specifications, but I don’t think anything fundamental needs to change in the core semantics of RDF.

Vocabulary Designers

Under the “punning” approach, the property used within an RDF statement determines how the URI given as its subject or object should be interpreted. A consumer that discovers a URI by looking at the properties associated with it needs to be able to tell from the properties themselves whether it can associate those properties to particular content that it locates by requesting the URI or not.

Some properties have a defined domain or range that precludes the property from being used to annotate content. For example, the foaf:nick property has a domain of foaf:Person, and a foaf:Person cannot be a web page. Given this domain, an application can tell that the URI http://www.jenitennison.com/ used in a statement such as:


  foaf:nick "JeniT" ;
  .

cannot be being used to locate the content of http://www.jenitennison.com/, even if http://www.jenitennison.com/ responds with a 200 OK response.

Note that this inference doesn’t work the other way around. The property cc:license has a domain of a cc:Work but without additional information about the property an application could not infer that in a statement such as


  cc:license  ;
  .

the URI http://www.amazon.com/gp/product/B004TRXX7C was being used to locate the content of http://www.amazon.com/gp/product/B004TRXX7C: it could equally be being used to refer to some sense of the resource (for example the novel Moby Dick).

To support “punning”, therefore, RDF vocabulary designers would need to have additional properties that could be applied to RDF Properties to indicate how their subject (and object where applicable) should be interpreted. For example, the Creative Commons vocabulary might include (warning: made up property names and instances):

cc:license
  rdfs:subjectUri rdf:sense ;
  rdfs:objectUri rdf:content ;
  .

with the implication that URIs used as the subject of cc:license should be understood as referring to the sense of the URI, while those used as the object of cc:license should be understood as referring to the content retrieved from the URI.

Even if properties like rdfs:subjectUri or rdfs:objectUri are defined, there are going to be RDF properties for which the interpretation of subject and/or object URIs isn’t specified, and thus consumers of RDF content need to have a default interpretation. What that should be is, I think, a matter for the RDF community to decide.

Inference

The major difficulties with the “punning” approach and the current use of RDF comes when reasoning is used across RDF statements in which the same URI is used in different ways, particularly with properties where the interpretation of the subject and/or object isn’t specified.

For example, if a consumer finds the following triples at http://www.imdb.com/title/tt1334573/:


  og:url "http://www.imdb.com/title/tt1334573/" ;
  og:title "Moby Dick (TV Series 2010)" ;
  og:type "video.tv_show" ;
  og:image "http://i.media-imdb.com/images/SFc0774313bf9ccbfe22050c8bb4029e41/imdb-share-logo.gif" ;
  og:site_name "IMDb" ;
  fb:app_id "115109575169727" ;
  .

and the following triples at https://en.wikipedia.org/wiki/Moby_Dick_(miniseries):


  og:url "https://en.wikipedia.org/wiki/Moby_Dick_(miniseries)" ;
  og:title "Moby Dick (miniseries)" ;
  og:type "video.tv_show" ;
  og:site_name "Wikipedia" ;
  .

and then the assertion:


  owl:sameAs  ;
  .

then the result of inference will be that all the statements made about http://www.imdb.com/title/tt1334573/ apply equally to https://en.wikipedia.org/wiki/Moby_Dick_(miniseries), which is not the case.

To enable publishers to make assertions about equality of sense and equality of content separately, we will need new relationships. For example:


  owl:sameSenseAs  ;
  .

would only infer that those properties whose subject is the sense of http://www.imdb.com/title/tt1334573/ apply equally to the sense of https://en.wikipedia.org/wiki/Moby_Dick_(miniseries). A owl:sameContentAs property could similarly assert equality between the content of two URIs.

The impact is not limited to reasoning with owl:sameAs: all inference in RDFS and OWL is based on the assumption that a single URI identifies a single entity. This works in situations where the RDF over which inferences are being made is all trusted (for example if it is all made available by the same publisher), and a lot of current use of OWL is precisely in these kinds of closed environments. The same inferences can be made even with information gleaned from the web at large, if that information is selected carefully.

Another approach, where publishers have mixed properties about the sense referred to by a URI and those about the content located by a URI is to pre-process those RDF statements to create separate (blank node) RDF resources. For example, if og:url, og:title, og:type and og:image are defined to have a subject that refers to the sense of the URI, and og:site_name and fb:app_id to have a subject that locates the content of the URI, the statements about http://www.imdb.com/title/tt1334573/ above could be translated into:

_:imdbSubstance
  rdf:senseUri "http://www.imdb.com/title/tt1334573/" ;
  og:url "http://www.imdb.com/title/tt1334573/" ;
  og:title "Moby Dick (TV Series 2010)" ;
  og:type "video.tv_show" ;
  og:image "http://i.media-imdb.com/images/SFc0774313bf9ccbfe22050c8bb4029e41/imdb-share-logo.gif" ;
  .

_:imdbContent
  rdf:contentUri "http://www.imdb.com/title/tt1334573/" ;
  og:site_name "IMDb" ;
  fb:app_id "115109575169727" ;
  .

Here, the (putative) rdf:senseUri property is an inverse functional property that provides the URI for which the individual is a sense, and the rdf:contentUri property is an inverse functional properties that provides the URI for which the individual is the content.

This separation would then allow existing inference to take place on the separate entities.

Publishers

There are many advantages offered by the “punning” approach for linked data publishers:

it supplies an easy on-ramp for suppliers who want to annotate their pages with HTML data such as RDFa and microdata: suppliers can use URIs that they already support to refer to things other than documents, if they choose to, which means all they need to do is add metadata to their pages (as they are currently using OGP and schema.org)
suppliers do not have to have access to server configuration in order to promote the use of particular URIs to mean things that do not have content (such as people or organisations)
publishers can copy and paste URIs from the location bar of their browsers (a familiar activity for people who wish to provide a pointer to something) rather than inspecting pages for a recommended URI to be used to refer to a particular sense
organisations such as schema.org can easily recommend the reuse of URIs published by other people, such as Wikipedia, without requiring those publishers to alter their server configuration or requiring developers that use schema.org markup to add fragment identifiers to their URIs
explicit describedby and describes links can be made between URIs rather than using an HTTP status code where necessary; these can be incorporated directly in data and do not require a network connection to be discovered

Provenance

The “punning” approach that I’ve described here has as its core the recognition that different consumers will trust different sources of information to different levels. Knowledge of the provenance of a particular source of information is one way in which consumers can work out what to trust and how to resolve conflicts sources.

The work of the Provenance Working Group is important here both in identifying the provenance of particular content located at a given URI and in providing a vocabulary for describing the processing that a consumer performs to retrieve and process that content in order to extract data from it (for example, the time of the retrieval and the HTTP headers used may lead to the consumer receiving different content; the particular version of software used may lead to different information being gleaned from that content).

Linked Data Platform

The particular issues around what URIs actually identify within RDF only become an issue when the URIs are resolved — when RDF is used within linked data. The new Linked Data Platform Working Group is a great opportunity to standardise around these practices, in collaboration with the other relevant working groups.

Final Thoughts

People use the terms “resource”, “identifies” and “representation” both within specifications and in common parlance as if there is a shared understanding of what they mean, when in fact different people use the terms in subtly but meaningfully different ways. This would be fine, except that the different understandings lead to different assumptions and engineering decisions, and friction for developers trying to build applications that publish and consume data whose assumptions differ.

We need to find a way forward that, even if not everyone’s ideal, is realistic, explicable and palatable. The “punning” approach that I’ve described above might not be it, but the analysis that Jonathan’s done of the various proposals and use cases suggests to me that it’s the closest we have. The main questions I have are:

what use cases cannot be satisfied using this approach?
what specifications would have to change if this approach was adopted, and would it be realistic to make those changes?
what existing applications would break if this approach was adopted, and how might that breakage be mitigated?

At the very least, I hope that the vocabulary I’ve laid out in this post might be helpful in further discussions.

Of course any other comments are most welcome.

UK Open Standards Consultation

2012-04-14T23:44:51+01:00

Over the last few months, the UK Government has been running a consultation on its Open Standards policy. The outcome of this consultation is incredibly important not only for organisations and individuals who want to work with government but also because of its potential knock-on effects on the publication of Open Data and the use of Open Source software within public sector organisations.

Unsurprisingly, Microsoft, Qualcomm and other organisations who have a vested interest in keeping the UK Government locked in to their products are responding vociferously to the consultation. They risk not only losing business to smaller enterprises within the UK but also, if the policy is successfully adopted here, in other countries in Europe and internationally that follow suit.

If we want our Government to be Open — to use Open Standards, to publish Open Data, to adopt Open Source — then we must respond to this consultation in numbers.

There are three things that you can do:

Respond to the consultation — made even easier by this response form developed by Ric Harvey
Attend the events — these seem pretty full now, but try to get in if you can
Spread the message — blog and tweet and write to raise awareness of the importance and impact that this consultation could have

The consultation is quite long and there are a lot of questions to answer. In the hope of making this easier for everyone, I’m publishing my response below. Please consider these responses public domain, and feel free to copy as much or as little as you like from them (though I recommend you omit the parts that are about my individual experience and substitute them with your own).

For extra background, read:

Of Microsoft, Netscape, Patents and Open Standards by Glyn Moody
Are open standards a closed barrier? by Linda Humphries
Open Standards at risk by Ric Harvey

Criteria for Open Standards

1. How does this definition of open standard compare to your view of what makes a standard ‘open’?

The definition in the consultation closely matches my view of what makes a standard open. The important factors are:

a documented, open process which enables participation not just from implementers but also from users of the standard, and that provides ongoing maintenance and development of the standard
publication of the standard such that anyone can read it
a royalty-free and non-discriminatory license such that anyone can implement the standard without cost

The one factor that does not match my view is the availability of multiple independent implementations of the standard. In some cases it may be that market pressures mean there are not currently multiple good implementations of an otherwise Open Standard, or several but only for one particular platform. Limiting the definition of Open Standards to only those with multiple cross-platform implementations is probably too constraining.

There are two examples from my work that demonstrate why the availability of multiple implementations should not be a factor.

First, one of the Open Standards that I use is XSLT, which for years has been dominated by a single implementation — Saxon — giving customers no real choice. Nevertheless, because it is an Open Standard, Saxon has had a lot of pressure to be completely conformant with that standard, and in the past year a number of other implementations have been started that can compete with it on different platforms, so the presence of a single implementation has proven to be a short-term issue.

Second, in some new technology areas such as Linked Data, there may be only single implementations simply because the area and the Open Standards on which they are built are not yet very mature. As the use of the technology grows, so do the number of implementations and their adherence to the standard. Thus number and quality of implementations does develop over time; government should concentrate on long-term adoption rather than short-term availability.

2. What will the Government be inhibited from doing if this definition of open standards is adopted for software interoperability, data and document formats across central government?

In some specialist areas, there may not be existing Open Standards available, or it may be that the Open Standards that are available do not match the specialist needs of the UK government. Where an Open Standard is imminent but not yet fully standardised or implemented, waiting for standardisation or implementation could delay government IT projects. However, the measures suggested in the rest of the proposed policy, including government involvement in standards creation and allowing for selecting other standards where there is no available Open Standard, mitigate against this risk.

The other mitigating factor is that where an Open Standard doesn’t exist, it is usually possible to build new work on top of Open Standards. In my experience working on legislation.gov.uk, even though good appropriate Open Standards for UK legislation weren’t available, we have been able to work using the underlying Open Standards such as XML, RDF and HTTP, so the gap between necessary custom work and Open Standards is minimised.

The definition of Open Standards may also prevent the Government from entering into contracts with companies which do not adopt Open Standards. It may hinder exchange of information with outside organisations that use software that doesn’t support Open Standards. These may be problems in the short term but over the longer term, an Open Standards policy will move the supplier market and other organisations towards Open Standards more generally.

3. For businesses attempting to break into the government IT market, would this policy make things easier or more difficult – does it help to level the playing field?

The policy does help to level the playing field for two main reasons:

using Open Standards reduces the cost of switching to new suppliers, because the new suppliers do not have to spend a lot of time reverse engineering existing processing and data; this means new suppliers can make more competitive bids
Open Standards are often implemented within Open Source Software, which has low or no cost; this helps smaller businesses because it lowers the cost of their entering the market

4. How would mandating open standards for use in government IT for software interoperability, data and document formats affect your organisation?

I work as an independent contractor who specialises in a variety of Open Web Standards. For me as a contractor, the adoption of Open Standards within government increases the potential opportunities for me to work on government projects.

I am currently contracted to work on the delivery of legislation.gov.uk and The National Archives’ Expert Participation Programme. This work is already built substantially on Open Standards. One of the main pain points in this work has been that the government organisations that provide data such as Bills, new Statutory Instruments or Tables of Effects for legislation.gov.uk are using proprietary technologies to do so, and converting from those proprietary data formats, or getting users to save in an open data format, can be both hard to do initially and difficult to maintain as new versions of software are rolled out. An Open Standards policy within government would greatly reduce the cost involved in those conversion processes and increase the ease of use for government users.

I am also a member of the Technical Architecture Group within the World Wide Web Consortium (W3C), which is the main standards body for Web Standards. From that perspective, the Open Standards policy would increase UK government, and UK government supplier, involvement in the development of Open Standards within the W3C, which can only improve the quality of the standards and the life of the organisation as a whole.

5. What effect would this policy have on improving value for money in the provision of government services?

This policy would greatly increase value for money in the provision of government services. Adopting Open Standards often means that the basic building blocks for a service can be selected off the shelf, and then fitted together and customised, rather than a proprietary solution built from scratch. This can reduce the cost of providing the service as a whole and improve the service as development effort can be directed on the unique features of the service.

6. Would this policy support innovation, competition and choice in delivery of government services?

The policy as written is well-framed to support innovation, competition and choice across the market in the medium and long term, in areas which are beneficial to the UK Government and to the rest of the economy.

Adopting Open Standards focuses innovation on novel areas (which are not currently covered by existing standards) and on providing better quality services (which may mean better performance, better user experience and so on). It also encourages innovation in public, and this greater exposure brings with it higher quality and a better focus on user requirements. The only innovation it prevents is that whose purpose is to lock the government in to individual suppliers, such as closed standards that largely repeat existing work but can only be implemented by one supplier.

As part of the work that I did on Linked Data for data.gov.uk, myself and several colleagues worked on new RDF vocabularies such as org, Data Cube and OPMV and processing standards such as the Linked Data API. We did this in the open, and it was all based on Open Standards: that approach did not prevent us from doing new and innovative things. The results of that innovation were then taken forward by the wider community, being made more rigorous and better suited to applicability across a wider audience, and are resulting in Open Standards from W3C. In addition, my colleagues have built new products that integrate that work, for example in Kasabi.

Adopting Open Standards focuses competition on the quality of service that is offered rather than on winning a single competition that will lock the government in to contracts for many years to come. It prevents supplier complacency: when the government can move easily to a new supplier, suppliers have to provide continuous improvements over the lifetime of a contract, because they cannot be guaranteed to win the next one simply because they are the only ones who have implementations that can process the data. Competition is thus focused into areas that matter to the customer, including cost, rather than areas that matter to the supplier.

I was involved tangentially in The Stationery Office’s (TSO) bid during the recent re-procurement of legislation services by The National Archives. Because legislation.gov.uk was built on Open Standards, TSO’s bid had to be based on quality of service, on continuing innovation over the lifetime of the contract, and on low cost of delivery.

Adopting Open Standards increases choice for the UK Government because it opens up competition to suppliers who would not otherwise be able to compete (as in Question 3). Of course some companies may not currently use Open Standards, and under this policy the UK Government would not be able to choose them as suppliers in the short term, but it is unlikely that companies would continue their use of closed standards in the long term, if they wish to compete for UK Government contracts.

Put another way, adopting Open Standards means companies are innovating and competing on the right things: on things that are important to the UK Government.

7. In what way do software copyright licences and standards patent licences interact to support or prevent interoperability?

It is possible to have a Open Standard implemented by software that is public domain or completely closed or anything in between: Open Standards do not necessarily lead to free software. On the other hand, standards patent licenses reduce the ability of developers to produce free (open licensed) software because they need to make enough money to pay to use the license. Thus standards patent licenses limit the number and type of implementations of a standard, effectively limiting the market to those with enough capital to enter it.

The fewer implementations of a standard, the less pressure there is on those implementations to be interoperable, because there are a known and limited number of other implementations with which they need to interoperate. The greater the market of implementations, the greater the drive for interoperability because it becomes increasingly likely that they will have to interoperate with each other.

For example, I have often had to move code that I have written based on an Open Standard from one implementation and use it in another. If it doesn’t work, I can work out which implementation is correct (by looking at the standard) and report the error to the implementation developers, which helps them improve the interoperability of their product. The ability to move between implementations is vital to ensure interoperability between them, and the broader the market the more likely that is to happen.

8. How could adopting (Fair) Reasonable and Non Discriminatory ((F)RAND) standards deliver a level playing field for open source and proprietary software solution providers?

Adopting (F)RAND standards does not deliver a level playing field across providers, because it limits the ability of open source providers to enter the market, as they have to recoup licensing cost. Only royalty-free licenses provide a completely level playing field across providers.

9. Does selecting open standards which are compatible with a free or open source software licence exclude certain suppliers or products?

Selecting Open Standards necessarily excludes those products that only use closed standards, and suppliers that only offer those products. In the short term, suppliers who have built their products and business models around closed standards and lock-in will be excluded.

However, there is no requirement for companies that adopt Open Standards to license their products using a free or open source software license. While there are often free or open source products built around Open Standards, these are only competitive when they are at the same quality as those with closed licenses.

An example from legislation.gov.uk is that our early development used eXist, which is an XML database available under an open source software license which implemented the Open Standards of XML and XQuery. It became clear that (at that time) eXist did not support the level of use that we needed, in its performance and its scalability. We therefore instead adopted MarkLogic, which uses the same Open Standards but is not available under an open source software license. This demonstrates how companies which adopt Open Standards can still offer competitive value even within a market with open source implementations.

Another interesting point to draw from this is that the presence of an open source implementation of an Open Standard enabled us to prototype and experiment using that implementation, knowing that should we need better performance and so on we would be able to move all our code to another interoperable implementation. If MarkLogic had implemented a custom method of querying XML, committing to paying for it early in the process would have been too high a risk. So the use of Open Standards effectively helped MarkLogic win that business.

10. Does a promise of non-assertion of a patent when used in open source software alleviate concerns relating to patents and royalty charging?

I would personally be very wary of implementing a standard which had a promise of non-assertion of a patent, and standards made available under those terms would seem more risky and costly to implement because any such promise would have to be checked by a lawyer and would likely constrain my future actions and the use of the software in other environments.

11. Should a different rationale be applied when purchasing off-the-shelf software solutions than is applied when purchasing bespoke solutions?

The same policy should be applied both to off-the-shelf and bespoke solutions, particularly as the difference between these is not at all clear cut: off-the-shelf solutions are often customised, and bespoke solutions built from off-the-shelf products. Whatever the type of solution, the crucial point is that it needs to interoperate with other products, using Open Standards.

12. In terms of standards for software interoperability, data and document formats, is there a need for the Government to engage with or provide funding for specific committees/bodies?

The Government should engage with those standards bodies that work on standards that the Government uses, so that it can shape the development of those standards and highlight new areas where standards work is needed to satisfy the UK Government’s requirements. In my own area of web standards, the main one is the W3C. In fact, given the UK Government’s transparency, open data and “digital by default” policies, all which require the use of W3C standards, the UK Government is under-represented within W3C, with only two agencies (Ordnance Survey (OS) and The National Archives (TNA)) being members. Other public-sector organisations which make heavy use of W3C standards, such as the BBC, have made a business decision be become more engaged in W3C activities in order to shape and influence them.

Membership of W3C and other standards bodies is particularly important where standards development impacts on the ability to achieve the goals of particular organisations. For example, TNA are taking particular interest in the provenance work being done at W3C; the Government Digital Service should be participating in the development of standards in web design and applications; the Office of National Statistics should be taking an interest in the development of the Data Cube vocabulary for statistical information and so on.

13. Are there any are other policy options which would meet the described outcomes more effectively?

I believe that the Open Standards policy described in the consultation is the best way to achieve lower cost, higher interoperability, reduced lock-in, increased innovation and competition in the right areas and to level the playing field both for open source software and for small and medium enterprises who wish to compete for government contracts.

Open Standards Mandation

1. What criteria should the Government consider when deciding whether it is appropriate to mandate particular standards?

The only time the Government should mandate a particular Open Standard for IT is when there is a clear and apparent cost in two competing Open Standards that offer equivalent functionality being adopted by different parts of Government due to poor interoperability between them. In most cases, central government should avoid mandating a particular Open Standard, but instead let individual public-sector organisations select an appropriate Open Standard based on their own requirements.

Government should mandate a particular Open Standard where:

there are competing Open Standards that cover the given functionality and
interoperability or conversion between these standards is lossy or difficult and
there are multiple organisations within government with which interoperability involving the standard is required

2. What effect would mandating particular open standards have on improving value for money in the provision of government services?

A central government authority mandating particular Open Standards could reduce value for money in the provision of government services, because that central authority is unlikely to fully understand the requirements of the particular service in detail. Public sector bodies who are actually procuring solutions should select particular Open Standards as part of the procurement process.

Where mandating particular Open Standards could improve value for money is if there are competing Open Standards and different public sector bodies are likely to adopt different ones. In those cases, there may be interoperability problems between the standards which make it more costly to provide services, and mandating the adoption of a particular Open Standard could help.

3. Are there any legal or procurement barriers to mandating specific open standards in the UK Government’s IT?

I do not know.

4. Could mandation of competing open standards for the same function deliver interoperable software and information at reduced cost?

It is unclear what this question means.

Mandating both of two competing Open Standards may increase the availability of interoperable software, but it will raise the cost of implementation and therefore of software, because implementing two standards takes more development effort than implementing one.

Mandating a single Open Standard from two competing standards may increase interoperability but comes with a possible risk and cost. It may be that two competing standards, while similar, have different target audiences and capabilities. If one is mandated, those public sector bodies whose requirements fit more closely with the other will suffer increased cost in trying to use the first standard to fit their requirements. In addition, the standards may evolve over time such that they either diverge in functionality or increase in interoperability (so that mandation is no longer necessary) or such that the original judgement about which to mandate no longer applies.

5. Could mandation of open standards promote anti-competitive behaviour in public procurement?

Anti-competitive behaviour arises when the number of potential competitors is artificially restricted due to the terms of the procurement. Whether mandating a particular Open Standard promotes anti-competitive behaviour depends on two factors.

First, it depends on the nature of the Open Standard and its implementations. Some Open Standards are small and easy to implement while others are large and complex to implement. Some have open source implementations across a number of platforms, others have few implementations, available under restricted terms or on restricted platforms. In the short term, mandating an Open Standard that is costly for a company to implement and for which there is no easily available implementation is going to favour those companies who have already implemented the Open Standard. In the longer term, by their nature, implementations of Open Standards tend to become more widely available, and companies are likely to invest in implementing them if they are industry standards. Crucially, because the standards are Open, they are freely able to read them and can implement them without royalty payments, so in the long term it is not anti-competitive.

Secondly, the reason for the mandation of a particular Open Standard should be clear within the procurement exercise and a particular Open Standard should not be mandated unless there are clear reasons for that mandation. As an example from my own experience, there are two Open Standards for embedding metadata within HTML pages: RDFa and microdata. Either could be used to provide largely the same functionality so there would generally not be a need to mandate one or the other during procurement, but there should be a requirement to show how the standard selected by the supplier would be used to achieve the aims of the system.

6. How would mandation of specific open standards for government IT software interoperability, data and document formats affect your organisation/business?

Mandating specific Open Standards could limit the approaches that I would be able to take in my work, which could mean that I was less able to select an appropriate technology based on the requirements of the system. Any technology selection requires balancing a large set of requirements, both in terms of functionality and in terms of performance, reliability, scalability and so on. As a developer, I and my direct customers have the clearest understanding of these requirements and their relative importance within the system. These are complex choices, and they should not be made by central authorities who do not understand the details.

It is much more important to me to have a clear understanding that Open Standards should be used wherever possible and which Open Standards are being used within the organisations that interact with the systems that I am responsible for, so that the systems I build interoperate with them more smoothly.

7. How should the Government best deal with the issue of change relating to legacy systems or incompatible updates to existing open standards?

Good Open Standards provide clear statements about both forwards and backwards compatibility and, as they are developed through open participation, the extent of changes and the reasons for them are usually clear, which makes the process of working out what needs to be upgraded easier than with closed standards. For example, I was involved in the development of the second version of XSLT, and those of us in the Working Group spent substantial time ensuring that the impact on existing users of XSLT were not too great and were thoroughly documented.

When adopting a particular Open Standard within a system, the Government or their suppliers should assess the impacts of future changes in the standard on the system and should be involved to an appropriate level in the development of the standard to ensure that it continues to meet requirements.

8. What should trigger the review of an open standard that has already been mandated?

The Government should continuously work with its suppliers during the contract term and with potential suppliers during procurement to assess the best Open Standard to use. Even when a given standard itself doesn’t change, the wider IT environment may alter: there may be more or fewer implementations over time, alternative standards, or a change in interoperating standards used by other organisations. Thus there are no particular trigger points at which an Open Standard should be reviewed, though re-procurement will naturally cause a re-exploration of a system’s environment and the best approach to its implementation.

9. How should the Government strike a balance between nurturing innovation and conforming to standards?

Good Open Standards have built-in extensibility points which provide the scope for implementer innovation while providing general interoperability. In the best cases these extensions gradually become standardised themselves: this is what has happened with XSLT, for example. Thus, conforming to standards does not prevent innovation; instead it focuses innovation on user requirements, and in particular on improving the quality of implementation. The Government’s Open Standards policy should ensure that when standards are selected they provide broad interoperability while giving scope for extension, and this is particularly important if the Government mandates particular Open Standards for general use.

10. How should the Government confirm that a solution claiming conformity to a standard is interoperable in practice?

Within the W3C, new standards must have an associated test suite, usually constructed both by the Working Group who develop the standard and from the test suites created by individual implementers of the standard. Running an implementation against such a test suite makes it possible to empirically test whether there is conformance to the standard. The results of running a given implementation against such a test suite are often available, for example see the RDFa test suite results, but the Government could also ask the solution provider for evidence of interoperability in the form of test suite results.

11. Are there any are other policy options which would meet the objective more effectively?

A general Government policy to use Open Standards will in most cases naturally lead to the best Open Standard being adopted across the public sector without Government needing to mandate the use of specific Open Standards. Systems should be procured on the basis of their ability to interoperate with those other systems with which they need to work; in such an environment the easiest approach for suppliers will be to adopt the standards used by other systems.

We have seen this happen within legislation.gov.uk, where we store and publish legislation using a particular XML vocabulary. Various parliaments and government departments draft legislation which is published on the site, and they have traditionally done so based on the particular requirements of the drafters themselves (for example, the UK parliament uses Framemaker while government departments use Word). We have to cater for these different formats, and converting them into the standard that we use within legislation.gov.uk. However, as these authoring systems are being re-procured, the ability to produce the XML vocabulary that we use within legislation.gov.uk has become part of the requirements on future authoring systems, because it gives greater fidelity between authoring and eventual publication. Thus we are naturally moving towards a more interoperable environment without any central mandation of the standard and without overriding the local requirements of particular authors.

International Alignment

1. Is the proposed UK policy compatible with European policies, directives and regulations (existing or planned) such as the European Interoperability Framework version 2.0 and the reform proposal for European Standardisation?

I do not know.

2. Will the open standards policy be beneficial or detrimental for innovation and competition in the UK and Europe?

The UK’s Open Standards policy will be beneficial to innovation and competition in the UK because it levels the playing field for a wider set of providers and focuses innovation and competition on the requirements of the users of software rather than the suppliers. These benefits apply as well to Europe as to the UK, and the successful adoption of an Open Standards policy within the UK will naturally aid the adoption of similar policies in other countries within Europe and internationally.

3. Are there any are other policy options which would meet the objectives described in this consultation paper more effectively?

The one part of the international alignment policy that needs to be rethought is the preference to international standards over local standards. While in general the wider and greater adoption of a given standard, the better, it is not always the case that international standards provide greater benefits than local ones.

Looking at providing data about legislation, for example, different jurisdictions have very different ways of identifying, creating, revising and formatting legislation. Any international standards for legislation are highly unlikely to take into account the complexities and special cases that are specific to UK legislation (such as the use of regnal years for identifying older items). A local standard can be better tailored to the requirements of the locality, which may be incredibly important in easing implementation cost and data fidelity.

I would therefore recommend a policy approach that emphasised interoperability with international standards without requiring their wholesale adoption, particularly where there are specific local requirements that are not met by the international standard.

Content and Descriptions of Web Resources

2012-03-31T22:27:08+01:00

Those readers who follow the TAG or public-lod mailing lists over the last couple of weeks cannot have failed to notice a large number of posts on a theme that recurs on roughly a 9-monthly cycle within these communities: httpRange-14.

The reason for this particular recurrence was a Call for Change Proposals on the resolution. The TAG meets on Monday, and discussion of this issue is one of the first items on our agenda. These are my thoughts going in to that discussion.

The Questions

The recent discussion on the lists has, I think, helped to refine the questions that lie at the core of the httpRange-14 issue. They are:

When you get a successful response from a URI, does that response, by definition, include the content of the resource identified by the URI?
How can you discover a description of the resource identified by a URI?

Knowing whether the response to a URI provides the content of the resource identified by that URI is important because when you have data about the thing identified by a URI, such as its author or the license that it is provided under, you need to know what information is actually being referred to so that you can tell what information you can reuse and whom you have to attribute.

For example, the GOV UK website has a license at the bottom of each page:


  Much of the information on this website is available for reuse under the 
  Open Government Licence

Seeing this, an application that knows the Open Government License enables free reuse can tell that it can lift content out of the page and use it on their own site. An application could automatically scrape out and republish the first paragraph of those news stories provided on this site and any others that were published with under this license.

The Conflict

There are vocal disagreements about particularly the first of the two questions I outlined above. What’s become clear to me is that the source of the arguments stem from a difference in world view about what kind of resources are available on the web.

Web of Data

Under the web of data view, the web consists of data, and all the resources on the web are information resources, defined as those resources whose essential characteristics can be conveyed in a message. Data, in other words.

URIs can still be used to name other resources, which are not on the web either because they are not information resources (such as a Person) or because they are not available yet (such as unscanned books). Under this world view, however, giving a successful HTTP response for such a resource is simply wrong, because these resources aren’t on the web.

The problem that this world view therefore needs to address is how to create URIs to identify resources that aren’t on the web. There are two answers:

Hash URIs

Hash URIs have the benefit that there is a direct relationship between the hash URI which identifies the resource and a resource on the web that describes it. An HTTP client naturally strips the fragment identifier from the URI in order to make the request to a server, which then delivers the description of the resource.

303 Redirections

If you identify a resource that isn’t on the web using an HTTP URI that is not a hash URI, you cannot get a successful response back because the resource you have asked for is, by definition in this world view, not on the web. The workaround is for the publisher to use the 303 See Other status code to point from the resource that you requested to its description on the web. (This is the essence of the httpRange-14 resolution.)

Web of Things

Under the web of things view, the web consists of things, and resources on the web could be anything: documents, people, films, teapots and so on. When a client makes an HTTP request for a resource, the response must reflect the state of the resource, but that state could be its content (if it’s an information resource) or it could be a description of the resource.

Under this world view, giving a successful HTTP response for a resource that isn’t an information resource is absolutely fine: the description of the resource is still a reflection of its state.

The problem that needs to be addressed when you have this world view becomes apparent when you think back to the licensing example above. Given an application knows that the resource identified by a given URI can be reused, how does it know whether the representation of that resource is reusable? It could be that the identified resource (for example an out-of-copyright book) has an open license, but that the representation of the resource holds only a description of that resource (some metadata about the book), and that description has a much more restrictive license. Or vice versa.

So to address this use case, you need some other mechanism to enable an application to tell that the representation is the content of the resource, rather than merely a description of it.

The Current State

To generalise, the linked data community operates within the web of data world view and the larger web community operates within the web of things world view.

What is happening increasingly, however, is that these two world views are rubbing up against each other, and while both are internally coherent, switching between the world views causes not only a cognitive disconnect for developers but practical problems when transforming or moving data published under one world view into the other world view.

In addition, publication of data on the web through APIs is growing all the time, particularly REST APIs supporting the Hypertext as the Engine of Application State (HATEOAS) principle. As we share more data on the web, and we use URIs in our APIs, the question of what those URIs mean and how we associate licenses and provenance information with data, will only become more important.

We have an obligation, therefore, to reflect on the experience from the linked data community over the last few years and how that experience might spread to the larger web community.

Discussions within the linked data community over the httpRange-14 resolution centre on two problems that people have encountered:

the web of data vs web of things disconnect hurting adoption
practical aspects of responding to requests with 303 redirections such as
- round-trip delays, particularly as in HTTP 1.1 303s can’t be cached
- inability for people to use 303s without server admin access

The Social Context

A bit of a side-point here. I think that the questions I posed at the start of this post are general questions about web architecture, so it puzzles me that the only people who seem to really care about them, and who debate them endlessly, are the linked data community. This is partly because the linked data community use URIs extensively to identify the things about which they provide data, but I think it’s also about the fundamental attitude of those within the community, which was characterised in a recent post by Hugh Glaser:

Personally, I never did agree with the solution [to httpRange-14], but have always aimed to carry out the implications of it in the systems I construct.

This is for two reasons:
a) as a member of a small community, it is destructive to do otherwise;
b) as a professional engineer, my ethical obligations require me to do so.

It is this second, the ethical obligations that are the most significant.
I should not digress from the standards, or even Best Practice, in my work.

The linked data community is jam packed with people who feel an ethical obligation to adhere to standards and best practices. We try to do what we are told is the Right Thing by individuals and standards organisations even when we don’t agree that it is the Right Thing and even if it turns out to be impractical.

In the larger web community, people who don’t agree with a standard or best practice, or who find it too impractical to implement, simply ignore it. There is no need to endlessly debate something that you can just ignore. And the httpRange-14 resolution is ignorable by the larger web community because so far it has had very little impact on any implementations at all, let alone widely-deployed implementations that work over the non-linked-data web.

The Choices

Going into the TAG meeting about this on Monday, the main decision that I see is whether to continue to assume a web of data world view. In the web of data world view, it is impossible for a URI to return a description of a resource, whereas in the web of things world view it is fine. Personally, I would prefer to design around the web of things world view as I think this would ease some of the disconnects between linked data and the wider web, but there are others on the TAG who adhere strongly to the web of data view, so I think that change is unlikely.

If we stick with the web of data view, the main issues are how to alleviate the current practical difficulties that people are encountering with its implementation and explanation. I think there are three measures that would help:

Determine a conventional syntax for fragment identifiers that are used to identify things that are not on the web, as opposed to fragments of content. I’m thinking something like hash-bang URIs: using a character after the hash character that just gives a quick indication that the fragment identifier is being used in a special way, to refer to something that isn’t on the web rather than a fragment of a document, for example #*.
Change to recommending a single best practice of using hash URIs for resources that aren’t on the web, and in particular recommending having a one-to-one correspondence between resources on the web and those not on the web, using one particular conventional hash URI. For example, http://www.whitehouse.gov/#* would identify the resource that http://www.whitehouse.gov/ is about: The Whitehouse. This ensures that new publishers of data won’t run into the problems with publishing using 303 redirections, because they won’t use that method of publication. It also removes choice, which helps adopters who can otherwise get overwhelmed with options and the trade-offs between them.
Allow publishers who are currently using 303 redirections to publish descriptions of resources identified using non-hash URIs to switch to providing a representation using a 200 status code, along with a method of indicating that the representation is the description of the resource rather than its content. This indicator could be:
- a new HTTP header or status code (though I’d prefer not)
- a Link: header with a particular relationship (eg ‘describedby’)
- a statement embedded in the response itself (eg a element in HTML)

If we did move to a web of things view, the main question would be how to provide an indicator that the representation of a particular resource is the content of that resource as opposed to being a description. It would help ease transition if this was a natural consequence of the current pattern of publication on httpRange-14-compliant sites, so for example, you’d want to consider the representation of a resource the content of the resource if you got to it:

when retrieving a hash URI, if it was the part of the URI before the hash
when following a 303 See Other redirection, if it was the target of the redirection
when following a ‘describedby’ link, if it was the target of the link

as well as if there was an explicit indicator within the representation that said the resource was an information resource.

Whichever decisions are made, I would personally like to see the concrete requirements on client behaviour that arise from these different publication practices, for example enabling a reuser to associate a license with a particular piece of content or a crawler to create RDF statements about URIs encountered on the web, to bring whatever decisions are made down to earth and less ignorable.

Precious Snowflakes

2012-03-10T11:56:36+00:00

Disclaimer: As usual, this post contains my personal opinion and does not reflect that of any organisation with which you might associate me.

The other day, I had a lovely conversation with some folks from the BBC about some of their future plans. In the course of the conversation, Michael Smethurst spoke about his frustration when dealing with people involved with particular programmes at the BBC, where every single one of them thinks their programme is a “precious snowflake”, completely unique, that simply can’t be treated in the same way as all the other programmes described on the site.

Michael’s point, of course, is that TV programmes have a hell of a lot of similarities with each other. They all have episodes and cast members and may have trailers or be available on iPlayer. When the BBC models them in the same way, they gain enormous efficiencies in their ability to store and access information about programmes: they can reuse code, share content between programmes, and perform analyses over the aggregated data set. It’s great for users too: they get the same fantastic user experience no matter which programme they are viewing information about, and can apply the experience they gain when navigating pages about one programme when they need to find information about another.

The ability to classify and categorise, to bring order to what seems like chaos, to create a model of the world, is one of the things that marks humans from animals. We can look at a hundred people, with different colour hair and skin; different height and build; smiling, talking, crying, and still call them all Person because the essential characteristics that govern how we interact with them are the same.

But if there’s one thing that the last five long, hard years working with legislation has taught me, it’s that in any vaguely interesting domain, this search for order will always fall down in the face of reality.

Surely, I thought in my naive early days, every piece of legislation is uniquely identified through its type, calendar year, and number? Not so! There are six items for which this is not the case, because prior to 1963 legislation was numbered based on the year of reign of the monarch rather than the calendar year.

Surely the year that is used to number legislation is dependent on the date it is made and written into law? Not so! Sometimes departments forget to register legislation they make until the following year, so it is numbered the year after it’s made.

Surely an item of legislation can only make changes to legislation from the day it is written into law? Not so! There is, rarely, legislation that rewrites history: that says other legislation should always have had different content to that which was originally written.

It has come to the point where I never (hah!) make any statements about legislation of the form “X never happens” or “Y is always true” because there is always, always, an exception.

What this has taught me, as a developer, is the power and necessity of escape hatches. For example, templating languages that provide a method of escaping to code are so much more valuable than those that do not. Similarly, I favour strongly, in the technologies that I use, the ability to extend a common data structure, be it through data-* attributes in HTML, through generic elements such as and

or through the essentially open-ended nature of RDF as a data model.

It has also given me a very different view of the world to Michael. Because when you accept that there are always exceptions, you do not see snowflakes as merely crystals of water, but as exceptional, beautiful and, yes, immensely precious.

And this is why I love the web. The web does not force every site to have the same structure or the same look and feel. It does not insist on consistency; it has space for every quirk. And it proves beyond all doubt that it is possible for all these precious snowflakes to exist in a single, global, interlinked information system in which people manage to find not only the information that they need, but also community and connection with each other.

Inside Government

So it is with these eyes that I look at the new Inside Government pages on the gov.uk site and am frankly horrified. Because we’re not just talking about a BBC programmes here, but about powerful institutions, many of them decades if not centuries old, that lie at the very heart of government and how our nation is run. And each of them is relegated to a subfolder of a subfolder of a subfolder, their unique histories and approaches and goals expressed through three pictures on a carousel.

It feels like some kind of Orwellian nightmare: the relentless focus on user needs leading to a future of identikit pages, with no individuality, no character, no clue that behind these pages — which, remember, under the Single Government Domain policy becomes the single authoritative view, the site that represents the department on the web — is a living and breathing institution that manages hugely important parts of our lives. A future in which what each department says and the way that it says it is governed through the Government Digital Service (GDS), in Cabinet Office, the hand of the prime minister. And now we’re talking about local government too?

Let us just look at one example. Last September, William Hague gave a speech in which he described the hollowing out of the Foreign and Commonwealth Office (FCO) by the previous government, a process that scrapped its language school, closed embassies and destroyed its library. He said:

Strong institutions are necessary in civil society, to encourage participation and keep in check an overmighty State; they are necessary to our judiciary and Parliament so that the law is upheld and the making of it respected; but they are also necessary within the State, a point tragically overlooked by those Prime Ministers who have created and abolished departments on a fancy or a whim, destroying as they did so the pride and continuity of thousands of public servants while rendering government incomprehensible to the average citizen. The whole country should know what the Foreign and Commonwealth Office is and what it does, and all those interested in foreign policy at home or abroad should see it as a centre of excellence with which they aspire to be associated.

For most UK citizens, the only point of access to the Foreign and Commonwealth Office is its website: they will not visit King Charles Street, nor any of the UK’s embassies. The department’s web presence is the only way that it makes itself, and its unique role, comprehensible to the average citizen, the only method of letting the whole country know what the FCO is and what it does. And they have content that is completely unique to them: a database of Treaties, a hugely rich set of information on travel and living abroad and a wealth of historical information about the Foreign Office. This simply doesn’t fit in a model of a department as a set of Ministers, Policies, Publications and so on. And if it doesn’t fit, will it simply be excluded, lost from its website like its language school, its embassies, its library?

I could have picked any government department here — each one has its unique characteristics and content — but Hague articulates the case around FCO so well. His message is not the expression of a simple conservative impulse to resist change and preserve the status quo, but about maintaining the integrity of an institution’s identity and independence to encourage participation and keep in check an overmighty State. If we believe in Open Government, Open Democracy and the power of the web to enhance civic engagement then we must, surely, enable each of these institutions to have their own independent voice on the web.

I am reminded of the XKCD:

The two sides of this Venn diagram illustrate two approaches to building a website for an organisation. On the left is the website as an expression of the identity of the institution, on the right the website as a means of satisfying the reason the user originally visited the site. My argument is not that the right side of this diagram is unimportant — in fact I believe it is absolutely essential — but that an institution’s website must cover the entire space: it must provide a mechanism for self-expression as well as catering for its user’s requirements. To enhance civic engagement, we do not need to simply answer the query that led the user to the site, but to encourage and lead them on to see more about the institution that has provided the answer.

It is only the institution itself that knows the self it wants to express, and because the real world is complex and organisations are unique, that self will not fit into any model that we devise. News, Policies, Consultations — of course these are all important to all departments, but they are the tip of an iceberg. Look at the space that FCO devotes to its history on its website: this shouts to the world the kind of reliable, solid and flexible organisation that they are and want to remain. Compare how DECC devotes space to statistics, emphasising its adherence to transparency and evidence-based policy. Self-expression is so much more than changing logos or backgrounds, more than having different content on an About page, it is about making space for the things that are important to you.

“But but but!” I know the arguments. We must cut costs, stop the uncontrolled proliferation of government websites; we must improve the quality of the government’s presence on the web, present a unified view, make it easy for users to locate content without knowing where to look. The vision we see expressed through Inside Government is but the natural conclusion, the end of that slippery slope. But it is the great slippery slope fallacy that everything must be taken to its natural conclusion, that because 750 websites is too many, one is enough.

Possibly the biggest irony of the gov.uk beta is that while it is delivering a Single Government Domain — everything is to be found under www.gov.uk — it does not seem to address the core reason stated for providing it. In Martha Lane-Fox’s letter to Francis Maude, which kicked off this whole endeavour, she said:

Government publishes millions of pages on the Web, via hundreds of different websites. Most of these sites are still run as silos within departments. This fragmentation leads to significant duplication of functions and technology, and means the overall user experience is highly inconsistent.

Try searching for ‘Single Government Domain’ on the main gov.uk site or for driving test centres on Inside Government. The searches do not work (though Inside Government does give you a link that enables you to search the other silo). The result pages are completely different in feel except for the top and bottom banners. The page on Arrest and Imprisonment Abroad mentions but does not link to the Foreign and Commonwealth Office’s page. Yes, yes, I know that it’s still beta, but these things lie at the heart of the stated rationale for a Single Government Domain: is this the extent of the consistency and integration that we are aiming for?

Yes, it is, because the Single Government Domain policy was never truly about either of these things. Read Martha Lane-Fox’s letter again carefully (my emphasis):

No1O feel it is preferable to go from 750 top level website domains (eg www.cabinetoffice.gov.uk) to a single top level website domain for all of central government.

The Single Government Domain policy, indeed GDS itself, is about control. It is “we will do it for you”, not “we will help you do it”. It is about managing the output of institutions that might keep in check an overmighty State. It is anti-web and it is anti-democracy and I cannot remain quiet about it any longer.

To my friends at GDS: I respect and admire you all. You are incredibly talented and able to do amazing things. You have behind you a level of financial and political support the like of which most civil servants will never see. I know you have joined GDS not just to do work that you love but to do good for the country. This is my plea to you: find a way to avoid this vision. Nurture the exceptions. Give institutions their voice. Treat them as precious snowflakes.

Microdata and RDFa Living Together in Harmony

2011-08-20T17:39:11+01:00

One of the options that the TAG put forward when it asked the W3C to put together task force on embedded data in HTML was the co-existence of RDFa and microdata. If that’s what we’re headed for, what might make things easier for consumers and publishers who have to live in that world?

In a situation where there are two competing standards, I think that developers — both on the publication and consumption sides — are going to want to hedge their bets. They will want to avoid being tied to one syntax in case it turns out that that syntax isn’t supported by the majority of publishers/consumers in the long term and they have to switch.

Publishers like us at legislation.gov.uk who are aiming to share their data to whoever is interested in it (rather than having a particular consumer in mind) are also likely to want to publish in both microdata and RDFa, rather than force potential consumers to adopt a particular processing model, and will therefore need to mix the syntaxes within their pages.

(Of course developers might just avoid embedded data altogether while they wait to see what happens, but let’s assume that they want to press ahead regardless of the lack of consensus from the standardistas.)

I’ve therefore embarked on a task of trying:

to identify the differences in approach and functionality of the two languages, which should help developers choose between them
to identify any guidelines for developers of vocabularies for use with both languages
to identify a subset of functionality that is common between the two languages, which developers might want to stick to to make switching and mixing easier
to identify mapping rules that might be applied to automatically or manually map from one language to another if the simple subset is used

I’ve done this by looking at converting microdata examples to RDFa and vice versa, and the lessons to be drawn from that exercise. I’ve broken down the result into three posts:

This is the last of these posts. It is probably the only one you will want to read :)

Please treat this as a draft on which I’d welcome comments. I have based what’s written here on the latest specifications of both microdata (in its WHAT WG and W3C variants) and RDFa Core and HTML+RDFa but I haven’t consulted with anyone involved in these efforts and may well have got things wrong. Plus the specs are changing all the time. I have only here considered the syntax of the two languages, not the features such as DOM APIs or drag-and-drop support, where there are also clear differences.

Please add comments if there are things that I’ve missed or got wrong, or just to have your say.

Philip Jägenstedt’s Live Microdata service and Gregg Kellogg’s Distiller service have both proved invaluable for testing — thank you to both for making these services available. I heartily recommend them.

Mapping Rules

The first problem is how to judge equivalence when microdata and RDFa have different data models. Microdata essentially uses a JSON data model: there are objects (items) with properties that have values that are strings, other objects, or arrays of strings or objects or both. RDFa naturally uses a RDF data model: there are resources with properties that have values that are literals (of some datatype or with a language) or other resources.

Underlying both is the same basic entity-attribute-value pattern, but there are various mismatches between the models that make some mappings more complicated than others, or in other cases mean that information is necessarily lost on conversion.

In performing the analysis, I’ve tried to map microdata into sensible RDF and then match that RDF output using RDFa, and to map RDFa into sensible microdata+JSON and then match that microdata+JSON using microdata. The microdata-to-RDF mapping rules that I’ve followed are basically those outlined in my post on Microdata + RDF. To create microdata JSON from RDFa, I’ve used the rule that the URI of the first type of a resource is processed to provide a namespace that is stripped from the URIs of the properties (to create simple names where possible). In addition, when a resource has no properties, it will be represented as a string (URI) value rather than as a nested item.

These rules need to be formalised, obviously, but the basics above work well enough for the examples from the specs.

Mismatched Features

The following features are problematic when mapping from microdata to RDFa or vice versa. I’ve described them roughly in an order from things where it might be relatively easy to address the problem by changing one or other specification, to places where the necessary changes would be difficult to make in the specs, which means that publishers and consumers need to be aware of the issue so that they can make an educated choice about how they proceed.

Local Property Names

Many of the microdata examples involve items with no type and local property names. I’ve assumed in the analysis below that this generates properties whose URI is based on the document in which they are found, but this is not a helpful solution for data sharing: if a whole site uses short property names across its pages, those properties really need to be recognised as being the same across the site for any kind of useful processing to occur.

What microdata actually creates here is a global namespace, shared by everyone, specifically for embedded data. There are three things that could be done at different levels here:

In a mapping from microdata to RDF, any short property names on items that don’t have a type could be assigned to a global namespace (eg http://w3.org/ns/global/). Of course there will be clashes in semantics within this namespace, but that is true in microdata generally and not having to create a new namespace makes the initial experimentation easier for those starting with embedded data. The W3C (or whoever operates the namespace) could operate a wiki at that location that would operate as an informal registry for the property names.
HTML+RDFa could change to use this global namespace as the default vocabulary URI (rather than not having one). This would make it a little easier for people to convert microdata to RDFa: if they don’t use types for their items, there would then be no need for a vocab attribute to be added to the HTML. It also makes it possible to use RDFa in a basic, lightweight way, which might help people get started with it.
Publishers can be advised to use itemtype within their microdata, reusing existing classes or creating their own, if they want to ensure that the embedded data within their pages isn’t misinterpreted by global consumers.

Interpretation of Element’s `datetime` Attribute

Interpreting the datetime attribute of the element to supply a value, rather than repeating that value in a content attribute, is ISSUE-97 on RDFa, and hopefully RDFa will be changed to use that value (or the content of the element if there is no datetime attribute), add a seconds component if necessary, and work out an appropriate date/time datatype for it based on its syntax.

Content Overrides

In RDFa, publishers can provide a machine-readable version of the content of an element (or even an entirely different value) using the content attribute. This can only be done for date/times in microdata. The ability to annotate non-date/time content with machine-readable values is a current issue on HTML5. Resolving this in favour of providing such annotation would make using RDFa and microdata in concert, or converting between them, easier, particularly if HTML5 uses the attribute content or RDFa adopts whatever attribute is introduced to HTML5.

and Elements in Flow Content

The ability to use and elements in flow content is only supported in microdata: it’s support that’s added by the microdata specification (in the Editor’s Draft since May 31st; the text allowing this didn’t make it into the Last Call version of the spec), in which it’s limited to and elements with an itemprop attribute.

It would be possible for the RDFa specification to similarly make the statement that and elements are allowed in flow content as long as they have particular attributes. This would ease the transition between the two formats, and works a lot better than empty elements which crop up fairly commonly in RDFa content.

(One oddity here is that because date/time values have to be on a element in microdata, publishers cannot replace empty elements with elements as they might an empty .)

Identifiers without Types

Many of the RDFa examples are of resources that have a URI identifier but for which no type is supplied. Microdata, on the other hand, states that itemid is only allowed on elements that also have an itemtype (and an itemscope). The reason given is because the itemid needs to be interpreted based on the itemtype. This would be understandable if it held a string, but given that the itemid provides a URI it seems a bit strange. Perhaps it’s an attempt to avoid the whole httpRange-14 / ambiguity in URIs issue.

If this restriction remains, the advice to RDFa users who might want to convert to microdata at a future date would be to always provide a type for your (non-blank-node) resources. It may be useful to define a http://w3.org/ns/global/Thing within the vocabulary that I propose above, given that the URI for rdfs:Resource is long and hard to recall.

Built-in Prefixes

The built-in profile for RDFa defines a number of prefixes for vocabularies that are either coined by the W3C or coined elsewhere but in common use on the web. This, coupled with vocab and the ability to directly use URIs in the relevant attributes, means that declaring prefixes within the document is increasingly unnecessary in RDFa.

In contrast, using existing vocabularies, even popular ones, within microdata is relatively difficult, particularly when vocabularies are mixed on the same item.

Most useful for publishers would be if both RDFa and microdata recognised the same set of prefixes. This would reduce the size of microdata created from existing RDFa content as well as making it easier to move between the languages. At the very least, it would be good to have rdf:, rdfs:, xsd: and xhv: built into both.

The list of popular vocabularies is likely to change over time; for example a prefix for the schema.org vocabulary might be useful at some point in the near future. The problem is that publishers and consumers need to be synchronised in their use of prefixes: it’s no good for a publisher to use the prefix sch: if there might be processors for the page that don’t recognise it. Equally, consumers shouldn’t be reliant on a network connection to retrieve the latest set of prefix mappings in order to parse the page. It’s not clear to me how best to manage this evolution, but even a fixed set of prefixes at the point the specs reach Recommendation is more usable than spelling out URIs all the time.

Literals Including Markup

RDFa supports literals that include markup (the innerHTML of an element) as well as those that don’t (the textContent of an element), whereas microdata only supports creating values from particular attributes or the textContent of the element. This makes it hard to create embedded microdata that includes values which contain things like mathematical or chemical formulae, ruby text, or multiple paragraphs.

A solution would be for microdata to introduce an itemhtml (or something) attribute that, when present, indicates that the value of the property should include markup. There is a current issue on microdata to support HTML values.

Itemref

RDFa can support a subset of itemref’s functionality, namely to have properties defined elsewhere in a document be associated with a given resource. What it doesn’t support is the sharing of properties defined in one place by two or more resources.

RDFa could add such support by adding an attribute that mirrors itemref (eg ref, I guess), with the referenced element being processed using the evaluation context inherited by the referencing element (which means that attributes such as vocab would sometimes have a scope that wasn’t based on the document tree). This would make it easier to tackle the use case for itemref using RDFa as well as making it easier to move between or mix RDFa and microdata.

Lists

It is easy for microdata to represent a property with a list of values, and really really hard to do the same in RDFa. This is in part because RDF views lists resources rather than a distinct data type, and in part because RDFa hasn’t added any syntax sugar to make creating rdf:List resources easy. Adding some syntax sugar for lists would make life a lot easier for anyone using RDFa, but especially if they are adapting existing microdata content to RDFa.

Datatypes

Microdata assumes that consumers will convert values to appropriate datatypes based on the property (which they understand) as a separate process after microdata processing, whereas RDFa supports the use of a datatype attribute to explicitly indicate the datatype of each value. This mismatch means that information is lost when RDFa is converted to microdata, and has to be added when microdata is converted to RDFa.

Bringing the languages completely into sync would mean either microdata adding a facility to support (at least some) datatypes, or deprecating the datatype attribute in RDFa. Alternatively, this may simply be an area where the differences in behaviour between the two specifications doesn’t matter because the data models that they use are distinct anyway.

Languages

Languages are similar to datatypes, in that RDF (and hence RDFa) supports annotating strings with the language that they are in whereas microdata doesn’t within its core data model or its JSON serialisation. However, the elements that represent properties within the HTML, used within the DOM API access to microdata, will have a language.

It may be that in practice consumers need to base their microdata processing on the DOM API rather than the core microdata data model or JSON extracted through a standalone process, and thus pick up the language from the property elements, I don’t know. In any case, the microdata JSON serialisation, used for drag-and-drop, is lossy and could be extended to include the language of each value when available, at fairly substantial complexity cost.

For publishers, it doesn’t much matter either way; if they are dealing with multi-lingual text they will want to include a lang attribute in the HTML anyway, regardless of the impact on embedded data.

Multiple Types

RDFa supports having multiple types named in the typeof attribute whereas microdata only supports one type per item. In any mapping from RDFa to microdata, publishers have to choose which type is the primary type for the item and move the others to be expressed via rdf:type properties. Consumers who want to support publishers who might not choose their type as the primary type have to detect items that have the type they are interested in within the rdf:type property as well as those which have the type as the main type. Given that the rdf:type URI is long and (naturally) associated with RDF, it might be better to define a property such as http://w3.org/ns/global/type for this use.

Microdata could be extended to allow multiple values in the itemtype attribute, with the first being used to interpret any properties that aren’t full URIs. This would make it easier for both consumers to detect when a type they were interested in was used and for publishers to use RDFa and microdata in tandem or move between them.

The `src` Attribute

RDFa and microdata interpret the src attribute in opposite ways. In RDFa, it provides the identifier for a new resource (equivalent to itemid in microdata); in microdata, it provides a URL value of a property on elements that support it (equivalent to resource or href in RDFa).

RDFa interprets src in this way to make it easier to make assertions about an image, but it’s of limited effect as even in RDFa its only possible to make three such assertions (through the typeof, rel and property attributes). So, for example, you can specify the type of the image, link to its license and give the name of its creator, with:

but this won’t help you if you also want to give the title for the image and when it was created (say). At that point, the microdata and RDFa start to look similar:

or:

and really, to make the markup consistent, you may as well not use the src of the image at all in the RDFa either:

So it’s not clear to me that interpreting the src attribute as the subject of triples offers such a huge advantage that it’s worth the inconvenience that it brings for the simple things, such as having to use:

rather than:

Link relations

This isn’t so much a clash between RDFa and microdata as between the interpretation that RDFa has for the rel attribute and that specified in HTML.

The built-in rel values in HTML are a bit of a mix. Some of them, like alternate, prev and next encode relationships between the document in which the link appears and another document. Others, such as bookmark and help, create relationships between the context in which the link is found and the referenced document. Still others, like nofollow, noreferrer and prefetch, are really instructions to the client about how to manage the act of traversing the link.

It doesn’t seem semantically correct to automatically create relationships based on the built-in HTML rel values, unless you are deliberately trying to extract document semantics from the page. This is a problem for RDFa, which reuses the rel attribute to provide property values for the embedded data.

One thing that could be done would be for RDFa to consistently use the property attribute everywhere rather than the rel attribute. This would not only ease the overloading but also reduce the confusion for users, who currently have to work out which attribute to use based on whether the value is a resource or a literal.

Possible Subset of RDFa

When mapping from microdata to RDFa, the only attributes that are really needed are:

vocab to define a vocabulary for the types and properties within its scope (not technically necessary, but keeps the markup simple compared to spelling out URIs for everything)
typeof to define the type of a resource or indicate a new blank node
about to provide a URI for a resource or a local identifier for a blank node
property and rel to define property names (though see above for discussion about dropping rel)
href, src and content to provide values (and datetime assuming that is supported)

In the mappings in the analysis below, I did also use the resource attribute, but only to create a reference to a blank node that was described elsewhere, when replicating the functionality of itemref. If RDFa were to enable and in content in the same way as microdata, resource functionality could be replicated using ; as it is, you can get away with using an empty element.

Similarly, I only used datatype when providing a datatype for date/time values, something that could be done automatically by RDFa. But this isn’t surprising given that microdata doesn’t support datatypes at all and the examples I was using for the mapping were from the microdata specification.

There was no need for:

prefix which defines prefixes to simplify references to properties and classes; this is hardly surprising as few of the microdata examples involved mixing namespaces, but it’s notable that the built-in prefixes of rdf: and xsd: were useful
profile which is a pointer to an external document that defines a set of terms; this is being dropped from RDFa in any case

I also kept to a simplified version of the syntax in which each property element only provided one value. This subset is basically:

resource elements can have about (equivalent to itemid) and typeof (equivalent to itemtype) attributes on them
property elements can have property or rel (equivalent to itemprop), and a value-providing attribute on them such as href or content
no element is both a resource element and a property element; to provide a property whose value is a resource, nest the resource element within the property element (using “hanging rel” processing)
no property element should provide more than one value for a property; in particular, a “hanging rel” should only have a single resource element child

This simplified profile of RDFa is fairly easy to remember and maps easily to and from microdata: most attributes can be simply renamed; the only attribute that needs to be moved as well as renamed is the “hanging rel”, which moves onto the resource element and is renamed to itemprop. Note that it also means avoiding using the src attribute to encode embedded data.

In addition to sticking to this subset of attributes, developers might be advised that using HTML link relations may lead to clashes with browser or search engine interpretation of the links in the page.

Possible Subset of Microdata

Microdata is pretty minimalistic already. The only feature that developers need to be warned about is itemref, which has no RDFa equivalent at the moment.

Guidelines for Vocabulary Authors

There are a several guidelines that come out of this comparison for people putting together vocabularies that aim to be usable in both RDFa and microdata:

The classes in the vocabulary should be distinct, or subclasses created with any relevant combinations of superclasses, so that publishers don’t have to assign more than one type to an item/resource. This restriction helps with using the vocabulary with microdata, which assumes that every item has a single type.
Provide explicit classes for everything which you anticipate might be given an identifier, as microdata doesn’t (currently) enable items to have an identifier without also having a type.
Put classes and properties in the same namespace, but do not name classes and properties with the same local name; while this doesn’t matter in microdata because the properties are interpreted relative to the class, standard conversions to RDF will create a class and a property with the same URI. URIs are case-sensitive to a simple way of ensuring that there aren’t clashes is to follow the usual RDF convention of beginning class names with an upper-case letter and property names with a lower-case letter.
Avoid property names that contain dots, as these aren’t allowed in non-URI property names in microdata.
Ensure that properties either only expect one type of value or expect values whose type can be sniffed based on the syntax of the value. If publishers use microdata, they will not be able to indicate the type of a value through the markup.
Be aware that consumers of microdata using your vocabulary will have to use the DOM API to identify the language used in any strings, and that language information won’t be carried through the standard microdata JSON serialisation (used by drag-and-drop, for example). If you anticipate multi-lingual use of your vocabulary, you may way to define a MultiLingual class with value and language properties that people can use as nested items. (It may be useful for this class and properties to be defined in the proposed ‘global’ W3C namespace so that it can be used anywhere.) If you know what languages will be used then provide separate properties for each language (eg for UK legislation I know the languages are English and Welsh so on a vocabulary for UK Legislation I could have title-en and title-cy properties).
To make markup cleaner, only reuse properties from other vocabularies on your classes if they have built-in prefixes (eg unless rdfs: is built-in to microdata as well as RDFa, don’t use rdfs:label to provide a label, but create your own label property). On the other hand, do reuse classes from other vocabularies if you don’t need to add any specialised properties to them. Note that avoiding reuse has the unfortunate side-effect of not enabling processors that understand these other vocabularies to process your data.
Avoid having properties whose values need to be retrieved in order, as these are hard to represent in RDFa. Instead, use properties with distinct names when position is important. (Yes, I know this sucks.)

Choosing Between Microdata and RDFa

The choices developers make between microdata and RDFa will, I suspect, be largely dictated by what their consumers/toolsets/publishers will support. Nevertheless, there are some features that are better supported by one or other format and might therefore sway developers one way or another:

multi-lingual embedded data is better supported in RDF than microdata+JSON
explicit datatypes for values can be provided by RDFa but not microdata
resources with multiple types are a lot easier to describe in RDFa
property values that include markup are a lot easier to write in RDFa
mixed vocabulary use is a bit easier in RDFa than in microdata
HTML5 link relations may be misinterpreted by RDFa processors
properties with list values are much easier to support in microdata
common content adopted by multiple entities is much easier in microdata

Final Words

I have no doubt that developers would be better off if there were only one recommended way of embedding data in HTML (so long as it met their requirements of course). But realistically that is, and always has been, a long shot, given the entrenched positions of the microdata and RDFa communities.

Regardless, there are lessons that RDFa and microdata could learn from each other, and changes to both languages that would help developers use them on their own, switch between them and mix them in the same document. I expect and welcome debate about the viability and effectiveness of the changes and guidelines that I’ve suggested here.

Investigating those lessons, documenting those changes and generating those guidelines was something that I had hoped the microdata/RDFa task force would be able to do. The other question to ask, given the argument that there shouldn’t be a task force at all if it’s not going to be able to bring the languages together, is whether this kind of analysis is worthwhile, and worth publishing as something more official than a blog post?

Mapping RDFa to Microdata

2011-08-20T17:38:38+01:00

This post is part of a three-part series that analyses the differences in features and syntax between microdata and RDFa. The series attempts:

to identify the differences in approach and functionality of the two languages, which should help developers choose between them
to identify any guidelines for developers of vocabularies for use with both languages
to identify a subset of functionality that is common between the two languages, which developers might want to stick to to make switching and mixing easier
to identify mapping rules that might be applied to automatically or manually map from one language to another if the simple subset is used

I’ve done this by looking at converting microdata examples to RDFa and vice versa, and the lessons to be drawn from that exercise. The three posts are on:

This post is the second of these, which looks at how RDFa might be mapped to microdata. In this case, I will aim to express the RDF created from the RDFa as the equivalent microdata JSON, and aim to create that JSON with the microdata.

To create the microdata JSON, I’ve used the rule that the URI of the first type of a resource is processed to provide a namespace that is stripped from the URIs of the properties (to create simple names where possible). In addition, when a resource has no properties, it will be represented as a string (URI) value rather than as a nested item. Other than that I hope the mapping will be obvious; I’ll point out where it involves a loss of information. I’m assuming that the document is at http://example.org/ throughout.

I have based what’s written here on the latest specifications of both microdata (in its WHAT WG and W3C variants) and RDFa Core and HTML+RDFa but I haven’t consulted with anyone involved in these efforts and may well have got things wrong. Philip Jägenstedt’s Live Microdata service has proved invaluable for testing, so many thanks to him for providing that service.

The post is rather heavy going and you might want to just skip to the summary instead of reading the whole thing.

The post goes through the examples from the RDFa specification plus one additional example from the wild. I haven’t included examples that don’t illustrate anything new, so there are some that are skipped. Other examples would be welcome.

Page Metadata

When parsing begins, the current subject will be the IRI of the document being parsed, or a value as set by a Host Language-provided mechanism (e.g., the base element in (X)HTML). This means that by default any metadata found in the document will concern the document itself:
  
    Jo's Friends and Family Blog
    
    
  
  
    ...
  

The equivalent microdata JSON for this document that we’d want to create is (note: invalid example):

{ "items": [
  {
    "id": "http://example.org/" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://example.org/#bbq" ],
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  }
]}

and we’d want to create it with (note: invalid example):


  
    Jo's Friends and Family Blog
    
    
  
  
    ...

However, this is is not valid according to the microdata specification. In microdata, only items that have types are allowed to have identifiers. Rather than losing the identifier, we’ll add a type; I’m going to use rdfs:Resource. It’s not the nicest of URIs to type, but it’s got something close to the correct semantics. So we’ll aim for the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://example.org/#bbq" ] ,
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  }
]}

which means we need:


  
    Jo's Friends and Family Blog
    
    
  
  
    ...

Notes:

The itemscope is necessary for the page to be recognised as containing any data at all.
The itemid can’t just be empty: the . is the shortest URI you can use to reference the page itself.
I put the itemscope, itemtype and itemid on the element rather than the element so that they wouldn’t be inherited into the : it seems to make sense for any data within the to be about the page itself.
The foaf: and dc: prefixes are built-in to RDFa, so it’s easy for people to use classes and properties in those common vocabularies without having to remember their full URI. In microdata, that URI and the one for the rdfs:Resource class have to be spelled out in full.

Base URI

In (X)HTML the value of base may change the initial value of current subject:
  
    
    Jo's Friends and Family Blog
    
    
  
  
    ...
  

This changes the id of the item generated:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://www.example.org/jo/blog" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://www.example.org/jo/blog#bbq" ] ,
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  }
]}

In the microdata, the itemid can still be . as the base URI set by the element is used to resolve it:


  
    
    Jo's Friends and Family Blog
    
    
  
  
    ...

Explicit Subjects / ItemIds

To illustrate how this affects the statements, note in this markup how the properties inside the (X)HTML body element become part of a new calendar event object, rather than referring to the document as they do in the head of the document:
  
    Jo's Friends and Family Blog
    
    
  
  
    
      I'm holding
      
        one last summer barbecue
      ,
      on
      
        September 16th at 4pm
      .
    
  

In microdata JSON, this would be:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://example.org/#bbq" ],
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  } ,
  {
    "type": "http://www.w3.org/2002/12/cal/ical#Vevent" ,
    "id": "http://example.org/#bbq" ,
    "properties": {
      "summary": [ "one last summer barbecue" ] ,
      "dtstart": [ "2015-09-16T16:00:00-05:00" ] ,
    }
  }
]}

Note that this mapping loses the fact that the value of the dtstart property is a date-and-time. Processors of this JSON are expected to know that the dtstart property takes a date/time value and would have to sniff the value to work out that it’s a date-and-time rather than a date.

In-browser microdata processors can identify the value as a date/time value because the property element itself is accessed through the element.properties IDL attribute; processors that work with this DOM API can tell that it’s a element, get hold of the date/time itself and access the content of the element for the human-readable representation used on the page. However, this information isn’t part of the core microdata data model.

To create this JSON from microdata you need:


  
    Jo's Friends and Family Blog
    
    
  
  
    
      I'm holding
      
        one last summer barbecue
      ,
      on
      
        September 16th at 4pm
      .

Notes:

There are no prefix definitions in microdata, so the type has to be spelled out in full. However, with the mapping I’m assuming from RDFa to microdata JSON, the properties in that same namespace for items in that class don’t.
The itemscope has to be added despite the
element having both an itemid and an itemtype; if the itemscope is forgotten, the item isn’t recognised.
The original element has to be changed to a element because it isn’t conformant microdata for a date/time value to be supplied by any other element.

Items from the `src` Attribute

If @about is not present, then @src is next in priority order, for setting the subject of a statement. A typical use would be to indicate the licensing type of an image:

This should generate the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/photo1.jpg" ,
    "properties": {
      "http://www.w3.org/1999/xhtml/vocab#license": [ "http://creativecommons.org/licenses/by/2.0/" ] ,
    }
  }
]}

The src attribute in microdata is only used for a value, so creating the microdata about the image means a wrapper element and a separate element:

Note:

The license property is part of the built-in set of link relationships in HTML, but there is no easy way to refer to that property from microdata; they have to be spelled out as full URLs.

Additional Properties for Images

Since there is no difference between @src and @about, then the information expressed in the last example in the section on @about (the creator of an image), could be expressed as follows:

This is a simple additional property in the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/photo1.jpg" ,
    "properties": {
      "http://www.w3.org/1999/xhtml/vocab#license": [ "http://creativecommons.org/licenses/by/2.0/" ] ,
      "http://purl.org/dc/terms/creator": [ "Mark Birbeck" ]
    }
  }
]}

which can be created through a element within the :

Nested Images

Since normal chaining rules will apply, the image IRI can also be used to complete hanging triples:
  

This should generate the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://www.blogger.com/profile/1109404" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/img": [{
        "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
        "id": "http://example.org/photo1.jpg" ,
        "properties": {
          "http://www.w3.org/1999/xhtml/vocab#license": [ "http://creativecommons.org/licenses/by/2.0/" ] ,
          "http://purl.org/dc/terms/creator": [ "Mark Birbeck" ]
        }
      }]
    }
  }
]}

The microdata to generate this is:

Notes:

The big gotcha in this conversion is that in microdata, the foaf:img property has to be moved onto the item that is a value of that property; there’s no equivalent to the “hanging rel” processing in RDFa. A disadvantage of this is that anyone copying-and-pasting the element to embed the same information about the image within their own page will have the itemprop attribute carried along with the image, into a context where the foaf:img property might not be relevant.

Types with Blank Nodes

For example, an author may wish to create markup for a person using the FOAF vocabulary, but without having a clear identifier for the item:
  Albert Einstein
  Albert

Now we have an explicit type, we can create microdata JSON that uses short names:

{ "items": [
  {
    "type": "http://xmlns.com/foaf/0.1/Person" ,
    "properties": {
      "name": [ "Albert Einstein" ] ,
      "givenName": [ "Albert" ] ,
    }
  }
]}

This can be generated with the microdata:


  Albert Einstein
  Albert

which is nice and simple.

Inherited Subject

The most usual way that an inherited subject might get set would be when the parent statement has an object that is a resource. Returning to the earlier example, in which the long name for the German_Empire was added, the following markup was used:
  Albert Einstein
  1879-03-14
  
  the German Empire

The equivalent microdata JSON for this would be:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Albert_Einstein" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
      "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ] ,
      "http://dbpedia.org/property/birthPlace": [ "http://dbpedia.org/resource/German_Empire" ]
    }
  } , {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/German_Empire" ,
    "properties": {
      "http://dbpedia.org/property/conventionalLongName": [ "the German Empire" ] ,
    }
  }
]}

Note that this microdata JSON could only be generated syntactically from the RDFa, not via RDF, because going via RDF would make it impossible to know whether to give the dbp:birthPlace property a string (which is a URI) value or a nested item. We’ll see the alternative version of the microdata RDF in the next example.

To create this microdata JSON, we need:


  Albert Einstein
  1879-03-14
  
  
    the German Empire

Notes:

I’ve had to change two elements here: the for the date of birth has become a element as the value of the property is a date, and the
for the birth place has become a element because the value of that property is a URL.
I’ve also had to add a nested element as it’s not possible in microdata to have a single element describe both an item and a property for that item as it is in RDFa.

In an earlier illustration the subject and object for the German Empire were connected by removing the @resource, relying on the @about to set the object:
  Albert Einstein
  1879-03-14
  
    the German Empire
  

While this generates the same RDF as the previous example, the microdata JSON that it generates should probably be different: this time, the item for the German Empire is nested within the item for Albert Einstein:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Albert_Einstein" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
      "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ] ,
      "http://dbpedia.org/property/birthPlace": [{
        "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
        "id": "http://dbpedia.org/resource/German_Empire" ,
        "properties": {
          "http://dbpedia.org/property/conventionalLongName": [ "the German Empire" ]
        }
      }
    }
  }
]}

To create this, the microdata needs to look like:


  Albert Einstein
  1879-03-14
  
    the German Empire

Note that while this looks quite similar to the RDFa version, in fact the itemid attribute that holds the URI for the German Empire is on a different element from the about attribute in the RDFa.

The third RDFa example around this same content is:

but it is also possible for authors to achieve the same effect by removing the @about and leaving the @resource:
  Albert Einstein
  1879-03-14
  
    the German Empire
  

This should lead to the same microdata JSON, so I won’t bother repeating the microdata. What’s interesting is that this pattern: the wrapper element containing the property (rel) and identifier for the item that is the value for that property (resource) is a lot closer to the microdata pattern of expressing nested items. The big distinction here is that while in microdata, the itemtype also resides on that element, if you tried adding a typeof attribute to the inner

in RDFa, you’d end up with a new blank node.

Anonymous Nested Resources

However, an author could just as easily say that Spinoza influenced something by the name of Albert Einstein, that was born on March 14th, 1879:
  
    Albert Einstein
    1879-03-14
  

This should generate the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Baruch_Spinoza" ,
    "properties": {
      "http://dbpedia.org/ontology/influenced": [{
        "properties": {
          "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ]
        }
      }]
    }
  }
]}

which again means moving an attribute in microdata:


  
    Albert Einstein
    1879-03-14

It is generally harder to move to microdata from RDFa when the RDFa has an element that both provides a subject and provides a property.

The RDFa spec provides a couple of additional methods of marking up the same content to give exactly the same RDF (and microdata JSON):

Note that the div is superfluous, and an RDFa Processor will create the intermediate object even if the element is removed:
  Albert Einstein
  1879-03-14
An alternative pattern is to keep the div and move the @rel onto it:
  
    Albert Einstein
    1879-03-14
  
From the point of view of the markup, this latter layout is to be preferred, since it draws attention to the ‘hanging rel’. But from the point of view of an RDFa Processor, all of these permutations need to be supported.

Interestingly, it’s this latter permutation that is the one that’s closest to the microdata method of expressing the data, though as we will see in the next section, the “hanging rel” is not exactly equivalent to the itemprop on the wrapper element.

Hanging Rels

Note that each occurrence of @about will complete any incomplete triples. For example, to mark up the fact that Albert Einstein had a residence both in the German Empire and Switzerland, an author need only specify one @rel value that is then used with multiple @about values:
  
  

The data embedded here gives two values for the dbp-owl:residence property:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Albert_Einstein" ,
    "properties": {
      "http://dbpedia.org/ontology/residence": [
        "http://dbpedia.org/resource/German_Empire" ,
        "http://dbpedia.org/resource/Switzerland"
      ]
    }
  }
]}

In microdata, the itemprop attribute has to appear on both the nested elements to make it clear that they both provide values for that property:

The next example illustrates this with nested items rather than strings:

To illustrate, to indicate that Spinoza influenced both Einstein and Schopenhauer, the following markup could be used:
  
    
      Albert Einstein
      1879-03-14
    
    
      Arthur Schopenhauer
      1788-02-22
              
  

This should generate:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Baruch_Spinoza" ,
    "properties": {
      "http://dbpedia.org/ontology/influenced": [{
        "type": "http://xmlns.com/foaf/0.1/Person" ,
        "properties": {
          "name": [ "Albert Einstein" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ]
        }
      }, {
        "type": "http://xmlns.com/foaf/0.1/Person" ,
        "properties": {
          "name": [ "Arthur Schopenhauer" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1788-02-22" ]
        }
      }]
    }
  }
]}

In this case, the itemprop that is equivalent to the RDFa rel has to move down onto the elements representing the items:


  
    
      Albert Einstein
      1879-03-14
    
    
      Arthur Schopenhauer
      1788-02-22

The wrapper

around both items isn’t necessary; I’ve left it to stay as close to the markup of the original RDFa as possible.

Implicit Resources

Triples are also ‘completed’ if any one of @property, @rel or @rev are present. However, unlike the situation when @about or @typeof are present, all predicates are attached to one bnode:
  Albert Einstein
  1879-03-14
  
    
    
  

To be equivalent to the RDF generated from this markup, the microdata JSON would be:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Baruch_Spinoza" ,
    "properties": {
      "http://dbpedia.org/ontology/influenced": [{
        "properties": {
          "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ]
          "http://dbpedia.org/ontology/residence": [
            "http://dbpedia.org/resource/German_Empire" ,
            "http://dbpedia.org/resource/Switzerland"
          ]
        }
      }]
    }
  }
]}

Microdata is a lot more explicit about when items get created, and consequently requires a bit more markup:


  
    Albert Einstein
    1879-03-14

Overriding Text Content

The value of @content is given precedence over any element content, so the following would give exactly the same triple as shown above:
John Doe

The equivalent microdata should generate the JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://internet-apps.blogspot.com/" ,
    "properties": {
      "http://purl.org/dc/terms/creator": [ "Mark Birbeck" ]
    }
  }
]}

Only the element and links override the content of an element in microdata. So a mirror of this example needs a separate element:

  
    
    John Doe

Language Support

In RDFa the Host Language may provide a mechanism for setting the language tag. In XHTML+RDFa [XHTML-RDFA], for example, the XML language attribute @xml:lang or the attribute @lang is used to add this information, whether the plain literal is designated by @content, or by the inline text of the element:

Like the datatype of a value, the language of a value isn’t captured by the microdata data model or the JSON representation of that data model. So the fact that ‘chat’ is French is lost:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/node" ,
    "properties": {
      "http://example.org/property": [ "chat" ]
    }
  }
]}

The equivalent microdata is thus:

with the language only accessible if you are using the DOM to process the microdata.

Literals that Include Markup

RDFa therefore supports the use of normal markup to express XML literals, by using @datatype:
  E = mc²: The Most Urgent Problem of Our Time

The datatype="rdf:XMLLiteral" acts like a flag to indicate that the serialised content of the element (innerHTML) needs to be used as the value of the property, rather than the textContent, which includes markup, can be expressed in microdata JSON as follows:

{ "http://purl.org/dc/terms/title": "E = mc²: The Most Urgent Problem of Our Time" }

There’s no way to generate this in microdata except by repeating the escaped version of the content in a content attribute:


  E = mc²: The Most Urgent Problem of Our Time

This is hardly ideal. It’s tedious enough with a short string like this one; for larger amounts of information such as long descriptions of an event, it would be very tedious.

The `resource` Attribute

RDFa provides the @resource attribute as a way to set the object of statements. This is particularly useful when referring to resources that are not themselves navigable links:
  
    On Crime and Punishment
    
  
  
    
      
        Rodion Romanovitch! My dear friend! If you go on in this way
        you will go mad, I am positive! Drink, pray, if only a few drops!
      
    
  

This should produce:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://www.example.com/candp.xhtml#q1" ,
    "properties": {
      "http://purl.org/dc/terms/source": [ "urn:ISBN:0140449132" ]
    }
  }
]}

which is expressed through:


  
    On Crime and Punishment
    
  
  
    
      
      
        Rodion Romanovitch! My dear friend! If you go on in this way
        you will go mad, I am positive! Drink, pray, if only a few drops!

The property and value have to be moved onto a nested , but this is a more extensible pattern than the RDFa method as it enables other properties to be expressed in the same way.

Multiple Types

This last example comes from the wild rather than being an example in the specification. At http://bitmunk.com/browse we find:

Bitmunk

This shows the use of multiple types and of multiple properties with the same value, because the pages are attempting to use multiple vocabularies that cover the same domain (organisations) to different depths. In the equivalent microdata, we have to choose one of the types; I’m going to assume that it should just use the first one from the typeof attribute:

{ "items": [
  {
    "type": "http://www.w3.org/2006/vcard/ns#VCard" ,
    "id": "http://bitmunk.com/about#service" ,
    "properties": {
      "http://www.w3.org/1999/02/22-rdf-syntax-ns#type": [
        "http://purl.org/commerce#Business" ,
        "http://purl.org/goodrelations/v1#BusinessEntity"
      ] ,
      "http://www.w3.org/2000/01/rdf-schema#label": [ "Bitmunk" ] ,
      "fn": [ "Bitmunk" ]
    }
  }
]}

The microdata equivalent is:


  
  
  Bitmunk

Notes:

Technically, the RDFa doesn’t place any ordering on the three classes, but I’m picking the first for the purpose of the microdata conversion. The other classes are harder to get at in the JSON: they have to be referenced via the rdf:type microdata property rather than the type JSON property. Consumers that are on the lookout for items of the type gr:BusinessEntity wouldn’t spot these items.

Mapping Microdata to RDFa

2011-08-20T17:35:28+01:00

This post is part of a three-part series that analyses the differences in features and syntax between microdata and RDFa. The series attempts:

to identify the differences in approach and functionality of the two languages, which should help developers choose between them
to identify any guidelines for developers of vocabularies for use with both languages
to identify a subset of functionality that is common between the two languages, which developers might want to stick to to make switching and mixing easier
to identify mapping rules that might be applied to automatically or manually map from one language to another if the simple subset is used

I’ve done this by looking at converting microdata examples to RDFa and vice versa, and the lessons to be drawn from that exercise. The three posts are on:

This post is the first of these, which looks at how microdata might be mapped to RDFa, in terms of generating the same RDF according to the microdata-to-RDF mapping rules that I outlined in my post on Microdata + RDF.

I have based what’s written here on the latest specifications of both microdata (in its WHAT WG and W3C variants) and RDFa Core and HTML+RDFa but I haven’t consulted with anyone involved in these efforts and may well have got things wrong. Gregg Kellogg’s Distiller service has proved invaluable for testing, so many thanks to him for providing that service.

The post is rather heavy going and you might want to just skip to the summary instead of reading the whole thing.

The post goes through the examples from the microdata specification (most of them are in both versions, the only exceptions being those that use the vCard vocabulary). I haven’t included examples that don’t illustrate anything new, so there are some that are skipped. Other examples would be welcome.

Unidentified Items / Blank Node Subjects

Here there are two items, each of which has the property “name”:
 My name is Elizabeth.



 My name is Daniel.

The first challenge is to map this into RDFa because the properties are tokens rather than URIs and there is no type for either of the items. What I’ll assume here is that the name properties are local to the document itself and thus the equivalent RDF is:

[ <#name> "Elizabeth" ] .
[ <#name> "Daniel" ] .

This can be achieved in RDFa through either:


  My name is Elizabeth.



  My name is Daniel.

or:


  My name is Elizabeth.



  My name is Daniel.

Notes:

The vocab="#" sets the vocabulary to the location of the current document (plus an empty fragment identifier); this URI is then concatenated to the property token (name) to create a URI that is unique to the document. In a document such as this it would make sense to put the vocab="#" attribute on the element rather than on every single item.
With no type in sight, blank nodes can either be created by having an empty typeof attribute or through an about attributes whose value starts with _:. The latter has the advantage of providing an identifier for the blank node that can be used elsewhere in the document, but the former is shorter so will be used where possible in the remaining examples of this post.

Values from the `src` Attribute

The next example introduces the use of the src attribute to set the value of the property.

In this example, the item has one property, “image”, whose value is a URL:
 

This should probably be mapped to the RDF:

[ <#image>  ] .

The difficulty with this is that in RDFa, the src attribute is used for the subject of a statement (equivalent to a microdata item) rather than the object (equivalent to a microdata value). So we have two choices for equivalent RDFa. One is to use a similar pattern to that used above, but introduce a wrapper element that provides the property:

Another is to provide what would normally be an object through a resource attribute and then use a rev attribute (rather than the usual rel) attribute to reverse the relationship:

This has three disadvantages over the first option:

the resource attribute that creates the item is on the element rather than on the wrapper
which makes it hard to create other properties for that item
we have to use a rev attribute, reversing the normal flow of relationships; I (at least) find this hard to figure out when there’s not a rel attribute as well
we have to make up an id for the blank node we want to generate

I’ll note that it took me five or six failed attempts to generate the above options. If I hadn’t had the RDF Distiller to test with, I would have got it wrong. Note that at least through the RDF Distiller, to be recognised, the resource attribute has to have an (empty) value — it is not enough for it to simply be present, unlike with the typeof attribute. Note that the resource attribute has to explicitly point to a blank node to create a blank node rather than having the property be associated with the document in which this appears.

Values from the `datetime` Attribute

The next example illustrates the use of the element to provide a date/time value for a property.

In this example, the item has one property, “birthday”, whose value is a date:
 I was born on May 10th 2009.

I’m assuming this should map to the RDF:

[ <#birthday> "2009-05-10"^^ ]

There is an open issue (ISSUE-97) about this on RDFa, which currently requires the use of the content attribute to provide the value as follows:


 I was born on May 10th 2009.

Note that the xsd: prefix is built-in within RDFa so there’s on need for any declaration for it, which makes it fairly easy to specify the standard date/time datatypes.

If ISSUE-97 were resolved nicely it would be possible to instead do:


 I was born on May 10th 2009.

Notes:

To make this work, RDFa processors would have to look at the syntax of the datetime attribute to work out what datatype the value should be matched to.
The syntax permitted in the datetime attribute isn’t exactly the same as that permitted by the XML Schema time and dateTime types usually used in RDF (and XML), in that the seconds component is optional within HTML. The resolution to ISSUE-97 will need to take this into account. Otherwise, anyone mapping from microdata to RDFa manually will need to ensure that the content attribute includes the seconds component.

Nested Items / Object Properties

In this example, the outer item represents a person, and the inner one represents a band:
 Name: Amanda
 Band:  Jazz Band (12 players)
The outer item here has two properties, “name” and “band”. The “name” is “Amanda”, and the “band” is an item in its own right, with two properties, “name” and “size”. The “name” of the band is “Jazz Band”, and the “size” is “12”.

The equivalent RDF for this example would be:

[
  <#name> "Amanda" ;
  <#band> [
    <#name> "Jazz Band" ;
    <#size> "12"
  ]
]

Note that the size property is just a plain literal value; unlike with date/times, there’s no way to tell from the microdata that the value is a number.

In RDFa this could be done with:


 Name: Amanda
 Band:  Jazz Band (12 players)

This follows the microdata fairly closely but note that the nested resource doesn’t need an empty typeof attribute: it’s only the top-level items that do. It might be easier, for consistency and extensibility, to always include an explicit nested element (with an empty typeof attribute in this case) to represent the nested resource:


 Name: Amanda
 Band:  Jazz Band (12 players)

The other thing that people have to watch out for is that because the value of the band property is a resource rather than a literal, we have to use the rel attribute rather than the property attribute as we do elsewhere.

Itemref

This example is the same as the previous one, but all the properties are separated from their items:
Name: Amanda


 Band: Jazz Band
 Size: 12 players

This should create the same RDF as the previous example:

[
  <#name> "Amanda" ;
  <#band> [
    <#name> "Jazz Band" ;
    <#size> "12"
  ]
]

changing the markup as little as possible. The RDFa equivalent is:


Name: Amanda


 Band: Jazz Band
 Size: 12 players

In microdata, the itemref attribute is a method of an item adopting name/value pairs described in a separate location within the page. In RDFa, the equivalent is to say that the name/value pairs are all related to the same resource by consistently referring to the resource as the subject of the statements. In the above case, there are two blank nodes labelled _:amanda and _:c, and the about attribute is used on the same elements that provide the properties (or a wrapper element) to indicate the identity of the subject of the statements.

Notes:

The resource attribute has to be used to indicate the blank node for the band.
As before, the rel attribute has to be used for the band property, rather than the property attribute, because the object of the statement is a resource. The rule is that if you’re using resource, you should use rel. (I used property erroneously the first time I tried to write this mapping. I will never learn.)

There is another example of itemref in use later in the microdata specification:



 
  Photo gallery
 
 
  My photos
  
   
   The house I found.
  
  
   
   The mailbox.
  
  
   All images licensed under the MIT
   license.

This is equivalent to the RDF:

[
  a  ;
    ;
    ;
   "The house I found." ;
] .
[
  a  ;
    ;
    ;
   "The mailbox." ;
] .

Note that the license property is adopted by both the items in the microdata. In this particular example, the two items have the same type, and thus the license property has the same meaning in each item. It’s also possible for itemref to be used on two items that have different types, pointing to the same element, in which case the shared properties defined within that element could mean different things for the two items.

There is no way that I am aware of within RDFa to support shared use of portions of content. There could be a rough equivalent that would work in the case where the shared properties had the same semantics if RDFa allowed the about attribute to take multiple values (note: invalid example):



 
  Photo gallery
 
 
  My photos
  
   
   The house I found.
  
  
   
   The mailbox.
  
  
   All images licensed under the MIT
   license.

but this wouldn’t support the possibility of the same property having different semantics (and therefore different URIs) for the separate resources.

It’s also worth noting in this example that the mapping to RDF that I’m assuming results, in this example, in http://n.whatwg.org/work being both a class and a property. The creators of RDF vocabularies tend to name classes with an Uppercase initial letter and properties with a lowercase initial letter, and thus avoid these kinds of clashes. Vocabulary designers who are mindful of mappings to RDF may want to take the same approach.

Multiple Values

This example describes an ice cream, with two flavors:
 Flavors in my favorite ice cream:
 
  Lemon sorbet
  Apricot sorbet
 
This thus results in an item with two properties, both “flavor”, having the values “Lemon sorbet” and “Apricot sorbet”.

This example highlights one of the real nightmares of RDF: lists. In microdata, the order of the values ‘Lemon sorbet’ and ‘Apricot sorbet’ is naturally retained. There are three possible mappings to RDF.

Creating Multiple Statements

If the order of the flavours of ice cream in this example don’t actually matter, the equivalent RDF is:

[ <#flavor> "Lemon sorbet" , "Apricot sorbet" ]

which is equivalent to:

[ <#flavor> "Apricot sorbet" , "Lemon sorbet" ]

In this case, the RDFa is straight-forward:


 Flavors in my favorite ice cream:
 
  Lemon sorbet
  Apricot sorbet

It’s surprising how common it is that order doesn’t actually matter when there are multiple values for a property, such that this mapping is quite sufficient. But I’m absolutely not going to pretend that order is never important…

Creating an `rdf:Seq`

If the order of the flavours does matter, there are two ways of representing that order using RDF. The first is to use an rdf:Seq resource. This method was the original method of representing lists in RDF and is very natural to do in RDF/XML, but has largely fallen out of favour for the second method which I’ll describe below.

Using the rdf:Seq method, the equivalent RDF for the microdata would be:

@prefix rdf:  .
[
  <#flavor> [
    a rdf:Seq ;
    rdf:_1 "Lemon sorbet" ;
    rdf:_2 "Apricot sorbet"
  ]
]

which can be generated with:


 Flavors in my favorite ice cream:
 
  
   Lemon sorbet
   Apricot sorbet

Notes:

There are various other ways in which the namespace for the rdf:Seq could be created, but since the rdf: prefix is built-in to RDFa 1.1, it seems easier to use that than anything that explicitly writes out the full (ugly) RDF namespace.
The
wrapper for the
Creating a rdf:List

The current recommended way to create a list in RDF is to use a rdf:List resource. This essentially uses a linked list model to represent lists, with the rdf:first item of a list being a value and the rdf:rest being either another rdf:List or rdf:nil. Spelled out, the RDF would look like:
```
@prefix rdf:  .
[ 
  <#flavor> [
    a rdf:List ;
    rdf:first "Lemon sorbet" ;
    rdf:rest [
      rdf:first "Apricot sorbet" ;
      rdf:rest rdf:nil
    ]
  ]
]
```
but of course Turtle lets you write it:
```
[] <#flavor> ( "Lemon sorbet" "Apricot sorbet" ) .
```
Unfortunately, RDFa has no such syntax sugar. Which means:
```
 Flavors in my favorite ice cream:
 
  
   Lemon sorbet
   
    
     Apricot sorbet
     
    
   
  
 
```
Yep, horrific. Verbose and easy to get wrong, and that’s just for two items. If a third is added, the pattern is to add an about attribute on the middle items of the list so that the rdf:rest property which covers the next item in the list can be assigned to it. For example:
```
 Flavors in my favorite ice cream:
 
  
   Lemon sorbet
   
    
     Apricot sorbet
    
   
   
     
     Raspberry sorbet
     
     
   
  
 
```
Note:
- I’ve used an empty element with a href attribute to point to the rdf:nil resource. An alternative would be to use the resource attribute, which would have the advantage of not having to spell out the full URI for rdf:nil, but I’m trying to stick to using as few attributes as possible.
- Using an empty element for a link isn’t ideal; it would be neater to use a element, but these aren’t allowed in flow content within HTML5 ( and are only permitted within the microdata specification, and then only if they have an itemprop attribute). The RDFa specification could likewise allow them.
Multiple Properties Sharing a Value
Here we see an item with two properties, “favorite-color” and “favorite-fruit”, both set to the value “orange”:
```
 orange
```
This should map to the RDF:
```
[
  <#favorite-color> "orange" ;
  <#favorite-fruit> "orange"
]
```
Like itemprop, property can take multiple values, so the RDFa equivalent is simply:
```
 orange
```
Types
Here, the item’s type is “http://example.org/animals#cat”:
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
In this example the “http://example.org/animals#cat” item has three properties, a “name” (“Hedral”), a “desc” (“Hedral is…”), and an “img” (“hedral.jpeg”).
I’ll assume that this should be mapped to the RDF:
```
[
  a  ;
   "Hedral" ;
   "Hedral is a male american domestic shorthair, with a fluffy black fur with white paws and belly." ;
   
]
```
In this case, the vocab can be set to http://example.org/animals# and both the itemtype and the various property and rel attributes will use that as the basis for their identifying URIs:
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
Global Identifiers
Here, an item is talking about a particular book:
```
 Title
 
The Reality Dysfunction
 
Author
 
Peter F. Hamilton
 
Publication date
 
26 January 1996
```
Here, the item has an identifier so unlike the previous examples, the subject of the statements in the RDF is no longer a blank node:
```
@prefix xsd:  .

  a  ;
   "The Reality Dysfunction\n " ;
   "Peter F. Hamilton\n " ;
   "1996-01-26"^^xsd:date ;
  .
```
In RDFa, the subject is provided using the about attribute:
```
 Title
 
The Reality Dysfunction
 
Author
 
Peter F. Hamilton
 
Publication date
 
26 January 1996
```
Global Property Names
Here, an item is an “http://example.org/animals#cat”, and most of the properties have names that are words defined in the context of that type. There are also a few additional properties whose names come from other vocabularies.
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
The RDF equivalent to this is:
```
[
  a  ;
   "Hedral" ;
   "Hedral" ;
   "Hedral is a male american domestic shorthair, with a fluffy black fur with white paws and belly." ;
   "black" , "white" ;
   
]
```
To create this, we need the RDFa:
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
Link Relations
Here is an example of a page that uses the vEvent vocabulary to mark up an event:

... Bluesday Tuesday: Money Road ... May 5th @ 7pm (until 9pm) ...Link to this page ... Location: The RoadHouse ... ...

This example is interesting because it contains, in the natural markup of the page, a rel attribute with the value bookmark, which is used for links that go to the page or section of the page within which the link is found. In this case, it’s the page. The RDF that should be generated from the page is:
```
[
  a  ;
   "Bluesday Tuesday: Money Road" ;
   "2009-05-05T19:00:00Z"^^xsd:dateTime ;
   "2009-05-05T21:00:00Z"^^xsd:dateTime ;
    ;
   "The RoadHouse" ;
   "via livebrum.co.uk"
] .
```
The following statement could legitimately be generated as well:
```
<> 
    ;
  .
```
but the item representing the event should definitely not have the http://www.w3.org/1999/xhtml/vocab#bookmark property.

Achieving this without significantly changing the HTML markup is problematic in RDFa because RDFa uses the rel attribute to provide properties for the resources that it describes within the page, overloading its standard use in HTML which is to describe properties of the page or sections within the page. The following involves the least amount of repetition:
```
 
  ...
  Bluesday Tuesday: Money Road
  ...
  May 5th @ 7pm
  (until 9pm)
  ...
  
  Link to this page
  ...
  Location: The RoadHouse
  ...
  
  ...
  
 
```
Notes:
- In the above, the typeof attribute has been moved onto a wrapper
  that encompasses the entirety of the page because if it resides on the element, it’s assumed to apply to the document itself rather than a blank node. An alternative mapping would use about="_:event" to create a blank node for the event.
- There’s no way to avoid creating a statement for the rel="bookmark" link, so the best we can do is make sure that the statement is accurate, and relates the current document to the provided URI. Unfortunately, that means creating a separate element for the url property, repeating that URL within the page, and adding an empty about attribute; here I’ve used an empty element to express the relationship; a element would do the same job if it were allowed in flow content.