Jeni's Musings

Using "Punning" to Answer httpRange-14

2012-05-11T21:11:43+01:00

As part of the TAG’s work on httpRange-14, Jonathan Rees has assessed how a variety of use cases could be met by various proposals put before the TAG. The results of the assessment are a matrix which shows that “punning” is the most promising method, unique in not failing on either ease of use (use case J) or HTTP consistency (use case M).

In normal use, “punning” is about making jokes based around a word that has two meanings. In this context, “punning” is about using the same URI to mean two (or more) different things. It’s most commonly used as a term of art in OWL but normal people don’t need to worry particularly about that use. Here I’ll explore what that might actually mean as an approach to the httpRange-14 issue.

Note: The material here is a summary of what I think is the best way forward following various discussions within and outside the TAG, in particular with Jonathan, Henry Thompson and TimBL. Not all these people agree with or endorse the approach described here, but neither do all the ideas in this post originate from me.

Background

Five things recently make me more convinced than ever that the TAG must either provide some direction to the community, and soon, or get out of the way.

The proposed Linked Data Platform Working Group charter and the Submission that is the main input to the group specifically brings together linked data and REST, and the only mention of 303 redirections so far is to do with paging.
A recent thread on the W3C public-vocabs mailing list, raised the question of whether to embed schema.org markup about the page itself within a given page, or only about the thing that the page is about. I wonder how many pages are being described as schema:WebPage as well as things like schema:Organisation, and how people choose which class to use.
The initial version of Dan’s proposal for handling external enumerations within schema.org talked about minting new URIs in the ext.schema.org domain specifically to proxy existing URIs so that they can be guaranteed to provide the right (303) HTTP response. I can see the reasoning (persuading people to use 303 redirections is difficult) but it would be frustrating if the end result were a centralisation of the URI space.
Talking with my colleague John Sheridan about updating the UK government’s guidance on Designing URI Sets for the UK Public Sector, I really don’t know what to advise. Should the guidance continue to be to use 303 redirections, when I know from experience that these can be impractically slow? Should it change to recommend using hash URIs to identify things?
The very first message on the Technical Architecture Community Group was in part about how to identify people with URIs.

Of course the httpRange-14 issue has been running for so long now that I’d estimate that currently 80% of the discussion about it is meta-discussion about whether there it needs to be discussed, how much it should be discussed, how to raise the quality of the discussion, how anyone who discusses it is a time-wasting idiot and should just shut up and so on. It’s a terrible destructive cycle: the more this goes on, the higher the proportion of time being spent on the meta-discussion, and the longer any discussion takes.

But I believe that we can get to a point where we don’t have to discuss it any more (except to reminisce about what a waste of time it was), and I believe that the only way to get to that point is for the TAG to push through and provide a practical way forward.

Terminology

Let’s start off with some terminology. The basic scenario is a three-way interaction between three agents:

a supplier who manages the information that is accessible at a URI; it’s worth noting that the supplier for a particular URI might change over time and what exactly is provided at a URI is controlled by multiple parties, as there may be many service providers involved in routing the resolution of a URI, others in constructing what’s shown on the page served from the origin web server, and still others who transform that content en route to the consumer
a third-party publisher who publishes some information about a URI; unlike the supplier, they generally have no control and incomplete knowledge about the information available at a particular URI, its stability over time or consistency across representations
a consumer, typically an application of some kind, who has discovered the information published by the supplier or third-party publisher and wants to do something with it

It’s also useful to include terms for three things that are passed around or referenced during the interaction, which are defined with the URI specification and HTTPbis. (The current HTTP specification, RFC 2616 is also of interest, of course, but HTTPbis is a better reflection of current practical use of HTTP, and is close to complete, at which point it will replace RFC 2616.)

a URI which is a string of characters matching the syntax in the URI specification that identifies a resource; we only care about http: URIs here, although similar considerations may apply to URIs that use other schemes
a resource which is identified by the URI; the debate over whether HTTP constrains the nature of the resource is at the heart of some discussions around httpRange-14; here, as in the URI specification and HTTPbis, a resource could be anything
representations which are media-typed sequences of bytes (often characters) encoded within the response to an HTTP GET request on a given URI; per HTTPbis, the response to a GET request contains a representation of the current state of the resource identified by the URI

Content and Meaning

Now I will introduce a few new terms that aren’t used in the URI specification of HTTPbis but which are useful for discussion.

The content located by a http: URI is whatever core information a consumer interacts with through the HTTP interface provided by the server for that URI. This is the information that is common across all the representations that are returned through a GET on a given URI (through content-negotiated variants). We can say that the http: URI locates the content of the resource that it identifies, because you can get hold of the content of a resource by performing an HTTP GET request.
The sense referred to by a http: URI is a social construct that arises from the properties associated with the URI by publishers and the way that these invoke action in consumers. We can say that a http: URI refers to a sense. While you can GET content from a URI, determining its sense can only be achieved by examining the way in which the URI is used within data published on the web. The sense referred to by a URI might vary in different contexts, but equally a single sense may emerge in the use of the URI. One sense referred to by a URI may be its content, and for some types of information that may be the only sense referred to by the URI by anyone.

Here is a diagram that shows how these different terms hang together.

For example, take the URI http://www.amazon.com/gp/product/B004TRXX7C. The content located by this URI is the core information in the web pages we GET from the URI. The sense referred to by the URI could the same as the content, or it could be the novel Moby Dick, or the particular Kindle edition of the book. We can’t tell from any interaction at the level of the HTTP protocol what the sense of the resource is: that information has to come from the application level.

Like the meaning of a word, the sense that a URI refers to is a social understanding which emerges from use of the URI across the web, and a given URI may be used to refer to different senses in different sources of information or over time. Consumers interpret the information that uses a URI and is made available to them on the web in order to draw conclusions and perform a task. Different consumers will have different levels of trust in the particular interpretation of the URI that a given publisher provides; in particular, the information published by the supplier of the URI might be given a higher weight than that from third-party publishers. Tools like sig.ma illustrate how information can be combined from multiple sources with different weights, by associating metadata about the location of the data with the data itself; unpicking commonalities between groups of sources may help to work out the different senses referred to by these different sources.

The content located by a URI is more concrete, and is important because certain classes of application may infer something about what they can do with the content found at a given URI based on information published about the URI. The canonical example of this is a consumer that searches the web for public-domain pages, based on information published about the licensing of those pages, and displays a portion of one each day within a feed. This application can’t work properly if it doesn’t know what actual content is public domain. In the Amazon example above, if there is a statement saying http://www.amazon.com/gp/product/B004TRXX7C is public domain (referring to the novel “Moby Dick”, which is one possible sense referred by the URI, one on which the copyright has expired), a consumer that assumes the URI is being used to locate the content at that URI will assume that the representation retrieved through a GET on that URI is public domain. The consumer might pick out the first major paragraph from the HTML page for display, but that first paragraph is actually an editorial review that is marked with a separate copyright which therefore shouldn’t be displayed in a feed of public-domain content.

Of course interpreting assertions made by publishers about particular content can be just as complex as interpreting statements about a particular sense of a URI, especially when those assertions come from a third party. The content located by a URI may change over time, and potentially in dramatic ways if the domain name of the URI changes hands, so any statements that a consumer discovers about content needs to be assessed by a consumer in that context. That said, while the validity or truthfulness of a given piece of information about content may be variable, the bytes that are located through a URI by a consumer are tangible and discoverable in a way that the sense referred to by a URI can never be.

The core disagreements around httpRange-14 arise from whether you view the content located by a hash-less http: URI which provides a successful (2XX) response to be the only valid sense referred to by that URI, or whether you think they could be different things, and if you think they can be different things then which of those you think the URI identifies when it is used.

Current State

The URI and HTTP specifications talk only about URIs identifying resources and being able to make requests using URIs to get a representation of the resource, they do not talk about senses or content.

The httpRange-14 decision is based on a design where if you can successfully GET a representation using a URI (ie you receive a 200 OK response) then the sense referred to by the URI is the content located by that URI. This is pictured in the diagram below.

Sometimes, a server simply doesn’t store the content for something that it wishes to provide information about (for example, the Amazon website doesn’t store the content of Moby Dick), and sometimes the sense that a publisher wishes to confer on a URI is such that no content can be transmitted over the wire, such as a Person. In these cases, under this design, the server cannot give a 200 OK response because it does not have the content for the URI. There are then two patterns a publisher can use to assign a http: URI to something in these cases.

One pattern is to use a 303 See Other redirection to a URI whose content (which is the only sense of the URI in this design, remember) describes the original URI. This is pictured below.

The second pattern is to use a hash URI. This gives a very similar pattern, as you can see from the diagram below; the only difference is that instead of following a 303 See Other redirection to get from one URI to the other, you can use URI parsing: you chop off the fragment part of the URI and perform a GET on the resulting URI to get a description. Often this leads to several hash URIs being described by the same content, as illustrated here:

However, hash URIs are also used to identify fragments within pages, which are bits of content. For hash URIs that identify fragments of a page, the picture looks more like this:

A consumer can’t tell just from looking at a hash URI whether it identifies a fragment of content or is being used to refer to something described by the content located by its base URI: the consumer has to make a request and understand the fragment identifier as it applies to the media type of the representation that it gets back. Also, if there’s any content negotiation going on, even if the fragment identifier doesn’t make sense for the media type, or doesn’t locate a fragment within the representation that the consumer retrieves, it might still locate a fragment of content within a different representation of the resource than that the consumer has retrieved.

In any case, common usage of hash-less http: URIs differs from this model of content and sense being one and the same for all URIs that give a successful response. Often URIs are used in data formats such as JSON, XML, RDF or HTML where the content and the sense of the URI are different things. For example, on the Flickr the content located through the URI http://www.flickr.com/photos/45701084@N08/7051652969/ is a landing page that provides a bunch of information about an image, but the data within the page includes a statement about its license:


  cc:license  ;
  .

in which that same URI refers to the photograph itself: the sense referred to by the URI. The only way that you can tell this is by being a human: reading the page and the context in which the text describing the license is used.

This mismatch between the design specified by RFC 2616 and the httpRange-14 decision, and practice on the web today results in arguments back and fro with people saying, in essence, that the resource a URI identifies is the sense conferred by the URI’s supplier, or that a URI should always be taken as identifying the content of the resource, and then discussions about how to signal to an application that in particular cases the supplier really does mean the URI to identify some content or really does mean the URI to identify a particular sense, and if so which sense is being referred to.

Punning

“Punning” approaches attempt to cut through these disagreements by saying that the context in which the URI is used determines whether it is locating content or referring to a sense.

If we look at some Open Graph Protocol (OGP) statements on http://www.imdb.com/title/tt1334573/, we see:

Some of these properties — the url, title, type and image — are about the Moby Dick TV Series — the sense referred to by the URI http://www.imdb.com/title/tt1334573/. Others — the site name and Facebook application id — are about the content located by the URI. The properties that are provided by this data are all related to the same URI, but they aren’t all properties of the same thing. In natural language we might say:

a URL of the thing described by the page is http://www.imdb.com/title/tt1334573/
a title of the thing described by the page is “Moby Dick (TV Series 2010)”
a type of the thing described by the page is “video.tv_show”
an image of the thing described by the page is the content at http://i.media-imdb.com/images/SFc0774313bf9ccbfe22050c8bb4029e41/imdb-share-logo.gif
the site of the page is IMDb
the Facebook application to use with the page has the identifier 115109575169727

The property itself determines whether it applies to the content located by the URI (the page) or a sense referred to by the URI (in this case, the thing the page describes). Here’s a diagram that shows the distinction:

Defining URI Usage

The way in which a URI is interpreted — as referring to a sense or locating content — is dependent on where it is used. In XML, for example, an xmlns:* attribute contains a URI; when this is a hash-less http: URI, this refers to an XML namespace (a sense of that URI): it doesn’t matter what content you GET from dereferencing the URI, or even if it can be dereferenced at all. On the other hand, the href attribute on a xi:include element defined by XInclude is used to locate some content to be included within the referring XML.

It is really up to the format in which data is encoded to determine how the URI should be interpreted: as locating some content or referring to a sense. As with interpreting any information with which it’s presented, an application that needs to work out which is meant might use:

built-in knowledge (eg an application might know that the og:title property is always about the sense referred to by the subject URI, based on documentation about the property [this is essentially the same as if the information were embedded within a schema, but without the implication that every application must download and interpret a schema every time it happens across a property])
information encoded within a schema (eg a schema might classify the og:title property as a PropertyWhoseSubjectIsTheSubstanceOfAResource)
a default for the format of the data (eg given OGP uses RDFa, RDF could specify that by default URIs refer to a sense, and therefore barring other information to the contrary, properties cannot be assumed to be about the page itself)
a default for the web (eg we might say that barring overriding information, all hash-less http: URIs are assumed to locate content, as this is consistent with the current definition of HTTP in RFC 2616)

Thus an implication of this approach is that the people who define languages and vocabularies must specify what aspect of a resource a URI used in a particular way identifies. There are four possibilities for a given URI:

the URI is being used to locate some content
the URI is being used to refer to a sense
the URI is being used to identify either content or sense but it’s not specified which
the URI is being used to both locate content and refer to a sense (ie a property applies equally to both)

Equality

Now let’s consider what happens when there is more information available about something, but it uses a different URI. The page https://en.wikipedia.org/wiki/Moby_Dick_(miniseries) is about the same TV series as http://www.imdb.com/title/tt1334573/. Imagine that this similarly made available the information that it held using OGP. It might contain:

The two pages describe the same thing: the sense referred to by the two URIs is the same. However, the content of the two pages is different. If you simply smushed the properties together, ignoring the fact that some properties apply to the content and others the sense of the resource, you’d get some data that wasn’t quite right:

{
  url: [
    'http://www.imdb.com/title/tt1334573/',
    'https://en.wikipedia.org/wiki/Moby_Dick_(miniseries)'
  ],
  title: [
    'Moby Dick (TV Series 2010)',
    'Moby Dick (miniseries)'
  ],
  type: [
    'video.tv_show'
  ],
  image: [
    'http://i.media-imdb.com/images/SFc0774313bf9ccbfe22050c8bb4029e41/imdb-share-logo.gif'
  ],
  site_name: [
    'IMDb',
    'Wikipedia'
  ],
  app_id: [
    '115109575169727'
  ]
}

Having two URLs, two titles and so on is fine, but having two site names doesn’t make sense: the og:site_name property is related to the content located by the URI, and the content is different for the two URIs. This is illustrated below.

Conversely, imagine a situation in which there is a single document on a web server that is served up from the two URIs

http://example.org/gender/male
http://example.org/gender/female

In this case, the content located by the two URIs is exactly the same, but the sense referred to by the two URIs is different: one refers to the gender ‘male’ and the other to the gender ‘female’, as illustrated here.

So there are three types of equality that we have to be concerned with:

equality between the desired sense and content of a single URI, as described earlier
equality between the senses referred to by different URIs
equality between the content located by different URIs

In general equality between the resources identified between two URIs is a controversial thing to assert, because different contexts may refer to different senses of a particular URI, some of which may be equal with a sense referred to by another URI and some not. Like any statement made about URIs, the source of statements about equality must be considered.

Formats that wish to make assertions about equality between resources should provide ways of saying that the sense referred to by two URIs is the same, without implying that the content located at those two URIs are the same, and vice versa, and to assert that the sense and content of a URI are equal. What these properties are — how exactly these kinds of equality are asserted in a given format — is up to the format, but it’s important that the properties are kept distinct to enable people to articulate the full range of equality relationships between resources.

Implications for Linked Data

I have tried to keep the description above neutral in terms of technology choice, because I believe that the issue of how to interpret URIs within data is common across all languages that use URIs. However, as I’ve discussed previously, linked data is particularly affected by these issues both because URIs form a central part of the way it works as a data format and because culturally the community tries very hard to adhere to “good web architectural practice” in the hope that this will confer long-term benefits.

For that reason, I’ll look at what I think the impacts are on linked data practice of using the “punning” approach that I’ve described above.

RDF

The definition of RDF is currently in flux, as RDF 1.1 is developed, so now is a good time to consider its use of URIs.

RDF itself is not particularly concerned with what URIs identify: it is simply a model that can be used to associate properties between “resources”, where in the RDF context this term means anything that can be the subject or object of an RDF statement, including literals. (RDF’s use of the term “resource” is not the same as that used in the URI specification or HTTPbis.) The only real limitation in RDF 1.0 Concepts, is that a hash URI identifies something described by the RDF/XML representation retrieved when the URI is resolved. In the current Editor’s Draft of RDF 1.1, the same section is less specific about what such a fragment might denote. So if the language and concepts above were adopted, RDF 1.1 should be more careful in its use of terminology, and attempt to be consistent with the URI and HTTP specifications, but I don’t think anything fundamental needs to change in the core semantics of RDF.

Vocabulary Designers

Under the “punning” approach, the property used within an RDF statement determines how the URI given as its subject or object should be interpreted. A consumer that discovers a URI by looking at the properties associated with it needs to be able to tell from the properties themselves whether it can associate those properties to particular content that it locates by requesting the URI or not.

Some properties have a defined domain or range that precludes the property from being used to annotate content. For example, the foaf:nick property has a domain of foaf:Person, and a foaf:Person cannot be a web page. Given this domain, an application can tell that the URI http://www.jenitennison.com/ used in a statement such as:


  foaf:nick "JeniT" ;
  .

cannot be being used to locate the content of http://www.jenitennison.com/, even if http://www.jenitennison.com/ responds with a 200 OK response.

Note that this inference doesn’t work the other way around. The property cc:license has a domain of a cc:Work but without additional information about the property an application could not infer that in a statement such as


  cc:license  ;
  .

the URI http://www.amazon.com/gp/product/B004TRXX7C was being used to locate the content of http://www.amazon.com/gp/product/B004TRXX7C: it could equally be being used to refer to some sense of the resource (for example the novel Moby Dick).

To support “punning”, therefore, RDF vocabulary designers would need to have additional properties that could be applied to RDF Properties to indicate how their subject (and object where applicable) should be interpreted. For example, the Creative Commons vocabulary might include (warning: made up property names and instances):

cc:license
  rdfs:subjectUri rdf:sense ;
  rdfs:objectUri rdf:content ;
  .

with the implication that URIs used as the subject of cc:license should be understood as referring to the sense of the URI, while those used as the object of cc:license should be understood as referring to the content retrieved from the URI.

Even if properties like rdfs:subjectUri or rdfs:objectUri are defined, there are going to be RDF properties for which the interpretation of subject and/or object URIs isn’t specified, and thus consumers of RDF content need to have a default interpretation. What that should be is, I think, a matter for the RDF community to decide.

Inference

The major difficulties with the “punning” approach and the current use of RDF comes when reasoning is used across RDF statements in which the same URI is used in different ways, particularly with properties where the interpretation of the subject and/or object isn’t specified.

For example, if a consumer finds the following triples at http://www.imdb.com/title/tt1334573/:


  og:url "http://www.imdb.com/title/tt1334573/" ;
  og:title "Moby Dick (TV Series 2010)" ;
  og:type "video.tv_show" ;
  og:image "http://i.media-imdb.com/images/SFc0774313bf9ccbfe22050c8bb4029e41/imdb-share-logo.gif" ;
  og:site_name "IMDb" ;
  fb:app_id "115109575169727" ;
  .

and the following triples at https://en.wikipedia.org/wiki/Moby_Dick_(miniseries):


  og:url "https://en.wikipedia.org/wiki/Moby_Dick_(miniseries)" ;
  og:title "Moby Dick (miniseries)" ;
  og:type "video.tv_show" ;
  og:site_name "Wikipedia" ;
  .

and then the assertion:


  owl:sameAs  ;
  .

then the result of inference will be that all the statements made about http://www.imdb.com/title/tt1334573/ apply equally to https://en.wikipedia.org/wiki/Moby_Dick_(miniseries), which is not the case.

To enable publishers to make assertions about equality of sense and equality of content separately, we will need new relationships. For example:


  owl:sameSenseAs  ;
  .

would only infer that those properties whose subject is the sense of http://www.imdb.com/title/tt1334573/ apply equally to the sense of https://en.wikipedia.org/wiki/Moby_Dick_(miniseries). A owl:sameContentAs property could similarly assert equality between the content of two URIs.

The impact is not limited to reasoning with owl:sameAs: all inference in RDFS and OWL is based on the assumption that a single URI identifies a single entity. This works in situations where the RDF over which inferences are being made is all trusted (for example if it is all made available by the same publisher), and a lot of current use of OWL is precisely in these kinds of closed environments. The same inferences can be made even with information gleaned from the web at large, if that information is selected carefully.

Another approach, where publishers have mixed properties about the sense referred to by a URI and those about the content located by a URI is to pre-process those RDF statements to create separate (blank node) RDF resources. For example, if og:url, og:title, og:type and og:image are defined to have a subject that refers to the sense of the URI, and og:site_name and fb:app_id to have a subject that locates the content of the URI, the statements about http://www.imdb.com/title/tt1334573/ above could be translated into:

_:imdbSubstance
  rdf:senseUri "http://www.imdb.com/title/tt1334573/" ;
  og:url "http://www.imdb.com/title/tt1334573/" ;
  og:title "Moby Dick (TV Series 2010)" ;
  og:type "video.tv_show" ;
  og:image "http://i.media-imdb.com/images/SFc0774313bf9ccbfe22050c8bb4029e41/imdb-share-logo.gif" ;
  .

_:imdbContent
  rdf:contentUri "http://www.imdb.com/title/tt1334573/" ;
  og:site_name "IMDb" ;
  fb:app_id "115109575169727" ;
  .

Here, the (putative) rdf:senseUri property is an inverse functional property that provides the URI for which the individual is a sense, and the rdf:contentUri property is an inverse functional properties that provides the URI for which the individual is the content.

This separation would then allow existing inference to take place on the separate entities.

Publishers

There are many advantages offered by the “punning” approach for linked data publishers:

it supplies an easy on-ramp for suppliers who want to annotate their pages with HTML data such as RDFa and microdata: suppliers can use URIs that they already support to refer to things other than documents, if they choose to, which means all they need to do is add metadata to their pages (as they are currently using OGP and schema.org)
suppliers do not have to have access to server configuration in order to promote the use of particular URIs to mean things that do not have content (such as people or organisations)
publishers can copy and paste URIs from the location bar of their browsers (a familiar activity for people who wish to provide a pointer to something) rather than inspecting pages for a recommended URI to be used to refer to a particular sense
organisations such as schema.org can easily recommend the reuse of URIs published by other people, such as Wikipedia, without requiring those publishers to alter their server configuration or requiring developers that use schema.org markup to add fragment identifiers to their URIs
explicit describedby and describes links can be made between URIs rather than using an HTTP status code where necessary; these can be incorporated directly in data and do not require a network connection to be discovered

Provenance

The “punning” approach that I’ve described here has as its core the recognition that different consumers will trust different sources of information to different levels. Knowledge of the provenance of a particular source of information is one way in which consumers can work out what to trust and how to resolve conflicts sources.

The work of the Provenance Working Group is important here both in identifying the provenance of particular content located at a given URI and in providing a vocabulary for describing the processing that a consumer performs to retrieve and process that content in order to extract data from it (for example, the time of the retrieval and the HTTP headers used may lead to the consumer receiving different content; the particular version of software used may lead to different information being gleaned from that content).

Linked Data Platform

The particular issues around what URIs actually identify within RDF only become an issue when the URIs are resolved — when RDF is used within linked data. The new Linked Data Platform Working Group is a great opportunity to standardise around these practices, in collaboration with the other relevant working groups.

Final Thoughts

People use the terms “resource”, “identifies” and “representation” both within specifications and in common parlance as if there is a shared understanding of what they mean, when in fact different people use the terms in subtly but meaningfully different ways. This would be fine, except that the different understandings lead to different assumptions and engineering decisions, and friction for developers trying to build applications that publish and consume data whose assumptions differ.

We need to find a way forward that, even if not everyone’s ideal, is realistic, explicable and palatable. The “punning” approach that I’ve described above might not be it, but the analysis that Jonathan’s done of the various proposals and use cases suggests to me that it’s the closest we have. The main questions I have are:

what use cases cannot be satisfied using this approach?
what specifications would have to change if this approach was adopted, and would it be realistic to make those changes?
what existing applications would break if this approach was adopted, and how might that breakage be mitigated?

At the very least, I hope that the vocabulary I’ve laid out in this post might be helpful in further discussions.

Of course any other comments are most welcome.

UK Open Standards Consultation

2012-04-14T23:44:51+01:00

Over the last few months, the UK Government has been running a consultation on its Open Standards policy. The outcome of this consultation is incredibly important not only for organisations and individuals who want to work with government but also because of its potential knock-on effects on the publication of Open Data and the use of Open Source software within public sector organisations.

Unsurprisingly, Microsoft, Qualcomm and other organisations who have a vested interest in keeping the UK Government locked in to their products are responding vociferously to the consultation. They risk not only losing business to smaller enterprises within the UK but also, if the policy is successfully adopted here, in other countries in Europe and internationally that follow suit.

If we want our Government to be Open — to use Open Standards, to publish Open Data, to adopt Open Source — then we must respond to this consultation in numbers.

There are three things that you can do:

Respond to the consultation — made even easier by this response form developed by Ric Harvey
Attend the events — these seem pretty full now, but try to get in if you can
Spread the message — blog and tweet and write to raise awareness of the importance and impact that this consultation could have

The consultation is quite long and there are a lot of questions to answer. In the hope of making this easier for everyone, I’m publishing my response below. Please consider these responses public domain, and feel free to copy as much or as little as you like from them (though I recommend you omit the parts that are about my individual experience and substitute them with your own).

For extra background, read:

Of Microsoft, Netscape, Patents and Open Standards by Glyn Moody
Are open standards a closed barrier? by Linda Humphries
Open Standards at risk by Ric Harvey

Criteria for Open Standards

1. How does this definition of open standard compare to your view of what makes a standard ‘open’?

The definition in the consultation closely matches my view of what makes a standard open. The important factors are:

a documented, open process which enables participation not just from implementers but also from users of the standard, and that provides ongoing maintenance and development of the standard
publication of the standard such that anyone can read it
a royalty-free and non-discriminatory license such that anyone can implement the standard without cost

The one factor that does not match my view is the availability of multiple independent implementations of the standard. In some cases it may be that market pressures mean there are not currently multiple good implementations of an otherwise Open Standard, or several but only for one particular platform. Limiting the definition of Open Standards to only those with multiple cross-platform implementations is probably too constraining.

There are two examples from my work that demonstrate why the availability of multiple implementations should not be a factor.

First, one of the Open Standards that I use is XSLT, which for years has been dominated by a single implementation — Saxon — giving customers no real choice. Nevertheless, because it is an Open Standard, Saxon has had a lot of pressure to be completely conformant with that standard, and in the past year a number of other implementations have been started that can compete with it on different platforms, so the presence of a single implementation has proven to be a short-term issue.

Second, in some new technology areas such as Linked Data, there may be only single implementations simply because the area and the Open Standards on which they are built are not yet very mature. As the use of the technology grows, so do the number of implementations and their adherence to the standard. Thus number and quality of implementations does develop over time; government should concentrate on long-term adoption rather than short-term availability.

2. What will the Government be inhibited from doing if this definition of open standards is adopted for software interoperability, data and document formats across central government?

In some specialist areas, there may not be existing Open Standards available, or it may be that the Open Standards that are available do not match the specialist needs of the UK government. Where an Open Standard is imminent but not yet fully standardised or implemented, waiting for standardisation or implementation could delay government IT projects. However, the measures suggested in the rest of the proposed policy, including government involvement in standards creation and allowing for selecting other standards where there is no available Open Standard, mitigate against this risk.

The other mitigating factor is that where an Open Standard doesn’t exist, it is usually possible to build new work on top of Open Standards. In my experience working on legislation.gov.uk, even though good appropriate Open Standards for UK legislation weren’t available, we have been able to work using the underlying Open Standards such as XML, RDF and HTTP, so the gap between necessary custom work and Open Standards is minimised.

The definition of Open Standards may also prevent the Government from entering into contracts with companies which do not adopt Open Standards. It may hinder exchange of information with outside organisations that use software that doesn’t support Open Standards. These may be problems in the short term but over the longer term, an Open Standards policy will move the supplier market and other organisations towards Open Standards more generally.

3. For businesses attempting to break into the government IT market, would this policy make things easier or more difficult – does it help to level the playing field?

The policy does help to level the playing field for two main reasons:

using Open Standards reduces the cost of switching to new suppliers, because the new suppliers do not have to spend a lot of time reverse engineering existing processing and data; this means new suppliers can make more competitive bids
Open Standards are often implemented within Open Source Software, which has low or no cost; this helps smaller businesses because it lowers the cost of their entering the market

4. How would mandating open standards for use in government IT for software interoperability, data and document formats affect your organisation?

I work as an independent contractor who specialises in a variety of Open Web Standards. For me as a contractor, the adoption of Open Standards within government increases the potential opportunities for me to work on government projects.

I am currently contracted to work on the delivery of legislation.gov.uk and The National Archives’ Expert Participation Programme. This work is already built substantially on Open Standards. One of the main pain points in this work has been that the government organisations that provide data such as Bills, new Statutory Instruments or Tables of Effects for legislation.gov.uk are using proprietary technologies to do so, and converting from those proprietary data formats, or getting users to save in an open data format, can be both hard to do initially and difficult to maintain as new versions of software are rolled out. An Open Standards policy within government would greatly reduce the cost involved in those conversion processes and increase the ease of use for government users.

I am also a member of the Technical Architecture Group within the World Wide Web Consortium (W3C), which is the main standards body for Web Standards. From that perspective, the Open Standards policy would increase UK government, and UK government supplier, involvement in the development of Open Standards within the W3C, which can only improve the quality of the standards and the life of the organisation as a whole.

5. What effect would this policy have on improving value for money in the provision of government services?

This policy would greatly increase value for money in the provision of government services. Adopting Open Standards often means that the basic building blocks for a service can be selected off the shelf, and then fitted together and customised, rather than a proprietary solution built from scratch. This can reduce the cost of providing the service as a whole and improve the service as development effort can be directed on the unique features of the service.

6. Would this policy support innovation, competition and choice in delivery of government services?

The policy as written is well-framed to support innovation, competition and choice across the market in the medium and long term, in areas which are beneficial to the UK Government and to the rest of the economy.

Adopting Open Standards focuses innovation on novel areas (which are not currently covered by existing standards) and on providing better quality services (which may mean better performance, better user experience and so on). It also encourages innovation in public, and this greater exposure brings with it higher quality and a better focus on user requirements. The only innovation it prevents is that whose purpose is to lock the government in to individual suppliers, such as closed standards that largely repeat existing work but can only be implemented by one supplier.

As part of the work that I did on Linked Data for data.gov.uk, myself and several colleagues worked on new RDF vocabularies such as org, Data Cube and OPMV and processing standards such as the Linked Data API. We did this in the open, and it was all based on Open Standards: that approach did not prevent us from doing new and innovative things. The results of that innovation were then taken forward by the wider community, being made more rigorous and better suited to applicability across a wider audience, and are resulting in Open Standards from W3C. In addition, my colleagues have built new products that integrate that work, for example in Kasabi.

Adopting Open Standards focuses competition on the quality of service that is offered rather than on winning a single competition that will lock the government in to contracts for many years to come. It prevents supplier complacency: when the government can move easily to a new supplier, suppliers have to provide continuous improvements over the lifetime of a contract, because they cannot be guaranteed to win the next one simply because they are the only ones who have implementations that can process the data. Competition is thus focused into areas that matter to the customer, including cost, rather than areas that matter to the supplier.

I was involved tangentially in The Stationery Office’s (TSO) bid during the recent re-procurement of legislation services by The National Archives. Because legislation.gov.uk was built on Open Standards, TSO’s bid had to be based on quality of service, on continuing innovation over the lifetime of the contract, and on low cost of delivery.

Adopting Open Standards increases choice for the UK Government because it opens up competition to suppliers who would not otherwise be able to compete (as in Question 3). Of course some companies may not currently use Open Standards, and under this policy the UK Government would not be able to choose them as suppliers in the short term, but it is unlikely that companies would continue their use of closed standards in the long term, if they wish to compete for UK Government contracts.

Put another way, adopting Open Standards means companies are innovating and competing on the right things: on things that are important to the UK Government.

7. In what way do software copyright licences and standards patent licences interact to support or prevent interoperability?

It is possible to have a Open Standard implemented by software that is public domain or completely closed or anything in between: Open Standards do not necessarily lead to free software. On the other hand, standards patent licenses reduce the ability of developers to produce free (open licensed) software because they need to make enough money to pay to use the license. Thus standards patent licenses limit the number and type of implementations of a standard, effectively limiting the market to those with enough capital to enter it.

The fewer implementations of a standard, the less pressure there is on those implementations to be interoperable, because there are a known and limited number of other implementations with which they need to interoperate. The greater the market of implementations, the greater the drive for interoperability because it becomes increasingly likely that they will have to interoperate with each other.

For example, I have often had to move code that I have written based on an Open Standard from one implementation and use it in another. If it doesn’t work, I can work out which implementation is correct (by looking at the standard) and report the error to the implementation developers, which helps them improve the interoperability of their product. The ability to move between implementations is vital to ensure interoperability between them, and the broader the market the more likely that is to happen.

8. How could adopting (Fair) Reasonable and Non Discriminatory ((F)RAND) standards deliver a level playing field for open source and proprietary software solution providers?

Adopting (F)RAND standards does not deliver a level playing field across providers, because it limits the ability of open source providers to enter the market, as they have to recoup licensing cost. Only royalty-free licenses provide a completely level playing field across providers.

9. Does selecting open standards which are compatible with a free or open source software licence exclude certain suppliers or products?

Selecting Open Standards necessarily excludes those products that only use closed standards, and suppliers that only offer those products. In the short term, suppliers who have built their products and business models around closed standards and lock-in will be excluded.

However, there is no requirement for companies that adopt Open Standards to license their products using a free or open source software license. While there are often free or open source products built around Open Standards, these are only competitive when they are at the same quality as those with closed licenses.

An example from legislation.gov.uk is that our early development used eXist, which is an XML database available under an open source software license which implemented the Open Standards of XML and XQuery. It became clear that (at that time) eXist did not support the level of use that we needed, in its performance and its scalability. We therefore instead adopted MarkLogic, which uses the same Open Standards but is not available under an open source software license. This demonstrates how companies which adopt Open Standards can still offer competitive value even within a market with open source implementations.

Another interesting point to draw from this is that the presence of an open source implementation of an Open Standard enabled us to prototype and experiment using that implementation, knowing that should we need better performance and so on we would be able to move all our code to another interoperable implementation. If MarkLogic had implemented a custom method of querying XML, committing to paying for it early in the process would have been too high a risk. So the use of Open Standards effectively helped MarkLogic win that business.

10. Does a promise of non-assertion of a patent when used in open source software alleviate concerns relating to patents and royalty charging?

I would personally be very wary of implementing a standard which had a promise of non-assertion of a patent, and standards made available under those terms would seem more risky and costly to implement because any such promise would have to be checked by a lawyer and would likely constrain my future actions and the use of the software in other environments.

11. Should a different rationale be applied when purchasing off-the-shelf software solutions than is applied when purchasing bespoke solutions?

The same policy should be applied both to off-the-shelf and bespoke solutions, particularly as the difference between these is not at all clear cut: off-the-shelf solutions are often customised, and bespoke solutions built from off-the-shelf products. Whatever the type of solution, the crucial point is that it needs to interoperate with other products, using Open Standards.

12. In terms of standards for software interoperability, data and document formats, is there a need for the Government to engage with or provide funding for specific committees/bodies?

The Government should engage with those standards bodies that work on standards that the Government uses, so that it can shape the development of those standards and highlight new areas where standards work is needed to satisfy the UK Government’s requirements. In my own area of web standards, the main one is the W3C. In fact, given the UK Government’s transparency, open data and “digital by default” policies, all which require the use of W3C standards, the UK Government is under-represented within W3C, with only two agencies (Ordnance Survey (OS) and The National Archives (TNA)) being members. Other public-sector organisations which make heavy use of W3C standards, such as the BBC, have made a business decision be become more engaged in W3C activities in order to shape and influence them.

Membership of W3C and other standards bodies is particularly important where standards development impacts on the ability to achieve the goals of particular organisations. For example, TNA are taking particular interest in the provenance work being done at W3C; the Government Digital Service should be participating in the development of standards in web design and applications; the Office of National Statistics should be taking an interest in the development of the Data Cube vocabulary for statistical information and so on.

13. Are there any are other policy options which would meet the described outcomes more effectively?

I believe that the Open Standards policy described in the consultation is the best way to achieve lower cost, higher interoperability, reduced lock-in, increased innovation and competition in the right areas and to level the playing field both for open source software and for small and medium enterprises who wish to compete for government contracts.

Open Standards Mandation

1. What criteria should the Government consider when deciding whether it is appropriate to mandate particular standards?

The only time the Government should mandate a particular Open Standard for IT is when there is a clear and apparent cost in two competing Open Standards that offer equivalent functionality being adopted by different parts of Government due to poor interoperability between them. In most cases, central government should avoid mandating a particular Open Standard, but instead let individual public-sector organisations select an appropriate Open Standard based on their own requirements.

Government should mandate a particular Open Standard where:

there are competing Open Standards that cover the given functionality and
interoperability or conversion between these standards is lossy or difficult and
there are multiple organisations within government with which interoperability involving the standard is required

2. What effect would mandating particular open standards have on improving value for money in the provision of government services?

A central government authority mandating particular Open Standards could reduce value for money in the provision of government services, because that central authority is unlikely to fully understand the requirements of the particular service in detail. Public sector bodies who are actually procuring solutions should select particular Open Standards as part of the procurement process.

Where mandating particular Open Standards could improve value for money is if there are competing Open Standards and different public sector bodies are likely to adopt different ones. In those cases, there may be interoperability problems between the standards which make it more costly to provide services, and mandating the adoption of a particular Open Standard could help.

3. Are there any legal or procurement barriers to mandating specific open standards in the UK Government’s IT?

I do not know.

4. Could mandation of competing open standards for the same function deliver interoperable software and information at reduced cost?

It is unclear what this question means.

Mandating both of two competing Open Standards may increase the availability of interoperable software, but it will raise the cost of implementation and therefore of software, because implementing two standards takes more development effort than implementing one.

Mandating a single Open Standard from two competing standards may increase interoperability but comes with a possible risk and cost. It may be that two competing standards, while similar, have different target audiences and capabilities. If one is mandated, those public sector bodies whose requirements fit more closely with the other will suffer increased cost in trying to use the first standard to fit their requirements. In addition, the standards may evolve over time such that they either diverge in functionality or increase in interoperability (so that mandation is no longer necessary) or such that the original judgement about which to mandate no longer applies.

5. Could mandation of open standards promote anti-competitive behaviour in public procurement?

Anti-competitive behaviour arises when the number of potential competitors is artificially restricted due to the terms of the procurement. Whether mandating a particular Open Standard promotes anti-competitive behaviour depends on two factors.

First, it depends on the nature of the Open Standard and its implementations. Some Open Standards are small and easy to implement while others are large and complex to implement. Some have open source implementations across a number of platforms, others have few implementations, available under restricted terms or on restricted platforms. In the short term, mandating an Open Standard that is costly for a company to implement and for which there is no easily available implementation is going to favour those companies who have already implemented the Open Standard. In the longer term, by their nature, implementations of Open Standards tend to become more widely available, and companies are likely to invest in implementing them if they are industry standards. Crucially, because the standards are Open, they are freely able to read them and can implement them without royalty payments, so in the long term it is not anti-competitive.

Secondly, the reason for the mandation of a particular Open Standard should be clear within the procurement exercise and a particular Open Standard should not be mandated unless there are clear reasons for that mandation. As an example from my own experience, there are two Open Standards for embedding metadata within HTML pages: RDFa and microdata. Either could be used to provide largely the same functionality so there would generally not be a need to mandate one or the other during procurement, but there should be a requirement to show how the standard selected by the supplier would be used to achieve the aims of the system.

6. How would mandation of specific open standards for government IT software interoperability, data and document formats affect your organisation/business?

Mandating specific Open Standards could limit the approaches that I would be able to take in my work, which could mean that I was less able to select an appropriate technology based on the requirements of the system. Any technology selection requires balancing a large set of requirements, both in terms of functionality and in terms of performance, reliability, scalability and so on. As a developer, I and my direct customers have the clearest understanding of these requirements and their relative importance within the system. These are complex choices, and they should not be made by central authorities who do not understand the details.

It is much more important to me to have a clear understanding that Open Standards should be used wherever possible and which Open Standards are being used within the organisations that interact with the systems that I am responsible for, so that the systems I build interoperate with them more smoothly.

7. How should the Government best deal with the issue of change relating to legacy systems or incompatible updates to existing open standards?

Good Open Standards provide clear statements about both forwards and backwards compatibility and, as they are developed through open participation, the extent of changes and the reasons for them are usually clear, which makes the process of working out what needs to be upgraded easier than with closed standards. For example, I was involved in the development of the second version of XSLT, and those of us in the Working Group spent substantial time ensuring that the impact on existing users of XSLT were not too great and were thoroughly documented.

When adopting a particular Open Standard within a system, the Government or their suppliers should assess the impacts of future changes in the standard on the system and should be involved to an appropriate level in the development of the standard to ensure that it continues to meet requirements.

8. What should trigger the review of an open standard that has already been mandated?

The Government should continuously work with its suppliers during the contract term and with potential suppliers during procurement to assess the best Open Standard to use. Even when a given standard itself doesn’t change, the wider IT environment may alter: there may be more or fewer implementations over time, alternative standards, or a change in interoperating standards used by other organisations. Thus there are no particular trigger points at which an Open Standard should be reviewed, though re-procurement will naturally cause a re-exploration of a system’s environment and the best approach to its implementation.

9. How should the Government strike a balance between nurturing innovation and conforming to standards?

Good Open Standards have built-in extensibility points which provide the scope for implementer innovation while providing general interoperability. In the best cases these extensions gradually become standardised themselves: this is what has happened with XSLT, for example. Thus, conforming to standards does not prevent innovation; instead it focuses innovation on user requirements, and in particular on improving the quality of implementation. The Government’s Open Standards policy should ensure that when standards are selected they provide broad interoperability while giving scope for extension, and this is particularly important if the Government mandates particular Open Standards for general use.

10. How should the Government confirm that a solution claiming conformity to a standard is interoperable in practice?

Within the W3C, new standards must have an associated test suite, usually constructed both by the Working Group who develop the standard and from the test suites created by individual implementers of the standard. Running an implementation against such a test suite makes it possible to empirically test whether there is conformance to the standard. The results of running a given implementation against such a test suite are often available, for example see the RDFa test suite results, but the Government could also ask the solution provider for evidence of interoperability in the form of test suite results.

11. Are there any are other policy options which would meet the objective more effectively?

A general Government policy to use Open Standards will in most cases naturally lead to the best Open Standard being adopted across the public sector without Government needing to mandate the use of specific Open Standards. Systems should be procured on the basis of their ability to interoperate with those other systems with which they need to work; in such an environment the easiest approach for suppliers will be to adopt the standards used by other systems.

We have seen this happen within legislation.gov.uk, where we store and publish legislation using a particular XML vocabulary. Various parliaments and government departments draft legislation which is published on the site, and they have traditionally done so based on the particular requirements of the drafters themselves (for example, the UK parliament uses Framemaker while government departments use Word). We have to cater for these different formats, and converting them into the standard that we use within legislation.gov.uk. However, as these authoring systems are being re-procured, the ability to produce the XML vocabulary that we use within legislation.gov.uk has become part of the requirements on future authoring systems, because it gives greater fidelity between authoring and eventual publication. Thus we are naturally moving towards a more interoperable environment without any central mandation of the standard and without overriding the local requirements of particular authors.

International Alignment

1. Is the proposed UK policy compatible with European policies, directives and regulations (existing or planned) such as the European Interoperability Framework version 2.0 and the reform proposal for European Standardisation?

I do not know.

2. Will the open standards policy be beneficial or detrimental for innovation and competition in the UK and Europe?

The UK’s Open Standards policy will be beneficial to innovation and competition in the UK because it levels the playing field for a wider set of providers and focuses innovation and competition on the requirements of the users of software rather than the suppliers. These benefits apply as well to Europe as to the UK, and the successful adoption of an Open Standards policy within the UK will naturally aid the adoption of similar policies in other countries within Europe and internationally.

3. Are there any are other policy options which would meet the objectives described in this consultation paper more effectively?

The one part of the international alignment policy that needs to be rethought is the preference to international standards over local standards. While in general the wider and greater adoption of a given standard, the better, it is not always the case that international standards provide greater benefits than local ones.

Looking at providing data about legislation, for example, different jurisdictions have very different ways of identifying, creating, revising and formatting legislation. Any international standards for legislation are highly unlikely to take into account the complexities and special cases that are specific to UK legislation (such as the use of regnal years for identifying older items). A local standard can be better tailored to the requirements of the locality, which may be incredibly important in easing implementation cost and data fidelity.

I would therefore recommend a policy approach that emphasised interoperability with international standards without requiring their wholesale adoption, particularly where there are specific local requirements that are not met by the international standard.

Content and Descriptions of Web Resources

2012-03-31T22:27:08+01:00

Those readers who follow the TAG or public-lod mailing lists over the last couple of weeks cannot have failed to notice a large number of posts on a theme that recurs on roughly a 9-monthly cycle within these communities: httpRange-14.

The reason for this particular recurrence was a Call for Change Proposals on the resolution. The TAG meets on Monday, and discussion of this issue is one of the first items on our agenda. These are my thoughts going in to that discussion.

The Questions

The recent discussion on the lists has, I think, helped to refine the questions that lie at the core of the httpRange-14 issue. They are:

When you get a successful response from a URI, does that response, by definition, include the content of the resource identified by the URI?
How can you discover a description of the resource identified by a URI?

Knowing whether the response to a URI provides the content of the resource identified by that URI is important because when you have data about the thing identified by a URI, such as its author or the license that it is provided under, you need to know what information is actually being referred to so that you can tell what information you can reuse and whom you have to attribute.

For example, the GOV UK website has a license at the bottom of each page:


  Much of the information on this website is available for reuse under the 
  Open Government Licence

Seeing this, an application that knows the Open Government License enables free reuse can tell that it can lift content out of the page and use it on their own site. An application could automatically scrape out and republish the first paragraph of those news stories provided on this site and any others that were published with under this license.

The Conflict

There are vocal disagreements about particularly the first of the two questions I outlined above. What’s become clear to me is that the source of the arguments stem from a difference in world view about what kind of resources are available on the web.

Web of Data

Under the web of data view, the web consists of data, and all the resources on the web are information resources, defined as those resources whose essential characteristics can be conveyed in a message. Data, in other words.

URIs can still be used to name other resources, which are not on the web either because they are not information resources (such as a Person) or because they are not available yet (such as unscanned books). Under this world view, however, giving a successful HTTP response for such a resource is simply wrong, because these resources aren’t on the web.

The problem that this world view therefore needs to address is how to create URIs to identify resources that aren’t on the web. There are two answers:

Hash URIs

Hash URIs have the benefit that there is a direct relationship between the hash URI which identifies the resource and a resource on the web that describes it. An HTTP client naturally strips the fragment identifier from the URI in order to make the request to a server, which then delivers the description of the resource.

303 Redirections

If you identify a resource that isn’t on the web using an HTTP URI that is not a hash URI, you cannot get a successful response back because the resource you have asked for is, by definition in this world view, not on the web. The workaround is for the publisher to use the 303 See Other status code to point from the resource that you requested to its description on the web. (This is the essence of the httpRange-14 resolution.)

Web of Things

Under the web of things view, the web consists of things, and resources on the web could be anything: documents, people, films, teapots and so on. When a client makes an HTTP request for a resource, the response must reflect the state of the resource, but that state could be its content (if it’s an information resource) or it could be a description of the resource.

Under this world view, giving a successful HTTP response for a resource that isn’t an information resource is absolutely fine: the description of the resource is still a reflection of its state.

The problem that needs to be addressed when you have this world view becomes apparent when you think back to the licensing example above. Given an application knows that the resource identified by a given URI can be reused, how does it know whether the representation of that resource is reusable? It could be that the identified resource (for example an out-of-copyright book) has an open license, but that the representation of the resource holds only a description of that resource (some metadata about the book), and that description has a much more restrictive license. Or vice versa.

So to address this use case, you need some other mechanism to enable an application to tell that the representation is the content of the resource, rather than merely a description of it.

The Current State

To generalise, the linked data community operates within the web of data world view and the larger web community operates within the web of things world view.

What is happening increasingly, however, is that these two world views are rubbing up against each other, and while both are internally coherent, switching between the world views causes not only a cognitive disconnect for developers but practical problems when transforming or moving data published under one world view into the other world view.

In addition, publication of data on the web through APIs is growing all the time, particularly REST APIs supporting the Hypertext as the Engine of Application State (HATEOAS) principle. As we share more data on the web, and we use URIs in our APIs, the question of what those URIs mean and how we associate licenses and provenance information with data, will only become more important.

We have an obligation, therefore, to reflect on the experience from the linked data community over the last few years and how that experience might spread to the larger web community.

Discussions within the linked data community over the httpRange-14 resolution centre on two problems that people have encountered:

the web of data vs web of things disconnect hurting adoption
practical aspects of responding to requests with 303 redirections such as
- round-trip delays, particularly as in HTTP 1.1 303s can’t be cached
- inability for people to use 303s without server admin access

The Social Context

A bit of a side-point here. I think that the questions I posed at the start of this post are general questions about web architecture, so it puzzles me that the only people who seem to really care about them, and who debate them endlessly, are the linked data community. This is partly because the linked data community use URIs extensively to identify the things about which they provide data, but I think it’s also about the fundamental attitude of those within the community, which was characterised in a recent post by Hugh Glaser:

Personally, I never did agree with the solution [to httpRange-14], but have always aimed to carry out the implications of it in the systems I construct.

This is for two reasons:
a) as a member of a small community, it is destructive to do otherwise;
b) as a professional engineer, my ethical obligations require me to do so.

It is this second, the ethical obligations that are the most significant.
I should not digress from the standards, or even Best Practice, in my work.

The linked data community is jam packed with people who feel an ethical obligation to adhere to standards and best practices. We try to do what we are told is the Right Thing by individuals and standards organisations even when we don’t agree that it is the Right Thing and even if it turns out to be impractical.

In the larger web community, people who don’t agree with a standard or best practice, or who find it too impractical to implement, simply ignore it. There is no need to endlessly debate something that you can just ignore. And the httpRange-14 resolution is ignorable by the larger web community because so far it has had very little impact on any implementations at all, let alone widely-deployed implementations that work over the non-linked-data web.

The Choices

Going into the TAG meeting about this on Monday, the main decision that I see is whether to continue to assume a web of data world view. In the web of data world view, it is impossible for a URI to return a description of a resource, whereas in the web of things world view it is fine. Personally, I would prefer to design around the web of things world view as I think this would ease some of the disconnects between linked data and the wider web, but there are others on the TAG who adhere strongly to the web of data view, so I think that change is unlikely.

If we stick with the web of data view, the main issues are how to alleviate the current practical difficulties that people are encountering with its implementation and explanation. I think there are three measures that would help:

Determine a conventional syntax for fragment identifiers that are used to identify things that are not on the web, as opposed to fragments of content. I’m thinking something like hash-bang URIs: using a character after the hash character that just gives a quick indication that the fragment identifier is being used in a special way, to refer to something that isn’t on the web rather than a fragment of a document, for example #*.
Change to recommending a single best practice of using hash URIs for resources that aren’t on the web, and in particular recommending having a one-to-one correspondence between resources on the web and those not on the web, using one particular conventional hash URI. For example, http://www.whitehouse.gov/#* would identify the resource that http://www.whitehouse.gov/ is about: The Whitehouse. This ensures that new publishers of data won’t run into the problems with publishing using 303 redirections, because they won’t use that method of publication. It also removes choice, which helps adopters who can otherwise get overwhelmed with options and the trade-offs between them.
Allow publishers who are currently using 303 redirections to publish descriptions of resources identified using non-hash URIs to switch to providing a representation using a 200 status code, along with a method of indicating that the representation is the description of the resource rather than its content. This indicator could be:
- a new HTTP header or status code (though I’d prefer not)
- a Link: header with a particular relationship (eg ‘describedby’)
- a statement embedded in the response itself (eg a element in HTML)

If we did move to a web of things view, the main question would be how to provide an indicator that the representation of a particular resource is the content of that resource as opposed to being a description. It would help ease transition if this was a natural consequence of the current pattern of publication on httpRange-14-compliant sites, so for example, you’d want to consider the representation of a resource the content of the resource if you got to it:

when retrieving a hash URI, if it was the part of the URI before the hash
when following a 303 See Other redirection, if it was the target of the redirection
when following a ‘describedby’ link, if it was the target of the link

as well as if there was an explicit indicator within the representation that said the resource was an information resource.

Whichever decisions are made, I would personally like to see the concrete requirements on client behaviour that arise from these different publication practices, for example enabling a reuser to associate a license with a particular piece of content or a crawler to create RDF statements about URIs encountered on the web, to bring whatever decisions are made down to earth and less ignorable.

Precious Snowflakes

2012-03-10T11:56:36+00:00

Disclaimer: As usual, this post contains my personal opinion and does not reflect that of any organisation with which you might associate me.

The other day, I had a lovely conversation with some folks from the BBC about some of their future plans. In the course of the conversation, Michael Smethurst spoke about his frustration when dealing with people involved with particular programmes at the BBC, where every single one of them thinks their programme is a “precious snowflake”, completely unique, that simply can’t be treated in the same way as all the other programmes described on the site.

Michael’s point, of course, is that TV programmes have a hell of a lot of similarities with each other. They all have episodes and cast members and may have trailers or be available on iPlayer. When the BBC models them in the same way, they gain enormous efficiencies in their ability to store and access information about programmes: they can reuse code, share content between programmes, and perform analyses over the aggregated data set. It’s great for users too: they get the same fantastic user experience no matter which programme they are viewing information about, and can apply the experience they gain when navigating pages about one programme when they need to find information about another.

The ability to classify and categorise, to bring order to what seems like chaos, to create a model of the world, is one of the things that marks humans from animals. We can look at a hundred people, with different colour hair and skin; different height and build; smiling, talking, crying, and still call them all Person because the essential characteristics that govern how we interact with them are the same.

But if there’s one thing that the last five long, hard years working with legislation has taught me, it’s that in any vaguely interesting domain, this search for order will always fall down in the face of reality.

Surely, I thought in my naive early days, every piece of legislation is uniquely identified through its type, calendar year, and number? Not so! There are six items for which this is not the case, because prior to 1963 legislation was numbered based on the year of reign of the monarch rather than the calendar year.

Surely the year that is used to number legislation is dependent on the date it is made and written into law? Not so! Sometimes departments forget to register legislation they make until the following year, so it is numbered the year after it’s made.

Surely an item of legislation can only make changes to legislation from the day it is written into law? Not so! There is, rarely, legislation that rewrites history: that says other legislation should always have had different content to that which was originally written.

It has come to the point where I never (hah!) make any statements about legislation of the form “X never happens” or “Y is always true” because there is always, always, an exception.

What this has taught me, as a developer, is the power and necessity of escape hatches. For example, templating languages that provide a method of escaping to code are so much more valuable than those that do not. Similarly, I favour strongly, in the technologies that I use, the ability to extend a common data structure, be it through data-* attributes in HTML, through generic elements such as and

or through the essentially open-ended nature of RDF as a data model.

It has also given me a very different view of the world to Michael. Because when you accept that there are always exceptions, you do not see snowflakes as merely crystals of water, but as exceptional, beautiful and, yes, immensely precious.

And this is why I love the web. The web does not force every site to have the same structure or the same look and feel. It does not insist on consistency; it has space for every quirk. And it proves beyond all doubt that it is possible for all these precious snowflakes to exist in a single, global, interlinked information system in which people manage to find not only the information that they need, but also community and connection with each other.

Inside Government

So it is with these eyes that I look at the new Inside Government pages on the gov.uk site and am frankly horrified. Because we’re not just talking about a BBC programmes here, but about powerful institutions, many of them decades if not centuries old, that lie at the very heart of government and how our nation is run. And each of them is relegated to a subfolder of a subfolder of a subfolder, their unique histories and approaches and goals expressed through three pictures on a carousel.

It feels like some kind of Orwellian nightmare: the relentless focus on user needs leading to a future of identikit pages, with no individuality, no character, no clue that behind these pages — which, remember, under the Single Government Domain policy becomes the single authoritative view, the site that represents the department on the web — is a living and breathing institution that manages hugely important parts of our lives. A future in which what each department says and the way that it says it is governed through the Government Digital Service (GDS), in Cabinet Office, the hand of the prime minister. And now we’re talking about local government too?

Let us just look at one example. Last September, William Hague gave a speech in which he described the hollowing out of the Foreign and Commonwealth Office (FCO) by the previous government, a process that scrapped its language school, closed embassies and destroyed its library. He said:

Strong institutions are necessary in civil society, to encourage participation and keep in check an overmighty State; they are necessary to our judiciary and Parliament so that the law is upheld and the making of it respected; but they are also necessary within the State, a point tragically overlooked by those Prime Ministers who have created and abolished departments on a fancy or a whim, destroying as they did so the pride and continuity of thousands of public servants while rendering government incomprehensible to the average citizen. The whole country should know what the Foreign and Commonwealth Office is and what it does, and all those interested in foreign policy at home or abroad should see it as a centre of excellence with which they aspire to be associated.

For most UK citizens, the only point of access to the Foreign and Commonwealth Office is its website: they will not visit King Charles Street, nor any of the UK’s embassies. The department’s web presence is the only way that it makes itself, and its unique role, comprehensible to the average citizen, the only method of letting the whole country know what the FCO is and what it does. And they have content that is completely unique to them: a database of Treaties, a hugely rich set of information on travel and living abroad and a wealth of historical information about the Foreign Office. This simply doesn’t fit in a model of a department as a set of Ministers, Policies, Publications and so on. And if it doesn’t fit, will it simply be excluded, lost from its website like its language school, its embassies, its library?

I could have picked any government department here — each one has its unique characteristics and content — but Hague articulates the case around FCO so well. His message is not the expression of a simple conservative impulse to resist change and preserve the status quo, but about maintaining the integrity of an institution’s identity and independence to encourage participation and keep in check an overmighty State. If we believe in Open Government, Open Democracy and the power of the web to enhance civic engagement then we must, surely, enable each of these institutions to have their own independent voice on the web.

I am reminded of the XKCD:

The two sides of this Venn diagram illustrate two approaches to building a website for an organisation. On the left is the website as an expression of the identity of the institution, on the right the website as a means of satisfying the reason the user originally visited the site. My argument is not that the right side of this diagram is unimportant — in fact I believe it is absolutely essential — but that an institution’s website must cover the entire space: it must provide a mechanism for self-expression as well as catering for its user’s requirements. To enhance civic engagement, we do not need to simply answer the query that led the user to the site, but to encourage and lead them on to see more about the institution that has provided the answer.

It is only the institution itself that knows the self it wants to express, and because the real world is complex and organisations are unique, that self will not fit into any model that we devise. News, Policies, Consultations — of course these are all important to all departments, but they are the tip of an iceberg. Look at the space that FCO devotes to its history on its website: this shouts to the world the kind of reliable, solid and flexible organisation that they are and want to remain. Compare how DECC devotes space to statistics, emphasising its adherence to transparency and evidence-based policy. Self-expression is so much more than changing logos or backgrounds, more than having different content on an About page, it is about making space for the things that are important to you.

“But but but!” I know the arguments. We must cut costs, stop the uncontrolled proliferation of government websites; we must improve the quality of the government’s presence on the web, present a unified view, make it easy for users to locate content without knowing where to look. The vision we see expressed through Inside Government is but the natural conclusion, the end of that slippery slope. But it is the great slippery slope fallacy that everything must be taken to its natural conclusion, that because 750 websites is too many, one is enough.

Possibly the biggest irony of the gov.uk beta is that while it is delivering a Single Government Domain — everything is to be found under www.gov.uk — it does not seem to address the core reason stated for providing it. In Martha Lane-Fox’s letter to Francis Maude, which kicked off this whole endeavour, she said:

Government publishes millions of pages on the Web, via hundreds of different websites. Most of these sites are still run as silos within departments. This fragmentation leads to significant duplication of functions and technology, and means the overall user experience is highly inconsistent.

Try searching for ‘Single Government Domain’ on the main gov.uk site or for driving test centres on Inside Government. The searches do not work (though Inside Government does give you a link that enables you to search the other silo). The result pages are completely different in feel except for the top and bottom banners. The page on Arrest and Imprisonment Abroad mentions but does not link to the Foreign and Commonwealth Office’s page. Yes, yes, I know that it’s still beta, but these things lie at the heart of the stated rationale for a Single Government Domain: is this the extent of the consistency and integration that we are aiming for?

Yes, it is, because the Single Government Domain policy was never truly about either of these things. Read Martha Lane-Fox’s letter again carefully (my emphasis):

No1O feel it is preferable to go from 750 top level website domains (eg www.cabinetoffice.gov.uk) to a single top level website domain for all of central government.

The Single Government Domain policy, indeed GDS itself, is about control. It is “we will do it for you”, not “we will help you do it”. It is about managing the output of institutions that might keep in check an overmighty State. It is anti-web and it is anti-democracy and I cannot remain quiet about it any longer.

To my friends at GDS: I respect and admire you all. You are incredibly talented and able to do amazing things. You have behind you a level of financial and political support the like of which most civil servants will never see. I know you have joined GDS not just to do work that you love but to do good for the country. This is my plea to you: find a way to avoid this vision. Nurture the exceptions. Give institutions their voice. Treat them as precious snowflakes.

Microdata and RDFa Living Together in Harmony

2011-08-20T17:39:11+01:00

One of the options that the TAG put forward when it asked the W3C to put together task force on embedded data in HTML was the co-existence of RDFa and microdata. If that’s what we’re headed for, what might make things easier for consumers and publishers who have to live in that world?

In a situation where there are two competing standards, I think that developers — both on the publication and consumption sides — are going to want to hedge their bets. They will want to avoid being tied to one syntax in case it turns out that that syntax isn’t supported by the majority of publishers/consumers in the long term and they have to switch.

Publishers like us at legislation.gov.uk who are aiming to share their data to whoever is interested in it (rather than having a particular consumer in mind) are also likely to want to publish in both microdata and RDFa, rather than force potential consumers to adopt a particular processing model, and will therefore need to mix the syntaxes within their pages.

(Of course developers might just avoid embedded data altogether while they wait to see what happens, but let’s assume that they want to press ahead regardless of the lack of consensus from the standardistas.)

I’ve therefore embarked on a task of trying:

to identify the differences in approach and functionality of the two languages, which should help developers choose between them
to identify any guidelines for developers of vocabularies for use with both languages
to identify a subset of functionality that is common between the two languages, which developers might want to stick to to make switching and mixing easier
to identify mapping rules that might be applied to automatically or manually map from one language to another if the simple subset is used

I’ve done this by looking at converting microdata examples to RDFa and vice versa, and the lessons to be drawn from that exercise. I’ve broken down the result into three posts:

This is the last of these posts. It is probably the only one you will want to read :)

Please treat this as a draft on which I’d welcome comments. I have based what’s written here on the latest specifications of both microdata (in its WHAT WG and W3C variants) and RDFa Core and HTML+RDFa but I haven’t consulted with anyone involved in these efforts and may well have got things wrong. Plus the specs are changing all the time. I have only here considered the syntax of the two languages, not the features such as DOM APIs or drag-and-drop support, where there are also clear differences.

Please add comments if there are things that I’ve missed or got wrong, or just to have your say.

Philip Jägenstedt’s Live Microdata service and Gregg Kellogg’s Distiller service have both proved invaluable for testing — thank you to both for making these services available. I heartily recommend them.

Mapping Rules

The first problem is how to judge equivalence when microdata and RDFa have different data models. Microdata essentially uses a JSON data model: there are objects (items) with properties that have values that are strings, other objects, or arrays of strings or objects or both. RDFa naturally uses a RDF data model: there are resources with properties that have values that are literals (of some datatype or with a language) or other resources.

Underlying both is the same basic entity-attribute-value pattern, but there are various mismatches between the models that make some mappings more complicated than others, or in other cases mean that information is necessarily lost on conversion.

In performing the analysis, I’ve tried to map microdata into sensible RDF and then match that RDF output using RDFa, and to map RDFa into sensible microdata+JSON and then match that microdata+JSON using microdata. The microdata-to-RDF mapping rules that I’ve followed are basically those outlined in my post on Microdata + RDF. To create microdata JSON from RDFa, I’ve used the rule that the URI of the first type of a resource is processed to provide a namespace that is stripped from the URIs of the properties (to create simple names where possible). In addition, when a resource has no properties, it will be represented as a string (URI) value rather than as a nested item.

These rules need to be formalised, obviously, but the basics above work well enough for the examples from the specs.

Mismatched Features

The following features are problematic when mapping from microdata to RDFa or vice versa. I’ve described them roughly in an order from things where it might be relatively easy to address the problem by changing one or other specification, to places where the necessary changes would be difficult to make in the specs, which means that publishers and consumers need to be aware of the issue so that they can make an educated choice about how they proceed.

Local Property Names

Many of the microdata examples involve items with no type and local property names. I’ve assumed in the analysis below that this generates properties whose URI is based on the document in which they are found, but this is not a helpful solution for data sharing: if a whole site uses short property names across its pages, those properties really need to be recognised as being the same across the site for any kind of useful processing to occur.

What microdata actually creates here is a global namespace, shared by everyone, specifically for embedded data. There are three things that could be done at different levels here:

In a mapping from microdata to RDF, any short property names on items that don’t have a type could be assigned to a global namespace (eg http://w3.org/ns/global/). Of course there will be clashes in semantics within this namespace, but that is true in microdata generally and not having to create a new namespace makes the initial experimentation easier for those starting with embedded data. The W3C (or whoever operates the namespace) could operate a wiki at that location that would operate as an informal registry for the property names.
HTML+RDFa could change to use this global namespace as the default vocabulary URI (rather than not having one). This would make it a little easier for people to convert microdata to RDFa: if they don’t use types for their items, there would then be no need for a vocab attribute to be added to the HTML. It also makes it possible to use RDFa in a basic, lightweight way, which might help people get started with it.
Publishers can be advised to use itemtype within their microdata, reusing existing classes or creating their own, if they want to ensure that the embedded data within their pages isn’t misinterpreted by global consumers.

Interpretation of Element’s `datetime` Attribute

Interpreting the datetime attribute of the element to supply a value, rather than repeating that value in a content attribute, is ISSUE-97 on RDFa, and hopefully RDFa will be changed to use that value (or the content of the element if there is no datetime attribute), add a seconds component if necessary, and work out an appropriate date/time datatype for it based on its syntax.

Content Overrides

In RDFa, publishers can provide a machine-readable version of the content of an element (or even an entirely different value) using the content attribute. This can only be done for date/times in microdata. The ability to annotate non-date/time content with machine-readable values is a current issue on HTML5. Resolving this in favour of providing such annotation would make using RDFa and microdata in concert, or converting between them, easier, particularly if HTML5 uses the attribute content or RDFa adopts whatever attribute is introduced to HTML5.

and Elements in Flow Content

The ability to use and elements in flow content is only supported in microdata: it’s support that’s added by the microdata specification (in the Editor’s Draft since May 31st; the text allowing this didn’t make it into the Last Call version of the spec), in which it’s limited to and elements with an itemprop attribute.

It would be possible for the RDFa specification to similarly make the statement that and elements are allowed in flow content as long as they have particular attributes. This would ease the transition between the two formats, and works a lot better than empty elements which crop up fairly commonly in RDFa content.

(One oddity here is that because date/time values have to be on a element in microdata, publishers cannot replace empty elements with elements as they might an empty .)

Identifiers without Types

Many of the RDFa examples are of resources that have a URI identifier but for which no type is supplied. Microdata, on the other hand, states that itemid is only allowed on elements that also have an itemtype (and an itemscope). The reason given is because the itemid needs to be interpreted based on the itemtype. This would be understandable if it held a string, but given that the itemid provides a URI it seems a bit strange. Perhaps it’s an attempt to avoid the whole httpRange-14 / ambiguity in URIs issue.

If this restriction remains, the advice to RDFa users who might want to convert to microdata at a future date would be to always provide a type for your (non-blank-node) resources. It may be useful to define a http://w3.org/ns/global/Thing within the vocabulary that I propose above, given that the URI for rdfs:Resource is long and hard to recall.

Built-in Prefixes

The built-in profile for RDFa defines a number of prefixes for vocabularies that are either coined by the W3C or coined elsewhere but in common use on the web. This, coupled with vocab and the ability to directly use URIs in the relevant attributes, means that declaring prefixes within the document is increasingly unnecessary in RDFa.

In contrast, using existing vocabularies, even popular ones, within microdata is relatively difficult, particularly when vocabularies are mixed on the same item.

Most useful for publishers would be if both RDFa and microdata recognised the same set of prefixes. This would reduce the size of microdata created from existing RDFa content as well as making it easier to move between the languages. At the very least, it would be good to have rdf:, rdfs:, xsd: and xhv: built into both.

The list of popular vocabularies is likely to change over time; for example a prefix for the schema.org vocabulary might be useful at some point in the near future. The problem is that publishers and consumers need to be synchronised in their use of prefixes: it’s no good for a publisher to use the prefix sch: if there might be processors for the page that don’t recognise it. Equally, consumers shouldn’t be reliant on a network connection to retrieve the latest set of prefix mappings in order to parse the page. It’s not clear to me how best to manage this evolution, but even a fixed set of prefixes at the point the specs reach Recommendation is more usable than spelling out URIs all the time.

Literals Including Markup

RDFa supports literals that include markup (the innerHTML of an element) as well as those that don’t (the textContent of an element), whereas microdata only supports creating values from particular attributes or the textContent of the element. This makes it hard to create embedded microdata that includes values which contain things like mathematical or chemical formulae, ruby text, or multiple paragraphs.

A solution would be for microdata to introduce an itemhtml (or something) attribute that, when present, indicates that the value of the property should include markup. There is a current issue on microdata to support HTML values.

Itemref

RDFa can support a subset of itemref’s functionality, namely to have properties defined elsewhere in a document be associated with a given resource. What it doesn’t support is the sharing of properties defined in one place by two or more resources.

RDFa could add such support by adding an attribute that mirrors itemref (eg ref, I guess), with the referenced element being processed using the evaluation context inherited by the referencing element (which means that attributes such as vocab would sometimes have a scope that wasn’t based on the document tree). This would make it easier to tackle the use case for itemref using RDFa as well as making it easier to move between or mix RDFa and microdata.

Lists

It is easy for microdata to represent a property with a list of values, and really really hard to do the same in RDFa. This is in part because RDF views lists resources rather than a distinct data type, and in part because RDFa hasn’t added any syntax sugar to make creating rdf:List resources easy. Adding some syntax sugar for lists would make life a lot easier for anyone using RDFa, but especially if they are adapting existing microdata content to RDFa.

Datatypes

Microdata assumes that consumers will convert values to appropriate datatypes based on the property (which they understand) as a separate process after microdata processing, whereas RDFa supports the use of a datatype attribute to explicitly indicate the datatype of each value. This mismatch means that information is lost when RDFa is converted to microdata, and has to be added when microdata is converted to RDFa.

Bringing the languages completely into sync would mean either microdata adding a facility to support (at least some) datatypes, or deprecating the datatype attribute in RDFa. Alternatively, this may simply be an area where the differences in behaviour between the two specifications doesn’t matter because the data models that they use are distinct anyway.

Languages

Languages are similar to datatypes, in that RDF (and hence RDFa) supports annotating strings with the language that they are in whereas microdata doesn’t within its core data model or its JSON serialisation. However, the elements that represent properties within the HTML, used within the DOM API access to microdata, will have a language.

It may be that in practice consumers need to base their microdata processing on the DOM API rather than the core microdata data model or JSON extracted through a standalone process, and thus pick up the language from the property elements, I don’t know. In any case, the microdata JSON serialisation, used for drag-and-drop, is lossy and could be extended to include the language of each value when available, at fairly substantial complexity cost.

For publishers, it doesn’t much matter either way; if they are dealing with multi-lingual text they will want to include a lang attribute in the HTML anyway, regardless of the impact on embedded data.

Multiple Types

RDFa supports having multiple types named in the typeof attribute whereas microdata only supports one type per item. In any mapping from RDFa to microdata, publishers have to choose which type is the primary type for the item and move the others to be expressed via rdf:type properties. Consumers who want to support publishers who might not choose their type as the primary type have to detect items that have the type they are interested in within the rdf:type property as well as those which have the type as the main type. Given that the rdf:type URI is long and (naturally) associated with RDF, it might be better to define a property such as http://w3.org/ns/global/type for this use.

Microdata could be extended to allow multiple values in the itemtype attribute, with the first being used to interpret any properties that aren’t full URIs. This would make it easier for both consumers to detect when a type they were interested in was used and for publishers to use RDFa and microdata in tandem or move between them.

The `src` Attribute

RDFa and microdata interpret the src attribute in opposite ways. In RDFa, it provides the identifier for a new resource (equivalent to itemid in microdata); in microdata, it provides a URL value of a property on elements that support it (equivalent to resource or href in RDFa).

RDFa interprets src in this way to make it easier to make assertions about an image, but it’s of limited effect as even in RDFa its only possible to make three such assertions (through the typeof, rel and property attributes). So, for example, you can specify the type of the image, link to its license and give the name of its creator, with:

but this won’t help you if you also want to give the title for the image and when it was created (say). At that point, the microdata and RDFa start to look similar:

or:

and really, to make the markup consistent, you may as well not use the src of the image at all in the RDFa either:

So it’s not clear to me that interpreting the src attribute as the subject of triples offers such a huge advantage that it’s worth the inconvenience that it brings for the simple things, such as having to use:

rather than:

Link relations

This isn’t so much a clash between RDFa and microdata as between the interpretation that RDFa has for the rel attribute and that specified in HTML.

The built-in rel values in HTML are a bit of a mix. Some of them, like alternate, prev and next encode relationships between the document in which the link appears and another document. Others, such as bookmark and help, create relationships between the context in which the link is found and the referenced document. Still others, like nofollow, noreferrer and prefetch, are really instructions to the client about how to manage the act of traversing the link.

It doesn’t seem semantically correct to automatically create relationships based on the built-in HTML rel values, unless you are deliberately trying to extract document semantics from the page. This is a problem for RDFa, which reuses the rel attribute to provide property values for the embedded data.

One thing that could be done would be for RDFa to consistently use the property attribute everywhere rather than the rel attribute. This would not only ease the overloading but also reduce the confusion for users, who currently have to work out which attribute to use based on whether the value is a resource or a literal.

Possible Subset of RDFa

When mapping from microdata to RDFa, the only attributes that are really needed are:

vocab to define a vocabulary for the types and properties within its scope (not technically necessary, but keeps the markup simple compared to spelling out URIs for everything)
typeof to define the type of a resource or indicate a new blank node
about to provide a URI for a resource or a local identifier for a blank node
property and rel to define property names (though see above for discussion about dropping rel)
href, src and content to provide values (and datetime assuming that is supported)

In the mappings in the analysis below, I did also use the resource attribute, but only to create a reference to a blank node that was described elsewhere, when replicating the functionality of itemref. If RDFa were to enable and in content in the same way as microdata, resource functionality could be replicated using ; as it is, you can get away with using an empty element.

Similarly, I only used datatype when providing a datatype for date/time values, something that could be done automatically by RDFa. But this isn’t surprising given that microdata doesn’t support datatypes at all and the examples I was using for the mapping were from the microdata specification.

There was no need for:

prefix which defines prefixes to simplify references to properties and classes; this is hardly surprising as few of the microdata examples involved mixing namespaces, but it’s notable that the built-in prefixes of rdf: and xsd: were useful
profile which is a pointer to an external document that defines a set of terms; this is being dropped from RDFa in any case

I also kept to a simplified version of the syntax in which each property element only provided one value. This subset is basically:

resource elements can have about (equivalent to itemid) and typeof (equivalent to itemtype) attributes on them
property elements can have property or rel (equivalent to itemprop), and a value-providing attribute on them such as href or content
no element is both a resource element and a property element; to provide a property whose value is a resource, nest the resource element within the property element (using “hanging rel” processing)
no property element should provide more than one value for a property; in particular, a “hanging rel” should only have a single resource element child

This simplified profile of RDFa is fairly easy to remember and maps easily to and from microdata: most attributes can be simply renamed; the only attribute that needs to be moved as well as renamed is the “hanging rel”, which moves onto the resource element and is renamed to itemprop. Note that it also means avoiding using the src attribute to encode embedded data.

In addition to sticking to this subset of attributes, developers might be advised that using HTML link relations may lead to clashes with browser or search engine interpretation of the links in the page.

Possible Subset of Microdata

Microdata is pretty minimalistic already. The only feature that developers need to be warned about is itemref, which has no RDFa equivalent at the moment.

Guidelines for Vocabulary Authors

There are a several guidelines that come out of this comparison for people putting together vocabularies that aim to be usable in both RDFa and microdata:

The classes in the vocabulary should be distinct, or subclasses created with any relevant combinations of superclasses, so that publishers don’t have to assign more than one type to an item/resource. This restriction helps with using the vocabulary with microdata, which assumes that every item has a single type.
Provide explicit classes for everything which you anticipate might be given an identifier, as microdata doesn’t (currently) enable items to have an identifier without also having a type.
Put classes and properties in the same namespace, but do not name classes and properties with the same local name; while this doesn’t matter in microdata because the properties are interpreted relative to the class, standard conversions to RDF will create a class and a property with the same URI. URIs are case-sensitive to a simple way of ensuring that there aren’t clashes is to follow the usual RDF convention of beginning class names with an upper-case letter and property names with a lower-case letter.
Avoid property names that contain dots, as these aren’t allowed in non-URI property names in microdata.
Ensure that properties either only expect one type of value or expect values whose type can be sniffed based on the syntax of the value. If publishers use microdata, they will not be able to indicate the type of a value through the markup.
Be aware that consumers of microdata using your vocabulary will have to use the DOM API to identify the language used in any strings, and that language information won’t be carried through the standard microdata JSON serialisation (used by drag-and-drop, for example). If you anticipate multi-lingual use of your vocabulary, you may way to define a MultiLingual class with value and language properties that people can use as nested items. (It may be useful for this class and properties to be defined in the proposed ‘global’ W3C namespace so that it can be used anywhere.) If you know what languages will be used then provide separate properties for each language (eg for UK legislation I know the languages are English and Welsh so on a vocabulary for UK Legislation I could have title-en and title-cy properties).
To make markup cleaner, only reuse properties from other vocabularies on your classes if they have built-in prefixes (eg unless rdfs: is built-in to microdata as well as RDFa, don’t use rdfs:label to provide a label, but create your own label property). On the other hand, do reuse classes from other vocabularies if you don’t need to add any specialised properties to them. Note that avoiding reuse has the unfortunate side-effect of not enabling processors that understand these other vocabularies to process your data.
Avoid having properties whose values need to be retrieved in order, as these are hard to represent in RDFa. Instead, use properties with distinct names when position is important. (Yes, I know this sucks.)

Choosing Between Microdata and RDFa

The choices developers make between microdata and RDFa will, I suspect, be largely dictated by what their consumers/toolsets/publishers will support. Nevertheless, there are some features that are better supported by one or other format and might therefore sway developers one way or another:

multi-lingual embedded data is better supported in RDF than microdata+JSON
explicit datatypes for values can be provided by RDFa but not microdata
resources with multiple types are a lot easier to describe in RDFa
property values that include markup are a lot easier to write in RDFa
mixed vocabulary use is a bit easier in RDFa than in microdata
HTML5 link relations may be misinterpreted by RDFa processors
properties with list values are much easier to support in microdata
common content adopted by multiple entities is much easier in microdata

Final Words

I have no doubt that developers would be better off if there were only one recommended way of embedding data in HTML (so long as it met their requirements of course). But realistically that is, and always has been, a long shot, given the entrenched positions of the microdata and RDFa communities.

Regardless, there are lessons that RDFa and microdata could learn from each other, and changes to both languages that would help developers use them on their own, switch between them and mix them in the same document. I expect and welcome debate about the viability and effectiveness of the changes and guidelines that I’ve suggested here.

Investigating those lessons, documenting those changes and generating those guidelines was something that I had hoped the microdata/RDFa task force would be able to do. The other question to ask, given the argument that there shouldn’t be a task force at all if it’s not going to be able to bring the languages together, is whether this kind of analysis is worthwhile, and worth publishing as something more official than a blog post?

Mapping RDFa to Microdata

2011-08-20T17:38:38+01:00

This post is part of a three-part series that analyses the differences in features and syntax between microdata and RDFa. The series attempts:

to identify the differences in approach and functionality of the two languages, which should help developers choose between them
to identify any guidelines for developers of vocabularies for use with both languages
to identify a subset of functionality that is common between the two languages, which developers might want to stick to to make switching and mixing easier
to identify mapping rules that might be applied to automatically or manually map from one language to another if the simple subset is used

I’ve done this by looking at converting microdata examples to RDFa and vice versa, and the lessons to be drawn from that exercise. The three posts are on:

This post is the second of these, which looks at how RDFa might be mapped to microdata. In this case, I will aim to express the RDF created from the RDFa as the equivalent microdata JSON, and aim to create that JSON with the microdata.

To create the microdata JSON, I’ve used the rule that the URI of the first type of a resource is processed to provide a namespace that is stripped from the URIs of the properties (to create simple names where possible). In addition, when a resource has no properties, it will be represented as a string (URI) value rather than as a nested item. Other than that I hope the mapping will be obvious; I’ll point out where it involves a loss of information. I’m assuming that the document is at http://example.org/ throughout.

I have based what’s written here on the latest specifications of both microdata (in its WHAT WG and W3C variants) and RDFa Core and HTML+RDFa but I haven’t consulted with anyone involved in these efforts and may well have got things wrong. Philip Jägenstedt’s Live Microdata service has proved invaluable for testing, so many thanks to him for providing that service.

The post is rather heavy going and you might want to just skip to the summary instead of reading the whole thing.

The post goes through the examples from the RDFa specification plus one additional example from the wild. I haven’t included examples that don’t illustrate anything new, so there are some that are skipped. Other examples would be welcome.

Page Metadata

When parsing begins, the current subject will be the IRI of the document being parsed, or a value as set by a Host Language-provided mechanism (e.g., the base element in (X)HTML). This means that by default any metadata found in the document will concern the document itself:
  
    Jo's Friends and Family Blog
    
    
  
  
    ...
  

The equivalent microdata JSON for this document that we’d want to create is (note: invalid example):

{ "items": [
  {
    "id": "http://example.org/" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://example.org/#bbq" ],
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  }
]}

and we’d want to create it with (note: invalid example):


  
    Jo's Friends and Family Blog
    
    
  
  
    ...

However, this is is not valid according to the microdata specification. In microdata, only items that have types are allowed to have identifiers. Rather than losing the identifier, we’ll add a type; I’m going to use rdfs:Resource. It’s not the nicest of URIs to type, but it’s got something close to the correct semantics. So we’ll aim for the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://example.org/#bbq" ] ,
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  }
]}

which means we need:


  
    Jo's Friends and Family Blog
    
    
  
  
    ...

Notes:

The itemscope is necessary for the page to be recognised as containing any data at all.
The itemid can’t just be empty: the . is the shortest URI you can use to reference the page itself.
I put the itemscope, itemtype and itemid on the element rather than the element so that they wouldn’t be inherited into the : it seems to make sense for any data within the to be about the page itself.
The foaf: and dc: prefixes are built-in to RDFa, so it’s easy for people to use classes and properties in those common vocabularies without having to remember their full URI. In microdata, that URI and the one for the rdfs:Resource class have to be spelled out in full.

Base URI

In (X)HTML the value of base may change the initial value of current subject:
  
    
    Jo's Friends and Family Blog
    
    
  
  
    ...
  

This changes the id of the item generated:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://www.example.org/jo/blog" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://www.example.org/jo/blog#bbq" ] ,
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  }
]}

In the microdata, the itemid can still be . as the base URI set by the element is used to resolve it:


  
    
    Jo's Friends and Family Blog
    
    
  
  
    ...

Explicit Subjects / ItemIds

To illustrate how this affects the statements, note in this markup how the properties inside the (X)HTML body element become part of a new calendar event object, rather than referring to the document as they do in the head of the document:
  
    Jo's Friends and Family Blog
    
    
  
  
    
      I'm holding
      
        one last summer barbecue
      ,
      on
      
        September 16th at 4pm
      .
    
  

In microdata JSON, this would be:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://example.org/#bbq" ],
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  } ,
  {
    "type": "http://www.w3.org/2002/12/cal/ical#Vevent" ,
    "id": "http://example.org/#bbq" ,
    "properties": {
      "summary": [ "one last summer barbecue" ] ,
      "dtstart": [ "2015-09-16T16:00:00-05:00" ] ,
    }
  }
]}

Note that this mapping loses the fact that the value of the dtstart property is a date-and-time. Processors of this JSON are expected to know that the dtstart property takes a date/time value and would have to sniff the value to work out that it’s a date-and-time rather than a date.

In-browser microdata processors can identify the value as a date/time value because the property element itself is accessed through the element.properties IDL attribute; processors that work with this DOM API can tell that it’s a element, get hold of the date/time itself and access the content of the element for the human-readable representation used on the page. However, this information isn’t part of the core microdata data model.

To create this JSON from microdata you need:


  
    Jo's Friends and Family Blog
    
    
  
  
    
      I'm holding
      
        one last summer barbecue
      ,
      on
      
        September 16th at 4pm
      .

Notes:

There are no prefix definitions in microdata, so the type has to be spelled out in full. However, with the mapping I’m assuming from RDFa to microdata JSON, the properties in that same namespace for items in that class don’t.
The itemscope has to be added despite the
element having both an itemid and an itemtype; if the itemscope is forgotten, the item isn’t recognised.
The original element has to be changed to a element because it isn’t conformant microdata for a date/time value to be supplied by any other element.

Items from the `src` Attribute

If @about is not present, then @src is next in priority order, for setting the subject of a statement. A typical use would be to indicate the licensing type of an image:

This should generate the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/photo1.jpg" ,
    "properties": {
      "http://www.w3.org/1999/xhtml/vocab#license": [ "http://creativecommons.org/licenses/by/2.0/" ] ,
    }
  }
]}

The src attribute in microdata is only used for a value, so creating the microdata about the image means a wrapper element and a separate element:

Note:

The license property is part of the built-in set of link relationships in HTML, but there is no easy way to refer to that property from microdata; they have to be spelled out as full URLs.

Additional Properties for Images

Since there is no difference between @src and @about, then the information expressed in the last example in the section on @about (the creator of an image), could be expressed as follows:

This is a simple additional property in the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/photo1.jpg" ,
    "properties": {
      "http://www.w3.org/1999/xhtml/vocab#license": [ "http://creativecommons.org/licenses/by/2.0/" ] ,
      "http://purl.org/dc/terms/creator": [ "Mark Birbeck" ]
    }
  }
]}

which can be created through a element within the :

Nested Images

Since normal chaining rules will apply, the image IRI can also be used to complete hanging triples:
  

This should generate the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://www.blogger.com/profile/1109404" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/img": [{
        "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
        "id": "http://example.org/photo1.jpg" ,
        "properties": {
          "http://www.w3.org/1999/xhtml/vocab#license": [ "http://creativecommons.org/licenses/by/2.0/" ] ,
          "http://purl.org/dc/terms/creator": [ "Mark Birbeck" ]
        }
      }]
    }
  }
]}

The microdata to generate this is:

Notes:

The big gotcha in this conversion is that in microdata, the foaf:img property has to be moved onto the item that is a value of that property; there’s no equivalent to the “hanging rel” processing in RDFa. A disadvantage of this is that anyone copying-and-pasting the element to embed the same information about the image within their own page will have the itemprop attribute carried along with the image, into a context where the foaf:img property might not be relevant.

Types with Blank Nodes

For example, an author may wish to create markup for a person using the FOAF vocabulary, but without having a clear identifier for the item:
  Albert Einstein
  Albert

Now we have an explicit type, we can create microdata JSON that uses short names:

{ "items": [
  {
    "type": "http://xmlns.com/foaf/0.1/Person" ,
    "properties": {
      "name": [ "Albert Einstein" ] ,
      "givenName": [ "Albert" ] ,
    }
  }
]}

This can be generated with the microdata:


  Albert Einstein
  Albert

which is nice and simple.

Inherited Subject

The most usual way that an inherited subject might get set would be when the parent statement has an object that is a resource. Returning to the earlier example, in which the long name for the German_Empire was added, the following markup was used:
  Albert Einstein
  1879-03-14
  
  the German Empire

The equivalent microdata JSON for this would be:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Albert_Einstein" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
      "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ] ,
      "http://dbpedia.org/property/birthPlace": [ "http://dbpedia.org/resource/German_Empire" ]
    }
  } , {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/German_Empire" ,
    "properties": {
      "http://dbpedia.org/property/conventionalLongName": [ "the German Empire" ] ,
    }
  }
]}

Note that this microdata JSON could only be generated syntactically from the RDFa, not via RDF, because going via RDF would make it impossible to know whether to give the dbp:birthPlace property a string (which is a URI) value or a nested item. We’ll see the alternative version of the microdata RDF in the next example.

To create this microdata JSON, we need:


  Albert Einstein
  1879-03-14
  
  
    the German Empire

Notes:

I’ve had to change two elements here: the for the date of birth has become a element as the value of the property is a date, and the
for the birth place has become a element because the value of that property is a URL.
I’ve also had to add a nested element as it’s not possible in microdata to have a single element describe both an item and a property for that item as it is in RDFa.

In an earlier illustration the subject and object for the German Empire were connected by removing the @resource, relying on the @about to set the object:
  Albert Einstein
  1879-03-14
  
    the German Empire
  

While this generates the same RDF as the previous example, the microdata JSON that it generates should probably be different: this time, the item for the German Empire is nested within the item for Albert Einstein:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Albert_Einstein" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
      "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ] ,
      "http://dbpedia.org/property/birthPlace": [{
        "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
        "id": "http://dbpedia.org/resource/German_Empire" ,
        "properties": {
          "http://dbpedia.org/property/conventionalLongName": [ "the German Empire" ]
        }
      }
    }
  }
]}

To create this, the microdata needs to look like:


  Albert Einstein
  1879-03-14
  
    the German Empire

Note that while this looks quite similar to the RDFa version, in fact the itemid attribute that holds the URI for the German Empire is on a different element from the about attribute in the RDFa.

The third RDFa example around this same content is:

but it is also possible for authors to achieve the same effect by removing the @about and leaving the @resource:
  Albert Einstein
  1879-03-14
  
    the German Empire
  

This should lead to the same microdata JSON, so I won’t bother repeating the microdata. What’s interesting is that this pattern: the wrapper element containing the property (rel) and identifier for the item that is the value for that property (resource) is a lot closer to the microdata pattern of expressing nested items. The big distinction here is that while in microdata, the itemtype also resides on that element, if you tried adding a typeof attribute to the inner

in RDFa, you’d end up with a new blank node.

Anonymous Nested Resources

However, an author could just as easily say that Spinoza influenced something by the name of Albert Einstein, that was born on March 14th, 1879:
  
    Albert Einstein
    1879-03-14
  

This should generate the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Baruch_Spinoza" ,
    "properties": {
      "http://dbpedia.org/ontology/influenced": [{
        "properties": {
          "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ]
        }
      }]
    }
  }
]}

which again means moving an attribute in microdata:


  
    Albert Einstein
    1879-03-14

It is generally harder to move to microdata from RDFa when the RDFa has an element that both provides a subject and provides a property.

The RDFa spec provides a couple of additional methods of marking up the same content to give exactly the same RDF (and microdata JSON):

Note that the div is superfluous, and an RDFa Processor will create the intermediate object even if the element is removed:
  Albert Einstein
  1879-03-14
An alternative pattern is to keep the div and move the @rel onto it:
  
    Albert Einstein
    1879-03-14
  
From the point of view of the markup, this latter layout is to be preferred, since it draws attention to the ‘hanging rel’. But from the point of view of an RDFa Processor, all of these permutations need to be supported.

Interestingly, it’s this latter permutation that is the one that’s closest to the microdata method of expressing the data, though as we will see in the next section, the “hanging rel” is not exactly equivalent to the itemprop on the wrapper element.

Hanging Rels

Note that each occurrence of @about will complete any incomplete triples. For example, to mark up the fact that Albert Einstein had a residence both in the German Empire and Switzerland, an author need only specify one @rel value that is then used with multiple @about values:
  
  

The data embedded here gives two values for the dbp-owl:residence property:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Albert_Einstein" ,
    "properties": {
      "http://dbpedia.org/ontology/residence": [
        "http://dbpedia.org/resource/German_Empire" ,
        "http://dbpedia.org/resource/Switzerland"
      ]
    }
  }
]}

In microdata, the itemprop attribute has to appear on both the nested elements to make it clear that they both provide values for that property:

The next example illustrates this with nested items rather than strings:

To illustrate, to indicate that Spinoza influenced both Einstein and Schopenhauer, the following markup could be used:
  
    
      Albert Einstein
      1879-03-14
    
    
      Arthur Schopenhauer
      1788-02-22
              
  

This should generate:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Baruch_Spinoza" ,
    "properties": {
      "http://dbpedia.org/ontology/influenced": [{
        "type": "http://xmlns.com/foaf/0.1/Person" ,
        "properties": {
          "name": [ "Albert Einstein" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ]
        }
      }, {
        "type": "http://xmlns.com/foaf/0.1/Person" ,
        "properties": {
          "name": [ "Arthur Schopenhauer" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1788-02-22" ]
        }
      }]
    }
  }
]}

In this case, the itemprop that is equivalent to the RDFa rel has to move down onto the elements representing the items:


  
    
      Albert Einstein
      1879-03-14
    
    
      Arthur Schopenhauer
      1788-02-22

The wrapper

around both items isn’t necessary; I’ve left it to stay as close to the markup of the original RDFa as possible.

Implicit Resources

Triples are also ‘completed’ if any one of @property, @rel or @rev are present. However, unlike the situation when @about or @typeof are present, all predicates are attached to one bnode:
  Albert Einstein
  1879-03-14
  
    
    
  

To be equivalent to the RDF generated from this markup, the microdata JSON would be:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Baruch_Spinoza" ,
    "properties": {
      "http://dbpedia.org/ontology/influenced": [{
        "properties": {
          "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ]
          "http://dbpedia.org/ontology/residence": [
            "http://dbpedia.org/resource/German_Empire" ,
            "http://dbpedia.org/resource/Switzerland"
          ]
        }
      }]
    }
  }
]}

Microdata is a lot more explicit about when items get created, and consequently requires a bit more markup:


  
    Albert Einstein
    1879-03-14

Overriding Text Content

The value of @content is given precedence over any element content, so the following would give exactly the same triple as shown above:
John Doe

The equivalent microdata should generate the JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://internet-apps.blogspot.com/" ,
    "properties": {
      "http://purl.org/dc/terms/creator": [ "Mark Birbeck" ]
    }
  }
]}

Only the element and links override the content of an element in microdata. So a mirror of this example needs a separate element:

  
    
    John Doe

Language Support

In RDFa the Host Language may provide a mechanism for setting the language tag. In XHTML+RDFa [XHTML-RDFA], for example, the XML language attribute @xml:lang or the attribute @lang is used to add this information, whether the plain literal is designated by @content, or by the inline text of the element:

Like the datatype of a value, the language of a value isn’t captured by the microdata data model or the JSON representation of that data model. So the fact that ‘chat’ is French is lost:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/node" ,
    "properties": {
      "http://example.org/property": [ "chat" ]
    }
  }
]}

The equivalent microdata is thus:

with the language only accessible if you are using the DOM to process the microdata.

Literals that Include Markup

RDFa therefore supports the use of normal markup to express XML literals, by using @datatype:
  E = mc²: The Most Urgent Problem of Our Time

The datatype="rdf:XMLLiteral" acts like a flag to indicate that the serialised content of the element (innerHTML) needs to be used as the value of the property, rather than the textContent, which includes markup, can be expressed in microdata JSON as follows:

{ "http://purl.org/dc/terms/title": "E = mc²: The Most Urgent Problem of Our Time" }

There’s no way to generate this in microdata except by repeating the escaped version of the content in a content attribute:


  E = mc²: The Most Urgent Problem of Our Time

This is hardly ideal. It’s tedious enough with a short string like this one; for larger amounts of information such as long descriptions of an event, it would be very tedious.

The `resource` Attribute

RDFa provides the @resource attribute as a way to set the object of statements. This is particularly useful when referring to resources that are not themselves navigable links:
  
    On Crime and Punishment
    
  
  
    
      
        Rodion Romanovitch! My dear friend! If you go on in this way
        you will go mad, I am positive! Drink, pray, if only a few drops!
      
    
  

This should produce:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://www.example.com/candp.xhtml#q1" ,
    "properties": {
      "http://purl.org/dc/terms/source": [ "urn:ISBN:0140449132" ]
    }
  }
]}

which is expressed through:


  
    On Crime and Punishment
    
  
  
    
      
      
        Rodion Romanovitch! My dear friend! If you go on in this way
        you will go mad, I am positive! Drink, pray, if only a few drops!

The property and value have to be moved onto a nested , but this is a more extensible pattern than the RDFa method as it enables other properties to be expressed in the same way.

Multiple Types

This last example comes from the wild rather than being an example in the specification. At http://bitmunk.com/browse we find:

Bitmunk

This shows the use of multiple types and of multiple properties with the same value, because the pages are attempting to use multiple vocabularies that cover the same domain (organisations) to different depths. In the equivalent microdata, we have to choose one of the types; I’m going to assume that it should just use the first one from the typeof attribute:

{ "items": [
  {
    "type": "http://www.w3.org/2006/vcard/ns#VCard" ,
    "id": "http://bitmunk.com/about#service" ,
    "properties": {
      "http://www.w3.org/1999/02/22-rdf-syntax-ns#type": [
        "http://purl.org/commerce#Business" ,
        "http://purl.org/goodrelations/v1#BusinessEntity"
      ] ,
      "http://www.w3.org/2000/01/rdf-schema#label": [ "Bitmunk" ] ,
      "fn": [ "Bitmunk" ]
    }
  }
]}

The microdata equivalent is:


  
  
  Bitmunk

Notes:

Technically, the RDFa doesn’t place any ordering on the three classes, but I’m picking the first for the purpose of the microdata conversion. The other classes are harder to get at in the JSON: they have to be referenced via the rdf:type microdata property rather than the type JSON property. Consumers that are on the lookout for items of the type gr:BusinessEntity wouldn’t spot these items.

Mapping Microdata to RDFa

2011-08-20T17:35:28+01:00

This post is part of a three-part series that analyses the differences in features and syntax between microdata and RDFa. The series attempts:

to identify the differences in approach and functionality of the two languages, which should help developers choose between them
to identify any guidelines for developers of vocabularies for use with both languages
to identify a subset of functionality that is common between the two languages, which developers might want to stick to to make switching and mixing easier
to identify mapping rules that might be applied to automatically or manually map from one language to another if the simple subset is used

I’ve done this by looking at converting microdata examples to RDFa and vice versa, and the lessons to be drawn from that exercise. The three posts are on:

This post is the first of these, which looks at how microdata might be mapped to RDFa, in terms of generating the same RDF according to the microdata-to-RDF mapping rules that I outlined in my post on Microdata + RDF.

I have based what’s written here on the latest specifications of both microdata (in its WHAT WG and W3C variants) and RDFa Core and HTML+RDFa but I haven’t consulted with anyone involved in these efforts and may well have got things wrong. Gregg Kellogg’s Distiller service has proved invaluable for testing, so many thanks to him for providing that service.

The post is rather heavy going and you might want to just skip to the summary instead of reading the whole thing.

The post goes through the examples from the microdata specification (most of them are in both versions, the only exceptions being those that use the vCard vocabulary). I haven’t included examples that don’t illustrate anything new, so there are some that are skipped. Other examples would be welcome.

Unidentified Items / Blank Node Subjects

Here there are two items, each of which has the property “name”:
 My name is Elizabeth.



 My name is Daniel.

The first challenge is to map this into RDFa because the properties are tokens rather than URIs and there is no type for either of the items. What I’ll assume here is that the name properties are local to the document itself and thus the equivalent RDF is:

[ <#name> "Elizabeth" ] .
[ <#name> "Daniel" ] .

This can be achieved in RDFa through either:


  My name is Elizabeth.



  My name is Daniel.

or:


  My name is Elizabeth.



  My name is Daniel.

Notes:

The vocab="#" sets the vocabulary to the location of the current document (plus an empty fragment identifier); this URI is then concatenated to the property token (name) to create a URI that is unique to the document. In a document such as this it would make sense to put the vocab="#" attribute on the element rather than on every single item.
With no type in sight, blank nodes can either be created by having an empty typeof attribute or through an about attributes whose value starts with _:. The latter has the advantage of providing an identifier for the blank node that can be used elsewhere in the document, but the former is shorter so will be used where possible in the remaining examples of this post.

Values from the `src` Attribute

The next example introduces the use of the src attribute to set the value of the property.

In this example, the item has one property, “image”, whose value is a URL:
 

This should probably be mapped to the RDF:

[ <#image>  ] .

The difficulty with this is that in RDFa, the src attribute is used for the subject of a statement (equivalent to a microdata item) rather than the object (equivalent to a microdata value). So we have two choices for equivalent RDFa. One is to use a similar pattern to that used above, but introduce a wrapper element that provides the property:

Another is to provide what would normally be an object through a resource attribute and then use a rev attribute (rather than the usual rel) attribute to reverse the relationship:

This has three disadvantages over the first option:

the resource attribute that creates the item is on the element rather than on the wrapper
which makes it hard to create other properties for that item
we have to use a rev attribute, reversing the normal flow of relationships; I (at least) find this hard to figure out when there’s not a rel attribute as well
we have to make up an id for the blank node we want to generate

I’ll note that it took me five or six failed attempts to generate the above options. If I hadn’t had the RDF Distiller to test with, I would have got it wrong. Note that at least through the RDF Distiller, to be recognised, the resource attribute has to have an (empty) value — it is not enough for it to simply be present, unlike with the typeof attribute. Note that the resource attribute has to explicitly point to a blank node to create a blank node rather than having the property be associated with the document in which this appears.

Values from the `datetime` Attribute

The next example illustrates the use of the element to provide a date/time value for a property.

In this example, the item has one property, “birthday”, whose value is a date:
 I was born on May 10th 2009.

I’m assuming this should map to the RDF:

[ <#birthday> "2009-05-10"^^ ]

There is an open issue (ISSUE-97) about this on RDFa, which currently requires the use of the content attribute to provide the value as follows:


 I was born on May 10th 2009.

Note that the xsd: prefix is built-in within RDFa so there’s on need for any declaration for it, which makes it fairly easy to specify the standard date/time datatypes.

If ISSUE-97 were resolved nicely it would be possible to instead do:


 I was born on May 10th 2009.

Notes:

To make this work, RDFa processors would have to look at the syntax of the datetime attribute to work out what datatype the value should be matched to.
The syntax permitted in the datetime attribute isn’t exactly the same as that permitted by the XML Schema time and dateTime types usually used in RDF (and XML), in that the seconds component is optional within HTML. The resolution to ISSUE-97 will need to take this into account. Otherwise, anyone mapping from microdata to RDFa manually will need to ensure that the content attribute includes the seconds component.

Nested Items / Object Properties

In this example, the outer item represents a person, and the inner one represents a band:
 Name: Amanda
 Band:  Jazz Band (12 players)
The outer item here has two properties, “name” and “band”. The “name” is “Amanda”, and the “band” is an item in its own right, with two properties, “name” and “size”. The “name” of the band is “Jazz Band”, and the “size” is “12”.

The equivalent RDF for this example would be:

[
  <#name> "Amanda" ;
  <#band> [
    <#name> "Jazz Band" ;
    <#size> "12"
  ]
]

Note that the size property is just a plain literal value; unlike with date/times, there’s no way to tell from the microdata that the value is a number.

In RDFa this could be done with:


 Name: Amanda
 Band:  Jazz Band (12 players)

This follows the microdata fairly closely but note that the nested resource doesn’t need an empty typeof attribute: it’s only the top-level items that do. It might be easier, for consistency and extensibility, to always include an explicit nested element (with an empty typeof attribute in this case) to represent the nested resource:


 Name: Amanda
 Band:  Jazz Band (12 players)

The other thing that people have to watch out for is that because the value of the band property is a resource rather than a literal, we have to use the rel attribute rather than the property attribute as we do elsewhere.

Itemref

This example is the same as the previous one, but all the properties are separated from their items:
Name: Amanda


 Band: Jazz Band
 Size: 12 players

This should create the same RDF as the previous example:

[
  <#name> "Amanda" ;
  <#band> [
    <#name> "Jazz Band" ;
    <#size> "12"
  ]
]

changing the markup as little as possible. The RDFa equivalent is:


Name: Amanda


 Band: Jazz Band
 Size: 12 players

In microdata, the itemref attribute is a method of an item adopting name/value pairs described in a separate location within the page. In RDFa, the equivalent is to say that the name/value pairs are all related to the same resource by consistently referring to the resource as the subject of the statements. In the above case, there are two blank nodes labelled _:amanda and _:c, and the about attribute is used on the same elements that provide the properties (or a wrapper element) to indicate the identity of the subject of the statements.

Notes:

The resource attribute has to be used to indicate the blank node for the band.
As before, the rel attribute has to be used for the band property, rather than the property attribute, because the object of the statement is a resource. The rule is that if you’re using resource, you should use rel. (I used property erroneously the first time I tried to write this mapping. I will never learn.)

There is another example of itemref in use later in the microdata specification:



 
  Photo gallery
 
 
  My photos
  
   
   The house I found.
  
  
   
   The mailbox.
  
  
   All images licensed under the MIT
   license.

This is equivalent to the RDF:

[
  a  ;
    ;
    ;
   "The house I found." ;
] .
[
  a  ;
    ;
    ;
   "The mailbox." ;
] .

Note that the license property is adopted by both the items in the microdata. In this particular example, the two items have the same type, and thus the license property has the same meaning in each item. It’s also possible for itemref to be used on two items that have different types, pointing to the same element, in which case the shared properties defined within that element could mean different things for the two items.

There is no way that I am aware of within RDFa to support shared use of portions of content. There could be a rough equivalent that would work in the case where the shared properties had the same semantics if RDFa allowed the about attribute to take multiple values (note: invalid example):



 
  Photo gallery
 
 
  My photos
  
   
   The house I found.
  
  
   
   The mailbox.
  
  
   All images licensed under the MIT
   license.

but this wouldn’t support the possibility of the same property having different semantics (and therefore different URIs) for the separate resources.

It’s also worth noting in this example that the mapping to RDF that I’m assuming results, in this example, in http://n.whatwg.org/work being both a class and a property. The creators of RDF vocabularies tend to name classes with an Uppercase initial letter and properties with a lowercase initial letter, and thus avoid these kinds of clashes. Vocabulary designers who are mindful of mappings to RDF may want to take the same approach.

Multiple Values

This example describes an ice cream, with two flavors:
 Flavors in my favorite ice cream:
 
  Lemon sorbet
  Apricot sorbet
 
This thus results in an item with two properties, both “flavor”, having the values “Lemon sorbet” and “Apricot sorbet”.

This example highlights one of the real nightmares of RDF: lists. In microdata, the order of the values ‘Lemon sorbet’ and ‘Apricot sorbet’ is naturally retained. There are three possible mappings to RDF.

Creating Multiple Statements

If the order of the flavours of ice cream in this example don’t actually matter, the equivalent RDF is:

[ <#flavor> "Lemon sorbet" , "Apricot sorbet" ]

which is equivalent to:

[ <#flavor> "Apricot sorbet" , "Lemon sorbet" ]

In this case, the RDFa is straight-forward:


 Flavors in my favorite ice cream:
 
  Lemon sorbet
  Apricot sorbet

It’s surprising how common it is that order doesn’t actually matter when there are multiple values for a property, such that this mapping is quite sufficient. But I’m absolutely not going to pretend that order is never important…

Creating an `rdf:Seq`

If the order of the flavours does matter, there are two ways of representing that order using RDF. The first is to use an rdf:Seq resource. This method was the original method of representing lists in RDF and is very natural to do in RDF/XML, but has largely fallen out of favour for the second method which I’ll describe below.

Using the rdf:Seq method, the equivalent RDF for the microdata would be:

@prefix rdf:  .
[
  <#flavor> [
    a rdf:Seq ;
    rdf:_1 "Lemon sorbet" ;
    rdf:_2 "Apricot sorbet"
  ]
]

which can be generated with:


 Flavors in my favorite ice cream:
 
  
   Lemon sorbet
   Apricot sorbet

Notes:

There are various other ways in which the namespace for the rdf:Seq could be created, but since the rdf: prefix is built-in to RDFa 1.1, it seems easier to use that than anything that explicitly writes out the full (ugly) RDF namespace.
The
wrapper for the
Creating a rdf:List

The current recommended way to create a list in RDF is to use a rdf:List resource. This essentially uses a linked list model to represent lists, with the rdf:first item of a list being a value and the rdf:rest being either another rdf:List or rdf:nil. Spelled out, the RDF would look like:
```
@prefix rdf:  .
[ 
  <#flavor> [
    a rdf:List ;
    rdf:first "Lemon sorbet" ;
    rdf:rest [
      rdf:first "Apricot sorbet" ;
      rdf:rest rdf:nil
    ]
  ]
]
```
but of course Turtle lets you write it:
```
[] <#flavor> ( "Lemon sorbet" "Apricot sorbet" ) .
```
Unfortunately, RDFa has no such syntax sugar. Which means:
```
 Flavors in my favorite ice cream:
 
  
   Lemon sorbet
   
    
     Apricot sorbet
     
    
   
  
 
```
Yep, horrific. Verbose and easy to get wrong, and that’s just for two items. If a third is added, the pattern is to add an about attribute on the middle items of the list so that the rdf:rest property which covers the next item in the list can be assigned to it. For example:
```
 Flavors in my favorite ice cream:
 
  
   Lemon sorbet
   
    
     Apricot sorbet
    
   
   
     
     Raspberry sorbet
     
     
   
  
 
```
Note:
- I’ve used an empty element with a href attribute to point to the rdf:nil resource. An alternative would be to use the resource attribute, which would have the advantage of not having to spell out the full URI for rdf:nil, but I’m trying to stick to using as few attributes as possible.
- Using an empty element for a link isn’t ideal; it would be neater to use a element, but these aren’t allowed in flow content within HTML5 ( and are only permitted within the microdata specification, and then only if they have an itemprop attribute). The RDFa specification could likewise allow them.
Multiple Properties Sharing a Value
Here we see an item with two properties, “favorite-color” and “favorite-fruit”, both set to the value “orange”:
```
 orange
```
This should map to the RDF:
```
[
  <#favorite-color> "orange" ;
  <#favorite-fruit> "orange"
]
```
Like itemprop, property can take multiple values, so the RDFa equivalent is simply:
```
 orange
```
Types
Here, the item’s type is “http://example.org/animals#cat”:
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
In this example the “http://example.org/animals#cat” item has three properties, a “name” (“Hedral”), a “desc” (“Hedral is…”), and an “img” (“hedral.jpeg”).
I’ll assume that this should be mapped to the RDF:
```
[
  a  ;
   "Hedral" ;
   "Hedral is a male american domestic shorthair, with a fluffy black fur with white paws and belly." ;
   
]
```
In this case, the vocab can be set to http://example.org/animals# and both the itemtype and the various property and rel attributes will use that as the basis for their identifying URIs:
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
Global Identifiers
Here, an item is talking about a particular book:
```
 Title
 
The Reality Dysfunction
 
Author
 
Peter F. Hamilton
 
Publication date
 
26 January 1996
```
Here, the item has an identifier so unlike the previous examples, the subject of the statements in the RDF is no longer a blank node:
```
@prefix xsd:  .

  a  ;
   "The Reality Dysfunction\n " ;
   "Peter F. Hamilton\n " ;
   "1996-01-26"^^xsd:date ;
  .
```
In RDFa, the subject is provided using the about attribute:
```
 Title
 
The Reality Dysfunction
 
Author
 
Peter F. Hamilton
 
Publication date
 
26 January 1996
```
Global Property Names
Here, an item is an “http://example.org/animals#cat”, and most of the properties have names that are words defined in the context of that type. There are also a few additional properties whose names come from other vocabularies.
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
The RDF equivalent to this is:
```
[
  a  ;
   "Hedral" ;
   "Hedral" ;
   "Hedral is a male american domestic shorthair, with a fluffy black fur with white paws and belly." ;
   "black" , "white" ;
   
]
```
To create this, we need the RDFa:
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
Link Relations
Here is an example of a page that uses the vEvent vocabulary to mark up an event:

... Bluesday Tuesday: Money Road ... May 5th @ 7pm (until 9pm) ...Link to this page ... Location: The RoadHouse ... ...

This example is interesting because it contains, in the natural markup of the page, a rel attribute with the value bookmark, which is used for links that go to the page or section of the page within which the link is found. In this case, it’s the page. The RDF that should be generated from the page is:
```
[
  a  ;
   "Bluesday Tuesday: Money Road" ;
   "2009-05-05T19:00:00Z"^^xsd:dateTime ;
   "2009-05-05T21:00:00Z"^^xsd:dateTime ;
    ;
   "The RoadHouse" ;
   "via livebrum.co.uk"
] .
```
The following statement could legitimately be generated as well:
```
<> 
    ;
  .
```
but the item representing the event should definitely not have the http://www.w3.org/1999/xhtml/vocab#bookmark property.

Achieving this without significantly changing the HTML markup is problematic in RDFa because RDFa uses the rel attribute to provide properties for the resources that it describes within the page, overloading its standard use in HTML which is to describe properties of the page or sections within the page. The following involves the least amount of repetition:
```
 
  ...
  Bluesday Tuesday: Money Road
  ...
  May 5th @ 7pm
  (until 9pm)
  ...
  
  Link to this page
  ...
  Location: The RoadHouse
  ...
  
  ...
  
 
```
Notes:
- In the above, the typeof attribute has been moved onto a wrapper
  that encompasses the entirety of the page because if it resides on the element, it’s assumed to apply to the document itself rather than a blank node. An alternative mapping would use about="_:event" to create a blank node for the event.
- There’s no way to avoid creating a statement for the rel="bookmark" link, so the best we can do is make sure that the statement is accurate, and relates the current document to the provided URI. Unfortunately, that means creating a separate element for the url property, repeating that URL within the page, and adding an empty about attribute; here I’ve used an empty element to express the relationship; a element would do the same job if it were allowed in flow content.

Microdata + RDF

2011-07-31T20:55:44+01:00

As part of the ongoing discussion about how to reconcile RDFa and microdata (if at all), Nathan Rixham has put together a suggested Microdata RDFa Merge which brings together parts of microdata and parts of RDFa, creating a completely new set of attributes, but a parsing model that more or less follows microdata’s.

I want here to put forward another possibility to the debate. I should say that this is just some noodling on my part as a way of exploring options, not any kind of official position on the behalf of the W3C or the TAG or any other body that you might associate me with, nor even a decided position on my part.

Simplifying RDFa

As I’ve said before, RDFa, in my experience, is complicated not primarily because of the whole namespaces/CURIEs issue but because its processing model tries to be too clever. RDFa was designed to largely fit in with existing markup and turn it into embedded data “just” by adding a few attributes here and there. Thus a simple image like:

is first marked up to indicate that it’s an image:

then to provide its license:

and finally to add a title:

all by adding attributes to the one element. The trouble with this approach is that the rules about how statements are made become extremely complex, dependent on context (eg what other attributes are present, what the parent element has on it, what content it has) and default in ways that are hard to remember.

Even having written an RDFa parser, having written code to mark up documents with RDFa, having taught it, I still cannot write RDFa past a trivial example and be 100% sure that it will produce what I was aiming to produce.

If we were to look at really simplifying RDFa, rather than making cosmetic changes, we need to address this complexity. It would certainly mean backwards-incompatible changes, such as dropping the use of particular attributes and revising the way the processing model works, such that future RDFa processors couldn’t be used on RDFa 1.0. There are two possible ways of approaching this:

retaining some backwards compatibility, and aiming for a simplified subset of RDFa 1.0 such that RDFa 1.0 processor will still get the intended triples out of data marked up with RDFa 1.1
dropping backwards compatibility entirely and using completely different attributes, essentially creating a new language

I do not know which of these routes is the best one to take.

My instinct is that the first will be hard to do. For example, there are already certain simplifications in RDFa 1.1 — such as assuming an element with no datatype attribute is giving a string value rather than looking to see if there are any non-text-nodes in the content of the element — which lead to markup that will not be processed correctly by RDFa 1.0 processors. Perhaps that could be addressed by rewriting history: creating a RDFa 1.0 Second Edition that includes any changes that are needed to make a simple subset viable.

What I want to explore here is what the second route — using entirely different attributes from those currently used in RDFa 1.0 — might mean. I think that in this case the substantial difference between microdata and this new language would be support for that much-derided requirement: decentralised extensibility.

Adding Decentralised Extensibility to Microdata

As I discussed earlier in the week, microdata is simply not designed for use in a web where publishers might want to use multiple vocabularies to mark up the same thing for different consumers. This focus is very probably the right one for the majority of uses, where publishers address single consumers or everyone has standardised on a single vocabulary. It’s certainly an assumption that keeps the markup simple.

However, there is a larger data web out there. It’s not just browsers and search engines who might look for and process data embedded within a page. Unlike with HTML, those few, large consumers don’t have to understand a particular vocabulary for other consumers to get valuable information from it. If you operate in a world of multiple consumers with different requirements, you need decentralised extensibility. And support for decentralised extensibility is RDF’s niche as a data model, its unique selling point.

Given that a new language would have to use a different processing model from RDFa 1.0, I would suggest that it simply uses microdata’s as a starting point. Using attributes from RDFa 1.0 would only cause conflicts with RDFa 1.0 processors. Microdata processing is there, already defined, already implemented. It isn’t going to go away. And you know, it’s pretty good.

The ‘new language’ would then not so much a ‘new language’ as an enhancement on something that already exists. It would be a set of additions that augment the data that is generated from normal microdata processing with a few extra features that are useful in a world where there are multiple vocabularies for the same domain, where publishers have to provide data to multiple consumers, where an RDF view of data is useful. Call it microdata+RDF.

So what would we need to add? Well, there are three things, I think, that make microdata hard to use in a decentralised world, and make it hard to generate good RDF from microdata markup:

lack of support for multiple types
scoping of properties by type
lack of datatypes

We would need to find a way to add these for use within the RDF extracted from the microdata markup such that a basic microdata parser would still generate the same JSON, and such that microdata’s DOM API would work as specified in the microdata spec. So we can’t change the types of values that are possible in microdata’s attributes or how they’re interpreted in the DOM API.

Multiple Types

Because of the restrictions I just mentioned in not touching microdata itself, we can’t simply make itemtype take multiple URLs. We could rely on itemprop="http://www.w3.org/1999/02/22-rdf-syntax-ns#type" as a mechanism of providing types for use by RDF processors, but I think that the types of something is such a fundamental property that it makes sense to have a dedicated attribute.

I suggest itemclass. It would only be allowed on elements with an itemscope attribute and would take a space-separated set of values in exactly the same way as the itemprop attribute. The values would be turned into URIs in the same way as for the itemprop attribute, which I’ll describe below.

Microdata+RDF would add a method to the existing microdata DOM API to enable people to access items by class rather than their single type. So:

document . getItemsByClass( classes )
Returns a NodeList of the elements in the Document that create items, that are not 
part of other items, and that have one or more of the types or classes given in the 
argument.

The classes argument is interpreted as a space-separated list of classes.

Note that for simplicity, because they are interpreted in the same way within the RDF model, this returns items whose itemtype is listed in the argument list of classes as well as those whose itemclass is listed.

Within the DOM API, the itemClass IDL attribute on HTML elements would reflect the itemclass attribute.

The itemclass attribute would be ignored for the purpose of creating JSON from microdata, and only be used when creating RDF.

An example would be:

...

The JSON generated from this would look like:

{
  "type": "http://schema.org/Event" ,
  "id": "http://lanyrd.com/2011/oscon/",
  "properties": {}
}

The RDF would look like:


  a  ,
     ,
     ;
  .

Disambiguating Properties

To work with the RDF model, properties have to have URIs. We need to have a way of easily creating the URIs for the short-name properties without people changing their existing microdata markup.

Note: I’ve substantially revised this section following discussion with Philip Jägenstedt. Old text is struck through, new text underlined.

The way that this is done in RDFa 1.1 is through a vocab attribute, which provides a URI prefix that is concatenated to any short-name properties or types. ~~We could use the same approach here, but call the attribute itemvocab to fit in with the general method of naming attributes in microdata.~~ Using this with microdata would be tedious for users however, and it would be easy for the itemtype and itemvocab to get out of sync in weird ways.

itemvocab would only be allowed on elements with an itemscope. The scope of itemvocab would be limited to the item itself, so that it’s not forgotten when it’s needed, particularly in copy-and-paste scenarios. However, to make it easier to use I think it should probably be given a default value if it isn’t present, as follows:

Instead, the vocabulary for the properties could be identified as follows:

set vocab to the itemtype of the item if it is present, and the URL of the document if not
use a substring of vocab:
1. if vocab contains a #, the substring of vocab up to and including the #
2. otherwise, the substring of vocab up to and including its final /

For example, if you have:

...

then ~~itemvocab~~ the item vocabulary would default to http://schema.org/.

There could be an extra restriction that if itemtype is specified, itemvocab must be in the same domain as that type; that could help prevent the weird situation where in the generated RDF the properties would be interpreted as being in a completely different vocabulary from the itemtype.

~~Within the DOM API, the itemVocab IDL attribute on HTML elements would reflect the itemvocab attribute.~~

Note: the following example has been altered in place.

For example, take the following markup:


  
    
      OSCON 2011
    
  
  
    United States / Portland
  
  
    25th–
    29th July 2011
  
  ...

The vocabulary for the

element defaults to http://schema.org/ based on the value of the itemtype. The short-named properties and classes within that item are turned into URIs by pre-pending http://schema.org/ to their name. Similarly, the properties on the nested http://schema.org/Place are pre-pended with http://schema.org/Place/. The resulting RDF would be:

@prefix s: 


  a s:Event ,
    s:SocialEvent ,
    s:BusinessEvent ,
    s:EducationEvent ;
  s:url  ;
  s:name "OSCON 2011" ;
  s:location  ;
  s:startDate "2011-07-25"^^xsd:date ;
  s:endDate "2011-07-29"^^xsd:date ;
  .


  a s:Place ;
  s:url  ;
  s:name "United States / Portland" ;
  .

Note: see below for how the values are created in this example.

The JSON would be just the same as from a standard microdata processor; there’s no mapping to URIs for that output:

{
  "type": "http://schema.org/Event",
  "id": "http://lanyrd.com/2011/oscon/",
  "properties": {
    "url": [
      "http://lanyrd.com/2011/oscon/"
    ],
    "name": [
      "OSCON 2011"
    ],
    "location": [
      {
        "type": "http://schema.org/Place",
        "id": "http://lanyrd.com/places/portland/",
        "properties": {
          "name": [
            "United States / Portland"
          ],
          "url": [
            "http://lanyrd.com/places/portland/"
          ]
        }
      }
    ],
    "startDate": [
      "2011-07-25"
    ],
    "endDate": [
      "2011-07-29"
    ]
  }
}

Adding Datatypes

How to manage datatypes in RDF generated from microdata is something where the best approach is not at all clear to me. A couple of years ago I talked about some frustrations with RDF datatyping, and datatypes in RDF still frustrate me by being hard to use in sensible ways throughout the RDF toolchain. Nevertheless, it’s what we have.

The possibilities I can see for microdata+RDF are:

Use plain literals for everything, including URIs, equivalent to using strings as microdata does. This makes things simple for the publisher and keeps the markup in the page clean, but makes it difficult for consumers who are using RDF toolchains: they will usually have to do some kind of processing of the RDF generated from microdata+RDF to add appropriate datatypes to the values. There are two issues with this approach:
- I have a feeling that microdata+RDF processors will make up their own rules to add datatypes to the data extracted from a page (using rules like those described below and/or sniffing of values and/or using information from known built-in vocabularies), in an effort to add value for their users. But if different processors do that in different ways, we have an interoperability problem.
- In some vocabularies, the datatype of a value is not derivable from the property. The most important/common example of this is skos:notation, which uses values with different datatypes to supply different identifiers from different identification schemes for a given concept.
Assign datatypes based on the element type in the HTML. If the property value has come from a URL attribute, assume that it’s a resource rather than a literal; if the element is a element, work out the datatype based on the syntax of the datetime attribute; otherwise assume it’s a string and give it a language in the case that one is specified. This gives some information but leads to a somewhat strange situation where you can mark up something as a date/time but not as a number.
Supplement the processing described in 2. with some basic datatype sniffing. Basically, if the value looks like a number or a boolean value then assign it a numeric or boolean datatype based on its syntax. This could reuse the rules for recognising different literals from Turtle. This wouldn’t be perfect; in particular, it would guess that strings that consist purely of numbers such as zip codes were numbers. I’m inclined not to go down this path.
Supplement the processing described in 2. with a itemvaltype attribute that takes a token from the list of built-in XML Schema Datatypes or the token ‘literal’. The ‘literal’ token would be used to override the normal processing of URL attributes in the case where those really should be literals rather than resources. In this design, it would be easy to create literals using one of the most usual datatypes, but not possible to use datatypes that are specific to a given vocabulary.
Supplement the processing described in 4. by allowing the itemvaltype to take either a token or a URL. The thing I don’t like about this design is that the token would be interpreted as being within the XML Schema Datatypes vocabulary rather than the vocabulary specified for itemvocab (used for tokens in itemprop and itemclass). This seems like it might turn into a source of confusion, but if we went the other way and had itemvaltype being interpreted based on itemvocab, it would be harder to give a value the more common datatypes such as numbers and boolean values.

My inclination, somewhat reluctantly as it’s the most complex, would be to use the last of these, because it provides for decentralised extensibility of datatypes, and support for decentralised extensibility is the core aim of these extensions. In other words, have a itemvaltype attribute that can hold either a token, which must be one of literal or the local name of an XML Schema datatype, or a URL. On a element, this would default to the appropriate type based on the syntax of the value of the datetime attribute.

To be conformant, the itemvaltype would have to be an allowed value type for the properties given in itemprop and the value of the property must be a legal value for the datatype. (In keeping with the style of the microdata specification, the mechanisms for working out what value types are allowed and what the legal values are for non-XML Schema datatypes would be left undefined — a consuming application would look at the definition of the vocabulary.)

Within the DOM API, the itemValType IDL attribute on HTML elements would reflect the itemvaltype attribute. The value of itemvaltype wouldn’t change the types of the values returned by element.itemValue or in the JSON mapping from microdata; it would purely be used when generating RDF from that data.

For example, if someone started with some markup like:


  Priced from: $35
  1938 tickets left

it might be supplemented with some type information like:


  Priced from: $35
  1938 tickets left

which would generate RDF like:

@prefix s: 

[] a s:AggregateOffer ;
  s:lowPrice "$35"^^s:Price ;
  s:offerCount 1938 ;
  .

(Note: Here I’m assuming that schema.org defines a http://schema.org/Price datatype which includes a currency and a number. They don’t currently.)

The JSON would still be:

{
  "type": "http://schema.org/AggregateOffer",
  "properties": {
    "lowPrice": [
      "$35"
    ],
    "offerCount": [
      "1938"
    ]
  }
}

Non-Additions

When I wrote a couple of years ago about what microdata can’t do, one of the things that I identified was not being able to express XML Literals. Having thought about this more, what’s actually missing isn’t to do with RDF, but is the ability to use the innerHTML of an element to provide a value for a property rather than its textContent.

For example, the description of an event might run over several paragraphs, or even in a single paragraph include other markup such as emphasised text, ruby markup, or links to additional information. People who are working from the DOM API can capture this information when they need it by getting the innerHTML of the element rather than its itemValue, but in the JSON mapping, the value is always the itemValue — the text content of the element.

So this is a general microdata simplifying limitation. I’d argue that we shouldn’t add any special handling to plug this hole at the microdata+RDF level. If it turns out that having values that contain markup is useful then it will be added to microdata, and the microdata+RDF mapping would then be extended to create rdf:XMLLiterals or HTML literals (for which there is no defined datatype in RDF at the moment) for such values.

Similarly, I haven’t said anything in this post about providing machine-readable values to override the text content of an element. There is an open bug about whether and how that capability might be added to HTML/microdata. I happen to think that it’s useful, but that utility isn’t limited to RDF processing. Whichever route is chosen there, I think it’s important to keep the property values used by basic microdata and microdata+RDF aligned.

Summary

To summarise, one direction that we could take in aligning microdata and RDFa would be to define an extension to microdata to add support for decentralised extensibility and the RDF data model. I think that would entail adding attributes such as:

itemclass to make it easy to define multiple types for an item
itemvocab and some default processing to provide nice mappings for short-name properties into URIs
itemvaltype and some default processing to assign datatypes to values

For publishers and consumers, a single language with optional extensions greatly simplifies the use of embedded data. Property names don’t have to be repeated or balancing acts made between different processing models.

RDFa proponents get a syntax that can be used to generate a natural RDF model against which they can build RDF-oriented APIs and map to other formats such as JSON-LD.

For microdata proponents, this approach doesn’t pollute microdata with requirements that they see as superfluous, and doesn’t change the behaviour of core microdata processors. Browsers, search engines and other consumers can continue to use the JSON output and only those who really want to support RDF need to do so.

I’m sure that there are things that I’ve missed in my outline above, issues that I haven’t thought of. But if there is to be any kind of convergence between microdata/RDFa, this layered approach seems to me to be the kind of convergence that is most likely to eventually result in one language for embedding data in HTML rather than two or three.

Note: if you prefer to comment on Google+, please add your comment to my announcement post there

Using Multiple Vocabularies in Microdata

2011-07-28T09:25:21+01:00

I wrote the other day about how legislation.gov.uk needs to share data at three levels to satisfy its goals as a website:

large-scale consumers such as search engines
small-scale consumers that provide us with a useful service
specialist consumers that are interested specifically in our data

and the requirement to use multiple, incrementally more specialised, vocabularies to describe the same things as a result.

What I want to do here is explore how a publisher might handle this kind of situation using microdata. The ground has already been substantially covered by Stéphane Corlosquet; what I do here is work through an example where the consumers are microdata’s primary targets — search engines and browsers — look at why it’s hard to fix this within microdata itself, and discuss how people who create vocabularies to be used with microdata might help publishers who find themselves in this situation by designing those vocabularies to be used together as well as on their own.

Use Case

I’m going to use Lanyrd from Simon Willison and Natalie Downe as an example. Lanyrd is “the social conference directory”: it keeps track of conferences that you’re attending or speaking at, and lets you know about ones that your friends (or at least the people you follow on Twitter) are going to as well, as well as providing a bunch of other useful facilities.

Lanyrd currently uses microformats to mark up events so that nice summaries appear within search engine results. Here’s a (slightly simplified for concision) example from the front page:


  OSCON 2011
  
    United States / Portland
  
  
    25th–
    29th July 2011
  
  ...

This is easy enough to understand: OSCON 2011 has a URL of http://lanyrd.com/2011/oscon/, is located in Portland (US), starts on 25th July and ends on 29th July 2011.

Say that Lanyrd decided to switch to using schema.org microdata. The markup would change to something like the following:


  
    
      OSCON 2011
    
  
  
    
      United States / Portland
    
  
  
    25th–
    29th July 2011
  
  ...

A few notes, because there were some design decisions involved in this mapping:

I’ve used a plain http://schema.org/Event because I wasn’t sure how to classify a conference — is it a SocialEvent or a BusinessEvent or an EducationEvent? Depends on the conference, I guess
I’ve assumed that the URIs for both the conference and its location are also item identifiers
I’ve changed the markup a bit to add elements where necessary to get the desired data out, namely around the names of the conference and the place; I could have used separate or elements instead but that would have meant repetition of data within the page

All well and good.

Now let’s say that browsers start to support the vEvent vocabulary defined within the WHATWG HTML microdata specification and offer some really nice functionality: because there’s a clear mapping to iCalendar, they enable users to drag an event from the browser to a calendar application, and have it create an entry within the calendar.

Say Lanyrd really want to take advantage of this. It means marking up their pages in something like a mix between the two examples we’ve looked at so far — microdata syntax but with the vEvent vocabulary (which is based on the hCalendar microformat vocabulary) rather than the schema.org vocabulary:


  
    
      OSCON 2011
    
  
  
    United States / Portland
  
  
    25th–
    29th July 2011
  
  ...

But now Lanyrd have a dilemma. If they mark up their pages using the schema.org vocabulary, they can’t take advantage of the browser drag-and-drop support; if they mark up their pages using the vEvent vocabulary they won’t get their pages displaying nicely in search engine results. They can get the benefits from one consumer or the other but not both at the same time. What to do?

Publisher Workarounds

What could Lanyrd do to work around this problem?

Different Syntaxes

The first, eminently pragmatic, workaround, would be to use different syntaxes to encode the event information for the two different consumers. Since schema.org is likely to continue to understand microformats for the forseeable future, Lanyrd could stick to their original microformat markup and just add similar microdata for browsers to pull out to create iCalendar data. The page would look like:


  
    
      OSCON 2011
    
  
  
    United States / Portland
  
  
    25th–
    29th July 2011
  
  ...

In other words, they could handle their requirement by not using microdata for one of the vocabularies. I don’t think this is a particularly acceptable solution, given that schema.org specifically wants publishers to use microdata, but it would work.

Repeated Data

A second workaround that Lanyrd could use would be to have some shadow markup for the data targeted at schema.org; the visible event information in the page itself should still be marked up using the vEvent vocabulary because it gives an area of the page that users can drag and drop. The basic version of this would look like:


  
  
    
      
        OSCON 2011
      
    
    
      United States / Portland
    
    
      25th–
      29th July 2011
    
    ...
  

  
  
    
    
    
      
      
    
    
    
    ...

Note: I’ve used empty elements to mark up the dates for the conference in the schema.org shadow data because the microdata spec says “If a property’s value represents a date, time, or global date and time, the property must be specified using the datetime attribute of a time element.” They’re empty, though, so they won’t be displayed on the page.

There are a few issues with this workaround:

it repeats content and thus bloats the page
in the microdata DOM API, there now appear to be two items when really there’s one conference; this might not be a big deal if scripts access items by type rather than just getting all the items
search engines might (wild speculation follows) be more suspicious of data that isn’t visible within the page; there’s no way for schema.org to know that the same data appears visibly elsewhere with equivalent markup

Use `itemref`

A third possibility for Lanyrd would be to something similar to the previous example but use the itemref attribute to point to any shared data. Unfortunately in this case, there’s only one property that’s actually shared (with the same semantics) between the two vocabularies — url — so using this technique doesn’t improve the markup all that much from the previous example:


  
  
    
      
        OSCON 2011
      
    
    
      United States / Portland
    
    
      25th–
      29th July 2011
    
    ...
  

  
  
    
    
      
      
    
    
    
    ...

In other situations, where there is more overlap in the property names used by the two types, there might be more advantage in this approach.

Content Negotiation

A final workaround would be for Lanyrd to serve up an HTML page that uses the schema.org vocabulary to search engines and an HTML page that uses the vEvent vocabulary to browsers, by sniffing the User-Agent header.

This has the advantage of not having to try to cram two conflicting vocabularies into a single page but the disadvantage of having to code for the content negotiation. Essentially, it shifts the complexity and repetition from the HTML page to the code that generates the HTML page, but does address the three disadvantages that I listed for the ‘repeated content’ solution described above.

Publisher Workarounds

Lanyrd could also lobby schema.org and/or WHATWG to make changes to what data they consume.

Lobby for Convergence

Lanyrd could lobby schema.org to understand the vEvent vocabulary and/or WHATWG to specify browser handling of the schema.org vocabulary.

This might work, but the vocabularies do have different goals and requirements, which might make it hard to unify them: vEvent maps neatly and easily to iCalendar, schema.org is oriented around Rich Snippets in search engine results. The modelling of the location property in each shows this different emphasis: it only needs to map to a string in iCalendar so there’s no need to model the location as an item itself, but in search engine results it’s useful to link to the location, display a map and so on, which is only possible if the location is modelled as an item in its own right.

Lobby for Different Processing

Finally, Lanyrd could lobby schema.org and/or WHATWG to trigger their recognition of an event based on something other than the itemtype of an item, and to interpret full URIs for properties in the same way as equivalent short names.

For example, currently the conversion of vEvent to iCalendar defined in the WHATWG HTML specification is triggered by the presence of an item that is an vEvent:

If none of the nodes in nodes are items with the type http://microformats.org/profile/hcalendar#vevent, then there is no vEvent data. Abort the algorithm, returning nothing.

Let’s say that it were instead triggered by items with either:

an itemtype of http://microformats.org/profile/hcalendar#vevent or
a http://microformats.org/profile/hcalendar#type of vevent

and that in the former case, it would read short name properties but in the latter case it would read properties with URIs like http://microformats.org/profile/hcalendar#location.

In that case, Lanyrd could use the schema.org vocabulary for the type given in the itemtype attribute, but markup extra properties for the item using the vEvent property URIs. The markup would be something like:


  
  
    
      OSCON 2011
    
  
  
    
      United States / Portland
    
  
  
  
    25th–
    29th July 2011
  
  ...

Note that here the location has to be repeated because vEvent expects a string while schema.org expects an item.

The same kind of pattern could work the other way around: schema.org could recognised events based on a http://schema.org/type property with the value Event, and understand property URIs that were equivalent to each of the short-name properties that it uses. (Such URIs for schema.org properties already exist.)

Multiple Types for Microdata Items

Earlier this year there was some discussion on the WHATWG mailing list about the requirement for multiple types for items.

The use cases there were not the same multiple-consumers use case that I have outlined above, but around support for inheritance in types. For example, schema.org lets people define their own types in the schema.org domain, such as http://schema.org/Event/Conference. There’s no way of exposing the type hierarchy explicitly (that all Conferences are Events) so scripts that use microdata’s DOM API to search for items of the type http://schema.org/Event won’t find conferences. The discussion was about alleviating this by allowing publishers to put both http://schema.org/Event and http://schema.org/Event/Conference within the itemtype attribute. Alternatively, a conference could be typed as a SocialEvent, BusinessEvent and a EducationEvent, enabling it to take properties from all three.

The conclusion of the discussion was that it just wasn’t possible to use what would seem to be the obvious method of assigning multiple types to an item: having a space-separated list in the itemtype attribute. If we look at the markup that you would get for this example, we can see why there’s a problem:


  
    
      OSCON 2011
    
  
  
    
      United States / Portland
    
  
  
  
    25th–
    29th July 2011
  
  ...

There are two issues with this markup. First, the definitions of the two types (in prose within the two specs) have different expectations about:

what properties will be present: schema.org expects a name property and not a summary property, and vice versa for vEvent; similarly for startDate/dtstart and endDate/dtend
what values the properties will have: schema.org expects location to have an item value whereas vEvent expects a string

The result is that the mixed markup isn’t conformant with either vocabulary, and hence not a conformant HTML document. (Whether microdata consumers do anything about that non-conformance is a different question — they could just ignore properties that they don’t understand, or with value types that they don’t expect.)

Second, if the data is turned into any kind of format that needs full URIs for properties, such as RDF through microdata’s RDF mapping, it’s impossible to tell what type URI to use as the basis of that URI. If the item is assigned just the schema.org type, the name property would map to the URI:

http://www.w3.org/1999/xhtml/microdata#http://schema.org/Event%23:name

If there is just the vEvent type, the summary property would map to the URI:

http://www.w3.org/1999/xhtml/microdata#http://microformats.org/profile/hcalendar%23vevent:summary

When the item has more than one type, there is no way to know which type should be used as the basis of the URI generated for the property, or even if both should be used, as in the properties used by both vocabularies such as url.

(This issue isn’t specific to the RDF mapping defined in the microdata specification; it would arise in any RDF mapping from microdata, or any mapping in which short names for properties needed to be turned into globally unique terms.)

Guidance for Microdata Vocabularies

Having short names for properties makes writing microdata simple. They are much easier on the fingers and the eye than URIs, and, because they are scoped by type rather than vocabulary, they can be given simple names while still being specified tightly in terms of the types on which they can appear and the values that they can take. For example, the location of an Event can be limited to a geographical place while the location of a Click can be specified in terms of a point on a web page, rather than having to use more complex property names like placeLocation and pointLocation.

This could be a particular advantage in large and wide-ranging vocabularies such as schema.org’s where it’s likely that at some point there will be a clash in meaning between properties with the same name for different things. (Though the flip side for schema.org is that it has lots of inherited properties which really do have the same meaning across subtypes.)

The biggest problem with short names arise when you want to provide data to different consumers that use different vocabularies for that data. My guess is that in real life, in many cases this won’t be an issue, and certainly microdata has been designed with that assumption. Realistically, the majority of websites will probably only care about embedding data in web pages to the extent that search engines will read it, and will therefore only use one vocabulary — schema.org’s. Where more than one vocabulary is used in the page, it may well be that they are used in different locations (eg OGP for Facebook in the head of a page, schema.org in the body), or to mark up data about completely different kinds of things.

However, if you’re a publisher who wants to provide data to multiple consumers who understand different vocabularies — search engines and browsers as in the Lanyrd example above, for example — and those consumers define what they will consume solely based on the itemtype of an item, then you’re going to have to either workaround consumer’s behaviour as I described above, or ask those consumers to change how they work.

The most promising direction I can see at the moment would be to ask consumers to define their vocabularies such that they include

a property that is used to identify the in-vocabulary type of items whose itemtype is not in that vocabulary
defined URIs for properties that are equivalent (and processed in the same way as) the short name properties for a given type

The type-defining property could be http://www.w3.org/1999/02/22-rdf-syntax-ns#type, with the value being the URI of the relevant type. However,

is a lot more verbose than:

so I imagine that the designers of microdata vocabularies would prefer to ask publishers to do the latter. On the other hand, if publishers are using multiple vocabularies, they might find it easier to use a consistent type-defining property across vocabularies; it’s hard to tell what the global usability payoffs might be here.

Or microdata could standardise the pattern by adding an attribute (eg itemkind, iteminherit, itemothertype, itemmixin, I dunno) which would list additional types. These could be exposed within the DOM API (which would be a big advantage for in-page scripts) but not used in the interpretation of short-name properties.

Vocabularies that don’t support processing of items based on a type-defining property and property URIs are effectively indicating that they don’t anticipate being mixed with others that have also made the same assumption that they won’t be mixed with others. Currently, for example, schema.org and the vocabularies defined within the WHATWG microdata specification both make this assumption. Working with one vocabulary that makes that assumption for a particular type is fine; working with two in microdata is much harder.

My Experience of Web Standards

2011-07-24T17:24:00+01:00

One of the things that’s been niggling at the back of my mind since the schema.org announcement is how small a role search engine results plays in the wider data sharing efforts that I’m more familiar with in my work on legislation.gov.uk, and more generally how my day job experience differs from (what seem to be) more common experiences of development on the web. In this post, I’m going to talk about that experience, and about the particular problems that I see with the coexistence of microdata and RDFa as a result.

My day job (the one I actually get paid for) is web development. The site I spend most of my time and effort on is legislation.gov.uk. This deals with complex content (UK legislation) that has to be presented in multiple formats (users love PDFs of legislation). Our aim is to make the data as reusable as possible by third parties through good, RESTful, web architecture, and we want to use open standards and open source technologies as part of the UK government’s general strategy.

legislation.gov.uk is not a global website like Amazon or eBay, but it’s not small either: it covers 60,000 changing items of legislation, providing point-in-time views for many of them, and with more added every day. It’s one of the top ten most used UK Government websites, with 2 million visits (about 10-12 million page views) each month and typically about 120 requests/second during the active times of the day. Legislation might sound like a highly specialist interest, but if you search for legislation.gov.uk on Twitter you’ll see it being referenced over and over by people who want to share what the law says.

I do not by any means claim that my experience is representative of the wider web. I know that there are large numbers of sites that deal only in data, not documents, and certainly not documents with the kind of rich semantic structure that legislation has. I offer the following discussion as a data point, partly because I can’t quite believe that legislation.gov.uk is completely unique in its requirements and partly because obviously my perspective on a bunch of issues arises from this experience.

Technology Stacks

Legislation items are complex, semi-structured documents. Their natural fit is XML (well, that’s not quite true — their natural fit would be something that allowed overlapping markup — but XML is the closest that we have). So we store it in XML in a native XML database and we use an XML toolset to query it (XQuery) and transform it (XSLT) into various formats including rendering it as PDF (through XSL-FO).

Our next step for the development of the site involves looking at legislative effects. These form a graph: one item of legislation affects other items of legislation which may in turn affect other items and so on. There are all sorts of other links between items of legislation in terms of commencements, conferred powers and so on. Particularly because we already have well-thought-through URIs for legislation, the natural fit is to use RDF to represent this graph. We already offer a SPARQL endpoint for accessing some aspects of our data, but we expect to expand and develop this over the next few months and to use it as a layer under the website and exposed for reusers, in much the same way as we use the XML database.

As a government site, we have fairly strict limits on what we can do within our web pages: we have to make sure that they’re accessible by everyone who wants to view them. We aren’t able to use technologies that are only available in the latest browsers, but that’s OK because with the kind of content we deal with, we don’t have to do anything fancy anyway. So we use pretty basic HTML and CSS and Javascript, because that’s how you deliver content to end-users on the web (as well as exposing the underlying XML and RDF, to enable others to reuse the data).

In other words, we use three web stacks for delivering legislation.gov.uk:

the XML stack, which is great for single-source publishing of documents that have more semantic structures than those supported by HTML
the RDF stack, which is well-suited for metadata about things that are identified by URIs
the HTML stack, which is absolutely necessary for delivering human-accessible content on the web

What bemuses me, because of this experience, is that sometimes it appears that the narrative around these technologies is framed in terms of an exclusive choice between them. For example, @mattur asked:

It is as if, if you use XML you cannot appreciate the utility of error-handling in HTML; or if you use RDF you cannot understand the need to represent documents in XML; or if you want to utilise HTML fully, you cannot adopt RDF’s view of data on the web. That’s simply not my experience. They each have their role on the web; supporting the use of one does not necessitate rejecting the use of the others.

It’s interesting that some of the standards that are most reviled are those that arise at the intersections, where it appears that one technology is trying to encroach on the space of another:

XHTML at the border of XML and HTML
RDF/XML at the border of RDF and XML
RDFa at the border of all three

At the same time, within legislation.gov.uk, we publish XHTML (because it’s the natural output from an XML toolchain) and create and process RDF/XML (because it gives us access to that data from within the XML toolchain). We use a small bit of RDFa in the XHTML to indicate the rights under which our information is avaialble, and don’t yet, but are thinking about using RDFa to mark up non-document semantics within our XML (to enable the XML markup to focus on the document structures that it’s good at). For all their imperfections, these intersection technologies are useful for managing cross-overs; the problems arise when they overstep their remit and people start to think that all HTML must be XHTML or all XML must be RDF/XML or all RDF must be RDFa.

Sharing Scenarios

The second thing that I wanted to explore is the experience from legislation.gov.uk of what it’s like to be a publisher who actively wants to share their data. We need to operate simultaneously at three levels in our data sharing efforts.

Large-Scale Consumer-Driven Data Sharing

The first target for our data sharing efforts are the search engines. Obviously we’re not selling anything, but we want people to be able to locate legislation easily when they want it, and we want people who have done the search to be able to see some information about the legislation so that they know that they’ve located the right item.

This is large-scale consumer (search engine) driven data sharing, typified by schema.org and Facebook’s Open Graph Protocol (OGP). There are a few very big data consumers (Google, Microsoft, Yahoo!, Facebook etc) who need to consume data from large numbers of data providers. These consumers obviously can’t understand everything, so they determine and document what syntaxes and vocabularies they do understand and expect publishers to follow.

The benefits that publishers get from a particular consumer determines which syntax/vocabulary they use; publishers who are particularly keen to show up prettily within search results will target schema.org whereas those who want to be sharable within Facebook will target OGP. Many publishers will want to target both. There is probably a driver towards eventual convergence:

publishers might push back about inserting two lots of very similar data in their pages
consumers might want to include data from publishers who haven’t specifically targeted them

although there’s likely to be a period where they coexist, much as there was for VHS and Betamax (and V2000, I know, dad) during the early days of video players.

As I discussed previously, these large-scale consumers will be driven by the data that they find in the wild, in all its messy variety. They get relatively little benefit directly from using a generic syntax, as they are really interested in only a few, pretty generic, vocabularies for which they have hardwired processing. Indirectly, adopting a generic syntax has benefits in that publishers might find it easier to find tools that enable them to generate it, tutorials about how to use it, and feel that they aren’t being quite as locked in to something proprietary. However, rejecting data that isn’t marked up properly using that syntax has no benefit for consumers except in so far as it makes them feel that they are being good community members.

This is the pattern we see with schema.org (which accepts microdata but, based on its documentation, won’t reject data that isn’t fully compliant with it) and with OGP (which accepts a subset of RDFa but doesn’t reject data that hasn’t got prefixes properly bound, for example).

Another point to mention is that there is very little trust in this scenario. The communication between consumers and publishers is very limited, and the consumers will want to protect themselves against accidental or malicious errors that are evident in mismatches between explicit metadata and that which is parsed from the visible content of the page.

The parallels to HTML and browser vendors are very strong in this type of data sharing.

Small-Scale Consumer-Driven Data Sharing

A second type of data sharing is again driven by consumers, but this time at a lot smaller and more specialised scale. For legislation.gov.uk, these are services such as GLIN, which is a global legislation registry. Other examples are the recent work that we’ve done to publish UK Government organograms or Chris Taggart’s Open Election Data project. In these cases, there’s a single, relatively small and specialised consumer and a small number of publishers which are closely coordinated together.

As in the large-scale case, the consumer ultimately determines the syntax/vocabulary that it recognises, and communicates that to the publishers. However, small-scale consumers typically have close coordination with the publishers, which has a number of side-effects:

consumers may be more able to both apply pressure to and help publishers to do well in their markup
publishers have the opportunity to feed back directly to the consumer any suggestions that they have about changes to the syntax/vocabulary
publishers are likely to gain an immediate and tangible benefit from their cooperation, such as visualisations of their data that they otherwise wouldn’t have seen

Another noteworthy point about small-scale consumers is that they’re unlikely to have the engineering capability to create a custom parser for a particular syntax, but will instead want to use something off-the-shelf to extract data from pages and into their own backend systems. This, coupled with the closer coordination with publishers, means that they’re much more likely to stick to a specification, assuming that the off-the-shelf tools do.

Publisher-Driven Data Sharing

The final type of data sharing is driven by publishers. At legislation.gov.uk, we’re motivated to make our data available for reuse for transparency/accountability reasons (to help citizens understand the law), efficiency reasons (to help parliament and government departments to publish new legislation better) and economic reasons (to foster innovation in legal publishing). We don’t have any individual consumers in mind when we publish our data, but have found that simply by publishing it well, we foster reuse.

In this case, we as publishers are highly motivated to ensure that the data we publish is easily parsed with something off-the-shelf, since that lowers the barrier for potential consumers. Publishers like us are very likely to have unique, specialised, content and need to use a vocabulary that fits closely to our internal data structures as this lowers implementation cost. Consumers can also trust publishers like us: we simply have no motivation to lie in the data that we provide for reuse.

Mixed Markup

As I’ve outlined above, publishers like legislation.gov.uk need to target several potential consumers at the same time:

large-scale consumers such as search engines
small-scale consumers that provide us with a useful service
specialist consumers that are interested specifically in our data

We cannot use a single vocabulary for all these different purposes. (Well, we could write our own vocabulary and describe mappings to other vocabularies using RDFS, but search engines wouldn’t read it.)

We must therefore use a mix of vocabularies:

generic vocabularies about things that search engines care about
specialised vocabularies for particular small consumers
site-specific vocabularies for sharing our unique data

It’s repetitive, but it’s manageable so long as we have a syntax that enables us to say that an item of legislation is a http://scheme.org/CreativeWork and a http://purl.org/dc/dcmitype/Text and a http://www.legislation.gov.uk/def/legislation/Legislation and allows us to give multiple properties the same value.

The way things are going at the moment, we might well end up having to use multiple syntaxes on the same page, as some consumers understand microdata, others consume RDFa, and still others will parse microformats. This leads to more repetition: adding itemprop for microdata, property for RDFa and specialised class attributes for microformats. But worse (much worse), each of the syntaxes uses a different parsing model to create an entity-property-value structure, so not only do we have to learn substantially different markup patterns but our pages quickly become some kind of hideous polyglot mess trying to balance between them.

Looking Forward

As I said at the start, I’m fairly sure that my experience at legislation.gov.uk isn’t representative of the wider web, but I don’t have a clear idea about just how unrepresentative it is, in terms of technology use or motivations around data sharing. When I read my twitter stream or blogs, there’s a massive sampling bias, both in terms of who I follow and what I read, but also about who talks about what they’re doing. (I’m reminded of Jeff Atwood’s post on the Two Types of Programmers: the vast majority of web developers don’t make a noise about what they do.)

Taking part in web standardisation today often feels like being part on an ongoing cold war between distinct camps rather than a community working towards common aims. The underlying question seems to be “who’s side are you on?” Every decision and activity is cast as a victory or defeat. Time is wasted on attack and defence, or on raking over past slights and stupidities, rather than on progress. Valid criticism from outside a group cannot be listened to for fear of giving ground, cannot be made within a group where it seems like betrayal.

It is the Robbers Cave Experiment played out in web standards. As a psychologist, I find it fascinating. As a developer, and particularly one who doesn’t self-identify with any single group, it is frustrating. As a TAG member, trying to work for the longer-term good of the web, it is worrying, because situations of intergroup conflict lead to groupthink and non-optimal solutions.

As I described above, a non-optimal outcome seems to be the most likely result of the particular microdata vs RDFa conflict for us at legislation.gov.uk. While I know we are not generally representative, I believe that it will be similarly bad for other developers: publishers, consumers and tool implementers.

This is a problem for all who want to foster data sharing on the web using open standards; it is not one that any one group can fix on their own. It’s my hope that a balanced task force of individuals with a variety of experience and backgrounds can provide a focus for us all to work together to solve it. If we can’t, then we have let our prejudice and bias overcome our judgement.

Jeni's Musings

Using "Punning" to Answer httpRange-14

Background

Terminology

Content and Meaning

Current State

Punning

Defining URI Usage

Equality

Implications for Linked Data

RDF

Vocabulary Designers

Inference

Publishers

Provenance

Linked Data Platform

Final Thoughts

UK Open Standards Consultation

Criteria for Open Standards

1. How does this definition of open standard compare to your view of what makes a standard ‘open’?

2. What will the Government be inhibited from doing if this definition of open standards is adopted for software interoperability, data and document formats across central government?

3. For businesses attempting to break into the government IT market, would this policy make things easier or more difficult – does it help to level the playing field?

4. How would mandating open standards for use in government IT for software interoperability, data and document formats affect your organisation?

5. What effect would this policy have on improving value for money in the provision of government services?

6. Would this policy support innovation, competition and choice in delivery of government services?

7. In what way do software copyright licences and standards patent licences interact to support or prevent interoperability?

8. How could adopting (Fair) Reasonable and Non Discriminatory ((F)RAND) standards deliver a level playing field for open source and proprietary software solution providers?

9. Does selecting open standards which are compatible with a free or open source software licence exclude certain suppliers or products?

10. Does a promise of non-assertion of a patent when used in open source software alleviate concerns relating to patents and royalty charging?

11. Should a different rationale be applied when purchasing off-the-shelf software solutions than is applied when purchasing bespoke solutions?

12. In terms of standards for software interoperability, data and document formats, is there a need for the Government to engage with or provide funding for specific committees/bodies?

13. Are there any are other policy options which would meet the described outcomes more effectively?

Open Standards Mandation

1. What criteria should the Government consider when deciding whether it is appropriate to mandate particular standards?

2. What effect would mandating particular open standards have on improving value for money in the provision of government services?

3. Are there any legal or procurement barriers to mandating specific open standards in the UK Government’s IT?

4. Could mandation of competing open standards for the same function deliver interoperable software and information at reduced cost?

5. Could mandation of open standards promote anti-competitive behaviour in public procurement?

6. How would mandation of specific open standards for government IT software interoperability, data and document formats affect your organisation/business?

7. How should the Government best deal with the issue of change relating to legacy systems or incompatible updates to existing open standards?

8. What should trigger the review of an open standard that has already been mandated?

9. How should the Government strike a balance between nurturing innovation and conforming to standards?

10. How should the Government confirm that a solution claiming conformity to a standard is interoperable in practice?

11. Are there any are other policy options which would meet the objective more effectively?

International Alignment

1. Is the proposed UK policy compatible with European policies, directives and regulations (existing or planned) such as the European Interoperability Framework version 2.0 and the reform proposal for European Standardisation?

2. Will the open standards policy be beneficial or detrimental for innovation and competition in the UK and Europe?

3. Are there any are other policy options which would meet the objectives described in this consultation paper more effectively?

Content and Descriptions of Web Resources

The Questions

The Conflict

Web of Data

Hash URIs

303 Redirections

Web of Things

The Current State

The Social Context

The Choices

Precious Snowflakes

Inside Government

Microdata and RDFa Living Together in Harmony

Mapping Rules

Mismatched Features

Local Property Names

Interpretation of Element’s datetime Attribute

Content Overrides

and Elements in Flow Content

Identifiers without Types

Built-in Prefixes

Literals Including Markup

Itemref

Lists

Datatypes

Languages

Multiple Types

The src Attribute

Link relations

Possible Subset of RDFa

Possible Subset of Microdata

Guidelines for Vocabulary Authors

Interpretation of Element’s `datetime` Attribute

The `src` Attribute

Items from the `src` Attribute

E = mc²: The Most Urgent Problem of Our Time

E = mc²: The Most Urgent Problem of Our Time

The `resource` Attribute

Values from the `src` Attribute

Values from the `datetime` Attribute

Creating an `rdf:Seq`

Creating a `rdf:List`

Use `itemref`