Jeni's Musings

Microdata and RDFa Living Together in Harmony

2011-08-20T16:39:11+00:00

One of the options that the TAG put forward when it asked the W3C to put together task force on embedded data in HTML was the co-existence of RDFa and microdata. If that’s what we’re headed for, what might make things easier for consumers and publishers who have to live in that world?

In a situation where there are two competing standards, I think that developers — both on the publication and consumption sides — are going to want to hedge their bets. They will want to avoid being tied to one syntax in case it turns out that that syntax isn’t supported by the majority of publishers/consumers in the long term and they have to switch.

Publishers like us at legislation.gov.uk who are aiming to share their data to whoever is interested in it (rather than having a particular consumer in mind) are also likely to want to publish in both microdata and RDFa, rather than force potential consumers to adopt a particular processing model, and will therefore need to mix the syntaxes within their pages.

(Of course developers might just avoid embedded data altogether while they wait to see what happens, but let’s assume that they want to press ahead regardless of the lack of consensus from the standardistas.)

I’ve therefore embarked on a task of trying:

to identify the differences in approach and functionality of the two languages, which should help developers choose between them
to identify any guidelines for developers of vocabularies for use with both languages
to identify a subset of functionality that is common between the two languages, which developers might want to stick to to make switching and mixing easier
to identify mapping rules that might be applied to automatically or manually map from one language to another if the simple subset is used

I’ve done this by looking at converting microdata examples to RDFa and vice versa, and the lessons to be drawn from that exercise. I’ve broken down the result into three posts:

This is the last of these posts. It is probably the only one you will want to read :)

Please treat this as a draft on which I’d welcome comments. I have based what’s written here on the latest specifications of both microdata (in its WHAT WG and W3C variants) and RDFa Core and HTML+RDFa but I haven’t consulted with anyone involved in these efforts and may well have got things wrong. Plus the specs are changing all the time. I have only here considered the syntax of the two languages, not the features such as DOM APIs or drag-and-drop support, where there are also clear differences.

Please add comments if there are things that I’ve missed or got wrong, or just to have your say.

Philip Jägenstedt’s Live Microdata service and Gregg Kellogg’s Distiller service have both proved invaluable for testing — thank you to both for making these services available. I heartily recommend them.

Mapping Rules

The first problem is how to judge equivalence when microdata and RDFa have different data models. Microdata essentially uses a JSON data model: there are objects (items) with properties that have values that are strings, other objects, or arrays of strings or objects or both. RDFa naturally uses a RDF data model: there are resources with properties that have values that are literals (of some datatype or with a language) or other resources.

Underlying both is the same basic entity-attribute-value pattern, but there are various mismatches between the models that make some mappings more complicated than others, or in other cases mean that information is necessarily lost on conversion.

In performing the analysis, I’ve tried to map microdata into sensible RDF and then match that RDF output using RDFa, and to map RDFa into sensible microdata+JSON and then match that microdata+JSON using microdata. The microdata-to-RDF mapping rules that I’ve followed are basically those outlined in my post on Microdata + RDF. To create microdata JSON from RDFa, I’ve used the rule that the URI of the first type of a resource is processed to provide a namespace that is stripped from the URIs of the properties (to create simple names where possible). In addition, when a resource has no properties, it will be represented as a string (URI) value rather than as a nested item.

These rules need to be formalised, obviously, but the basics above work well enough for the examples from the specs.

Mismatched Features

The following features are problematic when mapping from microdata to RDFa or vice versa. I’ve described them roughly in an order from things where it might be relatively easy to address the problem by changing one or other specification, to places where the necessary changes would be difficult to make in the specs, which means that publishers and consumers need to be aware of the issue so that they can make an educated choice about how they proceed.

Local Property Names

Many of the microdata examples involve items with no type and local property names. I’ve assumed in the analysis below that this generates properties whose URI is based on the document in which they are found, but this is not a helpful solution for data sharing: if a whole site uses short property names across its pages, those properties really need to be recognised as being the same across the site for any kind of useful processing to occur.

What microdata actually creates here is a global namespace, shared by everyone, specifically for embedded data. There are three things that could be done at different levels here:

In a mapping from microdata to RDF, any short property names on items that don’t have a type could be assigned to a global namespace (eg http://w3.org/ns/global/). Of course there will be clashes in semantics within this namespace, but that is true in microdata generally and not having to create a new namespace makes the initial experimentation easier for those starting with embedded data. The W3C (or whoever operates the namespace) could operate a wiki at that location that would operate as an informal registry for the property names.
HTML+RDFa could change to use this global namespace as the default vocabulary URI (rather than not having one). This would make it a little easier for people to convert microdata to RDFa: if they don’t use types for their items, there would then be no need for a vocab attribute to be added to the HTML. It also makes it possible to use RDFa in a basic, lightweight way, which might help people get started with it.
Publishers can be advised to use itemtype within their microdata, reusing existing classes or creating their own, if they want to ensure that the embedded data within their pages isn’t misinterpreted by global consumers.

Interpretation of Element’s `datetime` Attribute

Interpreting the datetime attribute of the element to supply a value, rather than repeating that value in a content attribute, is ISSUE-97 on RDFa, and hopefully RDFa will be changed to use that value (or the content of the element if there is no datetime attribute), add a seconds component if necessary, and work out an appropriate date/time datatype for it based on its syntax.

Content Overrides

In RDFa, publishers can provide a machine-readable version of the content of an element (or even an entirely different value) using the content attribute. This can only be done for date/times in microdata. The ability to annotate non-date/time content with machine-readable values is a current issue on HTML5. Resolving this in favour of providing such annotation would make using RDFa and microdata in concert, or converting between them, easier, particularly if HTML5 uses the attribute content or RDFa adopts whatever attribute is introduced to HTML5.

and Elements in Flow Content

The ability to use and elements in flow content is only supported in microdata: it’s support that’s added by the microdata specification (in the Editor’s Draft since May 31st; the text allowing this didn’t make it into the Last Call version of the spec), in which it’s limited to and elements with an itemprop attribute.

It would be possible for the RDFa specification to similarly make the statement that and elements are allowed in flow content as long as they have particular attributes. This would ease the transition between the two formats, and works a lot better than empty elements which crop up fairly commonly in RDFa content.

(One oddity here is that because date/time values have to be on a element in microdata, publishers cannot replace empty elements with elements as they might an empty .)

Identifiers without Types

Many of the RDFa examples are of resources that have a URI identifier but for which no type is supplied. Microdata, on the other hand, states that itemid is only allowed on elements that also have an itemtype (and an itemscope). The reason given is because the itemid needs to be interpreted based on the itemtype. This would be understandable if it held a string, but given that the itemid provides a URI it seems a bit strange. Perhaps it’s an attempt to avoid the whole httpRange-14 / ambiguity in URIs issue.

If this restriction remains, the advice to RDFa users who might want to convert to microdata at a future date would be to always provide a type for your (non-blank-node) resources. It may be useful to define a http://w3.org/ns/global/Thing within the vocabulary that I propose above, given that the URI for rdfs:Resource is long and hard to recall.

Built-in Prefixes

The built-in profile for RDFa defines a number of prefixes for vocabularies that are either coined by the W3C or coined elsewhere but in common use on the web. This, coupled with vocab and the ability to directly use URIs in the relevant attributes, means that declaring prefixes within the document is increasingly unnecessary in RDFa.

In contrast, using existing vocabularies, even popular ones, within microdata is relatively difficult, particularly when vocabularies are mixed on the same item.

Most useful for publishers would be if both RDFa and microdata recognised the same set of prefixes. This would reduce the size of microdata created from existing RDFa content as well as making it easier to move between the languages. At the very least, it would be good to have rdf:, rdfs:, xsd: and xhv: built into both.

The list of popular vocabularies is likely to change over time; for example a prefix for the schema.org vocabulary might be useful at some point in the near future. The problem is that publishers and consumers need to be synchronised in their use of prefixes: it’s no good for a publisher to use the prefix sch: if there might be processors for the page that don’t recognise it. Equally, consumers shouldn’t be reliant on a network connection to retrieve the latest set of prefix mappings in order to parse the page. It’s not clear to me how best to manage this evolution, but even a fixed set of prefixes at the point the specs reach Recommendation is more usable than spelling out URIs all the time.

Literals Including Markup

RDFa supports literals that include markup (the innerHTML of an element) as well as those that don’t (the textContent of an element), whereas microdata only supports creating values from particular attributes or the textContent of the element. This makes it hard to create embedded microdata that includes values which contain things like mathematical or chemical formulae, ruby text, or multiple paragraphs.

A solution would be for microdata to introduce an itemhtml (or something) attribute that, when present, indicates that the value of the property should include markup. There is a current issue on microdata to support HTML values.

Itemref

RDFa can support a subset of itemref’s functionality, namely to have properties defined elsewhere in a document be associated with a given resource. What it doesn’t support is the sharing of properties defined in one place by two or more resources.

RDFa could add such support by adding an attribute that mirrors itemref (eg ref, I guess), with the referenced element being processed using the evaluation context inherited by the referencing element (which means that attributes such as vocab would sometimes have a scope that wasn’t based on the document tree). This would make it easier to tackle the use case for itemref using RDFa as well as making it easier to move between or mix RDFa and microdata.

Lists

It is easy for microdata to represent a property with a list of values, and really really hard to do the same in RDFa. This is in part because RDF views lists resources rather than a distinct data type, and in part because RDFa hasn’t added any syntax sugar to make creating rdf:List resources easy. Adding some syntax sugar for lists would make life a lot easier for anyone using RDFa, but especially if they are adapting existing microdata content to RDFa.

Datatypes

Microdata assumes that consumers will convert values to appropriate datatypes based on the property (which they understand) as a separate process after microdata processing, whereas RDFa supports the use of a datatype attribute to explicitly indicate the datatype of each value. This mismatch means that information is lost when RDFa is converted to microdata, and has to be added when microdata is converted to RDFa.

Bringing the languages completely into sync would mean either microdata adding a facility to support (at least some) datatypes, or deprecating the datatype attribute in RDFa. Alternatively, this may simply be an area where the differences in behaviour between the two specifications doesn’t matter because the data models that they use are distinct anyway.

Languages

Languages are similar to datatypes, in that RDF (and hence RDFa) supports annotating strings with the language that they are in whereas microdata doesn’t within its core data model or its JSON serialisation. However, the elements that represent properties within the HTML, used within the DOM API access to microdata, will have a language.

It may be that in practice consumers need to base their microdata processing on the DOM API rather than the core microdata data model or JSON extracted through a standalone process, and thus pick up the language from the property elements, I don’t know. In any case, the microdata JSON serialisation, used for drag-and-drop, is lossy and could be extended to include the language of each value when available, at fairly substantial complexity cost.

For publishers, it doesn’t much matter either way; if they are dealing with multi-lingual text they will want to include a lang attribute in the HTML anyway, regardless of the impact on embedded data.

Multiple Types

RDFa supports having multiple types named in the typeof attribute whereas microdata only supports one type per item. In any mapping from RDFa to microdata, publishers have to choose which type is the primary type for the item and move the others to be expressed via rdf:type properties. Consumers who want to support publishers who might not choose their type as the primary type have to detect items that have the type they are interested in within the rdf:type property as well as those which have the type as the main type. Given that the rdf:type URI is long and (naturally) associated with RDF, it might be better to define a property such as http://w3.org/ns/global/type for this use.

Microdata could be extended to allow multiple values in the itemtype attribute, with the first being used to interpret any properties that aren’t full URIs. This would make it easier for both consumers to detect when a type they were interested in was used and for publishers to use RDFa and microdata in tandem or move between them.

The `src` Attribute

RDFa and microdata interpret the src attribute in opposite ways. In RDFa, it provides the identifier for a new resource (equivalent to itemid in microdata); in microdata, it provides a URL value of a property on elements that support it (equivalent to resource or href in RDFa).

RDFa interprets src in this way to make it easier to make assertions about an image, but it’s of limited effect as even in RDFa its only possible to make three such assertions (through the typeof, rel and property attributes). So, for example, you can specify the type of the image, link to its license and give the name of its creator, with:

but this won’t help you if you also want to give the title for the image and when it was created (say). At that point, the microdata and RDFa start to look similar:

or:

and really, to make the markup consistent, you may as well not use the src of the image at all in the RDFa either:

So it’s not clear to me that interpreting the src attribute as the subject of triples offers such a huge advantage that it’s worth the inconvenience that it brings for the simple things, such as having to use:

rather than:

Link relations

This isn’t so much a clash between RDFa and microdata as between the interpretation that RDFa has for the rel attribute and that specified in HTML.

The built-in rel values in HTML are a bit of a mix. Some of them, like alternate, prev and next encode relationships between the document in which the link appears and another document. Others, such as bookmark and help, create relationships between the context in which the link is found and the referenced document. Still others, like nofollow, noreferrer and prefetch, are really instructions to the client about how to manage the act of traversing the link.

It doesn’t seem semantically correct to automatically create relationships based on the built-in HTML rel values, unless you are deliberately trying to extract document semantics from the page. This is a problem for RDFa, which reuses the rel attribute to provide property values for the embedded data.

One thing that could be done would be for RDFa to consistently use the property attribute everywhere rather than the rel attribute. This would not only ease the overloading but also reduce the confusion for users, who currently have to work out which attribute to use based on whether the value is a resource or a literal.

Possible Subset of RDFa

When mapping from microdata to RDFa, the only attributes that are really needed are:

vocab to define a vocabulary for the types and properties within its scope (not technically necessary, but keeps the markup simple compared to spelling out URIs for everything)
typeof to define the type of a resource or indicate a new blank node
about to provide a URI for a resource or a local identifier for a blank node
property and rel to define property names (though see above for discussion about dropping rel)
href, src and content to provide values (and datetime assuming that is supported)

In the mappings in the analysis below, I did also use the resource attribute, but only to create a reference to a blank node that was described elsewhere, when replicating the functionality of itemref. If RDFa were to enable and in content in the same way as microdata, resource functionality could be replicated using ; as it is, you can get away with using an empty element.

Similarly, I only used datatype when providing a datatype for date/time values, something that could be done automatically by RDFa. But this isn’t surprising given that microdata doesn’t support datatypes at all and the examples I was using for the mapping were from the microdata specification.

There was no need for:

prefix which defines prefixes to simplify references to properties and classes; this is hardly surprising as few of the microdata examples involved mixing namespaces, but it’s notable that the built-in prefixes of rdf: and xsd: were useful
profile which is a pointer to an external document that defines a set of terms; this is being dropped from RDFa in any case

I also kept to a simplified version of the syntax in which each property element only provided one value. This subset is basically:

resource elements can have about (equivalent to itemid) and typeof (equivalent to itemtype) attributes on them
property elements can have property or rel (equivalent to itemprop), and a value-providing attribute on them such as href or content
no element is both a resource element and a property element; to provide a property whose value is a resource, nest the resource element within the property element (using “hanging rel” processing)
no property element should provide more than one value for a property; in particular, a “hanging rel” should only have a single resource element child

This simplified profile of RDFa is fairly easy to remember and maps easily to and from microdata: most attributes can be simply renamed; the only attribute that needs to be moved as well as renamed is the “hanging rel”, which moves onto the resource element and is renamed to itemprop. Note that it also means avoiding using the src attribute to encode embedded data.

In addition to sticking to this subset of attributes, developers might be advised that using HTML link relations may lead to clashes with browser or search engine interpretation of the links in the page.

Possible Subset of Microdata

Microdata is pretty minimalistic already. The only feature that developers need to be warned about is itemref, which has no RDFa equivalent at the moment.

Guidelines for Vocabulary Authors

There are a several guidelines that come out of this comparison for people putting together vocabularies that aim to be usable in both RDFa and microdata:

The classes in the vocabulary should be distinct, or subclasses created with any relevant combinations of superclasses, so that publishers don’t have to assign more than one type to an item/resource. This restriction helps with using the vocabulary with microdata, which assumes that every item has a single type.
Provide explicit classes for everything which you anticipate might be given an identifier, as microdata doesn’t (currently) enable items to have an identifier without also having a type.
Put classes and properties in the same namespace, but do not name classes and properties with the same local name; while this doesn’t matter in microdata because the properties are interpreted relative to the class, standard conversions to RDF will create a class and a property with the same URI. URIs are case-sensitive to a simple way of ensuring that there aren’t clashes is to follow the usual RDF convention of beginning class names with an upper-case letter and property names with a lower-case letter.
Avoid property names that contain dots, as these aren’t allowed in non-URI property names in microdata.
Ensure that properties either only expect one type of value or expect values whose type can be sniffed based on the syntax of the value. If publishers use microdata, they will not be able to indicate the type of a value through the markup.
Be aware that consumers of microdata using your vocabulary will have to use the DOM API to identify the language used in any strings, and that language information won’t be carried through the standard microdata JSON serialisation (used by drag-and-drop, for example). If you anticipate multi-lingual use of your vocabulary, you may way to define a MultiLingual class with value and language properties that people can use as nested items. (It may be useful for this class and properties to be defined in the proposed ‘global’ W3C namespace so that it can be used anywhere.) If you know what languages will be used then provide separate properties for each language (eg for UK legislation I know the languages are English and Welsh so on a vocabulary for UK Legislation I could have title-en and title-cy properties).
To make markup cleaner, only reuse properties from other vocabularies on your classes if they have built-in prefixes (eg unless rdfs: is built-in to microdata as well as RDFa, don’t use rdfs:label to provide a label, but create your own label property). On the other hand, do reuse classes from other vocabularies if you don’t need to add any specialised properties to them. Note that avoiding reuse has the unfortunate side-effect of not enabling processors that understand these other vocabularies to process your data.
Avoid having properties whose values need to be retrieved in order, as these are hard to represent in RDFa. Instead, use properties with distinct names when position is important. (Yes, I know this sucks.)

Choosing Between Microdata and RDFa

The choices developers make between microdata and RDFa will, I suspect, be largely dictated by what their consumers/toolsets/publishers will support. Nevertheless, there are some features that are better supported by one or other format and might therefore sway developers one way or another:

multi-lingual embedded data is better supported in RDF than microdata+JSON
explicit datatypes for values can be provided by RDFa but not microdata
resources with multiple types are a lot easier to describe in RDFa
property values that include markup are a lot easier to write in RDFa
mixed vocabulary use is a bit easier in RDFa than in microdata
HTML5 link relations may be misinterpreted by RDFa processors
properties with list values are much easier to support in microdata
common content adopted by multiple entities is much easier in microdata

Final Words

I have no doubt that developers would be better off if there were only one recommended way of embedding data in HTML (so long as it met their requirements of course). But realistically that is, and always has been, a long shot, given the entrenched positions of the microdata and RDFa communities.

Regardless, there are lessons that RDFa and microdata could learn from each other, and changes to both languages that would help developers use them on their own, switch between them and mix them in the same document. I expect and welcome debate about the viability and effectiveness of the changes and guidelines that I’ve suggested here.

Investigating those lessons, documenting those changes and generating those guidelines was something that I had hoped the microdata/RDFa task force would be able to do. The other question to ask, given the argument that there shouldn’t be a task force at all if it’s not going to be able to bring the languages together, is whether this kind of analysis is worthwhile, and worth publishing as something more official than a blog post?

Mapping RDFa to Microdata

2011-08-20T16:38:38+00:00

This post is part of a three-part series that analyses the differences in features and syntax between microdata and RDFa. The series attempts:

to identify the differences in approach and functionality of the two languages, which should help developers choose between them
to identify any guidelines for developers of vocabularies for use with both languages
to identify a subset of functionality that is common between the two languages, which developers might want to stick to to make switching and mixing easier
to identify mapping rules that might be applied to automatically or manually map from one language to another if the simple subset is used

I’ve done this by looking at converting microdata examples to RDFa and vice versa, and the lessons to be drawn from that exercise. The three posts are on:

This post is the second of these, which looks at how RDFa might be mapped to microdata. In this case, I will aim to express the RDF created from the RDFa as the equivalent microdata JSON, and aim to create that JSON with the microdata.

To create the microdata JSON, I’ve used the rule that the URI of the first type of a resource is processed to provide a namespace that is stripped from the URIs of the properties (to create simple names where possible). In addition, when a resource has no properties, it will be represented as a string (URI) value rather than as a nested item. Other than that I hope the mapping will be obvious; I’ll point out where it involves a loss of information. I’m assuming that the document is at http://example.org/ throughout.

I have based what’s written here on the latest specifications of both microdata (in its WHAT WG and W3C variants) and RDFa Core and HTML+RDFa but I haven’t consulted with anyone involved in these efforts and may well have got things wrong. Philip Jägenstedt’s Live Microdata service has proved invaluable for testing, so many thanks to him for providing that service.

The post is rather heavy going and you might want to just skip to the summary instead of reading the whole thing.

The post goes through the examples from the RDFa specification plus one additional example from the wild. I haven’t included examples that don’t illustrate anything new, so there are some that are skipped. Other examples would be welcome.

Page Metadata

When parsing begins, the current subject will be the IRI of the document being parsed, or a value as set by a Host Language-provided mechanism (e.g., the base element in (X)HTML). This means that by default any metadata found in the document will concern the document itself:
  
    Jo's Friends and Family Blog
    
    
  
  
    ...
  

The equivalent microdata JSON for this document that we’d want to create is (note: invalid example):

{ "items": [
  {
    "id": "http://example.org/" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://example.org/#bbq" ],
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  }
]}

and we’d want to create it with (note: invalid example):


  
    Jo's Friends and Family Blog
    
    
  
  
    ...

However, this is is not valid according to the microdata specification. In microdata, only items that have types are allowed to have identifiers. Rather than losing the identifier, we’ll add a type; I’m going to use rdfs:Resource. It’s not the nicest of URIs to type, but it’s got something close to the correct semantics. So we’ll aim for the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://example.org/#bbq" ] ,
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  }
]}

which means we need:


  
    Jo's Friends and Family Blog
    
    
  
  
    ...

Notes:

The itemscope is necessary for the page to be recognised as containing any data at all.
The itemid can’t just be empty: the . is the shortest URI you can use to reference the page itself.
I put the itemscope, itemtype and itemid on the element rather than the element so that they wouldn’t be inherited into the : it seems to make sense for any data within the to be about the page itself.
The foaf: and dc: prefixes are built-in to RDFa, so it’s easy for people to use classes and properties in those common vocabularies without having to remember their full URI. In microdata, that URI and the one for the rdfs:Resource class have to be spelled out in full.

Base URI

In (X)HTML the value of base may change the initial value of current subject:
  
    
    Jo's Friends and Family Blog
    
    
  
  
    ...
  

This changes the id of the item generated:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://www.example.org/jo/blog" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://www.example.org/jo/blog#bbq" ] ,
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  }
]}

In the microdata, the itemid can still be . as the base URI set by the element is used to resolve it:


  
    
    Jo's Friends and Family Blog
    
    
  
  
    ...

Explicit Subjects / ItemIds

To illustrate how this affects the statements, note in this markup how the properties inside the (X)HTML body element become part of a new calendar event object, rather than referring to the document as they do in the head of the document:
  
    Jo's Friends and Family Blog
    
    
  
  
    
      I'm holding
      
        one last summer barbecue
      ,
      on
      
        September 16th at 4pm
      .
    
  

In microdata JSON, this would be:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/primaryTopic": [ "http://example.org/#bbq" ],
      "http://purl.org/dc/terms/creator": [ "Jo" ]
    }
  } ,
  {
    "type": "http://www.w3.org/2002/12/cal/ical#Vevent" ,
    "id": "http://example.org/#bbq" ,
    "properties": {
      "summary": [ "one last summer barbecue" ] ,
      "dtstart": [ "2015-09-16T16:00:00-05:00" ] ,
    }
  }
]}

Note that this mapping loses the fact that the value of the dtstart property is a date-and-time. Processors of this JSON are expected to know that the dtstart property takes a date/time value and would have to sniff the value to work out that it’s a date-and-time rather than a date.

In-browser microdata processors can identify the value as a date/time value because the property element itself is accessed through the element.properties IDL attribute; processors that work with this DOM API can tell that it’s a element, get hold of the date/time itself and access the content of the element for the human-readable representation used on the page. However, this information isn’t part of the core microdata data model.

To create this JSON from microdata you need:


  
    Jo's Friends and Family Blog
    
    
  
  
    
      I'm holding
      
        one last summer barbecue
      ,
      on
      
        September 16th at 4pm
      .

Notes:

There are no prefix definitions in microdata, so the type has to be spelled out in full. However, with the mapping I’m assuming from RDFa to microdata JSON, the properties in that same namespace for items in that class don’t.
The itemscope has to be added despite the
element having both an itemid and an itemtype; if the itemscope is forgotten, the item isn’t recognised.
The original element has to be changed to a element because it isn’t conformant microdata for a date/time value to be supplied by any other element.

Items from the `src` Attribute

If @about is not present, then @src is next in priority order, for setting the subject of a statement. A typical use would be to indicate the licensing type of an image:

This should generate the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/photo1.jpg" ,
    "properties": {
      "http://www.w3.org/1999/xhtml/vocab#license": [ "http://creativecommons.org/licenses/by/2.0/" ] ,
    }
  }
]}

The src attribute in microdata is only used for a value, so creating the microdata about the image means a wrapper element and a separate element:

Note:

The license property is part of the built-in set of link relationships in HTML, but there is no easy way to refer to that property from microdata; they have to be spelled out as full URLs.

Additional Properties for Images

Since there is no difference between @src and @about, then the information expressed in the last example in the section on @about (the creator of an image), could be expressed as follows:

This is a simple additional property in the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/photo1.jpg" ,
    "properties": {
      "http://www.w3.org/1999/xhtml/vocab#license": [ "http://creativecommons.org/licenses/by/2.0/" ] ,
      "http://purl.org/dc/terms/creator": [ "Mark Birbeck" ]
    }
  }
]}

which can be created through a element within the :

Nested Images

Since normal chaining rules will apply, the image IRI can also be used to complete hanging triples:
  

This should generate the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://www.blogger.com/profile/1109404" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/img": [{
        "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
        "id": "http://example.org/photo1.jpg" ,
        "properties": {
          "http://www.w3.org/1999/xhtml/vocab#license": [ "http://creativecommons.org/licenses/by/2.0/" ] ,
          "http://purl.org/dc/terms/creator": [ "Mark Birbeck" ]
        }
      }]
    }
  }
]}

The microdata to generate this is:

Notes:

The big gotcha in this conversion is that in microdata, the foaf:img property has to be moved onto the item that is a value of that property; there’s no equivalent to the “hanging rel” processing in RDFa. A disadvantage of this is that anyone copying-and-pasting the element to embed the same information about the image within their own page will have the itemprop attribute carried along with the image, into a context where the foaf:img property might not be relevant.

Types with Blank Nodes

For example, an author may wish to create markup for a person using the FOAF vocabulary, but without having a clear identifier for the item:
  Albert Einstein
  Albert

Now we have an explicit type, we can create microdata JSON that uses short names:

{ "items": [
  {
    "type": "http://xmlns.com/foaf/0.1/Person" ,
    "properties": {
      "name": [ "Albert Einstein" ] ,
      "givenName": [ "Albert" ] ,
    }
  }
]}

This can be generated with the microdata:


  Albert Einstein
  Albert

which is nice and simple.

Inherited Subject

The most usual way that an inherited subject might get set would be when the parent statement has an object that is a resource. Returning to the earlier example, in which the long name for the German_Empire was added, the following markup was used:
  Albert Einstein
  1879-03-14
  
  the German Empire

The equivalent microdata JSON for this would be:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Albert_Einstein" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
      "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ] ,
      "http://dbpedia.org/property/birthPlace": [ "http://dbpedia.org/resource/German_Empire" ]
    }
  } , {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/German_Empire" ,
    "properties": {
      "http://dbpedia.org/property/conventionalLongName": [ "the German Empire" ] ,
    }
  }
]}

Note that this microdata JSON could only be generated syntactically from the RDFa, not via RDF, because going via RDF would make it impossible to know whether to give the dbp:birthPlace property a string (which is a URI) value or a nested item. We’ll see the alternative version of the microdata RDF in the next example.

To create this microdata JSON, we need:


  Albert Einstein
  1879-03-14
  
  
    the German Empire

Notes:

I’ve had to change two elements here: the for the date of birth has become a element as the value of the property is a date, and the
for the birth place has become a element because the value of that property is a URL.
I’ve also had to add a nested element as it’s not possible in microdata to have a single element describe both an item and a property for that item as it is in RDFa.

In an earlier illustration the subject and object for the German Empire were connected by removing the @resource, relying on the @about to set the object:
  Albert Einstein
  1879-03-14
  
    the German Empire
  

While this generates the same RDF as the previous example, the microdata JSON that it generates should probably be different: this time, the item for the German Empire is nested within the item for Albert Einstein:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Albert_Einstein" ,
    "properties": {
      "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
      "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ] ,
      "http://dbpedia.org/property/birthPlace": [{
        "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
        "id": "http://dbpedia.org/resource/German_Empire" ,
        "properties": {
          "http://dbpedia.org/property/conventionalLongName": [ "the German Empire" ]
        }
      }
    }
  }
]}

To create this, the microdata needs to look like:


  Albert Einstein
  1879-03-14
  
    the German Empire

Note that while this looks quite similar to the RDFa version, in fact the itemid attribute that holds the URI for the German Empire is on a different element from the about attribute in the RDFa.

The third RDFa example around this same content is:

but it is also possible for authors to achieve the same effect by removing the @about and leaving the @resource:
  Albert Einstein
  1879-03-14
  
    the German Empire
  

This should lead to the same microdata JSON, so I won’t bother repeating the microdata. What’s interesting is that this pattern: the wrapper element containing the property (rel) and identifier for the item that is the value for that property (resource) is a lot closer to the microdata pattern of expressing nested items. The big distinction here is that while in microdata, the itemtype also resides on that element, if you tried adding a typeof attribute to the inner

in RDFa, you’d end up with a new blank node.

Anonymous Nested Resources

However, an author could just as easily say that Spinoza influenced something by the name of Albert Einstein, that was born on March 14th, 1879:
  
    Albert Einstein
    1879-03-14
  

This should generate the microdata JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Baruch_Spinoza" ,
    "properties": {
      "http://dbpedia.org/ontology/influenced": [{
        "properties": {
          "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ]
        }
      }]
    }
  }
]}

which again means moving an attribute in microdata:


  
    Albert Einstein
    1879-03-14

It is generally harder to move to microdata from RDFa when the RDFa has an element that both provides a subject and provides a property.

The RDFa spec provides a couple of additional methods of marking up the same content to give exactly the same RDF (and microdata JSON):

Note that the div is superfluous, and an RDFa Processor will create the intermediate object even if the element is removed:
  Albert Einstein
  1879-03-14
An alternative pattern is to keep the div and move the @rel onto it:
  
    Albert Einstein
    1879-03-14
  
From the point of view of the markup, this latter layout is to be preferred, since it draws attention to the ‘hanging rel’. But from the point of view of an RDFa Processor, all of these permutations need to be supported.

Interestingly, it’s this latter permutation that is the one that’s closest to the microdata method of expressing the data, though as we will see in the next section, the “hanging rel” is not exactly equivalent to the itemprop on the wrapper element.

Hanging Rels

Note that each occurrence of @about will complete any incomplete triples. For example, to mark up the fact that Albert Einstein had a residence both in the German Empire and Switzerland, an author need only specify one @rel value that is then used with multiple @about values:
  
  

The data embedded here gives two values for the dbp-owl:residence property:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Albert_Einstein" ,
    "properties": {
      "http://dbpedia.org/ontology/residence": [
        "http://dbpedia.org/resource/German_Empire" ,
        "http://dbpedia.org/resource/Switzerland"
      ]
    }
  }
]}

In microdata, the itemprop attribute has to appear on both the nested elements to make it clear that they both provide values for that property:

The next example illustrates this with nested items rather than strings:

To illustrate, to indicate that Spinoza influenced both Einstein and Schopenhauer, the following markup could be used:
  
    
      Albert Einstein
      1879-03-14
    
    
      Arthur Schopenhauer
      1788-02-22
              
  

This should generate:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Baruch_Spinoza" ,
    "properties": {
      "http://dbpedia.org/ontology/influenced": [{
        "type": "http://xmlns.com/foaf/0.1/Person" ,
        "properties": {
          "name": [ "Albert Einstein" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ]
        }
      }, {
        "type": "http://xmlns.com/foaf/0.1/Person" ,
        "properties": {
          "name": [ "Arthur Schopenhauer" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1788-02-22" ]
        }
      }]
    }
  }
]}

In this case, the itemprop that is equivalent to the RDFa rel has to move down onto the elements representing the items:


  
    
      Albert Einstein
      1879-03-14
    
    
      Arthur Schopenhauer
      1788-02-22

The wrapper

around both items isn’t necessary; I’ve left it to stay as close to the markup of the original RDFa as possible.

Implicit Resources

Triples are also ‘completed’ if any one of @property, @rel or @rev are present. However, unlike the situation when @about or @typeof are present, all predicates are attached to one bnode:
  Albert Einstein
  1879-03-14
  
    
    
  

To be equivalent to the RDF generated from this markup, the microdata JSON would be:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://dbpedia.org/resource/Baruch_Spinoza" ,
    "properties": {
      "http://dbpedia.org/ontology/influenced": [{
        "properties": {
          "http://xmlns.com/foaf/0.1/name": [ "Albert Einstein" ] ,
          "http://dbpedia.org/property/dateOfBirth": [ "1879-03-14" ]
          "http://dbpedia.org/ontology/residence": [
            "http://dbpedia.org/resource/German_Empire" ,
            "http://dbpedia.org/resource/Switzerland"
          ]
        }
      }]
    }
  }
]}

Microdata is a lot more explicit about when items get created, and consequently requires a bit more markup:


  
    Albert Einstein
    1879-03-14

Overriding Text Content

The value of @content is given precedence over any element content, so the following would give exactly the same triple as shown above:
John Doe

The equivalent microdata should generate the JSON:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://internet-apps.blogspot.com/" ,
    "properties": {
      "http://purl.org/dc/terms/creator": [ "Mark Birbeck" ]
    }
  }
]}

Only the element and links override the content of an element in microdata. So a mirror of this example needs a separate element:

  
    
    John Doe

Language Support

In RDFa the Host Language may provide a mechanism for setting the language tag. In XHTML+RDFa [XHTML-RDFA], for example, the XML language attribute @xml:lang or the attribute @lang is used to add this information, whether the plain literal is designated by @content, or by the inline text of the element:

Like the datatype of a value, the language of a value isn’t captured by the microdata data model or the JSON representation of that data model. So the fact that ‘chat’ is French is lost:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://example.org/node" ,
    "properties": {
      "http://example.org/property": [ "chat" ]
    }
  }
]}

The equivalent microdata is thus:

with the language only accessible if you are using the DOM to process the microdata.

Literals that Include Markup

RDFa therefore supports the use of normal markup to express XML literals, by using @datatype:
  E = mc²: The Most Urgent Problem of Our Time

The datatype="rdf:XMLLiteral" acts like a flag to indicate that the serialised content of the element (innerHTML) needs to be used as the value of the property, rather than the textContent, which includes markup, can be expressed in microdata JSON as follows:

{ "http://purl.org/dc/terms/title": "E = mc²: The Most Urgent Problem of Our Time" }

There’s no way to generate this in microdata except by repeating the escaped version of the content in a content attribute:


  E = mc²: The Most Urgent Problem of Our Time

This is hardly ideal. It’s tedious enough with a short string like this one; for larger amounts of information such as long descriptions of an event, it would be very tedious.

The `resource` Attribute

RDFa provides the @resource attribute as a way to set the object of statements. This is particularly useful when referring to resources that are not themselves navigable links:
  
    On Crime and Punishment
    
  
  
    
      
        Rodion Romanovitch! My dear friend! If you go on in this way
        you will go mad, I am positive! Drink, pray, if only a few drops!
      
    
  

This should produce:

{ "items": [
  {
    "type": "http://www.w3.org/2000/01/rdf-schema#Resource" ,
    "id": "http://www.example.com/candp.xhtml#q1" ,
    "properties": {
      "http://purl.org/dc/terms/source": [ "urn:ISBN:0140449132" ]
    }
  }
]}

which is expressed through:


  
    On Crime and Punishment
    
  
  
    
      
      
        Rodion Romanovitch! My dear friend! If you go on in this way
        you will go mad, I am positive! Drink, pray, if only a few drops!

The property and value have to be moved onto a nested , but this is a more extensible pattern than the RDFa method as it enables other properties to be expressed in the same way.

Multiple Types

This last example comes from the wild rather than being an example in the specification. At http://bitmunk.com/browse we find:

Bitmunk

This shows the use of multiple types and of multiple properties with the same value, because the pages are attempting to use multiple vocabularies that cover the same domain (organisations) to different depths. In the equivalent microdata, we have to choose one of the types; I’m going to assume that it should just use the first one from the typeof attribute:

{ "items": [
  {
    "type": "http://www.w3.org/2006/vcard/ns#VCard" ,
    "id": "http://bitmunk.com/about#service" ,
    "properties": {
      "http://www.w3.org/1999/02/22-rdf-syntax-ns#type": [
        "http://purl.org/commerce#Business" ,
        "http://purl.org/goodrelations/v1#BusinessEntity"
      ] ,
      "http://www.w3.org/2000/01/rdf-schema#label": [ "Bitmunk" ] ,
      "fn": [ "Bitmunk" ]
    }
  }
]}

The microdata equivalent is:


  
  
  Bitmunk

Notes:

Technically, the RDFa doesn’t place any ordering on the three classes, but I’m picking the first for the purpose of the microdata conversion. The other classes are harder to get at in the JSON: they have to be referenced via the rdf:type microdata property rather than the type JSON property. Consumers that are on the lookout for items of the type gr:BusinessEntity wouldn’t spot these items.

Mapping Microdata to RDFa

2011-08-20T16:35:28+00:00

This post is part of a three-part series that analyses the differences in features and syntax between microdata and RDFa. The series attempts:

to identify the differences in approach and functionality of the two languages, which should help developers choose between them
to identify any guidelines for developers of vocabularies for use with both languages
to identify a subset of functionality that is common between the two languages, which developers might want to stick to to make switching and mixing easier
to identify mapping rules that might be applied to automatically or manually map from one language to another if the simple subset is used

I’ve done this by looking at converting microdata examples to RDFa and vice versa, and the lessons to be drawn from that exercise. The three posts are on:

This post is the first of these, which looks at how microdata might be mapped to RDFa, in terms of generating the same RDF according to the microdata-to-RDF mapping rules that I outlined in my post on Microdata + RDF.

I have based what’s written here on the latest specifications of both microdata (in its WHAT WG and W3C variants) and RDFa Core and HTML+RDFa but I haven’t consulted with anyone involved in these efforts and may well have got things wrong. Gregg Kellogg’s Distiller service has proved invaluable for testing, so many thanks to him for providing that service.

The post is rather heavy going and you might want to just skip to the summary instead of reading the whole thing.

The post goes through the examples from the microdata specification (most of them are in both versions, the only exceptions being those that use the vCard vocabulary). I haven’t included examples that don’t illustrate anything new, so there are some that are skipped. Other examples would be welcome.

Unidentified Items / Blank Node Subjects

Here there are two items, each of which has the property “name”:
 My name is Elizabeth.



 My name is Daniel.

The first challenge is to map this into RDFa because the properties are tokens rather than URIs and there is no type for either of the items. What I’ll assume here is that the name properties are local to the document itself and thus the equivalent RDF is:

[ <#name> "Elizabeth" ] .
[ <#name> "Daniel" ] .

This can be achieved in RDFa through either:


  My name is Elizabeth.



  My name is Daniel.

or:


  My name is Elizabeth.



  My name is Daniel.

Notes:

The vocab="#" sets the vocabulary to the location of the current document (plus an empty fragment identifier); this URI is then concatenated to the property token (name) to create a URI that is unique to the document. In a document such as this it would make sense to put the vocab="#" attribute on the element rather than on every single item.
With no type in sight, blank nodes can either be created by having an empty typeof attribute or through an about attributes whose value starts with _:. The latter has the advantage of providing an identifier for the blank node that can be used elsewhere in the document, but the former is shorter so will be used where possible in the remaining examples of this post.

Values from the `src` Attribute

The next example introduces the use of the src attribute to set the value of the property.

In this example, the item has one property, “image”, whose value is a URL:
 

This should probably be mapped to the RDF:

[ <#image>  ] .

The difficulty with this is that in RDFa, the src attribute is used for the subject of a statement (equivalent to a microdata item) rather than the object (equivalent to a microdata value). So we have two choices for equivalent RDFa. One is to use a similar pattern to that used above, but introduce a wrapper element that provides the property:

Another is to provide what would normally be an object through a resource attribute and then use a rev attribute (rather than the usual rel) attribute to reverse the relationship:

This has three disadvantages over the first option:

the resource attribute that creates the item is on the element rather than on the wrapper
which makes it hard to create other properties for that item
we have to use a rev attribute, reversing the normal flow of relationships; I (at least) find this hard to figure out when there’s not a rel attribute as well
we have to make up an id for the blank node we want to generate

I’ll note that it took me five or six failed attempts to generate the above options. If I hadn’t had the RDF Distiller to test with, I would have got it wrong. Note that at least through the RDF Distiller, to be recognised, the resource attribute has to have an (empty) value — it is not enough for it to simply be present, unlike with the typeof attribute. Note that the resource attribute has to explicitly point to a blank node to create a blank node rather than having the property be associated with the document in which this appears.

Values from the `datetime` Attribute

The next example illustrates the use of the element to provide a date/time value for a property.

In this example, the item has one property, “birthday”, whose value is a date:
 I was born on May 10th 2009.

I’m assuming this should map to the RDF:

[ <#birthday> "2009-05-10"^^ ]

There is an open issue (ISSUE-97) about this on RDFa, which currently requires the use of the content attribute to provide the value as follows:


 I was born on May 10th 2009.

Note that the xsd: prefix is built-in within RDFa so there’s on need for any declaration for it, which makes it fairly easy to specify the standard date/time datatypes.

If ISSUE-97 were resolved nicely it would be possible to instead do:


 I was born on May 10th 2009.

Notes:

To make this work, RDFa processors would have to look at the syntax of the datetime attribute to work out what datatype the value should be matched to.
The syntax permitted in the datetime attribute isn’t exactly the same as that permitted by the XML Schema time and dateTime types usually used in RDF (and XML), in that the seconds component is optional within HTML. The resolution to ISSUE-97 will need to take this into account. Otherwise, anyone mapping from microdata to RDFa manually will need to ensure that the content attribute includes the seconds component.

Nested Items / Object Properties

In this example, the outer item represents a person, and the inner one represents a band:
 Name: Amanda
 Band:  Jazz Band (12 players)
The outer item here has two properties, “name” and “band”. The “name” is “Amanda”, and the “band” is an item in its own right, with two properties, “name” and “size”. The “name” of the band is “Jazz Band”, and the “size” is “12”.

The equivalent RDF for this example would be:

[
  <#name> "Amanda" ;
  <#band> [
    <#name> "Jazz Band" ;
    <#size> "12"
  ]
]

Note that the size property is just a plain literal value; unlike with date/times, there’s no way to tell from the microdata that the value is a number.

In RDFa this could be done with:


 Name: Amanda
 Band:  Jazz Band (12 players)

This follows the microdata fairly closely but note that the nested resource doesn’t need an empty typeof attribute: it’s only the top-level items that do. It might be easier, for consistency and extensibility, to always include an explicit nested element (with an empty typeof attribute in this case) to represent the nested resource:


 Name: Amanda
 Band:  Jazz Band (12 players)

The other thing that people have to watch out for is that because the value of the band property is a resource rather than a literal, we have to use the rel attribute rather than the property attribute as we do elsewhere.

Itemref

This example is the same as the previous one, but all the properties are separated from their items:
Name: Amanda


 Band: Jazz Band
 Size: 12 players

This should create the same RDF as the previous example:

[
  <#name> "Amanda" ;
  <#band> [
    <#name> "Jazz Band" ;
    <#size> "12"
  ]
]

changing the markup as little as possible. The RDFa equivalent is:


Name: Amanda


 Band: Jazz Band
 Size: 12 players

In microdata, the itemref attribute is a method of an item adopting name/value pairs described in a separate location within the page. In RDFa, the equivalent is to say that the name/value pairs are all related to the same resource by consistently referring to the resource as the subject of the statements. In the above case, there are two blank nodes labelled _:amanda and _:c, and the about attribute is used on the same elements that provide the properties (or a wrapper element) to indicate the identity of the subject of the statements.

Notes:

The resource attribute has to be used to indicate the blank node for the band.
As before, the rel attribute has to be used for the band property, rather than the property attribute, because the object of the statement is a resource. The rule is that if you’re using resource, you should use rel. (I used property erroneously the first time I tried to write this mapping. I will never learn.)

There is another example of itemref in use later in the microdata specification:



 
  Photo gallery
 
 
  My photos
  
   
   The house I found.
  
  
   
   The mailbox.
  
  
   All images licensed under the MIT
   license.

This is equivalent to the RDF:

[
  a  ;
    ;
    ;
   "The house I found." ;
] .
[
  a  ;
    ;
    ;
   "The mailbox." ;
] .

Note that the license property is adopted by both the items in the microdata. In this particular example, the two items have the same type, and thus the license property has the same meaning in each item. It’s also possible for itemref to be used on two items that have different types, pointing to the same element, in which case the shared properties defined within that element could mean different things for the two items.

There is no way that I am aware of within RDFa to support shared use of portions of content. There could be a rough equivalent that would work in the case where the shared properties had the same semantics if RDFa allowed the about attribute to take multiple values (note: invalid example):



 
  Photo gallery
 
 
  My photos
  
   
   The house I found.
  
  
   
   The mailbox.
  
  
   All images licensed under the MIT
   license.

but this wouldn’t support the possibility of the same property having different semantics (and therefore different URIs) for the separate resources.

It’s also worth noting in this example that the mapping to RDF that I’m assuming results, in this example, in http://n.whatwg.org/work being both a class and a property. The creators of RDF vocabularies tend to name classes with an Uppercase initial letter and properties with a lowercase initial letter, and thus avoid these kinds of clashes. Vocabulary designers who are mindful of mappings to RDF may want to take the same approach.

Multiple Values

This example describes an ice cream, with two flavors:
 Flavors in my favorite ice cream:
 
  Lemon sorbet
  Apricot sorbet
 
This thus results in an item with two properties, both “flavor”, having the values “Lemon sorbet” and “Apricot sorbet”.

This example highlights one of the real nightmares of RDF: lists. In microdata, the order of the values ‘Lemon sorbet’ and ‘Apricot sorbet’ is naturally retained. There are three possible mappings to RDF.

Creating Multiple Statements

If the order of the flavours of ice cream in this example don’t actually matter, the equivalent RDF is:

[ <#flavor> "Lemon sorbet" , "Apricot sorbet" ]

which is equivalent to:

[ <#flavor> "Apricot sorbet" , "Lemon sorbet" ]

In this case, the RDFa is straight-forward:


 Flavors in my favorite ice cream:
 
  Lemon sorbet
  Apricot sorbet

It’s surprising how common it is that order doesn’t actually matter when there are multiple values for a property, such that this mapping is quite sufficient. But I’m absolutely not going to pretend that order is never important…

Creating an `rdf:Seq`

If the order of the flavours does matter, there are two ways of representing that order using RDF. The first is to use an rdf:Seq resource. This method was the original method of representing lists in RDF and is very natural to do in RDF/XML, but has largely fallen out of favour for the second method which I’ll describe below.

Using the rdf:Seq method, the equivalent RDF for the microdata would be:

@prefix rdf:  .
[
  <#flavor> [
    a rdf:Seq ;
    rdf:_1 "Lemon sorbet" ;
    rdf:_2 "Apricot sorbet"
  ]
]

which can be generated with:


 Flavors in my favorite ice cream:
 
  
   Lemon sorbet
   Apricot sorbet

Notes:

There are various other ways in which the namespace for the rdf:Seq could be created, but since the rdf: prefix is built-in to RDFa 1.1, it seems easier to use that than anything that explicitly writes out the full (ugly) RDF namespace.
The
wrapper for the
Creating a rdf:List

The current recommended way to create a list in RDF is to use a rdf:List resource. This essentially uses a linked list model to represent lists, with the rdf:first item of a list being a value and the rdf:rest being either another rdf:List or rdf:nil. Spelled out, the RDF would look like:
```
@prefix rdf:  .
[ 
  <#flavor> [
    a rdf:List ;
    rdf:first "Lemon sorbet" ;
    rdf:rest [
      rdf:first "Apricot sorbet" ;
      rdf:rest rdf:nil
    ]
  ]
]
```
but of course Turtle lets you write it:
```
[] <#flavor> ( "Lemon sorbet" "Apricot sorbet" ) .
```
Unfortunately, RDFa has no such syntax sugar. Which means:
```
 Flavors in my favorite ice cream:
 
  
   Lemon sorbet
   
    
     Apricot sorbet
     
    
   
  
 
```
Yep, horrific. Verbose and easy to get wrong, and that’s just for two items. If a third is added, the pattern is to add an about attribute on the middle items of the list so that the rdf:rest property which covers the next item in the list can be assigned to it. For example:
```
 Flavors in my favorite ice cream:
 
  
   Lemon sorbet
   
    
     Apricot sorbet
    
   
   
     
     Raspberry sorbet
     
     
   
  
 
```
Note:
- I’ve used an empty element with a href attribute to point to the rdf:nil resource. An alternative would be to use the resource attribute, which would have the advantage of not having to spell out the full URI for rdf:nil, but I’m trying to stick to using as few attributes as possible.
- Using an empty element for a link isn’t ideal; it would be neater to use a element, but these aren’t allowed in flow content within HTML5 ( and are only permitted within the microdata specification, and then only if they have an itemprop attribute). The RDFa specification could likewise allow them.
Multiple Properties Sharing a Value
Here we see an item with two properties, “favorite-color” and “favorite-fruit”, both set to the value “orange”:
```
 orange
```
This should map to the RDF:
```
[
  <#favorite-color> "orange" ;
  <#favorite-fruit> "orange"
]
```
Like itemprop, property can take multiple values, so the RDFa equivalent is simply:
```
 orange
```
Types
Here, the item’s type is “http://example.org/animals#cat”:
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
In this example the “http://example.org/animals#cat” item has three properties, a “name” (“Hedral”), a “desc” (“Hedral is…”), and an “img” (“hedral.jpeg”).
I’ll assume that this should be mapped to the RDF:
```
[
  a  ;
   "Hedral" ;
   "Hedral is a male american domestic shorthair, with a fluffy black fur with white paws and belly." ;
   
]
```
In this case, the vocab can be set to http://example.org/animals# and both the itemtype and the various property and rel attributes will use that as the basis for their identifying URIs:
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
Global Identifiers
Here, an item is talking about a particular book:
```
 Title
 
The Reality Dysfunction
 
Author
 
Peter F. Hamilton
 
Publication date
 
26 January 1996
```
Here, the item has an identifier so unlike the previous examples, the subject of the statements in the RDF is no longer a blank node:
```
@prefix xsd:  .

  a  ;
   "The Reality Dysfunction\n " ;
   "Peter F. Hamilton\n " ;
   "1996-01-26"^^xsd:date ;
  .
```
In RDFa, the subject is provided using the about attribute:
```
 Title
 
The Reality Dysfunction
 
Author
 
Peter F. Hamilton
 
Publication date
 
26 January 1996
```
Global Property Names
Here, an item is an “http://example.org/animals#cat”, and most of the properties have names that are words defined in the context of that type. There are also a few additional properties whose names come from other vocabularies.
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
The RDF equivalent to this is:
```
[
  a  ;
   "Hedral" ;
   "Hedral" ;
   "Hedral is a male american domestic shorthair, with a fluffy black fur with white paws and belly." ;
   "black" , "white" ;
   
]
```
To create this, we need the RDFa:
```
 Hedral
 Hedral is a male american domestic
 shorthair, with a fluffy black fur with white paws and belly.
 
```
Link Relations
Here is an example of a page that uses the vEvent vocabulary to mark up an event:

... Bluesday Tuesday: Money Road ... May 5th @ 7pm (until 9pm) ...Link to this page ... Location: The RoadHouse ... ...

This example is interesting because it contains, in the natural markup of the page, a rel attribute with the value bookmark, which is used for links that go to the page or section of the page within which the link is found. In this case, it’s the page. The RDF that should be generated from the page is:
```
[
  a  ;
   "Bluesday Tuesday: Money Road" ;
   "2009-05-05T19:00:00Z"^^xsd:dateTime ;
   "2009-05-05T21:00:00Z"^^xsd:dateTime ;
    ;
   "The RoadHouse" ;
   "via livebrum.co.uk"
] .
```
The following statement could legitimately be generated as well:
```
<> 
    ;
  .
```
but the item representing the event should definitely not have the http://www.w3.org/1999/xhtml/vocab#bookmark property.

Achieving this without significantly changing the HTML markup is problematic in RDFa because RDFa uses the rel attribute to provide properties for the resources that it describes within the page, overloading its standard use in HTML which is to describe properties of the page or sections within the page. The following involves the least amount of repetition:
```
 
  ...
  Bluesday Tuesday: Money Road
  ...
  May 5th @ 7pm
  (until 9pm)
  ...
  
  Link to this page
  ...
  Location: The RoadHouse
  ...
  
  ...
  
 
```
Notes:
- In the above, the typeof attribute has been moved onto a wrapper
  that encompasses the entirety of the page because if it resides on the element, it’s assumed to apply to the document itself rather than a blank node. An alternative mapping would use about="_:event" to create a blank node for the event.
- There’s no way to avoid creating a statement for the rel="bookmark" link, so the best we can do is make sure that the statement is accurate, and relates the current document to the provided URI. Unfortunately, that means creating a separate element for the url property, repeating that URL within the page, and adding an empty about attribute; here I’ve used an empty element to express the relationship; a element would do the same job if it were allowed in flow content.

Microdata + RDF

2011-07-31T19:55:44+00:00

As part of the ongoing discussion about how to reconcile RDFa and microdata (if at all), Nathan Rixham has put together a suggested Microdata RDFa Merge which brings together parts of microdata and parts of RDFa, creating a completely new set of attributes, but a parsing model that more or less follows microdata’s.

I want here to put forward another possibility to the debate. I should say that this is just some noodling on my part as a way of exploring options, not any kind of official position on the behalf of the W3C or the TAG or any other body that you might associate me with, nor even a decided position on my part.

Simplifying RDFa

As I’ve said before, RDFa, in my experience, is complicated not primarily because of the whole namespaces/CURIEs issue but because its processing model tries to be too clever. RDFa was designed to largely fit in with existing markup and turn it into embedded data “just” by adding a few attributes here and there. Thus a simple image like:

is first marked up to indicate that it’s an image:

then to provide its license:

and finally to add a title:

all by adding attributes to the one element. The trouble with this approach is that the rules about how statements are made become extremely complex, dependent on context (eg what other attributes are present, what the parent element has on it, what content it has) and default in ways that are hard to remember.

Even having written an RDFa parser, having written code to mark up documents with RDFa, having taught it, I still cannot write RDFa past a trivial example and be 100% sure that it will produce what I was aiming to produce.

If we were to look at really simplifying RDFa, rather than making cosmetic changes, we need to address this complexity. It would certainly mean backwards-incompatible changes, such as dropping the use of particular attributes and revising the way the processing model works, such that future RDFa processors couldn’t be used on RDFa 1.0. There are two possible ways of approaching this:

retaining some backwards compatibility, and aiming for a simplified subset of RDFa 1.0 such that RDFa 1.0 processor will still get the intended triples out of data marked up with RDFa 1.1
dropping backwards compatibility entirely and using completely different attributes, essentially creating a new language

I do not know which of these routes is the best one to take.

My instinct is that the first will be hard to do. For example, there are already certain simplifications in RDFa 1.1 — such as assuming an element with no datatype attribute is giving a string value rather than looking to see if there are any non-text-nodes in the content of the element — which lead to markup that will not be processed correctly by RDFa 1.0 processors. Perhaps that could be addressed by rewriting history: creating a RDFa 1.0 Second Edition that includes any changes that are needed to make a simple subset viable.

What I want to explore here is what the second route — using entirely different attributes from those currently used in RDFa 1.0 — might mean. I think that in this case the substantial difference between microdata and this new language would be support for that much-derided requirement: decentralised extensibility.

Adding Decentralised Extensibility to Microdata

As I discussed earlier in the week, microdata is simply not designed for use in a web where publishers might want to use multiple vocabularies to mark up the same thing for different consumers. This focus is very probably the right one for the majority of uses, where publishers address single consumers or everyone has standardised on a single vocabulary. It’s certainly an assumption that keeps the markup simple.

However, there is a larger data web out there. It’s not just browsers and search engines who might look for and process data embedded within a page. Unlike with HTML, those few, large consumers don’t have to understand a particular vocabulary for other consumers to get valuable information from it. If you operate in a world of multiple consumers with different requirements, you need decentralised extensibility. And support for decentralised extensibility is RDF’s niche as a data model, its unique selling point.

Given that a new language would have to use a different processing model from RDFa 1.0, I would suggest that it simply uses microdata’s as a starting point. Using attributes from RDFa 1.0 would only cause conflicts with RDFa 1.0 processors. Microdata processing is there, already defined, already implemented. It isn’t going to go away. And you know, it’s pretty good.

The ‘new language’ would then not so much a ‘new language’ as an enhancement on something that already exists. It would be a set of additions that augment the data that is generated from normal microdata processing with a few extra features that are useful in a world where there are multiple vocabularies for the same domain, where publishers have to provide data to multiple consumers, where an RDF view of data is useful. Call it microdata+RDF.

So what would we need to add? Well, there are three things, I think, that make microdata hard to use in a decentralised world, and make it hard to generate good RDF from microdata markup:

lack of support for multiple types
scoping of properties by type
lack of datatypes

We would need to find a way to add these for use within the RDF extracted from the microdata markup such that a basic microdata parser would still generate the same JSON, and such that microdata’s DOM API would work as specified in the microdata spec. So we can’t change the types of values that are possible in microdata’s attributes or how they’re interpreted in the DOM API.

Multiple Types

Because of the restrictions I just mentioned in not touching microdata itself, we can’t simply make itemtype take multiple URLs. We could rely on itemprop="http://www.w3.org/1999/02/22-rdf-syntax-ns#type" as a mechanism of providing types for use by RDF processors, but I think that the types of something is such a fundamental property that it makes sense to have a dedicated attribute.

I suggest itemclass. It would only be allowed on elements with an itemscope attribute and would take a space-separated set of values in exactly the same way as the itemprop attribute. The values would be turned into URIs in the same way as for the itemprop attribute, which I’ll describe below.

Microdata+RDF would add a method to the existing microdata DOM API to enable people to access items by class rather than their single type. So:

document . getItemsByClass( classes )
Returns a NodeList of the elements in the Document that create items, that are not 
part of other items, and that have one or more of the types or classes given in the 
argument.

The classes argument is interpreted as a space-separated list of classes.

Note that for simplicity, because they are interpreted in the same way within the RDF model, this returns items whose itemtype is listed in the argument list of classes as well as those whose itemclass is listed.

Within the DOM API, the itemClass IDL attribute on HTML elements would reflect the itemclass attribute.

The itemclass attribute would be ignored for the purpose of creating JSON from microdata, and only be used when creating RDF.

An example would be:

...

The JSON generated from this would look like:

{
  "type": "http://schema.org/Event" ,
  "id": "http://lanyrd.com/2011/oscon/",
  "properties": {}
}

The RDF would look like:


  a  ,
     ,
     ;
  .

Disambiguating Properties

To work with the RDF model, properties have to have URIs. We need to have a way of easily creating the URIs for the short-name properties without people changing their existing microdata markup.

Note: I’ve substantially revised this section following discussion with Philip Jägenstedt. Old text is struck through, new text underlined.

The way that this is done in RDFa 1.1 is through a vocab attribute, which provides a URI prefix that is concatenated to any short-name properties or types. ~~We could use the same approach here, but call the attribute itemvocab to fit in with the general method of naming attributes in microdata.~~ Using this with microdata would be tedious for users however, and it would be easy for the itemtype and itemvocab to get out of sync in weird ways.

itemvocab would only be allowed on elements with an itemscope. The scope of itemvocab would be limited to the item itself, so that it’s not forgotten when it’s needed, particularly in copy-and-paste scenarios. However, to make it easier to use I think it should probably be given a default value if it isn’t present, as follows:

Instead, the vocabulary for the properties could be identified as follows:

set vocab to the itemtype of the item if it is present, and the URL of the document if not
use a substring of vocab:
1. if vocab contains a #, the substring of vocab up to and including the #
2. otherwise, the substring of vocab up to and including its final /

For example, if you have:

...

then ~~itemvocab~~ the item vocabulary would default to http://schema.org/.

There could be an extra restriction that if itemtype is specified, itemvocab must be in the same domain as that type; that could help prevent the weird situation where in the generated RDF the properties would be interpreted as being in a completely different vocabulary from the itemtype.

~~Within the DOM API, the itemVocab IDL attribute on HTML elements would reflect the itemvocab attribute.~~

Note: the following example has been altered in place.

For example, take the following markup:


  
    
      OSCON 2011
    
  
  
    United States / Portland
  
  
    25th–
    29th July 2011
  
  ...

The vocabulary for the

element defaults to http://schema.org/ based on the value of the itemtype. The short-named properties and classes within that item are turned into URIs by pre-pending http://schema.org/ to their name. Similarly, the properties on the nested http://schema.org/Place are pre-pended with http://schema.org/Place/. The resulting RDF would be:

@prefix s: 


  a s:Event ,
    s:SocialEvent ,
    s:BusinessEvent ,
    s:EducationEvent ;
  s:url  ;
  s:name "OSCON 2011" ;
  s:location  ;
  s:startDate "2011-07-25"^^xsd:date ;
  s:endDate "2011-07-29"^^xsd:date ;
  .


  a s:Place ;
  s:url  ;
  s:name "United States / Portland" ;
  .

Note: see below for how the values are created in this example.

The JSON would be just the same as from a standard microdata processor; there’s no mapping to URIs for that output:

{
  "type": "http://schema.org/Event",
  "id": "http://lanyrd.com/2011/oscon/",
  "properties": {
    "url": [
      "http://lanyrd.com/2011/oscon/"
    ],
    "name": [
      "OSCON 2011"
    ],
    "location": [
      {
        "type": "http://schema.org/Place",
        "id": "http://lanyrd.com/places/portland/",
        "properties": {
          "name": [
            "United States / Portland"
          ],
          "url": [
            "http://lanyrd.com/places/portland/"
          ]
        }
      }
    ],
    "startDate": [
      "2011-07-25"
    ],
    "endDate": [
      "2011-07-29"
    ]
  }
}

Adding Datatypes

How to manage datatypes in RDF generated from microdata is something where the best approach is not at all clear to me. A couple of years ago I talked about some frustrations with RDF datatyping, and datatypes in RDF still frustrate me by being hard to use in sensible ways throughout the RDF toolchain. Nevertheless, it’s what we have.

The possibilities I can see for microdata+RDF are:

Use plain literals for everything, including URIs, equivalent to using strings as microdata does. This makes things simple for the publisher and keeps the markup in the page clean, but makes it difficult for consumers who are using RDF toolchains: they will usually have to do some kind of processing of the RDF generated from microdata+RDF to add appropriate datatypes to the values. There are two issues with this approach:
- I have a feeling that microdata+RDF processors will make up their own rules to add datatypes to the data extracted from a page (using rules like those described below and/or sniffing of values and/or using information from known built-in vocabularies), in an effort to add value for their users. But if different processors do that in different ways, we have an interoperability problem.
- In some vocabularies, the datatype of a value is not derivable from the property. The most important/common example of this is skos:notation, which uses values with different datatypes to supply different identifiers from different identification schemes for a given concept.
Assign datatypes based on the element type in the HTML. If the property value has come from a URL attribute, assume that it’s a resource rather than a literal; if the element is a element, work out the datatype based on the syntax of the datetime attribute; otherwise assume it’s a string and give it a language in the case that one is specified. This gives some information but leads to a somewhat strange situation where you can mark up something as a date/time but not as a number.
Supplement the processing described in 2. with some basic datatype sniffing. Basically, if the value looks like a number or a boolean value then assign it a numeric or boolean datatype based on its syntax. This could reuse the rules for recognising different literals from Turtle. This wouldn’t be perfect; in particular, it would guess that strings that consist purely of numbers such as zip codes were numbers. I’m inclined not to go down this path.
Supplement the processing described in 2. with a itemvaltype attribute that takes a token from the list of built-in XML Schema Datatypes or the token ‘literal’. The ‘literal’ token would be used to override the normal processing of URL attributes in the case where those really should be literals rather than resources. In this design, it would be easy to create literals using one of the most usual datatypes, but not possible to use datatypes that are specific to a given vocabulary.
Supplement the processing described in 4. by allowing the itemvaltype to take either a token or a URL. The thing I don’t like about this design is that the token would be interpreted as being within the XML Schema Datatypes vocabulary rather than the vocabulary specified for itemvocab (used for tokens in itemprop and itemclass). This seems like it might turn into a source of confusion, but if we went the other way and had itemvaltype being interpreted based on itemvocab, it would be harder to give a value the more common datatypes such as numbers and boolean values.

My inclination, somewhat reluctantly as it’s the most complex, would be to use the last of these, because it provides for decentralised extensibility of datatypes, and support for decentralised extensibility is the core aim of these extensions. In other words, have a itemvaltype attribute that can hold either a token, which must be one of literal or the local name of an XML Schema datatype, or a URL. On a element, this would default to the appropriate type based on the syntax of the value of the datetime attribute.

To be conformant, the itemvaltype would have to be an allowed value type for the properties given in itemprop and the value of the property must be a legal value for the datatype. (In keeping with the style of the microdata specification, the mechanisms for working out what value types are allowed and what the legal values are for non-XML Schema datatypes would be left undefined — a consuming application would look at the definition of the vocabulary.)

Within the DOM API, the itemValType IDL attribute on HTML elements would reflect the itemvaltype attribute. The value of itemvaltype wouldn’t change the types of the values returned by element.itemValue or in the JSON mapping from microdata; it would purely be used when generating RDF from that data.

For example, if someone started with some markup like:


  Priced from: $35
  1938 tickets left

it might be supplemented with some type information like:


  Priced from: $35
  1938 tickets left

which would generate RDF like:

@prefix s: 

[] a s:AggregateOffer ;
  s:lowPrice "$35"^^s:Price ;
  s:offerCount 1938 ;
  .

(Note: Here I’m assuming that schema.org defines a http://schema.org/Price datatype which includes a currency and a number. They don’t currently.)

The JSON would still be:

{
  "type": "http://schema.org/AggregateOffer",
  "properties": {
    "lowPrice": [
      "$35"
    ],
    "offerCount": [
      "1938"
    ]
  }
}

Non-Additions

When I wrote a couple of years ago about what microdata can’t do, one of the things that I identified was not being able to express XML Literals. Having thought about this more, what’s actually missing isn’t to do with RDF, but is the ability to use the innerHTML of an element to provide a value for a property rather than its textContent.

For example, the description of an event might run over several paragraphs, or even in a single paragraph include other markup such as emphasised text, ruby markup, or links to additional information. People who are working from the DOM API can capture this information when they need it by getting the innerHTML of the element rather than its itemValue, but in the JSON mapping, the value is always the itemValue — the text content of the element.

So this is a general microdata simplifying limitation. I’d argue that we shouldn’t add any special handling to plug this hole at the microdata+RDF level. If it turns out that having values that contain markup is useful then it will be added to microdata, and the microdata+RDF mapping would then be extended to create rdf:XMLLiterals or HTML literals (for which there is no defined datatype in RDF at the moment) for such values.

Similarly, I haven’t said anything in this post about providing machine-readable values to override the text content of an element. There is an open bug about whether and how that capability might be added to HTML/microdata. I happen to think that it’s useful, but that utility isn’t limited to RDF processing. Whichever route is chosen there, I think it’s important to keep the property values used by basic microdata and microdata+RDF aligned.

Summary

To summarise, one direction that we could take in aligning microdata and RDFa would be to define an extension to microdata to add support for decentralised extensibility and the RDF data model. I think that would entail adding attributes such as:

itemclass to make it easy to define multiple types for an item
itemvocab and some default processing to provide nice mappings for short-name properties into URIs
itemvaltype and some default processing to assign datatypes to values

For publishers and consumers, a single language with optional extensions greatly simplifies the use of embedded data. Property names don’t have to be repeated or balancing acts made between different processing models.

RDFa proponents get a syntax that can be used to generate a natural RDF model against which they can build RDF-oriented APIs and map to other formats such as JSON-LD.

For microdata proponents, this approach doesn’t pollute microdata with requirements that they see as superfluous, and doesn’t change the behaviour of core microdata processors. Browsers, search engines and other consumers can continue to use the JSON output and only those who really want to support RDF need to do so.

I’m sure that there are things that I’ve missed in my outline above, issues that I haven’t thought of. But if there is to be any kind of convergence between microdata/RDFa, this layered approach seems to me to be the kind of convergence that is most likely to eventually result in one language for embedding data in HTML rather than two or three.

Note: if you prefer to comment on Google+, please add your comment to my announcement post there

Using Multiple Vocabularies in Microdata

2011-07-28T08:25:21+00:00

I wrote the other day about how legislation.gov.uk needs to share data at three levels to satisfy its goals as a website:

large-scale consumers such as search engines
small-scale consumers that provide us with a useful service
specialist consumers that are interested specifically in our data

and the requirement to use multiple, incrementally more specialised, vocabularies to describe the same things as a result.

What I want to do here is explore how a publisher might handle this kind of situation using microdata. The ground has already been substantially covered by Stéphane Corlosquet; what I do here is work through an example where the consumers are microdata’s primary targets — search engines and browsers — look at why it’s hard to fix this within microdata itself, and discuss how people who create vocabularies to be used with microdata might help publishers who find themselves in this situation by designing those vocabularies to be used together as well as on their own.

Use Case

I’m going to use Lanyrd from Simon Willison and Natalie Downe as an example. Lanyrd is “the social conference directory”: it keeps track of conferences that you’re attending or speaking at, and lets you know about ones that your friends (or at least the people you follow on Twitter) are going to as well, as well as providing a bunch of other useful facilities.

Lanyrd currently uses microformats to mark up events so that nice summaries appear within search engine results. Here’s a (slightly simplified for concision) example from the front page:


  OSCON 2011
  
    United States / Portland
  
  
    25th–
    29th July 2011
  
  ...

This is easy enough to understand: OSCON 2011 has a URL of http://lanyrd.com/2011/oscon/, is located in Portland (US), starts on 25th July and ends on 29th July 2011.

Say that Lanyrd decided to switch to using schema.org microdata. The markup would change to something like the following:


  
    
      OSCON 2011
    
  
  
    
      United States / Portland
    
  
  
    25th–
    29th July 2011
  
  ...

A few notes, because there were some design decisions involved in this mapping:

I’ve used a plain http://schema.org/Event because I wasn’t sure how to classify a conference — is it a SocialEvent or a BusinessEvent or an EducationEvent? Depends on the conference, I guess
I’ve assumed that the URIs for both the conference and its location are also item identifiers
I’ve changed the markup a bit to add elements where necessary to get the desired data out, namely around the names of the conference and the place; I could have used separate or elements instead but that would have meant repetition of data within the page

All well and good.

Now let’s say that browsers start to support the vEvent vocabulary defined within the WHATWG HTML microdata specification and offer some really nice functionality: because there’s a clear mapping to iCalendar, they enable users to drag an event from the browser to a calendar application, and have it create an entry within the calendar.

Say Lanyrd really want to take advantage of this. It means marking up their pages in something like a mix between the two examples we’ve looked at so far — microdata syntax but with the vEvent vocabulary (which is based on the hCalendar microformat vocabulary) rather than the schema.org vocabulary:


  
    
      OSCON 2011
    
  
  
    United States / Portland
  
  
    25th–
    29th July 2011
  
  ...

But now Lanyrd have a dilemma. If they mark up their pages using the schema.org vocabulary, they can’t take advantage of the browser drag-and-drop support; if they mark up their pages using the vEvent vocabulary they won’t get their pages displaying nicely in search engine results. They can get the benefits from one consumer or the other but not both at the same time. What to do?

Publisher Workarounds

What could Lanyrd do to work around this problem?

Different Syntaxes

The first, eminently pragmatic, workaround, would be to use different syntaxes to encode the event information for the two different consumers. Since schema.org is likely to continue to understand microformats for the forseeable future, Lanyrd could stick to their original microformat markup and just add similar microdata for browsers to pull out to create iCalendar data. The page would look like:


  
    
      OSCON 2011
    
  
  
    United States / Portland
  
  
    25th–
    29th July 2011
  
  ...

In other words, they could handle their requirement by not using microdata for one of the vocabularies. I don’t think this is a particularly acceptable solution, given that schema.org specifically wants publishers to use microdata, but it would work.

Repeated Data

A second workaround that Lanyrd could use would be to have some shadow markup for the data targeted at schema.org; the visible event information in the page itself should still be marked up using the vEvent vocabulary because it gives an area of the page that users can drag and drop. The basic version of this would look like:


  
  
    
      
        OSCON 2011
      
    
    
      United States / Portland
    
    
      25th–
      29th July 2011
    
    ...
  

  
  
    
    
    
      
      
    
    
    
    ...

Note: I’ve used empty elements to mark up the dates for the conference in the schema.org shadow data because the microdata spec says “If a property’s value represents a date, time, or global date and time, the property must be specified using the datetime attribute of a time element.” They’re empty, though, so they won’t be displayed on the page.

There are a few issues with this workaround:

it repeats content and thus bloats the page
in the microdata DOM API, there now appear to be two items when really there’s one conference; this might not be a big deal if scripts access items by type rather than just getting all the items
search engines might (wild speculation follows) be more suspicious of data that isn’t visible within the page; there’s no way for schema.org to know that the same data appears visibly elsewhere with equivalent markup

Use `itemref`

A third possibility for Lanyrd would be to something similar to the previous example but use the itemref attribute to point to any shared data. Unfortunately in this case, there’s only one property that’s actually shared (with the same semantics) between the two vocabularies — url — so using this technique doesn’t improve the markup all that much from the previous example:


  
  
    
      
        OSCON 2011
      
    
    
      United States / Portland
    
    
      25th–
      29th July 2011
    
    ...
  

  
  
    
    
      
      
    
    
    
    ...

In other situations, where there is more overlap in the property names used by the two types, there might be more advantage in this approach.

Content Negotiation

A final workaround would be for Lanyrd to serve up an HTML page that uses the schema.org vocabulary to search engines and an HTML page that uses the vEvent vocabulary to browsers, by sniffing the User-Agent header.

This has the advantage of not having to try to cram two conflicting vocabularies into a single page but the disadvantage of having to code for the content negotiation. Essentially, it shifts the complexity and repetition from the HTML page to the code that generates the HTML page, but does address the three disadvantages that I listed for the ‘repeated content’ solution described above.

Publisher Workarounds

Lanyrd could also lobby schema.org and/or WHATWG to make changes to what data they consume.

Lobby for Convergence

Lanyrd could lobby schema.org to understand the vEvent vocabulary and/or WHATWG to specify browser handling of the schema.org vocabulary.

This might work, but the vocabularies do have different goals and requirements, which might make it hard to unify them: vEvent maps neatly and easily to iCalendar, schema.org is oriented around Rich Snippets in search engine results. The modelling of the location property in each shows this different emphasis: it only needs to map to a string in iCalendar so there’s no need to model the location as an item itself, but in search engine results it’s useful to link to the location, display a map and so on, which is only possible if the location is modelled as an item in its own right.

Lobby for Different Processing

Finally, Lanyrd could lobby schema.org and/or WHATWG to trigger their recognition of an event based on something other than the itemtype of an item, and to interpret full URIs for properties in the same way as equivalent short names.

For example, currently the conversion of vEvent to iCalendar defined in the WHATWG HTML specification is triggered by the presence of an item that is an vEvent:

If none of the nodes in nodes are items with the type http://microformats.org/profile/hcalendar#vevent, then there is no vEvent data. Abort the algorithm, returning nothing.

Let’s say that it were instead triggered by items with either:

an itemtype of http://microformats.org/profile/hcalendar#vevent or
a http://microformats.org/profile/hcalendar#type of vevent

and that in the former case, it would read short name properties but in the latter case it would read properties with URIs like http://microformats.org/profile/hcalendar#location.

In that case, Lanyrd could use the schema.org vocabulary for the type given in the itemtype attribute, but markup extra properties for the item using the vEvent property URIs. The markup would be something like:


  
  
    
      OSCON 2011
    
  
  
    
      United States / Portland
    
  
  
  
    25th–
    29th July 2011
  
  ...

Note that here the location has to be repeated because vEvent expects a string while schema.org expects an item.

The same kind of pattern could work the other way around: schema.org could recognised events based on a http://schema.org/type property with the value Event, and understand property URIs that were equivalent to each of the short-name properties that it uses. (Such URIs for schema.org properties already exist.)

Multiple Types for Microdata Items

Earlier this year there was some discussion on the WHATWG mailing list about the requirement for multiple types for items.

The use cases there were not the same multiple-consumers use case that I have outlined above, but around support for inheritance in types. For example, schema.org lets people define their own types in the schema.org domain, such as http://schema.org/Event/Conference. There’s no way of exposing the type hierarchy explicitly (that all Conferences are Events) so scripts that use microdata’s DOM API to search for items of the type http://schema.org/Event won’t find conferences. The discussion was about alleviating this by allowing publishers to put both http://schema.org/Event and http://schema.org/Event/Conference within the itemtype attribute. Alternatively, a conference could be typed as a SocialEvent, BusinessEvent and a EducationEvent, enabling it to take properties from all three.

The conclusion of the discussion was that it just wasn’t possible to use what would seem to be the obvious method of assigning multiple types to an item: having a space-separated list in the itemtype attribute. If we look at the markup that you would get for this example, we can see why there’s a problem:


  
    
      OSCON 2011
    
  
  
    
      United States / Portland
    
  
  
  
    25th–
    29th July 2011
  
  ...

There are two issues with this markup. First, the definitions of the two types (in prose within the two specs) have different expectations about:

what properties will be present: schema.org expects a name property and not a summary property, and vice versa for vEvent; similarly for startDate/dtstart and endDate/dtend
what values the properties will have: schema.org expects location to have an item value whereas vEvent expects a string

The result is that the mixed markup isn’t conformant with either vocabulary, and hence not a conformant HTML document. (Whether microdata consumers do anything about that non-conformance is a different question — they could just ignore properties that they don’t understand, or with value types that they don’t expect.)

Second, if the data is turned into any kind of format that needs full URIs for properties, such as RDF through microdata’s RDF mapping, it’s impossible to tell what type URI to use as the basis of that URI. If the item is assigned just the schema.org type, the name property would map to the URI:

http://www.w3.org/1999/xhtml/microdata#http://schema.org/Event%23:name

If there is just the vEvent type, the summary property would map to the URI:

http://www.w3.org/1999/xhtml/microdata#http://microformats.org/profile/hcalendar%23vevent:summary

When the item has more than one type, there is no way to know which type should be used as the basis of the URI generated for the property, or even if both should be used, as in the properties used by both vocabularies such as url.

(This issue isn’t specific to the RDF mapping defined in the microdata specification; it would arise in any RDF mapping from microdata, or any mapping in which short names for properties needed to be turned into globally unique terms.)

Guidance for Microdata Vocabularies

Having short names for properties makes writing microdata simple. They are much easier on the fingers and the eye than URIs, and, because they are scoped by type rather than vocabulary, they can be given simple names while still being specified tightly in terms of the types on which they can appear and the values that they can take. For example, the location of an Event can be limited to a geographical place while the location of a Click can be specified in terms of a point on a web page, rather than having to use more complex property names like placeLocation and pointLocation.

This could be a particular advantage in large and wide-ranging vocabularies such as schema.org’s where it’s likely that at some point there will be a clash in meaning between properties with the same name for different things. (Though the flip side for schema.org is that it has lots of inherited properties which really do have the same meaning across subtypes.)

The biggest problem with short names arise when you want to provide data to different consumers that use different vocabularies for that data. My guess is that in real life, in many cases this won’t be an issue, and certainly microdata has been designed with that assumption. Realistically, the majority of websites will probably only care about embedding data in web pages to the extent that search engines will read it, and will therefore only use one vocabulary — schema.org’s. Where more than one vocabulary is used in the page, it may well be that they are used in different locations (eg OGP for Facebook in the head of a page, schema.org in the body), or to mark up data about completely different kinds of things.

However, if you’re a publisher who wants to provide data to multiple consumers who understand different vocabularies — search engines and browsers as in the Lanyrd example above, for example — and those consumers define what they will consume solely based on the itemtype of an item, then you’re going to have to either workaround consumer’s behaviour as I described above, or ask those consumers to change how they work.

The most promising direction I can see at the moment would be to ask consumers to define their vocabularies such that they include

a property that is used to identify the in-vocabulary type of items whose itemtype is not in that vocabulary
defined URIs for properties that are equivalent (and processed in the same way as) the short name properties for a given type

The type-defining property could be http://www.w3.org/1999/02/22-rdf-syntax-ns#type, with the value being the URI of the relevant type. However,

is a lot more verbose than:

so I imagine that the designers of microdata vocabularies would prefer to ask publishers to do the latter. On the other hand, if publishers are using multiple vocabularies, they might find it easier to use a consistent type-defining property across vocabularies; it’s hard to tell what the global usability payoffs might be here.

Or microdata could standardise the pattern by adding an attribute (eg itemkind, iteminherit, itemothertype, itemmixin, I dunno) which would list additional types. These could be exposed within the DOM API (which would be a big advantage for in-page scripts) but not used in the interpretation of short-name properties.

Vocabularies that don’t support processing of items based on a type-defining property and property URIs are effectively indicating that they don’t anticipate being mixed with others that have also made the same assumption that they won’t be mixed with others. Currently, for example, schema.org and the vocabularies defined within the WHATWG microdata specification both make this assumption. Working with one vocabulary that makes that assumption for a particular type is fine; working with two in microdata is much harder.

My Experience of Web Standards

2011-07-24T16:24:00+00:00

One of the things that’s been niggling at the back of my mind since the schema.org announcement is how small a role search engine results plays in the wider data sharing efforts that I’m more familiar with in my work on legislation.gov.uk, and more generally how my day job experience differs from (what seem to be) more common experiences of development on the web. In this post, I’m going to talk about that experience, and about the particular problems that I see with the coexistence of microdata and RDFa as a result.

My day job (the one I actually get paid for) is web development. The site I spend most of my time and effort on is legislation.gov.uk. This deals with complex content (UK legislation) that has to be presented in multiple formats (users love PDFs of legislation). Our aim is to make the data as reusable as possible by third parties through good, RESTful, web architecture, and we want to use open standards and open source technologies as part of the UK government’s general strategy.

legislation.gov.uk is not a global website like Amazon or eBay, but it’s not small either: it covers 60,000 changing items of legislation, providing point-in-time views for many of them, and with more added every day. It’s one of the top ten most used UK Government websites, with 2 million visits (about 10-12 million page views) each month and typically about 120 requests/second during the active times of the day. Legislation might sound like a highly specialist interest, but if you search for legislation.gov.uk on Twitter you’ll see it being referenced over and over by people who want to share what the law says.

I do not by any means claim that my experience is representative of the wider web. I know that there are large numbers of sites that deal only in data, not documents, and certainly not documents with the kind of rich semantic structure that legislation has. I offer the following discussion as a data point, partly because I can’t quite believe that legislation.gov.uk is completely unique in its requirements and partly because obviously my perspective on a bunch of issues arises from this experience.

Technology Stacks

Legislation items are complex, semi-structured documents. Their natural fit is XML (well, that’s not quite true — their natural fit would be something that allowed overlapping markup — but XML is the closest that we have). So we store it in XML in a native XML database and we use an XML toolset to query it (XQuery) and transform it (XSLT) into various formats including rendering it as PDF (through XSL-FO).

Our next step for the development of the site involves looking at legislative effects. These form a graph: one item of legislation affects other items of legislation which may in turn affect other items and so on. There are all sorts of other links between items of legislation in terms of commencements, conferred powers and so on. Particularly because we already have well-thought-through URIs for legislation, the natural fit is to use RDF to represent this graph. We already offer a SPARQL endpoint for accessing some aspects of our data, but we expect to expand and develop this over the next few months and to use it as a layer under the website and exposed for reusers, in much the same way as we use the XML database.

As a government site, we have fairly strict limits on what we can do within our web pages: we have to make sure that they’re accessible by everyone who wants to view them. We aren’t able to use technologies that are only available in the latest browsers, but that’s OK because with the kind of content we deal with, we don’t have to do anything fancy anyway. So we use pretty basic HTML and CSS and Javascript, because that’s how you deliver content to end-users on the web (as well as exposing the underlying XML and RDF, to enable others to reuse the data).

In other words, we use three web stacks for delivering legislation.gov.uk:

the XML stack, which is great for single-source publishing of documents that have more semantic structures than those supported by HTML
the RDF stack, which is well-suited for metadata about things that are identified by URIs
the HTML stack, which is absolutely necessary for delivering human-accessible content on the web

What bemuses me, because of this experience, is that sometimes it appears that the narrative around these technologies is framed in terms of an exclusive choice between them. For example, @mattur asked:

It is as if, if you use XML you cannot appreciate the utility of error-handling in HTML; or if you use RDF you cannot understand the need to represent documents in XML; or if you want to utilise HTML fully, you cannot adopt RDF’s view of data on the web. That’s simply not my experience. They each have their role on the web; supporting the use of one does not necessitate rejecting the use of the others.

It’s interesting that some of the standards that are most reviled are those that arise at the intersections, where it appears that one technology is trying to encroach on the space of another:

XHTML at the border of XML and HTML
RDF/XML at the border of RDF and XML
RDFa at the border of all three

At the same time, within legislation.gov.uk, we publish XHTML (because it’s the natural output from an XML toolchain) and create and process RDF/XML (because it gives us access to that data from within the XML toolchain). We use a small bit of RDFa in the XHTML to indicate the rights under which our information is avaialble, and don’t yet, but are thinking about using RDFa to mark up non-document semantics within our XML (to enable the XML markup to focus on the document structures that it’s good at). For all their imperfections, these intersection technologies are useful for managing cross-overs; the problems arise when they overstep their remit and people start to think that all HTML must be XHTML or all XML must be RDF/XML or all RDF must be RDFa.

Sharing Scenarios

The second thing that I wanted to explore is the experience from legislation.gov.uk of what it’s like to be a publisher who actively wants to share their data. We need to operate simultaneously at three levels in our data sharing efforts.

Large-Scale Consumer-Driven Data Sharing

The first target for our data sharing efforts are the search engines. Obviously we’re not selling anything, but we want people to be able to locate legislation easily when they want it, and we want people who have done the search to be able to see some information about the legislation so that they know that they’ve located the right item.

This is large-scale consumer (search engine) driven data sharing, typified by schema.org and Facebook’s Open Graph Protocol (OGP). There are a few very big data consumers (Google, Microsoft, Yahoo!, Facebook etc) who need to consume data from large numbers of data providers. These consumers obviously can’t understand everything, so they determine and document what syntaxes and vocabularies they do understand and expect publishers to follow.

The benefits that publishers get from a particular consumer determines which syntax/vocabulary they use; publishers who are particularly keen to show up prettily within search results will target schema.org whereas those who want to be sharable within Facebook will target OGP. Many publishers will want to target both. There is probably a driver towards eventual convergence:

publishers might push back about inserting two lots of very similar data in their pages
consumers might want to include data from publishers who haven’t specifically targeted them

although there’s likely to be a period where they coexist, much as there was for VHS and Betamax (and V2000, I know, dad) during the early days of video players.

As I discussed previously, these large-scale consumers will be driven by the data that they find in the wild, in all its messy variety. They get relatively little benefit directly from using a generic syntax, as they are really interested in only a few, pretty generic, vocabularies for which they have hardwired processing. Indirectly, adopting a generic syntax has benefits in that publishers might find it easier to find tools that enable them to generate it, tutorials about how to use it, and feel that they aren’t being quite as locked in to something proprietary. However, rejecting data that isn’t marked up properly using that syntax has no benefit for consumers except in so far as it makes them feel that they are being good community members.

This is the pattern we see with schema.org (which accepts microdata but, based on its documentation, won’t reject data that isn’t fully compliant with it) and with OGP (which accepts a subset of RDFa but doesn’t reject data that hasn’t got prefixes properly bound, for example).

Another point to mention is that there is very little trust in this scenario. The communication between consumers and publishers is very limited, and the consumers will want to protect themselves against accidental or malicious errors that are evident in mismatches between explicit metadata and that which is parsed from the visible content of the page.

The parallels to HTML and browser vendors are very strong in this type of data sharing.

Small-Scale Consumer-Driven Data Sharing

A second type of data sharing is again driven by consumers, but this time at a lot smaller and more specialised scale. For legislation.gov.uk, these are services such as GLIN, which is a global legislation registry. Other examples are the recent work that we’ve done to publish UK Government organograms or Chris Taggart’s Open Election Data project. In these cases, there’s a single, relatively small and specialised consumer and a small number of publishers which are closely coordinated together.

As in the large-scale case, the consumer ultimately determines the syntax/vocabulary that it recognises, and communicates that to the publishers. However, small-scale consumers typically have close coordination with the publishers, which has a number of side-effects:

consumers may be more able to both apply pressure to and help publishers to do well in their markup
publishers have the opportunity to feed back directly to the consumer any suggestions that they have about changes to the syntax/vocabulary
publishers are likely to gain an immediate and tangible benefit from their cooperation, such as visualisations of their data that they otherwise wouldn’t have seen

Another noteworthy point about small-scale consumers is that they’re unlikely to have the engineering capability to create a custom parser for a particular syntax, but will instead want to use something off-the-shelf to extract data from pages and into their own backend systems. This, coupled with the closer coordination with publishers, means that they’re much more likely to stick to a specification, assuming that the off-the-shelf tools do.

Publisher-Driven Data Sharing

The final type of data sharing is driven by publishers. At legislation.gov.uk, we’re motivated to make our data available for reuse for transparency/accountability reasons (to help citizens understand the law), efficiency reasons (to help parliament and government departments to publish new legislation better) and economic reasons (to foster innovation in legal publishing). We don’t have any individual consumers in mind when we publish our data, but have found that simply by publishing it well, we foster reuse.

In this case, we as publishers are highly motivated to ensure that the data we publish is easily parsed with something off-the-shelf, since that lowers the barrier for potential consumers. Publishers like us are very likely to have unique, specialised, content and need to use a vocabulary that fits closely to our internal data structures as this lowers implementation cost. Consumers can also trust publishers like us: we simply have no motivation to lie in the data that we provide for reuse.

Mixed Markup

As I’ve outlined above, publishers like legislation.gov.uk need to target several potential consumers at the same time:

large-scale consumers such as search engines
small-scale consumers that provide us with a useful service
specialist consumers that are interested specifically in our data

We cannot use a single vocabulary for all these different purposes. (Well, we could write our own vocabulary and describe mappings to other vocabularies using RDFS, but search engines wouldn’t read it.)

We must therefore use a mix of vocabularies:

generic vocabularies about things that search engines care about
specialised vocabularies for particular small consumers
site-specific vocabularies for sharing our unique data

It’s repetitive, but it’s manageable so long as we have a syntax that enables us to say that an item of legislation is a http://scheme.org/CreativeWork and a http://purl.org/dc/dcmitype/Text and a http://www.legislation.gov.uk/def/legislation/Legislation and allows us to give multiple properties the same value.

The way things are going at the moment, we might well end up having to use multiple syntaxes on the same page, as some consumers understand microdata, others consume RDFa, and still others will parse microformats. This leads to more repetition: adding itemprop for microdata, property for RDFa and specialised class attributes for microformats. But worse (much worse), each of the syntaxes uses a different parsing model to create an entity-property-value structure, so not only do we have to learn substantially different markup patterns but our pages quickly become some kind of hideous polyglot mess trying to balance between them.

Looking Forward

As I said at the start, I’m fairly sure that my experience at legislation.gov.uk isn’t representative of the wider web, but I don’t have a clear idea about just how unrepresentative it is, in terms of technology use or motivations around data sharing. When I read my twitter stream or blogs, there’s a massive sampling bias, both in terms of who I follow and what I read, but also about who talks about what they’re doing. (I’m reminded of Jeff Atwood’s post on the Two Types of Programmers: the vast majority of web developers don’t make a noise about what they do.)

Taking part in web standardisation today often feels like being part on an ongoing cold war between distinct camps rather than a community working towards common aims. The underlying question seems to be “who’s side are you on?” Every decision and activity is cast as a victory or defeat. Time is wasted on attack and defence, or on raking over past slights and stupidities, rather than on progress. Valid criticism from outside a group cannot be listened to for fear of giving ground, cannot be made within a group where it seems like betrayal.

It is the Robbers Cave Experiment played out in web standards. As a psychologist, I find it fascinating. As a developer, and particularly one who doesn’t self-identify with any single group, it is frustrating. As a TAG member, trying to work for the longer-term good of the web, it is worrying, because situations of intergroup conflict lead to groupthink and non-optimal solutions.

As I described above, a non-optimal outcome seems to be the most likely result of the particular microdata vs RDFa conflict for us at legislation.gov.uk. While I know we are not generally representative, I believe that it will be similarly bad for other developers: publishers, consumers and tool implementers.

This is a problem for all who want to foster data sharing on the web using open standards; it is not one that any one group can fix on their own. It’s my hope that a balanced task force of individuals with a variety of experience and backgrounds can provide a focus for us all to work together to solve it. If we can’t, then we have let our prejudice and bias overcome our judgement.

What Do URIs Mean Anyway?

2011-07-05T22:06:33+00:00

If you’ve hung around in linked data circles for any amount of time, you’ll probably have come across the httpRange-14 issue. This was an issue placed before the W3C TAG years and years ago which has become a permathread on semantic web and linked data mailing lists. The basic question (or my interpretation of it) is:

Given that URIs can sometimes be used to name things that aren’t on the web (eg the novel Moby Dick) and sometimes things that are (eg the Wikipedia page about Moby Dick), how can you tell, for a given URI, how it’s being used so that you can work out what a statement (say, about its author) means?

One answer is to use a hash URI whenever you want to refer to something that doesn’t live on the web, with the base URI providing information about that thing. For example:

http://en.wikipedia.org/wiki/Moby-Dick is the URI for the Wikipedia page
http://en.wikipedia.org/wiki/Moby-Dick#thing is a URI for the novel itself

The problem some people (including me) have with this is that hash URIs are primarily used to indicate portions of a web page, and using them for things that aren’t page fragments overloads them. It’s also an inflexible method, because the server isn’t told what the fragment identifier is, and therefore it can’t be used as the basis for a redirection, for example.

The 2005 TAG resolution for people who wanted to use separate non-hash URIs, such as [warning, made-up URIs]

http://en.wikipedia.org/wiki/Moby-Dick is the URI for the Wikipedia page
http://wikipedia.org/thing/Moby-Dick is the URI for the novel itself

was:

if you get a 2XX response when you request a URI, that URI refers to a document (the document that you get back)
if you get a 303 response when you request a URI, that URI could refer to anything, and the resource you get by following the redirection describes that thing (hence if a URI should refer to something that isn’t on the web then requests to it should respond with a 303)
if you get a 4XX response when you request a URI, that URI could represent anything

This leads to the 303 pattern described for example within Cool URIs for the Semantic Web; in the example here, the response to http://wikipedia.org/thing/Moby-Dick would be a 303 redirection to http://en.wikipedia.org/wiki/Moby-Dick.

Six years later, we have a lot of experience about this technique of distinguishing between things that are or are not on the web, and it has a bunch of practical limitations.

it requires access to web server configuration (to add 303 redirections) that make life difficult for people without that level of access
URIs for things that aren’t on the web always require two round-trips to get hold of information, as the first always responds with a 303 redirection, which adds server load and slows things down (this is made worse as 303 responses can’t be cached — an oversight in the HTTP spec that I gather is fixed in HTTPbis)
using the 303 pattern requires a level of knowledge and understanding that is beyond most web developers, particularly if they get no benefit from taking care over their use of URIs (for example, Facebook, schema.org and so on all encourage the use of URIs for non-web things without a word about 303 redirections)
even people who do have this knowledge and understanding sometimes find it hard to work out whether a particular thing that they want to talk about is a thing-on-the-web or not and therefore whether the use of a 303 redirection is required
even people who do try to take care in their use of URIs easily make mistakes because we interact with URIs by copy-and-pasting them from browser address bars, and the only URIs that appear there are URIs for things on the web

Basically, while the web architectural principles behind the use of 303 redirections are (arguably!) sound, the collective experience of the past six years indicates that many publishers will not use it because they don’t know to, because they don’t care to, because they make mistakes or because they simply can’t while meeting the other practical constraints of their project.

A number of other approaches have been suggested, before and after the TAG decision, many of which are documented within the draft TAG finding Providing and discovering definitions of URIs.

The first observation that I want to make is that many of the objections to the 303 pattern are about the practicalities of publishers using it. Therefore, any suggestions to provide an alternative technique that involves

introducing new URI schemes (eg tdb)
introducing new HTTP methods (eg MGET)
introducing new HTTP status codes (eg 209)
using particular HTTP headers (eg Link or Content-Location or other specialist headers)

are not going to be widely used for exactly the same reason. I’m not at all persuaded that it’s worth spending time developing them.

My second observation is that there are three questions that are being conflated and we might make more progress if we separated them:

Must publishers provide separate URIs for things-on-the-web and the non-web-things that they describe?
How can you tell what a reference to a particular URI within a piece of data (eg an RDF statement) means?
How can you get from a URI to information about whatever that URI refers to?

Ambiguity in URIs

Both the hash URI pattern and the 303 pattern make the assumption that you need to have separate URIs for things that are not on the web (eg books) and documents on the web about them (eg pages about books). This is useful because it enables people to make separate statements about the author of a book:

 
  dct:creator  ;
  .

from the authors of the Wikipedia page about that book:


  dct:creator 
     ,
     ,
     ,
    ...
  .

If we only have the URI http://en.wikipedia.org/wiki/Moby-Dick then we run into difficulties interpreting statements made about that URI, and indeed different people might use the URI in different ways, or make some statements that use the URI to mean the novel and some to mean the Wikipedia page.

So there are good reasons to have two separate URIs in these cases.

But the fact is that many publishers currently have a one-URI-fits-all policy. And even if they don’t, people reusing those URIs will often make mistakes and use the wrong one. It would be nice if we could make the world see that this leads to all sorts of logical problems for the Semantic Web, but I just can’t see that happening.

This situation reminds me of one of the central innovations that the web had over previous hypertext systems. There is a great slide) by Dan Connolly which roughly looks like:

Web Semantic Web

Traditional Design hypertext logic/database

+ URIs

- link integrity ?

= viral growth

Are there parts of traditional logic and databases that, if we set them aside, will result in viral growth of the Semantic Web?

	Web	Semantic Web
Traditional Design	hypertext	logic/database
+	URIs
-	link integrity	?
=	viral growth

(By the way, in case my replication of this slide is interpreted incorrectly: I’m certainly not implying that viral growth of the Semantic Web as an end in itself, though I would like to see viral growth in data sharing.)

Dropping the requirement for link integrity, coping with the fact that sometimes links would break, was what made the web work. It would have been simply impossible to build the web as a decentralised system if there had been a requirement for links to always work.

Of course that doesn’t mean that we like it when links get broken. There’s oodles of best practice advice out there on making sure that you retain support for old URIs if you change your web space; we have backup systems in place in the form of web archives so we can work out what was once at the end of a particular URI; and the resolvability of links is something a linter will check about your website.

So it’s not that when he developed the web TimBL rejected entirely the very concept of link integrity, it’s that he recognised that we have to work with the imperfection of the real world. Links break. HTTP copes. Browsers cope. People cope.

The imperfection of the real world as it applies to linked data is that URIs will be used in ambiguous ways. We might not like it; we might write best practice documents that encourage people to have separate URIs for web-thing and non-web-thing, develop tools that help people detect when they’ve used the wrong URI, and so on. But it will still happen, and in my opinion we need to work out how to cope.

In fact, ambiguity in URIs goes much further than just a confusion between the Wikipedia page about Moby Dick and the novel Moby Dick itself. URIs are names, and names are used by different people to mean different things. The same URI might end up meaning:

the Wikipedia page about Moby Dick
the novel Moby Dick
the whale Moby Dick
the story Moby Dick (originally a novel but later adapted as a film)
and so on

Even if the publisher provides a clear and unambiguous definition about what the URI http://en.wikipedia.org/wiki/Moby-Dick means, other people will use it to mean something different because it’s close enough for what they want to say.

So I think the answer to the first question I posed — “Must publishers provide separate URIs for things-on-the-web and the non-web-things that they describe?” — has to be “No, though it is good practice to.” We can fight against ambiguity, but we have to accept that we cannot win.

Disambiguating Statements

As discussed above, in a perfect world, we would have separate URIs for things-on-the-web and non-web-things and any data that we published about Moby Dick would use the URI for the Wikipedia page to talk about things like the licence for that information, or how the information was created (its provenance), and the URI for the novel to talk about things like the licence for the novel and what characters appeared in it.

But the world is not perfect, and we are going to end up with situations where the same URI is used to refer to a whole range of different things. How do we cope?

Well, first let me say that I don’t see people merging data together willy-nilly and hoping to get something useful out of it. URIs give us connection points and RDF gives us a flexible data model, which means that merging data can be easier than the kinds of custom merging that you have to do with CSV and JSON, but I don’t think it can ever remove entirely the requirement for curation. We want to ensure that the need for intervention in merging two datasets is kept to a minimum, but we can’t expect it to be entirely removed.

So with that in mind, there are at least three techniques that can be used to get useful data out of a world in which the same URI is used to mean different things.

One-Step-Removed Properties

The first technique is to interpret particular properties as describing a one-or-more-step-removed relationship between a resource and a value. For example, the bib:author and dct:creator properties would be defined such that the RDF statements


  bib:author  ;
  dct:creator  ;
  .

would be interpreted as saying

The topic of the page http://en.wikipedia.org/wiki/Moby-Dick was authored by the topic of the page http://en.wikipedia.org/wiki/Herman_Melville. The creator of the page http://en.wikipedia.org/wiki/Moby-Dick is the topic of the page http://en.wikipedia.org/wiki/User:Aristophanes68.

The biggest problem with the global application of this approach is that there are a lot of existing properties defined in vocabularies such as FOAF or Dublin Core that aren’t defined as one-step-removed properties. One publisher might use dct:creator to link to “a page describing the creator of this page” and another might use it to point directly to a (non-web-thing) URI for the creator of the page. So practically, this approach requires the interpretation of properties to be done on a dataset-by-dataset basis. Which leads onto the next approach.

Named Graphs

A second technique would be to make the assumption that within a single dataset, a single URI has a single meaning, but that the meaning may differ between datasets. I suspect that this is true even when publishers attempt to take care about which URI they use, because, like names, the meaning of a URI is slightly different depending on its use.

Re-users of data need to work out whether the way URIs are used in one dataset is close enough to the way they are used in another dataset, to ascertain whether it’s appropriate to simply merge the datasets or whether something slightly more complicated needs to be done to bring the datasets together.

The problem with this approach is that it raises the barrier to joining together graphs: you can’t just bung the data into a triplestore and perform queries on it, you have to work out some kind of mapping between the datasets up front.

Duck Typing

The final technique that I’ll talk about here is to say that different applications need to access different properties, and can ignore any properties that don’t fit with how they want to use the data. It is relatively rarely useful to have generic RDF viewers; people (generally) build applications to answer questions and perform tasks, not to just browse around data.

For example, if a single dataset were to contain:


  a bib:Book ;
  bib:author  ;
  a foaf:Document ;
  dct:creator  ;
  .

then an application that was interested in gathering data about books would only care about the fact that http://en.wikipedia.org/wiki/Moby-Dick was a book with an author of http://en.wikipedia.org/wiki/Herman_Melville and wouldn’t care about the FOAF or Dublin Core classes or properties associated with the URI. An application that was interested in gathering information about the authorship of documents on the web, on the other hand, might look for the foaf:Document class and Dublin Core properties and ignore everything else.

To me, this approach seems the most promising way of retaining the core benefits of RDF. It seems more robust in the face of user error than the idea of defining one-step-removed properties, and retains the ease of mashing together data from different sources in a way that you wouldn’t get if you had to think about the URI usage within each of the datasets that you want to bring together.

Locating Data From URIs

And so we get to the final question: how should people be able to get from a URI to information about whatever the URI refers to?

I’ve discussed above how I think distinguishing between things-on-the-web and non-web-things has to be seen as a best practice. I think we should continue to recommend the 303 or hash URI methods as the best practice for accessing data from a URI. My reason for this is that introducing yet another method will just makes it harder for publishers to know which method to use when, plus I don’t want to see people who have adopted these techniques in good faith being told that they were doing the wrong thing all along. What I’d like to aim to do is to find a way of fitting these methods into a larger approach.

I also recognise the argument that articulating the relationships between on-the-web and not-on-the-web resources purely through HTTP responses isn’t ideal. It’s useful to have explicit links between resources within the data itself. Within the linked data work that I’ve done for data.gov.uk I’ve tried to adopt a pattern of explicitly using foaf:primaryTopic, foaf:primaryTopicOf and foaf:page to link together the different resources. Other people have suggested the wdrs:describedby property for pointers from a resource to information about that resource; rdfs:isDefinedBy performs a similar function for classes and properties within RDFS.

It would be nice to have one defined property or set of properties to describe these relationships, but we have to recognise that not everyone will use them, so the approach we take has to work when these links aren’t present. The majority of people and sites are going to start off by publishing data about something at a single URI, and simply return data about that thing (a 200 response) when the URI is requested. If they then progress to wanting to have separate URIs for that thing and the page about the thing, or indeed to disambiguate the URI that they’ve used in some other way, we need to make it easy for them to do so.

I think we need two properties: eg:describedBy and eg:couldBe. eg:describedBy describes the link between a resource (of any type) and a document that describes it; eg:couldBe is a disambiguation link that points from a URI to other possible, more precise, URIs.

Then I think we need some rules along the lines of (I don’t pretend these are entirely worked out):

if you get a 303 response redirecting to U' when you fetch a URI U then behave as if the response from U' included the triple U eg:describedBy U'
if the URI U is a hash URI whose base URI is U' then behave as if the response from U' included the triple U eg:describedBy U'
if you get a 2XX response in response to a URI U then:
- if there are multiple triples that match the pattern U eg:describedBy ?page then assume that the document you have is U' where U' eg:couldBe any of the ?pages
- otherwise, if there is a single triple that matches the pattern U eg:describedBy ?page then assume that the document that you have is ?page and it is about U (along with other things, possibly); statements about ?page might include information about the licence or provenance of the returned document
- if there are any triples that match the pattern ?thing eg:describedBy U then assume that the document you have is U and it is about (possibly multiple) ?things
- otherwise, behave as if there is a triple U eg:describedBy U; in this case, U is being used in an ambiguous way

We could go further and say:

if there are two triples that match the pattern U eg:couldBe ?page . ?thing eg:describedBy ?page then assume that the document you have is ?page and it is about ?thing
if there are two triples that match the pattern U eg:couldBe ?thing . ?thing eg:describedBy ?page then assume that the document you have is ?page and it is about ?thing

This way, if someone starts off using U in an ambiguous way, or to mean only the page or only the thing, they can later add eg:describedBy and eg:couldBe statements to disambiguate and add information about the page or thing the page describes.

It’s worth bearing in mind that we shouldn’t just be concerned about locating information about things that aren’t on the web, but about things that are on the web but that cannot have metadata embedded within them. For example, how do we discover the licence associated with a particular image? Although there are methods of embedding metadata within image and other binary formats, such as XMP, it’s still useful to be able to locate metadata about images based on their URI.

With a scheme such as that described above, publishers that used content negotiation to return some data about the image in another format could use eg:describedBy to indicate that the returned document is about the image (or set of images in different formats).

Summary

The summary of my thinking is:

we should learn to cope with ambiguity in URIs
we should not constrain how applications manage that ambiguity, though duck typing seems the most promising approach to me
we should define some specific properties that can be used to disambiguate URIs, describe their defaults with 303s and hash URIs and provide an easy upgrade path as publishers choose to add more specificity

The key will be how we find practical ways to cope with the real, imperfect, fuzzy web of data while providing an evolutionary path to greater clarity and specificity that publishers can take when they see the benefit of doing so.

TAG F2F, June 2011

2011-06-17T10:44:12+00:00

As you may know, I accepted an appointment to the W3C’s Technical Architecture Group earlier this year. Last week was the first face-to-face meeting that I attended, hosted in the Stata Center at MIT. As you can tell from the agenda (which was in fact revised as we went along) it was a packed three days.

What I intend to do here is to briefly report on the major areas that we discussed and give a tiny bit of my own personal take on them. In no way should any of what I write here be judged as revealing the official opinion of the TAG, it’s just me saying what I think, and I’m not going to go into anything in depth because they’re all incredibly gnarly and contentious topics and I’d not only be here all year but also end up in a tar pit.

Role of the TAG

Usefully for me as a newcomer, our first session was about the ongoing role of the TAG. The TAG occupies a unique position within the W3C. According to its charter it was set up

To improve the effectiveness of Working Groups, to reduce misunderstandings and overlapping work, and to improve the consistency of Web technologies developed inside and outside W3C

The TAG ultimately has three routes to do this:

by providing specific advice on issues that are brought to its attention
by writing documents on basic web architecture principles that go through community review, particularly through the general review of the W3C standards track and become Recommendations
by advising the W3C Director (Tim Berners-Lee) about what he should do on the extremely rare occasions when there are issues that he is supposed to adjudicate on

In none of these cases is there anything that binds the people receiving the advice of the TAG, or reading Findings or Recommendations made by the TAG, to accept them or do anything about them. The power and authority of the TAG depends solely on the quality and utility of its arguments, which is how it should be in my opinion.

Client-Side Application State

The first technical session was about client-side application state and was a review of the Identifying Application State draft that T.V. Raman began before he left the TAG and that Ashok Malhotra has been working on since. This should in the next few months or so be published as a TAG Finding (though it is currently on the Recommendation track).

This work is essentially about documenting the different ways in which you can identify application state within a URI, why that’s a useful thing to do, and some of the pitfalls of using hash URIs to do so. Most of the discussion was about details to do with wording within the document. One thing I thought particularly interesting was the point that URI-based application state is relevant in all ‘active content’, not just in HTML; for example, scripting in SVG or in PDFs bring the same considerations.

Buffer Bloat

Over lunch on Monday we listened to and discussed a presentation by Jim Gettys on buffer bloat. Basically (and all the errors here are introduced by me), TCP/IP is designed to route around network blockages, but it can only do so if it detects them quickly. When you have big buffers in place, as in the case of all modern operating systems and hardware, blockages aren’t detected quickly; they’re only detected when the buffers fill up. Then buffers empty and the data has to be sent again. The net result is that connections get really slow, not just for upload or download but for both, not just for you but for everyone using the network.

Jim talked about how this is exacerbated by the large amount of web traffic and the design of HTTP, particularly the lack of use of HTTP pipelining (whereby several HTTP requests and responses are sent over one long-term connection), because it leads to lots of small messages which can’t be handled effectively. There’s lots more about this on his blog.

Jim also talked about the failure of certificate authorities and how we should be looking at distributed protocols using digitally signed data, pointing us in particular to CCNx.

Fragment ID Semantics

First thing Tuesday was a session that I led on fragids, in particular the problems that are arising out of the mime type registration of +xml types (3023bis) clashing with those that are used for, say, images, and what happens when these come together in something like SVG.

The same issues arise whenever you have documents with types that ‘inherit’ fragid semantics from two directions. For example, XHTML documents are XML documents, so constraints on +xml mean you shouldn’t use interpreted fragids (eg hash-bangs) on them, but they are also ‘active content’ which makes interpreted fragids useful. Similarly, in linked data you shouldn’t really use a hash URI to mean a Person with a primary resource that provides as a response an XML document with embedded RDFa, because according to XML fragid semantics, such a URI should point to an XML element.

Basically the use of fragids has grown markedly outside their original scope and these situations aren’t really covered in the specs. I am now tasked to create a document that describes the issues and suggests ways forward. So that will be fun.

Telcon with IAB

The second session on Tuesday was a telcon with the IAB which has a similar role within the IETF as the TAG does within the W3C. This was a bit of a ‘getting to know you’ session, covering the work of the two groups on:

versioning and extensibility
security
privacy, including Do Not Track

and talking about opportunities to meet and work together on various topics like these.

URI Definition Discovery and Metadata Architecture

The afternoon session on Tuesday was spent on Jonathan Rees’s work on the Architecture of the World Wide Semantic Web, which covers, amongst other things, what people in semantic web circles call httpRange-14. At core, this is about the kinds of URIs we can use to refer to real-world things, what the response to HTTP requests on those URIs should be, and how we find out information about these resources.

Jonathan has put together a document called Providing and discovering definitions of URIs which covers the various ways that have been suggested over time, including the 303 method that was recommended by the TAG in 2005 and methods that have been suggested by various people since that time.

It’s clear that the 303 method has lots of practical shortcomings for people deploying linked data, and isn’t the way in which URIs are commonly used by Facebook and schema.org, who don’t currently care about using separate URIs for documents and the things those documents are about. We discussed these alongside concerns that we continue to support people who want to do things like describe the license or provenance of a document (as well as the facts that it contains) and don’t introduce anything that is incompatible with the ways in which people who have been following recommended practice are publishing their linked data. The general mood was that we need to support some kind of ‘punning’, whereby a single URI could be used to refer to both a document and a real-world thing, with different properties being assigned to different ‘views’ of that resource.

Jonathan is going to continue to work on the draft, incorporating some other possible approaches. It’s a very contentious topic within the linked data community. My opinion is while we need to provide some ‘good practice’ guides for linked data publishers, we can’t just stick to a theoretical ideal that experience has shown not to be practical. What I’d hope is that the TAG can help to pull together the various arguments for and against different options, and document whatever approach the wider community supports.

Can publication of hyperlinks cause copyright infringment?

The first session on Wednesday was another session that I led, discussing the Publishing and Linking on the Web draft that Dan Appelquist and I have been working on.

The aim of this document is to explain the tensions between terms that are commonly used in legal documents such as “possession”, “adaptation” and “distribution” and the way that publication works on the web, in which multiple servers may have copies of the same document (because they cache copies to make the ‘net go faster), automated agents may make changes to those documents (such as compressing or resizing documents, or merging Javascript) and people may refer others to those documents through linking.

We’re particularly keen to argue that linking to something is not the same thing as distributing it. The web’s power arises through its links, so it’s important that people are able to link to something without being worried about what happens when/if the domain they link to is taken over by something illegal.

Dan and I are going to continue to work on this document in response to various suggestions around organisation and terminology, with a view to getting some ‘friendly legal experts’ to look it over and then seeking wider review. The intention is for it to eventually become a Recommendation as this will give greater weight to it as a document for a legal audience.

API Minimisation and Client-Side Storage

There were then a couple of short sessions.

Dan talked about API Minimisation, which is the design principle that to increase privacy we should design APIs that enable people requesting information to say exactly what information they need, and return only that rather that everything known about a think. Dan’s put together an draft and should be calling for review for it soon.

Ashok then led discussion on client-side storage and we brainstormed around some of the architectural/design issues about which we might want to write if we were to put together a document. This work is at a very early stage.

TAG Priorities

After lunch, we had a session on TAG priorities where we discussed which of the various pieces of work that we’re doing should receive the most attention and had a quick review of who is doing what within the TAG.

Our basic problem is that a lot of this stuff feels quite urgent, and we want to be responsive, but with only 5-6 of us “actively involved” (which means 1 day/week) in drafting documents, and other TAG duties taking up our time, it feels like we have taken on too much work. Our focus for the next little while is going to be on responding to issues where our lack of response might either hold people up or cause longer term problems (for example the publication of contradictory mime type definitions), which means things like the document on publishing and linking on the web will need to bubble in the background rather than being the focus of activity.

HTML5 Last Call

Our final session, for which we were joined by Philippe Le Hégaret, was on the HTML5 Last Call documents. The TAG has raised various issues over the course of HTML5 development and want to follow up on how those issues have been addressed in the documents. Our role means that we’re responsible for making sure there’s consistency with other specifications, and that there isn’t anything that seems like it’s going to cause problems in the long term.

The part that we spent most discussion time on was the relationship between Microdata and RDFa. We talked about the precedents for having two specifications that do very similar things but with different approaches, such as CSS and XSL, and how this isn’t necessarily a bad thing so long as they don’t contradict each other and people can move between them easily (because they have the same conceptual foundations).

I’m going to save my opinion on this topic for another post. Suffice it to say that microdata and RDFa as currently specified don’t work well with each other but it’s not at all clear what the best path forward is. The TAG decided to recommend that the W3C set up a Task Force to look at what the best way forward might be.

Final Words

If you want links to the minutes of the TAG F2F, they’re available within the agenda or on separate pages for:

If you have anything to say on any of these topics, please send email to the TAG mailing list. Or you could comment here or email me directly if you like. Which leads me on to talking about what I’d like to do in the TAG.

One of the guidance notes for new members to the TAG says:

TAG members are elected or appointed not to represent their individual member organizations, but the Web community as a whole. We try to take that responsibility very seriously.

I do take that responsibility seriously. Web architecture has to be a combination of practice and theory, balancing approaches that work right now with a desire to not break anything long term. I do practical work developing web applications with HTML, CSS, Javascript, XML, RDF, XSLT, XQuery and so on and so on every day, but I know I don’t see all the difficult corners of the open web standard space: no one person can.

I can listen though, so that’s what I will try to do: listen, digest, reflect and act.

But I have limited resources. Unlike most of the members of the TAG, I am not employed by a large organisation that pays me for time I take on the work that I do for the TAG. The W3C kindly paid for my flights to and from F2Fs, but not hotels or expenses. I wouldn’t have taken this on if I wasn’t prepared to shoulder the financial burden, but if there is anyone out there who might sponsor my participation, I’d love to hear from you.

Schema.org and the Responsibility of Monopoly

2011-06-12T19:23:27+00:00

Update: This post has been translated to Italian on the Linked Open Data Italia blog.

In this post about schema.org I’m going to speculate about the economic drivers that affect how search engines use structured metadata on the web. I discuss how the technical features and choices within schema.org may cause wider long-term harm, and the role of open standards as a method for responsible companies to avoid the pitfalls of monopoly.

Before I launch into this, two things. The first is the standard disclaimer that I am speaking purely for myself. The second is that I recommend that you read Rufus Pollock’s paper Is Google the Next Microsoft? Competition, Welfare and Regulation in Internet Search. In it, he demonstrates how the search engine market will naturally tend to monopoly, and that because of the economic drivers in the search engine market, those monopolies will generally under-perform in terms of social good. In other words, if you are a search engine monopolist, you have to take positive steps to not be evil because all the market drivers force you in that direction.

Clearly schema.org is a significant move by our current search engine monopolist, Google, on several fronts and while I don’t pretend to have any particular insight, it’s fun to speculate about how schema.org fits with their wider goals, the extent to which they are avoiding monopolist traps, and what it might mean for the web in general.

Search engines serve their customers: advertisers. So why the interest in structured metadata? Structured metadata benefits search engines in at least three ways:

presenting richer information increases the utility of the search engine for users, thus attracting more of them (more users => more attention overall => more money from advertisers)
presenting richer information keeps users on the site for longer because search engines can present relevant information directly rather than users navigating away from the search engine’s site (more time on the site => more attention from individual users => more money from advertisers)
analysing social metadata extracted from web pages, such as social graphs and individual interests can aid the targeting of adverts to particular users (more targeted adverts => more effective adverts => more money from advertisers)

Clearly there’s a lot of potential for search engines in structured metadata. Their difficulty is in getting people to use it such that they don’t lie, don’t find it too much hassle, and don’t make too many mistakes, because that way lies metacrap.

So the drivers for search engines are towards making it as easy as it could possibly be for publishers to embed metadata in their pages. It is also in their interest to ensure that the information that they extract is based as much as possible on the visible content of the page as this reduces the opportunity for people to lie (or make honest mistakes) by providing one value in the metadata and another in the content of the page. And it is in their interest to correct for errors when publishers make them.

The trap is that blindly pursuing these interests can also lead to anti-competitive behaviour.

Raising Barriers to Entry

The Conformance section of the Data Model page says (my emphasis):

While we would like all the markup we get to follow the schema, in practice, we expect a lot of data that does not. We expect schema.org properties to be used with new types. We also expect that often, where we expect a property value of type Person, Place, Organization or some other subClassOf Thing, we will get a text string. In the spirit of “some data is better than none”, we will accept this markup and do the best we can.

Schema.org contains multiple examples of properties whose values should be interpreted as being of a particular type, such as dates, times, numbers, durations, and specialised micro-syntaxes such as for an EventVenue’s openingHours property or an Article’s interactionCount property which (from the examples, if not the text) expects a syntax like UserTweets:65. These seem clear enough.

However, looking in more detail at the examples, it seems that even putting aside the option of providing a string when the schema expects an item, there are a variety of ways of expressing values for properties in schema.org. There are examples where numbers contain commas or are preceded by currency signs. Distances are a number followed by a “unit of measurement” without any indication of what acceptable units of measurements are. Fat content seems to follow some kind of syntax that includes a number and a measure but various other text as well. Even when values have to adhere to a particular microsyntax, there are examples that are non-standard (such as initial ‘P’s missing from durations).

In other words, there is no documentation about the way in which the values of the schema.org properties will be interpreted by search engines and there is a clear intention on the part of the search engines behind schema.org to be generous in what they accept, so as to ensure that publishers can be lazy while search engines maximise the amount of data that they can understand on the web. Lacking a specification that describes how values are interpreted, the only way for publishers, validators and tool developers to work it out will be to try it out, see what happens, and attempt to find patterns that are generally interpreted in the same way by at least the major search engines, or more likely (because why bother with anyone else), try to work out what Google is going to do with it.

We have been here before, with HTML, pre-WHATWG. Then, IE, which dominated the browser market, had the clear intention to be generous in what it accepted, and there was no specification that described the various error handling quirks that had to be reproduced in bug-for-bug compatible user agents. WHATWG have had to work extremely hard to reverse engineer a specification that provides some kind of predictability and consistency for publishers as well as making it possible for new entrants to the browser market (such as Google’s Chrome), validators, and other tools to reproduce the behaviour of existing browsers. This work has paid off: over the past few years, browser market share has diversified somewhat, largely due to the rise of mobile browsers and Chrome taking market share from IE.

With structured metadata, Google is in an extremely dominant position. Concretely, it will be very hard for Google to reveal the methods by which they extract meaningful metadata from the huge variety of textual content on the web: they may have patents that cover some aspects, and in other cases (particularly when that interpretation depends on the analysis of their vast caches of web pages, as in the case of natural language translation) the behaviour simply might not be replicable by any third party.

None of this, by the way, would be helped by using a different syntax to express the data within the page. The only way it could be addressed is by much more clarity, detail, and conformance criteria within the schema.org vocabulary specification.

Without that specificity, we get into a world where Bing, Facebook and any other search engines will spend a lot of time and effort trying to reverse engineer Google behaviour to extract the same data as they do. They might even sometimes manage to introduce useful quirks of interpretation of their own, but that’s unlikely given that their constrained engineering effort will naturally be focused on matching Google. This also forms a massive barrier to entry (as if those weren’t already significant) to potential new search engines. Overall, the lack of specificity suppresses innovation in the market.

And of course publishers, writers and tool creators are left struggling to keep up.

Syntax Fixing

While both Google and Yahoo! have previously used information described using microformats and RDFa to provide similar functionality, in schema.org they deprecate that support, both by using microdata throughout the examples and by explicitly saying:

If you have already done markup and it is already being used by Google, Microsoft, or Yahoo!, the markup format will continue to be supported. Changing to the new markup format could be helpful over time because you will be switching to a standard that is accepted across all three companies, but you don’t have to do it.

Whichever technology they choose, the act of search engine monopolies making that choice and the consequent widespread adoption via SEO creates a large barrier to changes to the technology. Even if the specification for the technology changes, those changes will be likely to be ignored in practice as Google (and hence other search engines) seek to retain backwards compatibility with the examples and guidance published on schema.org as they stand now.

It is particularly damaging to have the choice be microdata because microdata is a relatively new technology that has only just reached W3C Last Call Working Draft. In my experience, Last Call is usually the first time that a wider community outside interested Working Groups start to look at a technology seriously. To create better technologies and better specifications, Working Groups must be able to change in response to this review.

The ultimate result is again standardisation-by-implementation, which has long term adverse consequences in restricting competition (not between technologies, but between organisations using those technologies) and leads us to a situation where we could end up using something that is less than optimal for any kind of wider purpose outside the interests of the monopolist.

Standards Bodies

The development of schema.org might seem like a very minor thing, only of interest to people interested in SEO and structured metadata, but it is part of a bigger picture of the kinds of ripple-through effects the dominant players on the internet can have. It is almost impossible for monopolies not to do harm, not because anyone within them sets out to, but simply because they are so large that their behaviour is that much more important than anyone else’s.

The kinds of effects described above — ones that result in an overall sub-optimal outcome for society as a whole — are why society has competition laws that constrain monopolies and cartels. Sooner or later, just as it did with Microsoft, society applies the corrective force of regulation. There are already rumblings of this storm approaching.

It is also why we have neutral standards bodies, such as the W3C or the IETF, which provide a royalty-free patent policy as well as a defined process for developing specifications. These might seem tedious to comply with, and it might seem beneficial to companies to form a small cabal in order to get things done more quickly without having to seek wide consensus, but the bigger picture is that open standards developed within standards bodies protect companies from antitrust actions. Companies can point to royalty-free standards developed through a defined and fair process as proof of good behaviour that demonstrates their understanding of a wider responsibility to society as a whole.

As Winston Churchill might have said:

Many ways of developing standards have been tried and will be tried in this world of sin and woe. No one pretends that standards bodies are perfect or all-wise, and it has been said that developing standards within standards bodies is the worst possible way to do it except all those other ways that have been tried from time to time.

Objections to schema.org may seem to be sour grapes because they didn’t use a particular existing syntax or vocabulary, but look deeper and the issues schema.org raises are all about the responsibilities of monopolies and the role of open standards. The parallels with HTML, IE and Microsoft are striking; it will be interesting to see if this turns out the same way.

Lessons for Microdata from schema.org

2011-06-10T20:27:38+00:00

There is (obviously, from the way my tweet stream, feed reader and email have filled up) lots to say at many levels about schema.org, a new collaboration between Google, Microsoft and Yahoo! that describes the next phase in search engines’ extraction of semantics from web pages. In this post I’m going to focus on what we can learn from schema.org about the design of microdata and how it might be improved.

Digging into the details of schema.org there are several examples of places where its recommended method of marking up metadata directly contradicts the HTML5 specs. Given the number of internal contradictions within schema.org, I’m assuming that these are mistakes that will be corrected as the material is reviewed and matures rather than deliberate forking of HTML5.

Note: What I say about HTML5 here is equally true — at least at time of writing — of the WHATWG version of HTML, which of course already diverges from HTML5.

One of the inputs to the design of microdata was to look at the mistakes that people make and try to design something to address the cause of those errors, so it’s interesting to apply that method to the errors made by schema.org. This doesn’t mean changing specs so that erroneous markup is conformant, but it does mean providing facilities that enable people to more easily do things in a conformant way, removing the temptation of non-conformance and lowering the likelihood of future mistakes.

Note: I am certain that there would also have been errors had schema.org used RDFa or microformats, indeed I gather that they are common in the documentation of Google’s Rich Snippets.

Use of element

The first example is one spotted by Tantek: the value of the openingHours property of EventVenue is described as:

The opening hours for a business. Opening hours can be specified as a weekly time range, starting with days, then times per day. Multiple days can be listed with commas ‘,’ separating each day. Day or time ranges are specified using a hyphen ‘-‘.

Days are specified using the following two-letter combinations: Mo, Tu, We, Th, Fr, Sa, Su.

Times are specified using 24:00 time. For example, 3pm is specified as 15:00.

Here is an example: Tuesdays and Thursdays 4-8pm

A similar error involving the element can be found on the Getting Started page which has an example in which the datetime attribute contains an ISO 8601 duration:

Durations can be specified in an analogous way using the time tag with the datetime attribute. Durations are prefixed with the letter P (stands for “period”). Here’s how you can specify a recipe cook time of 1 ½ hours:
1 1/2 hrs
H is used to designate the number of hours, and M is used to designate the number of minutes.

According to the HTML5 specification, the datetime attribute holds a valid date or time string which is specified as using the normal ISO 8601 syntaxes for dates and times. Conforming HTML5 document must not hold the syntax used in the above example. From what I can tell (please correct me if I have this wrong), conforming data-mining tools can process this syntax because they use the value of the datetime content attribute, rather than the value of the dateTime IDL attribute to provide the property’s value, but from an authoring perspective, no one should be encouraging people to create non-conformant HTML5.

In fact, given that values from microdata are never typed, it’s not clear why these examples use the element at all. The conformant way to provide the data would be to use a separate element to hold the data:


Tuesdays and Thursdays 4-8pm

But I can understand the itch to use the element here; the element above is before, rather than around, the textual content of the page which it reflects, whereas with the element it is more obviously an explicit machine-readable version of that content. In microformats, the pattern is to use a title attribute:


  Tuesdays and Thursdays 4-8pm

and in RDFa a content attribute:


  Tuesdays and Thursdays 4-8pm

So maybe this error is an indication that microdata needs an itemvalue attribute for those cases where the human-readable content can be expressed in a more formal machine-readable microsyntax, with special handling with the element for the case when the value is a date/time and href/src when the value is a URI.

String parsing of URIs

The second example of non-conformance with HTML5 in schema.org is the method by which they support extensibility. Schema.org provides a set of types for the things described by web pages. Naturally, this set does not cover everything, but search engines still want to be able to use the metadata within the page about a Person even when that Person is described as a Minister (say).

So schema.org says that to extend their type hierarchy, you simply append the name of the new type after a / at the end of the URI for the parent type. In this example, a Minister should be given the type http://schema.org/Person/Minister.

My guess (as I can’t see any other way in which they’d do it) is that the search engines intend to use string processing on the type URI in order to work out whether it’s a subtype of a known type (ie, does the URI start with the string http://schema.org/Person? If it does, it’s a Person of some kind).

The microdata specification states that:

The item type must be a type defined in an applicable specification.

and that:

Item types are opaque identifiers, and user agents must not dereference unknown item types, or otherwise deconstruct them, in order to determine how to process items that use them.

Treating URIs as opaque identifiers, and forbidding inferring semantic meaning through string processing, is a fairly fundamental web architectural principle as well as fitting with microdata’s constraint that types are specified somewhere.

Perhaps the gloss is that schema.org is an applicable specification that states that any URI that looks like http://schema.org/{known-type}/{extension-type} is a type that is defined in schema.org. But I think what’s actually happening is that schema.org wants to grow their vocabulary organically, responding to the data that the search engines find on the web. They recognise that people will want to use their own vocabularies for their own purposes (for example to provide data for scripts, reusers, or for browsers and other agents that aren’t search engines) but want to continue to be able to understand the semantics of that data.

Schema.org is constrained in the mechanisms that it could use to recognise these new types:

they can’t resolve the type URI and expect metadata at the end of it to indicate the parent schema.org type, because the HTML5 microdata specification forbids resolving unrecognised item type URIs; even if they were allowed to do so, working out inheritance by declarative mechanisms that involve resolving URIs and interpreting the results is computationally expensive compared to string munging; for search engines that expect to do this with billions of web pages, this may be a significant processing burden
they can’t tell people to use the schema.org type in addition to their own type because the HTML5 microdata specification doesn’t allow an item to have more than one type; if it did, they could encourage people to use the schema.org type as well as their more specific one, so rather than itemtype="http://schema.org/Person/Minister", enable publishers to do itemtype="http://schema.org/Person http://reference.data.gov.uk/def/central-government/Minister"

I suspect that having multiple types is currently disallowed in microdata because the semantics of the (non-URI) properties of an item are based on its type, but I think that microdata could handle multiple types for an item if instead of saying:

If the item is a typed item: [the property token must be] a defined property name allowed in this situation according to the specification that defines the relevant type for the item

it said:

If the item is a typed item: [the property token must be] a defined property name allowed in this situation according to any of the specifications that define the relevant types for the item; if the property is defined for more than one type, these definitions must be identical

It’s worth noting that, unlike itemtype, the itemprop attribute can take multiple values which, along with its ability to take URIs, provides for an easier inheritance mechanism. However, schema.org again recommends a string-based approach for extended properties; maybe that’s for consistency, or perhaps the real reason is that they want to become a centralised repository for acceptable microdata on the web.

Summary

Whatever method publishers use, creating structured machine-processable data is hard and embedding it in a page is even harder. It is a layer that necessarily sits above or alongside the content of a page, invisible to people simply looking at the page in a browser and therefore difficult to get right without additional tooling.

Microdata specifically aims to make this easy to do; schema.org demonstrates some of the ways in which it doesn’t quite meet that requirement. Perhaps the search engines behind the effort haven’t managed to (or bothered to) implement HTML5/microdata parsing correctly or perhaps the people writing the documentation thought they understood how HTML5/microdata works but actually didn’t.

Either way, the mistakes are worth learning from to improve the specs while they are not yet final. As I discussed above I think that means:

adding an itemvalue attribute for machine-readable versions of content
enabling itemtype to take multiple values to support extensibility

though I suspect the latter point at least will be contentious among those who don’t think decentralised extensibility is ever a desirable feature.

Jeni's Musings

Microdata and RDFa Living Together in Harmony

Mapping Rules

Mismatched Features

Local Property Names

Interpretation of Element’s datetime Attribute

Content Overrides

and Elements in Flow Content

Identifiers without Types

Built-in Prefixes

Literals Including Markup

Itemref

Lists

Datatypes

Languages

Multiple Types

The src Attribute

Link relations

Possible Subset of RDFa

Possible Subset of Microdata

Guidelines for Vocabulary Authors

Choosing Between Microdata and RDFa

Final Words

Mapping RDFa to Microdata

Page Metadata

Base URI

Explicit Subjects / ItemIds

Items from the src Attribute

Additional Properties for Images

Nested Images

Types with Blank Nodes

Inherited Subject

Anonymous Nested Resources

Hanging Rels

Implicit Resources

Overriding Text Content

Language Support

Literals that Include Markup

E = mc2: The Most Urgent Problem of Our Time

E = mc2: The Most Urgent Problem of Our Time

The resource Attribute

Multiple Types

Mapping Microdata to RDFa

Unidentified Items / Blank Node Subjects

Values from the src Attribute

Values from the datetime Attribute

Nested Items / Object Properties

Itemref

My photos

My photos

Multiple Values

Creating Multiple Statements

Creating an rdf:Seq

Creating a rdf:List

Multiple Properties Sharing a Value

Types

Hedral

Hedral

Global Identifiers

Global Property Names

Hedral

Hedral

Link Relations

Bluesday Tuesday: Money Road

Bluesday Tuesday: Money Road

Microdata + RDF

Simplifying RDFa

Adding Decentralised Extensibility to Microdata

Multiple Types

Disambiguating Properties

Adding Datatypes

Non-Additions

Summary

Using Multiple Vocabularies in Microdata

Use Case

Publisher Workarounds

Different Syntaxes

Repeated Data

Use itemref

Content Negotiation

Interpretation of Element’s `datetime` Attribute

The `src` Attribute

Items from the `src` Attribute

E = mc²: The Most Urgent Problem of Our Time

E = mc²: The Most Urgent Problem of Our Time

The `resource` Attribute

Values from the `src` Attribute

Values from the `datetime` Attribute

Creating an `rdf:Seq`

Creating a `rdf:List`

Use `itemref`