html5

My Experience of Web Standards

Jeni — Sun, 24 Jul 2011 16:24:00 +0000

One of the things that’s been niggling at the back of my mind since the schema.org announcement is how small a role search engine results plays in the wider data sharing efforts that I’m more familiar with in my work on legislation.gov.uk, and more generally how my day job experience differs from (what seem to be) more common experiences of development on the web. In this post, I’m going to talk about that experience, and about the particular problems that I see with the coexistence of microdata and RDFa as a result.

My day job (the one I actually get paid for) is web development. The site I spend most of my time and effort on is legislation.gov.uk. This deals with complex content (UK legislation) that has to be presented in multiple formats (users love PDFs of legislation). Our aim is to make the data as reusable as possible by third parties through good, RESTful, web architecture, and we want to use open standards and open source technologies as part of the UK government’s general strategy.

legislation.gov.uk is not a global website like Amazon or eBay, but it’s not small either: it covers 60,000 changing items of legislation, providing point-in-time views for many of them, and with more added every day. It’s one of the top ten most used UK Government websites, with 2 million visits (about 10-12 million page views) each month and typically about 120 requests/second during the active times of the day. Legislation might sound like a highly specialist interest, but if you search for legislation.gov.uk on Twitter you’ll see it being referenced over and over by people who want to share what the law says.

I do not by any means claim that my experience is representative of the wider web. I know that there are large numbers of sites that deal only in data, not documents, and certainly not documents with the kind of rich semantic structure that legislation has. I offer the following discussion as a data point, partly because I can’t quite believe that legislation.gov.uk is completely unique in its requirements and partly because obviously my perspective on a bunch of issues arises from this experience.

Technology Stacks

Legislation items are complex, semi-structured documents. Their natural fit is XML (well, that’s not quite true — their natural fit would be something that allowed overlapping markup — but XML is the closest that we have). So we store it in XML in a native XML database and we use an XML toolset to query it (XQuery) and transform it (XSLT) into various formats including rendering it as PDF (through XSL-FO).

Our next step for the development of the site involves looking at legislative effects. These form a graph: one item of legislation affects other items of legislation which may in turn affect other items and so on. There are all sorts of other links between items of legislation in terms of commencements, conferred powers and so on. Particularly because we already have well-thought-through URIs for legislation, the natural fit is to use RDF to represent this graph. We already offer a SPARQL endpoint for accessing some aspects of our data, but we expect to expand and develop this over the next few months and to use it as a layer under the website and exposed for reusers, in much the same way as we use the XML database.

As a government site, we have fairly strict limits on what we can do within our web pages: we have to make sure that they’re accessible by everyone who wants to view them. We aren’t able to use technologies that are only available in the latest browsers, but that’s OK because with the kind of content we deal with, we don’t have to do anything fancy anyway. So we use pretty basic HTML and CSS and Javascript, because that’s how you deliver content to end-users on the web (as well as exposing the underlying XML and RDF, to enable others to reuse the data).

In other words, we use three web stacks for delivering legislation.gov.uk:

the XML stack, which is great for single-source publishing of documents that have more semantic structures than those supported by HTML
the RDF stack, which is well-suited for metadata about things that are identified by URIs
the HTML stack, which is absolutely necessary for delivering human-accessible content on the web

What bemuses me, because of this experience, is that sometimes it appears that the narrative around these technologies is framed in terms of an exclusive choice between them. For example, @mattur asked:

It is as if, if you use XML you cannot appreciate the utility of error-handling in HTML; or if you use RDF you cannot understand the need to represent documents in XML; or if you want to utilise HTML fully, you cannot adopt RDF’s view of data on the web. That’s simply not my experience. They each have their role on the web; supporting the use of one does not necessitate rejecting the use of the others.

It’s interesting that some of the standards that are most reviled are those that arise at the intersections, where it appears that one technology is trying to encroach on the space of another:

XHTML at the border of XML and HTML
RDF/XML at the border of RDF and XML
RDFa at the border of all three

At the same time, within legislation.gov.uk, we publish XHTML (because it’s the natural output from an XML toolchain) and create and process RDF/XML (because it gives us access to that data from within the XML toolchain). We use a small bit of RDFa in the XHTML to indicate the rights under which our information is avaialble, and don’t yet, but are thinking about using RDFa to mark up non-document semantics within our XML (to enable the XML markup to focus on the document structures that it’s good at). For all their imperfections, these intersection technologies are useful for managing cross-overs; the problems arise when they overstep their remit and people start to think that all HTML must be XHTML or all XML must be RDF/XML or all RDF must be RDFa.

Sharing Scenarios

The second thing that I wanted to explore is the experience from legislation.gov.uk of what it’s like to be a publisher who actively wants to share their data. We need to operate simultaneously at three levels in our data sharing efforts.

Large-Scale Consumer-Driven Data Sharing

The first target for our data sharing efforts are the search engines. Obviously we’re not selling anything, but we want people to be able to locate legislation easily when they want it, and we want people who have done the search to be able to see some information about the legislation so that they know that they’ve located the right item.

This is large-scale consumer (search engine) driven data sharing, typified by schema.org and Facebook’s Open Graph Protocol (OGP). There are a few very big data consumers (Google, Microsoft, Yahoo!, Facebook etc) who need to consume data from large numbers of data providers. These consumers obviously can’t understand everything, so they determine and document what syntaxes and vocabularies they do understand and expect publishers to follow.

The benefits that publishers get from a particular consumer determines which syntax/vocabulary they use; publishers who are particularly keen to show up prettily within search results will target schema.org whereas those who want to be sharable within Facebook will target OGP. Many publishers will want to target both. There is probably a driver towards eventual convergence:

publishers might push back about inserting two lots of very similar data in their pages
consumers might want to include data from publishers who haven’t specifically targeted them

although there’s likely to be a period where they coexist, much as there was for VHS and Betamax (and V2000, I know, dad) during the early days of video players.

As I discussed previously, these large-scale consumers will be driven by the data that they find in the wild, in all its messy variety. They get relatively little benefit directly from using a generic syntax, as they are really interested in only a few, pretty generic, vocabularies for which they have hardwired processing. Indirectly, adopting a generic syntax has benefits in that publishers might find it easier to find tools that enable them to generate it, tutorials about how to use it, and feel that they aren’t being quite as locked in to something proprietary. However, rejecting data that isn’t marked up properly using that syntax has no benefit for consumers except in so far as it makes them feel that they are being good community members.

This is the pattern we see with schema.org (which accepts microdata but, based on its documentation, won’t reject data that isn’t fully compliant with it) and with OGP (which accepts a subset of RDFa but doesn’t reject data that hasn’t got prefixes properly bound, for example).

Another point to mention is that there is very little trust in this scenario. The communication between consumers and publishers is very limited, and the consumers will want to protect themselves against accidental or malicious errors that are evident in mismatches between explicit metadata and that which is parsed from the visible content of the page.

The parallels to HTML and browser vendors are very strong in this type of data sharing.

Small-Scale Consumer-Driven Data Sharing

A second type of data sharing is again driven by consumers, but this time at a lot smaller and more specialised scale. For legislation.gov.uk, these are services such as GLIN, which is a global legislation registry. Other examples are the recent work that we’ve done to publish UK Government organograms or Chris Taggart’s Open Election Data project. In these cases, there’s a single, relatively small and specialised consumer and a small number of publishers which are closely coordinated together.

As in the large-scale case, the consumer ultimately determines the syntax/vocabulary that it recognises, and communicates that to the publishers. However, small-scale consumers typically have close coordination with the publishers, which has a number of side-effects:

consumers may be more able to both apply pressure to and help publishers to do well in their markup
publishers have the opportunity to feed back directly to the consumer any suggestions that they have about changes to the syntax/vocabulary
publishers are likely to gain an immediate and tangible benefit from their cooperation, such as visualisations of their data that they otherwise wouldn’t have seen

Another noteworthy point about small-scale consumers is that they’re unlikely to have the engineering capability to create a custom parser for a particular syntax, but will instead want to use something off-the-shelf to extract data from pages and into their own backend systems. This, coupled with the closer coordination with publishers, means that they’re much more likely to stick to a specification, assuming that the off-the-shelf tools do.

Publisher-Driven Data Sharing

The final type of data sharing is driven by publishers. At legislation.gov.uk, we’re motivated to make our data available for reuse for transparency/accountability reasons (to help citizens understand the law), efficiency reasons (to help parliament and government departments to publish new legislation better) and economic reasons (to foster innovation in legal publishing). We don’t have any individual consumers in mind when we publish our data, but have found that simply by publishing it well, we foster reuse.

In this case, we as publishers are highly motivated to ensure that the data we publish is easily parsed with something off-the-shelf, since that lowers the barrier for potential consumers. Publishers like us are very likely to have unique, specialised, content and need to use a vocabulary that fits closely to our internal data structures as this lowers implementation cost. Consumers can also trust publishers like us: we simply have no motivation to lie in the data that we provide for reuse.

Mixed Markup

As I’ve outlined above, publishers like legislation.gov.uk need to target several potential consumers at the same time:

large-scale consumers such as search engines
small-scale consumers that provide us with a useful service
specialist consumers that are interested specifically in our data

We cannot use a single vocabulary for all these different purposes. (Well, we could write our own vocabulary and describe mappings to other vocabularies using RDFS, but search engines wouldn’t read it.)

We must therefore use a mix of vocabularies:

generic vocabularies about things that search engines care about
specialised vocabularies for particular small consumers
site-specific vocabularies for sharing our unique data

It’s repetitive, but it’s manageable so long as we have a syntax that enables us to say that an item of legislation is a http://scheme.org/CreativeWork and a http://purl.org/dc/dcmitype/Text and a http://www.legislation.gov.uk/def/legislation/Legislation and allows us to give multiple properties the same value.

The way things are going at the moment, we might well end up having to use multiple syntaxes on the same page, as some consumers understand microdata, others consume RDFa, and still others will parse microformats. This leads to more repetition: adding itemprop for microdata, property for RDFa and specialised class attributes for microformats. But worse (much worse), each of the syntaxes uses a different parsing model to create an entity-property-value structure, so not only do we have to learn substantially different markup patterns but our pages quickly become some kind of hideous polyglot mess trying to balance between them.

Looking Forward

As I said at the start, I’m fairly sure that my experience at legislation.gov.uk isn’t representative of the wider web, but I don’t have a clear idea about just how unrepresentative it is, in terms of technology use or motivations around data sharing. When I read my twitter stream or blogs, there’s a massive sampling bias, both in terms of who I follow and what I read, but also about who talks about what they’re doing. (I’m reminded of Jeff Atwood’s post on the Two Types of Programmers: the vast majority of web developers don’t make a noise about what they do.)

Taking part in web standardisation today often feels like being part on an ongoing cold war between distinct camps rather than a community working towards common aims. The underlying question seems to be “who’s side are you on?” Every decision and activity is cast as a victory or defeat. Time is wasted on attack and defence, or on raking over past slights and stupidities, rather than on progress. Valid criticism from outside a group cannot be listened to for fear of giving ground, cannot be made within a group where it seems like betrayal.

It is the Robbers Cave Experiment played out in web standards. As a psychologist, I find it fascinating. As a developer, and particularly one who doesn’t self-identify with any single group, it is frustrating. As a TAG member, trying to work for the longer-term good of the web, it is worrying, because situations of intergroup conflict lead to groupthink and non-optimal solutions.

As I described above, a non-optimal outcome seems to be the most likely result of the particular microdata vs RDFa conflict for us at legislation.gov.uk. While I know we are not generally representative, I believe that it will be similarly bad for other developers: publishers, consumers and tool implementers.

This is a problem for all who want to foster data sharing on the web using open standards; it is not one that any one group can fix on their own. It’s my hope that a balanced task force of individuals with a variety of experience and backgrounds can provide a focus for us all to work together to solve it. If we can’t, then we have let our prejudice and bias overcome our judgement.

TAG F2F, June 2011

Jeni — Fri, 17 Jun 2011 10:44:12 +0000

As you may know, I accepted an appointment to the W3C’s Technical Architecture Group earlier this year. Last week was the first face-to-face meeting that I attended, hosted in the Stata Center at MIT. As you can tell from the agenda (which was in fact revised as we went along) it was a packed three days.

What I intend to do here is to briefly report on the major areas that we discussed and give a tiny bit of my own personal take on them. In no way should any of what I write here be judged as revealing the official opinion of the TAG, it’s just me saying what I think, and I’m not going to go into anything in depth because they’re all incredibly gnarly and contentious topics and I’d not only be here all year but also end up in a tar pit.

Role of the TAG

Usefully for me as a newcomer, our first session was about the ongoing role of the TAG. The TAG occupies a unique position within the W3C. According to its charter it was set up

To improve the effectiveness of Working Groups, to reduce misunderstandings and overlapping work, and to improve the consistency of Web technologies developed inside and outside W3C

The TAG ultimately has three routes to do this:

by providing specific advice on issues that are brought to its attention
by writing documents on basic web architecture principles that go through community review, particularly through the general review of the W3C standards track and become Recommendations
by advising the W3C Director (Tim Berners-Lee) about what he should do on the extremely rare occasions when there are issues that he is supposed to adjudicate on

In none of these cases is there anything that binds the people receiving the advice of the TAG, or reading Findings or Recommendations made by the TAG, to accept them or do anything about them. The power and authority of the TAG depends solely on the quality and utility of its arguments, which is how it should be in my opinion.

Client-Side Application State

The first technical session was about client-side application state and was a review of the Identifying Application State draft that T.V. Raman began before he left the TAG and that Ashok Malhotra has been working on since. This should in the next few months or so be published as a TAG Finding (though it is currently on the Recommendation track).

This work is essentially about documenting the different ways in which you can identify application state within a URI, why that’s a useful thing to do, and some of the pitfalls of using hash URIs to do so. Most of the discussion was about details to do with wording within the document. One thing I thought particularly interesting was the point that URI-based application state is relevant in all ‘active content’, not just in HTML; for example, scripting in SVG or in PDFs bring the same considerations.

Buffer Bloat

Over lunch on Monday we listened to and discussed a presentation by Jim Gettys on buffer bloat. Basically (and all the errors here are introduced by me), TCP/IP is designed to route around network blockages, but it can only do so if it detects them quickly. When you have big buffers in place, as in the case of all modern operating systems and hardware, blockages aren’t detected quickly; they’re only detected when the buffers fill up. Then buffers empty and the data has to be sent again. The net result is that connections get really slow, not just for upload or download but for both, not just for you but for everyone using the network.

Jim talked about how this is exacerbated by the large amount of web traffic and the design of HTTP, particularly the lack of use of HTTP pipelining (whereby several HTTP requests and responses are sent over one long-term connection), because it leads to lots of small messages which can’t be handled effectively. There’s lots more about this on his blog.

Jim also talked about the failure of certificate authorities and how we should be looking at distributed protocols using digitally signed data, pointing us in particular to CCNx.

Fragment ID Semantics

First thing Tuesday was a session that I led on fragids, in particular the problems that are arising out of the mime type registration of +xml types (3023bis) clashing with those that are used for, say, images, and what happens when these come together in something like SVG.

The same issues arise whenever you have documents with types that ‘inherit’ fragid semantics from two directions. For example, XHTML documents are XML documents, so constraints on +xml mean you shouldn’t use interpreted fragids (eg hash-bangs) on them, but they are also ‘active content’ which makes interpreted fragids useful. Similarly, in linked data you shouldn’t really use a hash URI to mean a Person with a primary resource that provides as a response an XML document with embedded RDFa, because according to XML fragid semantics, such a URI should point to an XML element.

Basically the use of fragids has grown markedly outside their original scope and these situations aren’t really covered in the specs. I am now tasked to create a document that describes the issues and suggests ways forward. So that will be fun.

Telcon with IAB

The second session on Tuesday was a telcon with the IAB which has a similar role within the IETF as the TAG does within the W3C. This was a bit of a ‘getting to know you’ session, covering the work of the two groups on:

versioning and extensibility
security
privacy, including Do Not Track

and talking about opportunities to meet and work together on various topics like these.

URI Definition Discovery and Metadata Architecture

The afternoon session on Tuesday was spent on Jonathan Rees’s work on the Architecture of the World Wide Semantic Web, which covers, amongst other things, what people in semantic web circles call httpRange-14. At core, this is about the kinds of URIs we can use to refer to real-world things, what the response to HTTP requests on those URIs should be, and how we find out information about these resources.

Jonathan has put together a document called Providing and discovering definitions of URIs which covers the various ways that have been suggested over time, including the 303 method that was recommended by the TAG in 2005 and methods that have been suggested by various people since that time.

It’s clear that the 303 method has lots of practical shortcomings for people deploying linked data, and isn’t the way in which URIs are commonly used by Facebook and schema.org, who don’t currently care about using separate URIs for documents and the things those documents are about. We discussed these alongside concerns that we continue to support people who want to do things like describe the license or provenance of a document (as well as the facts that it contains) and don’t introduce anything that is incompatible with the ways in which people who have been following recommended practice are publishing their linked data. The general mood was that we need to support some kind of ‘punning’, whereby a single URI could be used to refer to both a document and a real-world thing, with different properties being assigned to different ‘views’ of that resource.

Jonathan is going to continue to work on the draft, incorporating some other possible approaches. It’s a very contentious topic within the linked data community. My opinion is while we need to provide some ‘good practice’ guides for linked data publishers, we can’t just stick to a theoretical ideal that experience has shown not to be practical. What I’d hope is that the TAG can help to pull together the various arguments for and against different options, and document whatever approach the wider community supports.

Can publication of hyperlinks cause copyright infringment?

The first session on Wednesday was another session that I led, discussing the Publishing and Linking on the Web draft that Dan Appelquist and I have been working on.

The aim of this document is to explain the tensions between terms that are commonly used in legal documents such as “possession”, “adaptation” and “distribution” and the way that publication works on the web, in which multiple servers may have copies of the same document (because they cache copies to make the ‘net go faster), automated agents may make changes to those documents (such as compressing or resizing documents, or merging Javascript) and people may refer others to those documents through linking.

We’re particularly keen to argue that linking to something is not the same thing as distributing it. The web’s power arises through its links, so it’s important that people are able to link to something without being worried about what happens when/if the domain they link to is taken over by something illegal.

Dan and I are going to continue to work on this document in response to various suggestions around organisation and terminology, with a view to getting some ‘friendly legal experts’ to look it over and then seeking wider review. The intention is for it to eventually become a Recommendation as this will give greater weight to it as a document for a legal audience.

API Minimisation and Client-Side Storage

There were then a couple of short sessions.

Dan talked about API Minimisation, which is the design principle that to increase privacy we should design APIs that enable people requesting information to say exactly what information they need, and return only that rather that everything known about a think. Dan’s put together an draft and should be calling for review for it soon.

Ashok then led discussion on client-side storage and we brainstormed around some of the architectural/design issues about which we might want to write if we were to put together a document. This work is at a very early stage.

TAG Priorities

After lunch, we had a session on TAG priorities where we discussed which of the various pieces of work that we’re doing should receive the most attention and had a quick review of who is doing what within the TAG.

Our basic problem is that a lot of this stuff feels quite urgent, and we want to be responsive, but with only 5-6 of us “actively involved” (which means 1 day/week) in drafting documents, and other TAG duties taking up our time, it feels like we have taken on too much work. Our focus for the next little while is going to be on responding to issues where our lack of response might either hold people up or cause longer term problems (for example the publication of contradictory mime type definitions), which means things like the document on publishing and linking on the web will need to bubble in the background rather than being the focus of activity.

HTML5 Last Call

Our final session, for which we were joined by Philippe Le Hégaret, was on the HTML5 Last Call documents. The TAG has raised various issues over the course of HTML5 development and want to follow up on how those issues have been addressed in the documents. Our role means that we’re responsible for making sure there’s consistency with other specifications, and that there isn’t anything that seems like it’s going to cause problems in the long term.

The part that we spent most discussion time on was the relationship between Microdata and RDFa. We talked about the precedents for having two specifications that do very similar things but with different approaches, such as CSS and XSL, and how this isn’t necessarily a bad thing so long as they don’t contradict each other and people can move between them easily (because they have the same conceptual foundations).

I’m going to save my opinion on this topic for another post. Suffice it to say that microdata and RDFa as currently specified don’t work well with each other but it’s not at all clear what the best path forward is. The TAG decided to recommend that the W3C set up a Task Force to look at what the best way forward might be.

Final Words

If you want links to the minutes of the TAG F2F, they’re available within the agenda or on separate pages for:

If you have anything to say on any of these topics, please send email to the TAG mailing list. Or you could comment here or email me directly if you like. Which leads me on to talking about what I’d like to do in the TAG.

One of the guidance notes for new members to the TAG says:

TAG members are elected or appointed not to represent their individual member organizations, but the Web community as a whole. We try to take that responsibility very seriously.

I do take that responsibility seriously. Web architecture has to be a combination of practice and theory, balancing approaches that work right now with a desire to not break anything long term. I do practical work developing web applications with HTML, CSS, Javascript, XML, RDF, XSLT, XQuery and so on and so on every day, but I know I don’t see all the difficult corners of the open web standard space: no one person can.

I can listen though, so that’s what I will try to do: listen, digest, reflect and act.

But I have limited resources. Unlike most of the members of the TAG, I am not employed by a large organisation that pays me for time I take on the work that I do for the TAG. The W3C kindly paid for my flights to and from F2Fs, but not hotels or expenses. I wouldn’t have taken this on if I wasn’t prepared to shoulder the financial burden, but if there is anyone out there who might sponsor my participation, I’d love to hear from you.

Lessons for Microdata from schema.org

Jeni — Fri, 10 Jun 2011 20:27:38 +0000

There is (obviously, from the way my tweet stream, feed reader and email have filled up) lots to say at many levels about schema.org, a new collaboration between Google, Microsoft and Yahoo! that describes the next phase in search engines’ extraction of semantics from web pages. In this post I’m going to focus on what we can learn from schema.org about the design of microdata and how it might be improved.

Digging into the details of schema.org there are several examples of places where its recommended method of marking up metadata directly contradicts the HTML5 specs. Given the number of internal contradictions within schema.org, I’m assuming that these are mistakes that will be corrected as the material is reviewed and matures rather than deliberate forking of HTML5.

Note: What I say about HTML5 here is equally true — at least at time of writing — of the WHATWG version of HTML, which of course already diverges from HTML5.

One of the inputs to the design of microdata was to look at the mistakes that people make and try to design something to address the cause of those errors, so it’s interesting to apply that method to the errors made by schema.org. This doesn’t mean changing specs so that erroneous markup is conformant, but it does mean providing facilities that enable people to more easily do things in a conformant way, removing the temptation of non-conformance and lowering the likelihood of future mistakes.

Note: I am certain that there would also have been errors had schema.org used RDFa or microformats, indeed I gather that they are common in the documentation of Google’s Rich Snippets.

Use of `<time>` element

The first example is one spotted by Tantek: the value of the openingHours property of EventVenue is described as:

The opening hours for a business. Opening hours can be specified as a weekly time range, starting with days, then times per day. Multiple days can be listed with commas ‘,’ separating each day. Day or time ranges are specified using a hyphen ‘-‘.

Days are specified using the following two-letter combinations: Mo, Tu, We, Th, Fr, Sa, Su.

Times are specified using 24:00 time. For example, 3pm is specified as 15:00.

Here is an example: <time itemprop="openingHours" datetime="Tu,Th 16:00-20:00">Tuesdays and Thursdays 4-8pm</time>

A similar error involving the <time> element can be found on the Getting Started page which has an example in which the datetime attribute contains an ISO 8601 duration:

Durations can be specified in an analogous way using the time tag with the datetime attribute. Durations are prefixed with the letter P (stands for “period”). Here’s how you can specify a recipe cook time of 1 ½ hours:
<time itemprop="cookTime" datetime="P1H30M">1 1/2 hrs</time>
H is used to designate the number of hours, and M is used to designate the number of minutes.

According to the HTML5 specification, the datetime attribute holds a valid date or time string which is specified as using the normal ISO 8601 syntaxes for dates and times. Conforming HTML5 document must not hold the syntax used in the above example. From what I can tell (please correct me if I have this wrong), conforming data-mining tools can process this syntax because they use the value of the datetime content attribute, rather than the value of the dateTime IDL attribute to provide the property’s value, but from an authoring perspective, no one should be encouraging people to create non-conformant HTML5.

In fact, given that values from microdata are never typed, it’s not clear why these examples use the <time> element at all. The conformant way to provide the data would be to use a separate <meta> element to hold the data:

<meta itemprop="openingHours" content="Tu,Th 16:00-20:00">
Tuesdays and Thursdays 4-8pm

But I can understand the itch to use the <time> element here; the <meta> element above is before, rather than around, the textual content of the page which it reflects, whereas with the <time> element it is more obviously an explicit machine-readable version of that content. In microformats, the pattern is to use a title attribute:

<span class="openingHours" title="Tu,Th 16:00-20:00">
  Tuesdays and Thursdays 4-8pm
</span>

and in RDFa a content attribute:

<span property="openingHours" content="Tu,Th 16:00-20:00">
  Tuesdays and Thursdays 4-8pm
</span>

So maybe this error is an indication that microdata needs an itemvalue attribute for those cases where the human-readable content can be expressed in a more formal machine-readable microsyntax, with special handling with the <time> element for the case when the value is a date/time and href/src when the value is a URI.

String parsing of URIs

The second example of non-conformance with HTML5 in schema.org is the method by which they support extensibility. Schema.org provides a set of types for the things described by web pages. Naturally, this set does not cover everything, but search engines still want to be able to use the metadata within the page about a Person even when that Person is described as a Minister (say).

So schema.org says that to extend their type hierarchy, you simply append the name of the new type after a / at the end of the URI for the parent type. In this example, a Minister should be given the type http://schema.org/Person/Minister.

My guess (as I can’t see any other way in which they’d do it) is that the search engines intend to use string processing on the type URI in order to work out whether it’s a subtype of a known type (ie, does the URI start with the string http://schema.org/Person? If it does, it’s a Person of some kind).

The microdata specification states that:

The item type must be a type defined in an applicable specification.

and that:

Item types are opaque identifiers, and user agents must not dereference unknown item types, or otherwise deconstruct them, in order to determine how to process items that use them.

Treating URIs as opaque identifiers, and forbidding inferring semantic meaning through string processing, is a fairly fundamental web architectural principle as well as fitting with microdata’s constraint that types are specified somewhere.

Perhaps the gloss is that schema.org is an applicable specification that states that any URI that looks like http://schema.org/{known-type}/{extension-type} is a type that is defined in schema.org. But I think what’s actually happening is that schema.org wants to grow their vocabulary organically, responding to the data that the search engines find on the web. They recognise that people will want to use their own vocabularies for their own purposes (for example to provide data for scripts, reusers, or for browsers and other agents that aren’t search engines) but want to continue to be able to understand the semantics of that data.

Schema.org is constrained in the mechanisms that it could use to recognise these new types:

they can’t resolve the type URI and expect metadata at the end of it to indicate the parent schema.org type, because the HTML5 microdata specification forbids resolving unrecognised item type URIs; even if they were allowed to do so, working out inheritance by declarative mechanisms that involve resolving URIs and interpreting the results is computationally expensive compared to string munging; for search engines that expect to do this with billions of web pages, this may be a significant processing burden
they can’t tell people to use the schema.org type in addition to their own type because the HTML5 microdata specification doesn’t allow an item to have more than one type; if it did, they could encourage people to use the schema.org type as well as their more specific one, so rather than itemtype="http://schema.org/Person/Minister", enable publishers to do itemtype="http://schema.org/Person http://reference.data.gov.uk/def/central-government/Minister"

I suspect that having multiple types is currently disallowed in microdata because the semantics of the (non-URI) properties of an item are based on its type, but I think that microdata could handle multiple types for an item if instead of saying:

If the item is a typed item: [the property token must be] a defined property name allowed in this situation according to the specification that defines the relevant type for the item

it said:

If the item is a typed item: [the property token must be] a defined property name allowed in this situation according to any of the specifications that define the relevant types for the item; if the property is defined for more than one type, these definitions must be identical

It’s worth noting that, unlike itemtype, the itemprop attribute can take multiple values which, along with its ability to take URIs, provides for an easier inheritance mechanism. However, schema.org again recommends a string-based approach for extended properties; maybe that’s for consistency, or perhaps the real reason is that they want to become a centralised repository for acceptable microdata on the web.

Summary

Whatever method publishers use, creating structured machine-processable data is hard and embedding it in a page is even harder. It is a layer that necessarily sits above or alongside the content of a page, invisible to people simply looking at the page in a browser and therefore difficult to get right without additional tooling.

Microdata specifically aims to make this easy to do; schema.org demonstrates some of the ways in which it doesn’t quite meet that requirement. Perhaps the search engines behind the effort haven’t managed to (or bothered to) implement HTML5/microdata parsing correctly or perhaps the people writing the documentation thought they understood how HTML5/microdata works but actually didn’t.

Either way, the mistakes are worth learning from to improve the specs while they are not yet final. As I discussed above I think that means:

adding an itemvalue attribute for machine-readable versions of content
enabling itemtype to take multiple values to support extensibility

though I suspect the latter point at least will be contentious among those who don’t think decentralised extensibility is ever a desirable feature.

The HTML5 DOM and RDFa

Jeni — Thu, 24 Sep 2009 08:05:03 +0000

One of the fundamental disconnects between HTML5 and previous versions of HTML is the way in which you answer the question “what is the structure of this page?”. Things that make use of that structure, such as RDFa, need to take this into account.

An example is the document:

<html>
  <head><title>HTML example</title></head>
  <body>
    <table>
      <span>Example title</span>
      <tr><td>Example table</td></tr>
    </table>
  </body>
</html>

There are two different ways in which you might interpret the structure of this document. First, you might view the structure to be as it is written, with the <span> element as a child of the <table> element and therefore a tree that looks like:

+- html
   +- head
   | +- title
   +- body
      +- table
         +- span
         +- tr
            +- td

Second, you might view the structure of the page to be the DOM as it is constructed by an HTML5 processor, which will move the <span> out from the table due to foster parenting, giving the result:

+- html
   +- head
   | +- title
   +- body
      +- span
      +- table
         +- tr
            +- td

Which you view it as doesn’t really matter at this point, but it does when you start to introduce markup that infers information based on the structure of the page, such as RDFa. Let me introduce some RDFa markup to the document:

<html xmlns:dc="http://purl.org/dc/elements/1.1/">
  <head><title>HTML+RDFa example</title></head>
  <body>
    <table about="http://example.com">
      <span property="dc:title">Example title</span>
      <tr><td>Example table</td></tr>
    </table>
  </body>
</html>

Now, if you view the structure to be as written, the <span> element is within the <table> element, and is therefore viewed as talking about whatever it is that the <table> element is talking about, namely http://example.com. So the RDF that you will glean from this page is:

<http://example.com> dc:title "Example title"

On the other hand, if you view the structure to be that constructed by an HTML5 processor, the <span> element is not within the <table> element, and is therefore viewed as talking about whatever the document is talking about, namely the document itself. So the RDF that you will glean from the page is:

<> dc:title "Example title"

This isn’t exactly a new problem. There has always been the possibility of Javascript embedded within a page changing the page by moving or inserting elements, making the page that a non-browser sees fundamentally different from the one that a browser sees. This has been used by SEO people and spam merchants to get search engines to direct people to pages which mutated into something different when they were actually visited by a browser. And this eventually lead to those people who cared about interpreting meaning from the structure of pages (ie search engines) to at least go some way towards evaluating the Javascript within the page in order to “see” the page as a human would.

So it’s not a new problem, but it’s still a problem.

For those people trying to define how RDFa is interpreted in HTML5, there are several unpleasant alternatives:

Define RDFa as operating over an HTML5 DOM. This would make things easy for Javascript implementations in as much as they can rely on being used with HTML5 DOMs, ie in HTML5 browsers. But it raises the implementation burden for other implementations, such as those based on XSLT or a simple tidy-then-interpret-as-XML approach: essentially every implementation will need to include an HTML5 parsing library.
Define RDFa as operating over a DOM, but leave the creation of that DOM as implementation-defined. This effectively passes the buck (“it’s not our fault that HTML5 processors will construct a different DOM from XML processors”) but makes it hard to test implementation conformance and for authors to know exactly how their page will be interpreted. For example, an implementation that constructed a DOM with randomly rearranged elements would be entirely conformant despite producing completely different triples from one that took the elements in the original order.
Define RDFa as operating over a serialisation, with precise wording that describes how that serialisation is mapped into a tree structure that is walked to process the RDFa within the page. This approach will prevent implementations that use other methods of constructing trees from being conformant; depending on how it’s defined that might include XSLT implementations and/or Javascript implementations and/or implementations that use standard (XML-based) libraries for parsing the documents.

Personally I lean towards the second of these: defining RDFa as operating over a DOM but placing no constraints on how that DOM is created. It leaves open the possibility of Javascript implementations to work on the DOM they see, which may be radically different from the one seen by other processors due both to HTML5 reordering of elements and the dynamic modification of the page through Javascript. (Several people use rdfQuery to do before-and-after parsing of RDFa within a page, turning browsers into semantic editors, for example.) But it also lets conformant implementations be constructed in other ways for implementation ease or user needs, supporting the use of XSLT through GRDDL and the static crawling of content with minimal processing.

Perhaps the set of permissible methods of DOM creation could be listed to prevent completely random processing, but I expect that it will be effectively limited through social and technological pressures. Implementations that build DOMs in random ways aren’t going to be as useful (to their users) as those that build them in expected ways; it’s also going to be far easier to implement RDFa processors using standard parsing libraries.

The approach is not without its downsides, of course. XSLT is similarly defined as operating over a tree model, with the question of how that tree model is constructed left to the implementation. Most processors decided to construct the tree using standard XML parsing, but famously MSXML would strip certain whitespace-only text nodes from the tree (unless you specified a parsing flag telling it not to), leading to incompatibilities and user confusion.

My guess is that the same kind of thing will happen with RDFa processors. It could very well be the case that an author will:

check their RDFa in an RDFa validator that constructs a static HTML5 DOM, revealing one set of triples
be confused when they then use a Javascript RDFa library within their page and get a slightly different set of triples because of some Javascript embedded in the page that changes its structure
be further confused when a search engine that uses a tidy-and-interpret-as-XML approach gleans yet another slightly different set of triples and displays it in the search result

So if this approach were chosen, I would expect wording in the specification that required implementations to state the method they used to create the DOM (ie it should be implementation-defined rather than implementation-dependent) and that warned authors of the most likely causes of differences between implementations (such as tree modifications performed by HTML5 processors and Javascript within the page). I’d also like to see tools that take an HTML page and indicate the triples that it generates under different common DOM construction methods, so that authors can see the variation in how their documents might be interpreted.

On Resolvability

Jeni — Fri, 28 Aug 2009 22:02:16 +0000

In my last post about RDFa and HTML I talked about how one of the gulfs that separates the HTML5 and Semantic Web communities is the attitude to the resolvability of property (and class) URIs.

I’m currently experimenting with introducing the ability to automatically locate information about properties and other resources that are referenced within triples to rdfQuery, so now is a good time, as far as I’m concerned, to look more closely at what the ability to resolve properties gives you and how to avoid problems if the property URI is (temporarily or permanently) unresolvable or resolvable to something new.

I’m going to attempt to answer:

How do or might applications use property and class URIs?
How can data and ontology publishers assist them in doing so?
What should frameworks (such as rdfQuery) do to help application developers?

Application Developers

We can divide applications using online data into three general categories:

data-specific applications are constructed around particular data sets that are known to the developer of the application; the visualisations that I’ve been doing are examples of data-specific applications
vocabulary-specific applications are constructed around particular vocabularies, wherever the data might be found that uses them; Google’s Social Graph API and Yahoo! SearchMonkey are examples
generic applications are constructed to navigate through any RDF that they find; Tabulator is one example

Most mashups are data-specific applications. When you, as a developer, create a data-specific application, the thing that you need to know most of all is what information the dataset contains. Part of that is working out the meaning of a particular property (or class). What the data publisher needs to do is make sure that the data they publish is documented.

There are three ways of locating the documentation about a particular property or class:

looking through the general documentation the data publisher has provided
resolving the URI of the class or property
searching

For a developer, it’s very useful to find out about a property by bunging its URI into a browser and hitting return. Want to know what http://xmlns.com/foaf/0.1/name means? Look up that URI. By comparison, if you want to know what a vevent is, your best bet is a search engine. In the results I get from Google, the microformat definition of vevent is currently second on the list. (The Microdata definition of vevent doesn’t even feature.) Even if a property isn’t available at its URI, its URI gives a more unique identifier to search for than an short term: you’re more likely to find relevant information if you search for http://xmlns.com/foaf/0.1/name than if you search for name.

But there’s no requirement for data-specific applications to use computer-readable information about properties or classes. If you know the data that’s available in a dataset, you can find out the semantics of the properties and classes it contains and hard-code those within your application. Most applications that reuse data are currently of this type, and it tends to be the only kind that non-Semantic Web people think about.

Vocabulary-specific and generic applications will have some vocabularies built in but may also operate with unknown vocabularies. For example, an application that cares about FOAF profiles is almost certainly going to want to hard-code information about FOAF rather than download its schema every time it’s used.

There are three reasons for building-in knowledge about particular vocabularies:

some information about a vocabulary simply can’t be represented in a schema or ontology; if you want special handling for particular properties, you’re going to want to hard-code it
downloading, parsing and interpreting a schema that you know you’re going to need every time you run the application is really inefficient
relying on the network to provide information about a vocabulary you know you’re going to need makes your application fragile, especially if you do not have control over the publication of the schema yourself

It’s worth noting that applications increasingly do rely on the availability of networked resources in order to operate — that’s what cloud computing is all about — but the resources are usually ones that the application developers have some kind of control over.

It helps to use URIs for properties and classes for well-known vocabularies only in as much as it means that property and class names from different vocabularies won’t clash, so you don’t have to worry about your application confusing http://xmlns.com/foaf/0.1/title with http://purl.org/dc/terms/title.

On the other hand, if data uses an unknown vocabulary, vocabulary-specific and generic applications would like to get hold of extra information. This falls into three categories:

human-readable information includes things that help with the display of data, such as human-readable labels for properties and classes; the expected datatype of the values of a property might also fall into this category
mapping information helps applications map unknown properties and classes onto known ones; for example, if http://people.example.org/ontology/fullName is defined as a sub-property of http://xmlns.com/foaf/0.1/name then the application can use or display the value of http://people.example.org/ontology/fullName in exactly the same way as the value of http://xmlns.com/foaf/0.1/name
reasoning information helps applications draw further conclusions about the resources for which there’s information based on what they already know; for example, if http://people.example.org/ontology/fullName has a domain of http://xmlns.com/foaf/0.1/Person then anything that has the property http://people.example.org/ontology/fullName must be a http://xmlns.com/foaf/0.1/Person

These are in descending order of priority: many applications will want to interact with the user in some way, in which case human-readable information is vital. Applications that have built-in knowledge about one or more vocabularies are likely to have special handling for those vocabularies, so being able to map unknown properties and classes into those known vocabularies will enhance the behaviour of the application, although it adds a bit of complexity in the implementation to do so. Further reasoning has the potential to increase the value of sparse data but again increases the complexity of implementation.

Using URIs for classes and properties provides a mechanism for applications to get hold of this extra information about unknown vocabularies. They might try four tactics, in order of priority:

look at the data they already know; the information they need about the unknown properties and classes may be included in the files they’ve already accessed (including those containing data)
look in an application-specific (possibly cloud-hosted) cache of vocabularies that the application has already downloaded
resolve the URI of the class or property by performing an HTTP GET (and add it to the application-specific cache)
look in a general-purpose cache, such as the Internet Archive or an ontology repository such as Swoogle

Robust applications will not break if they don’t manage to locate the definition of a property or class. They can certainly continue to parse any data that they come across. To create a human-readable label, they might use the part of the URI after the last # or /. It’s no loss (to the application) if they cannot perform other reasoning: they might display the data in some default way or simply ignore it.

It’s worth noting, because of the fear of DDoS attacks that some people have, that the majority of applications won’t need to actually GET property or class URIs, either because they are data-specific applications or because they only work with vocabularies that are hard-coded into them. Applications that are good web citizens will avoid DDoS attacks on popular vocabularies by hard-coding knowledge about those vocabularies and/or maintaining a cache, either locally or in the cloud, of vocabularies that have already been resolved.

Publishers

With what I’ve said above in mind, what can publishers do to help applications to understand the data that they provide?

If a publisher is only concerned about data-specific, point-to-point mashups, all they have to provide is the data itself. It will help developers if there is some documentation of the dataset and the properties and classes used within it. But data publishers who only want their data to be discoverable by people can rely on human intelligence for locating information, and for them using URIs for properties and classes may seem like overkill.

But in a linked data world, publishers should really support their data being discovered automatically via the links from other data. Here we’re talking about making life easier for vocabulary-specific and generic applications to use the data that you provide.

The vocabularies that you use within your data fall into three general categories:

well-known vocabularies are vocabularies that are commonly enough used that vocabulary-specific and generic applications are likely to have them built-in; these vocabularies tend to be useful across domains, such as FOAF, which is useful whenever you want to express information about people or organisations
local vocabularies are vocabularies that are specific to the dataset that you are publishing; you have as much control over their publication as you do over the publication of the data itself
reused vocabularies are vocabularies that you are using that are owned by other people but that do not have the take-up of well-known vocabularies; these are typically domain-specific; one example is Metalex, which is a vocabulary about legislation

As a data publisher, the first thing you can do is to use well-known vocabularies in your data wherever possible, even if you also use local or reused vocabularies to express the same properties or classes.

For example, say you have some data describing a cricket team and use http://cricket.example.org/ontology#name for the name of a member of a team, and that you mean it to be a sub-property of http://xmlns.com/foaf/0.1/name (which is itself a sub-property of http://www.w3.org/2000/01/rdf-schema#label). If you just publish the http://cricket.example.org/ontology#name property then the only way that a generic application can know that http://cricket.example.org/ontology#name can be used as a label for a resource (which is a person) is by attempting to resolve http://cricket.example.org/ontology and reasoning based on what it finds. On the other hand, if you also provide http://xmlns.com/foaf/0.1/name and http://www.w3.org/2000/01/rdf-schema#label properties, applications are no longer dependent on the network, nor on having the ability to reason, to use that information.

You could also provide mappings onto any reused vocabularies that you specialise, but this is less worthwhile given that vocabulary-specific and generic applications are unlikely to understand them either.

The second thing you can do is to include information about the properties that you use within the data that you publish. This isn’t important for well-known vocabularies (because they’re… uh… well-known) and it’s only useful for local vocabularies if you’re not publishing those vocabularies, because if someone can access your data, odds are they’re able to access your local vocabulary’s property URIs as well. But it is useful for reused vocabularies, where you can’t guarantee access, in just the same way as it’s useful to provide basic labelling information about any resources you reference.

If you’re publishing your data embeddded within a web page, as well as marking up the data, you can mark up the labels that you use for those values, which more than likely appear as headings in a table or something similar.

If you are publishing a schema or ontology that describes your properties and types, there are also things that you can do to help applications. The most important thing is to assist caches in their caching of the ontology, which will reduce the number of times that it needs to be accessed directly and help you avoid DDoS attacks: see Mark Nottingham’s Caching Tutorial. You can also reduce the number of hits on your server by using hash URIs for your property and class names and use standard load-balancing techniques to manage the traffic.

If you’re referring to reused vocabularies within your own, you can also embed information about the relevant properties and classes from those vocabularies within your own ontology. This can save applications an extra hop, and lessens the risk of the reused vocabulary disappearing (perhaps forever).

If you want to help people who might reuse your ontology, you can make the process of copying it easier by publishing it as a single file, or broken up into segments that are likely to be reused individually. At a non-technical level, it’s also a good idea to provide a announcement mailing list or a feed so that people who reuse your vocabulary can be kept up to date with any changes you make to it.

Framework Developers

Bearing all this in mind, what should I (and other framework developers) do to support the reusers of data? I think I need to make it easy for application developers to:

load in known ontologies from known locations
hard-code relevant semantics in the script
create catalogs that map known property or class names onto known locations of documents that contain details about them
use caching proxies when accessing unknown vocabularies
access vocabularies directly at the relevant URI
fallback on archives when the URI cannot be found

In other words, I need to make it easy for people to use a range of strategies for getting hold of information about a property or class, aside from simply trying to access it at its URI. I think that means that it’s better to provide a lightweight solution, giving developers the opportunity to be in control of which URIs get resolved rather than automatically downloading extra information from the URI that’s actually used for the property or class. It also means I need to provide hooks in the code that they can use to trigger that resolution.

It would also be useful, of course, for developers to be able to use information about properties and classes easily, in particular to reason with it. That kind of support is something I’ve been working on for rdfQuery. It’s not quite ready yet.

Conclusions

My (somewhat contentious) view is that we place too much emphasis on the resolvability of property and class names, and that this can put people off the idea of the Semantic Web. You can do useful things with data without resolving properties or classes. And for a large number of useful applications, being able to actually reason over the data you get at the end of a property URI would have a high implementation cost without providing a great deal of functional benefit.

Further, for data publishers, the requirement to enable the resolution of every property and class URI you use within your data just adds to the publishing burden, especially if you’re made to feel it has to resolve to some kind of grand OWL ontology.

There’s a concept in psychology of the zone of proximal development. The idea is that if someone is operating at a particular level then as a teacher you should help them to achieve something slightly above that level, rather than trying to get them to do everything straight away.

The same is true here. We need to help publishers make the small steps that they can make, one at a time, to gradually get them to full Semantic Web goodness:

publish a dataset in some kind of open format (CSV, XML etc) so that people can get hold of it
publish the data with distinct URIs for distinct resources so that people can reference them
publish the data in a machine-readable format so that people can easily reuse it
publish the data in a way that can be interpreted as RDF, with URIs for properties and types, to avoid conflicts with other vocabularies and so that the data can be “understood” even when discovered automatically
put some human-readable documentation at the end of the property/type URIs, so that developers can easily discover what your data’s about
embed machine-readable labels and descriptions for your properties/types within your data, so that applications can display it
embed rdfs:subPropertyOf/rdfs:subClassOf mappings from your properties/types to well-known properties/types within your data, so that it can be displayed in custom ways
put the machine-readable information about the properties/types at the end of the property/type URIs, so that you can update your vocabulary easily and so that other people can reuse it
add other RDFS and OWL statements about the properties/types, so that reasoners can add value to your data

The biggest leap, the one that requires the most persuasion and the most justification, is probably from simply publishing the data in a machine-readable format to using the RDF model with URIs for properties and types. But if you remove the cost of having to provide anything at the end of the URI and factor in the potential benefits you may reap in the future (as you step further up that ladder), the question becomes less “why?” and more “why not?”.

HTML5/RDFa Arguments

Jeni — Fri, 21 Aug 2009 18:28:16 +0000

When I came back from holiday, I caught up with the recent discussions around RDFa and HTML5. It’s exhausting reading so many posts repetitively reiterating the positions of people who all have the best of intentions but fundamentally different priorities. And such a shame that so much energy is spent on fruitless discussion when it could be spent at the very least improving specifications, if not testing, implementing, experimenting or otherwise in some very minor way changing the world.

The particular thread’s subject was the use of prefixes, which are used to provide a shorthand for URIs, which are used to name properties such as http://xmlns.com/foaf/0.1/firstName. It’s unquestionable, really, that prefixes are a source of problems:

with copy-and-paste, because it’s easy to lose the bindings that are used to construct the full URI
in general understandability, because people just don’t get the idea that the prefix foaf could be used for something other than http://xmlns.com/foaf/0.1/ or that http://xmlns.com/foaf/0.1/ might be bound to a prefix other than foaf
when declared using namespace declarations, because of differences in interpretation between HTML and XHTML and for the general namespaces-in-content problems that we see in XML

I think it’s generally the case that the Semantic Web community, who are used to using syntaxes such as RDF/XML and Turtle which use prefixes all the time, judge these as being less disadvantageous than the members of the HTML5 community, who are much more in touch with and concerned about the “common user”.

But underlying the arguments about the costs of prefixes are arguments about whether these disadvantages are important enough to stop

giving people shorthands for URIs and/or
using URIs when naming properties and/or
using RDF as the data model for data on the web

Fundamentally, members of the Semantic Web (capital S, capital W) community take it as a matter of faith that the correct data model to use for expressing data about things both on and off the web is RDF. If there’s anything that defines the Semantic Web community, it’s that underlying assumption. (Well, probably: I’m sure there are still some who hanker after Topic Maps.)

Further, they think it is absolutely essential that if a property such as ‘first name’ is used within a page, you can GET a URI to find interesting information about the property. For example, with http://xmlns.com/foaf/0.1/firstName, you will get

  <rdf:Property rdf:about="http://xmlns.com/foaf/0.1/firstName" 
    vs:term_status="testing" 
    rdfs:label="firstName" 
    rdfs:comment="The first name of a person.">
    <rdf:type rdf:resource="http://www.w3.org/2002/07/owl#DatatypeProperty"/>
    <rdfs:domain rdf:resource="http://xmlns.com/foaf/0.1/Person"/>
    <rdfs:range rdf:resource="http://www.w3.org/2000/01/rdf-schema#Literal"/>
    <rdfs:isDefinedBy rdf:resource="http://xmlns.com/foaf/0.1/"/>
  </rdf:Property>

which among other things includes a human-readable label that you might use in a display of information about things that have the property, and a statement about the domain of the property, from which you can tell that anything that has a http://xmlns.com/foaf/0.1/firstName property is a http://xmlns.com/foaf/0.1/Person.

Given that you need URIs for properties, members of the Semantic Web community generally think that you need a way to shorten those URIs to make them palatable to people who might be embedding metadata in their pages. The cost of that is that you have to use prefixes, with the disadvantages that I’ve outlined above, but it’s a cost that’s worth paying to gain resolvability and concision. And since it can be done in RDF/XML and Turtle and SPARQL and almost every other syntax in existence for expressing RDF, it seems unnatural not to be able to do it in the metadata embedded within webpages.

At an equally fundamental level, members of the HTML5 community are unconvinced that RDF is a necessary or useful model to use for data. They do not see how it offers significant advantages over Javascript object structures, for example.

Part of the reason for not being convinced of the utility of RDF is that members of the HTML5 community think it simply isn’t important for properties to be named with resolvable URIs. After all, microformats have illustrated that applications can derive meaning without having a machine-readable definition of the semantics of a property.

I haven’t heard them make these arguments, but they could also point out that there are vastly more mash-ups constructed with the developer having knowledge and understanding of the data that they are mashing up (and therefore requiring human-readable but not machine-readable definitions), than there are generic applications that could, or do, actually retrieve and reason with schemas or ontologies.

Some within the HTML5 community even think it is dangerous to use information from a property or class’s URI, because if the metadata within an HTML page can only be accurately understood when coupled with information from an external document, applications that are built on being able to locate that external information are in real trouble if (when) that document disappears, either temporarily or permanently.

For example, say that I have a page that contains the triple:

<http://www.jenitennison.com/#me> 
  <http://xmlns.com/foaf/0.1/firstName> "Jenifer"^^<http://www.w3.org/2001/XMLSchema#token> .

If it is the case that an application that resolves that triple also (a) retrieves the information available about the property and (b) reasons using it and any information derived from it, then having that triple in my page entails:

<http://www.jenitennison.com/#me> 
  a <http://xmlns.com/foaf/0.1/Person> ,
  a <http://xmlns.com/foaf/0.1/Agent> ,
  a <http://www.w3.org/2000/10/swap/pim/contact#Person> ,
  a <http://www.w3.org/2000/10/swap/pim/contact#SocialEntity> ,
  a <http://www.w3.org/2003/01/geo/wgs84_pos#SpatialThing> .

but this information can only be known by collecting the documents at:

http://xmlns.com/foaf/0.1/firstName
http://xmlns.com/foaf/0.1/Person (actually the same document as http://xmlns.com/foaf/0.1/firstName but I can’t know that)
http://xmlns.com/foaf/0.1/Agent (as with http://xmlns.com/foaf/0.1/Person, actually the same document as http://xmlns.com/foaf/0.1/firstName)
http://www.w3.org/2000/10/swap/pim/contact
http://www.w3.org/2003/01/geo/wgs84_pos (resolving this doesn’t actually tell me anything extra, but I have to do it all the same to check)

In these circumstances, if, say, http://xmlns.com/foaf/0.1/Person can’t be located (my connection dips out for a second or whatever) then suddenly the page’s metadata means something different, which is surely a problem in a Semantic Web.

Putting aside all that, even if you did need URIs for properties, the HTML5 community feels that the costs in usability of using prefixes to shorten URIs are simply too high to justify the benefit of concision. Why not simply use the full URI within an attribute?

In summary, you will not be able to persuade members of the HTML5 community that it’s worth paying the usability cost of prefixes until you have persuaded them:

that RDF has significant advantages as a way of modelling data over and above Javascript object structures
that the ability to resolve the URIs that are used to name properties is not dangerous, and can even be helpful
that it is ugly and tedious to have to use full URIs when naming properties

Really I’m just trying to draw attention to the fact that the HTML5 community has very reasonable concerns about things much more fundamental than using prefix bindings. After redrafting this concluding section many times, the things that I want to say are:

so wouldn’t things be better if we put as much effort into understanding each other as persuading each other (hah, what an idealist!)
so we will make more progress in discussions if we focus on the underlying arguments
so we need to talk in a balanced way about the advantages and disadvantages of RDF

or, in a more realistic frame of mind:

so it’s just not going to happen for HTML5
so why not just stop arguing and use the spare time and energy doing?
so why not demonstrate RDF’s power in real-world applications?

What You Can't Do with HTML5 Microdata

Jeni — Wed, 13 May 2009 00:57:46 +0000

Update: Fixed a couple of errors in the microdata code.

The HTML5 microdata proposal has hit the web, just days before Google announced its support for RDFa (or at least one vocabulary encoded using RDFa attributes). These are, indeed, “interesting times” for the semantic web.

Now, if you’re one of those weirdos who want to embed RDF triples within your web pages, what you’re going to care about is whether you can use microdata to do it. Those of us who have been using RDFa in anger, rather than in toy examples, know that it can be hard to map a particular set of RDF statements onto HTML content. I thought I’d take a look to see just what it would be like to create particular RDF with the HTML5 microdata proposal.

Basics

On the face of it, you can express any triple in microdata because a triple like this (Turtle):

<http://www.example.com/subject> <http://www.example.com/property> <http://www.example.com/object> .

can always, and anywhere, be expressed with (HTML5):

<span item>
  <link itemprop="about" href="http://www.example.com/subject">
  <link itemprop="http://www.example.com/property" href="http://www.example.com/object">
</span>

while a triple like:

<http://www.example.com/subject> <http://www.example.com/otherProperty> "value" .

can be expressed with:

<span item>
  <link itemprop="about" href="http://www.example.com/subject">
  <meta itemprop="http://www.example.com/otherProperty" content="value">
</span>

Of course having to use all those long, repetitive URIs is a bit of a pain and bloats out the markup, but we’d never expect this to be hand-authored, right? Right? And what we really care about is that we can express the RDF.

It’s not just the URIs that are long-winded, by the way. RDFa manages to cram a lot into each element, whereas microdata usually requires separate elements. This is an example from the RDFa specification:

<img src="photo1.jpg"
  rel="license" resource="http://creativecommons.org/licenses/by/2.0/"
  property="dc:creator" content="Mark Birbeck" />

which produces the triples:

<photo1.jpg> xhv:license <http://creativecommons.org/licenses/by/2.0/> .
<photo1.jpg> dc:creator "Mark Birbeck" .

In HTML5, I think this has to be done with:

<span item>
  <img itemprop="about" src="photo1.jpg">
  <link itemprop="http://www.w3.org/1999/xhtml/vocab#license" 
        href="http://creativecommons.org/licenses/by/2.0/">
  <meta itemprop="http://purl.org/dc/elements/1.1/creator" 
        content="Mark Birbeck">
</span>

It’s a bit more tedious, but also more obvious what’s going on. Even after handling RDFa as much as I have, I still struggle to work out when, for example, an href attribute is providing the object for a statement, and when the subject. And if you look at the London Gazette RDFa, you’ll notice many occasions where empty <span> elements are used to provide the equivalent of the inline <link> and <meta> elements shown above. (In fact, as far as I recall earlier drafts of RDFa allowed <link> and <meta> elements to be used this too.)

From what I can see, though, there are two things that the microdata proposal in its current form can’t handle: datatyping and XML literals.

Datatypes

Datatypes are important in RDF. Values of properties are often not just strings, but dates, times, integers and so on. The microdata proposal mentions using the <time> element to create values, and has this example:

<div item>
 I was born on <time itemprop="birthday" datetime="2009-05-10">May 10th 2009</time>.
</div>

The triple that you’d want to create from this is:

<> <http://www.w3.org/1999/xhtml/custom#birthday> "2009-05-10"^^xsd:date .

which makes it plain that the value is a date. However, the definition of the mapping from microdata to RDF makes it clear that the triple that’s created is:

<> <http://www.w3.org/1999/xhtml/custom#birthday> "2009-05-10" .

In other words, the value is a plain literal, not a date.

In RDFa, the datatype attribute is used to indicate the datatype of the value, so you can do:

<div xmlns:custom="http://www.w3.org/1999/xhtml/custom#">
  I was born on <span property="custom:birthday" content="2009-05-10" datatype="xsd:date">May 10th 2009</span>
</div>

It would be easy enough to say that the value of a <time> element has the datatype xsd:date, xsd:time or xsd:dateTime dependent on the syntax of its datetime attribute, but there are other times that you want typed values. We’ve used strings (as opposed to plain literals), integers and years. I wouldn’t want to rule out the use of custom datatypes such as colours (and RDF permits these). The JSON mapping could, perhaps, use an appropriate object if there is one, and otherwise use just the string value without too much loss of power.

XML Literals

Arguably less important is the lack of support for XML literals, which are values that contain markup. The example in the RDFa spec is:

<h2 property="dc:title">
  E = mc<sup>2</sup>: The Most Urgent Problem of Our Time
</h2>

which generates the triple (Turtle):

<> <http://purl.org/dc/elements/1.1/title> "E = mc<sup>2</sup>: The Most Urgent Problem of Our Time"^^rdf:XMLLiteral .

RDFa allows you to force a value as an XML literal or a plain literal using the datatype attribute. Otherwise, if the element has any element children then it’s assumed to be an XML literal, and if not, a plain literal. I think the microdata proposal could adopt the same course of action. The JSON mapping could, perhaps, result in a value which is an array or some other container for a sequence of text and element nodes.

Final Thoughts

To my mind, the HTML5 microdata proposal is unacceptable in its current form because, unlike RDFa, it can’t be used to represent all the statements that you might want to represent. If those issues were fixed, there would be pros and cons between it and RDFa. Microdata is more long-winded, but more explicit. RDFa is more arcane but doesn’t swamp the content of the page to quite the same extent.

Like a lot of people, I would have far rather seen a proposal which didn’t reinvent the wheel, but how does the old saying go: “The great thing about standards is that there are so many to choose from.” If the microdata proposal stays the course, I only hope that we’ll see consumers supporting both it and RDFa so that producers can choose which to use rather than being forced to embed both within their pages.

Evolving Standards

Jeni — Sun, 10 May 2009 20:36:52 +0000

I’ve been trying to finalise this post for a long time now, but today’s publication of an HTML5 draft that includes a new microdata section makes it all the more relevant. The long and short of it is that I am less and less concerned about the huge mess that is the HTML5 standardisation process. On the one hand, it’s a huge mess; on the other, it doesn’t matter.

I’ve been thinking about it this way: the web is an ecosystem, a rich, complex and ever-changing environment in which HTML documents live alongside browsers, developers, Javascript, CSS, spiders and robots and so on and on.

Every HTML document is like a creature living in this ecosystem. If you study the documents, you discover species and families and genera. Some may be like nautiluses — throwbacks to earlier eras when browser wars raged and nested tables were acceptable ways of laying out a page, but all the documents that you can GET are managing to survive in this modern environment in their own ways.

The kinds of documents on the web have evolved over time, as new browsers, legal regulations, and development practices open up new ecological niches. The documents we see now on the web are a snapshot that reflects that evolutionary process and the current web ecology.

What we know from the natural world is that the best way to survive in such an environment is to gradually adapt. In the natural world, the genetic make-up of an organism determines its survivability, and the mixing and mutation of genes allow organisms to react to changing conditions. The corollary to genes in the web world are elements and attributes. Some are evident across multiple organisms, essential parts of the way the web works (<p>) while others are only present in the rarest of specimens (<dir>). The elements and attributes that a page uses determine its utility to a large extent: who can view it, how its content can be processed.

What is the role of standards bodies in this rich, complex and ever-changing environment? They are like geneticists: documenting and manipulating genes in the hope of improving the survivability (utility) of the documents they effect. And only the kind of mad, hubristic geneticists you see in B-movies would attempt to

make large genetic deviations from known working organisms
commit specicide (there will always be an ecological niche for successful organisms)
create organisms that cannot evolve (because those that cannot adapt will die)

Translated to the web, I predict:

Creating grand new designs for HTML will not work (witness XHTML2); instead, it will develop through a series of small, gradual adjustments.
HTML5 cannot hope to eliminate the use of any particular markup (such as RDFa), if it’s useful for even the smallest population. Introducing an alternative that satisfies those same requirements in a better way might cause some change but the legacy will never completely die.
Even if HTML5 is not defined to be extensible, documents will be created that extend it to meet requirements we don’t even know about yet.

So, long term, I’m not worried. The web will evolve and there is space enough in it for us all. Short term, it’s a pity that so many intelligent, thoughtful and engaged people are wasting so much energy on a conflict that can’t be won.