xml

My Experience of Web Standards

Jeni — Sun, 24 Jul 2011 16:24:00 +0000

One of the things that’s been niggling at the back of my mind since the schema.org announcement is how small a role search engine results plays in the wider data sharing efforts that I’m more familiar with in my work on legislation.gov.uk, and more generally how my day job experience differs from (what seem to be) more common experiences of development on the web. In this post, I’m going to talk about that experience, and about the particular problems that I see with the coexistence of microdata and RDFa as a result.

My day job (the one I actually get paid for) is web development. The site I spend most of my time and effort on is legislation.gov.uk. This deals with complex content (UK legislation) that has to be presented in multiple formats (users love PDFs of legislation). Our aim is to make the data as reusable as possible by third parties through good, RESTful, web architecture, and we want to use open standards and open source technologies as part of the UK government’s general strategy.

legislation.gov.uk is not a global website like Amazon or eBay, but it’s not small either: it covers 60,000 changing items of legislation, providing point-in-time views for many of them, and with more added every day. It’s one of the top ten most used UK Government websites, with 2 million visits (about 10-12 million page views) each month and typically about 120 requests/second during the active times of the day. Legislation might sound like a highly specialist interest, but if you search for legislation.gov.uk on Twitter you’ll see it being referenced over and over by people who want to share what the law says.

I do not by any means claim that my experience is representative of the wider web. I know that there are large numbers of sites that deal only in data, not documents, and certainly not documents with the kind of rich semantic structure that legislation has. I offer the following discussion as a data point, partly because I can’t quite believe that legislation.gov.uk is completely unique in its requirements and partly because obviously my perspective on a bunch of issues arises from this experience.

Technology Stacks

Legislation items are complex, semi-structured documents. Their natural fit is XML (well, that’s not quite true — their natural fit would be something that allowed overlapping markup — but XML is the closest that we have). So we store it in XML in a native XML database and we use an XML toolset to query it (XQuery) and transform it (XSLT) into various formats including rendering it as PDF (through XSL-FO).

Our next step for the development of the site involves looking at legislative effects. These form a graph: one item of legislation affects other items of legislation which may in turn affect other items and so on. There are all sorts of other links between items of legislation in terms of commencements, conferred powers and so on. Particularly because we already have well-thought-through URIs for legislation, the natural fit is to use RDF to represent this graph. We already offer a SPARQL endpoint for accessing some aspects of our data, but we expect to expand and develop this over the next few months and to use it as a layer under the website and exposed for reusers, in much the same way as we use the XML database.

As a government site, we have fairly strict limits on what we can do within our web pages: we have to make sure that they’re accessible by everyone who wants to view them. We aren’t able to use technologies that are only available in the latest browsers, but that’s OK because with the kind of content we deal with, we don’t have to do anything fancy anyway. So we use pretty basic HTML and CSS and Javascript, because that’s how you deliver content to end-users on the web (as well as exposing the underlying XML and RDF, to enable others to reuse the data).

In other words, we use three web stacks for delivering legislation.gov.uk:

the XML stack, which is great for single-source publishing of documents that have more semantic structures than those supported by HTML
the RDF stack, which is well-suited for metadata about things that are identified by URIs
the HTML stack, which is absolutely necessary for delivering human-accessible content on the web

What bemuses me, because of this experience, is that sometimes it appears that the narrative around these technologies is framed in terms of an exclusive choice between them. For example, @mattur asked:

It is as if, if you use XML you cannot appreciate the utility of error-handling in HTML; or if you use RDF you cannot understand the need to represent documents in XML; or if you want to utilise HTML fully, you cannot adopt RDF’s view of data on the web. That’s simply not my experience. They each have their role on the web; supporting the use of one does not necessitate rejecting the use of the others.

It’s interesting that some of the standards that are most reviled are those that arise at the intersections, where it appears that one technology is trying to encroach on the space of another:

XHTML at the border of XML and HTML
RDF/XML at the border of RDF and XML
RDFa at the border of all three

At the same time, within legislation.gov.uk, we publish XHTML (because it’s the natural output from an XML toolchain) and create and process RDF/XML (because it gives us access to that data from within the XML toolchain). We use a small bit of RDFa in the XHTML to indicate the rights under which our information is avaialble, and don’t yet, but are thinking about using RDFa to mark up non-document semantics within our XML (to enable the XML markup to focus on the document structures that it’s good at). For all their imperfections, these intersection technologies are useful for managing cross-overs; the problems arise when they overstep their remit and people start to think that all HTML must be XHTML or all XML must be RDF/XML or all RDF must be RDFa.

Sharing Scenarios

The second thing that I wanted to explore is the experience from legislation.gov.uk of what it’s like to be a publisher who actively wants to share their data. We need to operate simultaneously at three levels in our data sharing efforts.

Large-Scale Consumer-Driven Data Sharing

The first target for our data sharing efforts are the search engines. Obviously we’re not selling anything, but we want people to be able to locate legislation easily when they want it, and we want people who have done the search to be able to see some information about the legislation so that they know that they’ve located the right item.

This is large-scale consumer (search engine) driven data sharing, typified by schema.org and Facebook’s Open Graph Protocol (OGP). There are a few very big data consumers (Google, Microsoft, Yahoo!, Facebook etc) who need to consume data from large numbers of data providers. These consumers obviously can’t understand everything, so they determine and document what syntaxes and vocabularies they do understand and expect publishers to follow.

The benefits that publishers get from a particular consumer determines which syntax/vocabulary they use; publishers who are particularly keen to show up prettily within search results will target schema.org whereas those who want to be sharable within Facebook will target OGP. Many publishers will want to target both. There is probably a driver towards eventual convergence:

publishers might push back about inserting two lots of very similar data in their pages
consumers might want to include data from publishers who haven’t specifically targeted them

although there’s likely to be a period where they coexist, much as there was for VHS and Betamax (and V2000, I know, dad) during the early days of video players.

As I discussed previously, these large-scale consumers will be driven by the data that they find in the wild, in all its messy variety. They get relatively little benefit directly from using a generic syntax, as they are really interested in only a few, pretty generic, vocabularies for which they have hardwired processing. Indirectly, adopting a generic syntax has benefits in that publishers might find it easier to find tools that enable them to generate it, tutorials about how to use it, and feel that they aren’t being quite as locked in to something proprietary. However, rejecting data that isn’t marked up properly using that syntax has no benefit for consumers except in so far as it makes them feel that they are being good community members.

This is the pattern we see with schema.org (which accepts microdata but, based on its documentation, won’t reject data that isn’t fully compliant with it) and with OGP (which accepts a subset of RDFa but doesn’t reject data that hasn’t got prefixes properly bound, for example).

Another point to mention is that there is very little trust in this scenario. The communication between consumers and publishers is very limited, and the consumers will want to protect themselves against accidental or malicious errors that are evident in mismatches between explicit metadata and that which is parsed from the visible content of the page.

The parallels to HTML and browser vendors are very strong in this type of data sharing.

Small-Scale Consumer-Driven Data Sharing

A second type of data sharing is again driven by consumers, but this time at a lot smaller and more specialised scale. For legislation.gov.uk, these are services such as GLIN, which is a global legislation registry. Other examples are the recent work that we’ve done to publish UK Government organograms or Chris Taggart’s Open Election Data project. In these cases, there’s a single, relatively small and specialised consumer and a small number of publishers which are closely coordinated together.

As in the large-scale case, the consumer ultimately determines the syntax/vocabulary that it recognises, and communicates that to the publishers. However, small-scale consumers typically have close coordination with the publishers, which has a number of side-effects:

consumers may be more able to both apply pressure to and help publishers to do well in their markup
publishers have the opportunity to feed back directly to the consumer any suggestions that they have about changes to the syntax/vocabulary
publishers are likely to gain an immediate and tangible benefit from their cooperation, such as visualisations of their data that they otherwise wouldn’t have seen

Another noteworthy point about small-scale consumers is that they’re unlikely to have the engineering capability to create a custom parser for a particular syntax, but will instead want to use something off-the-shelf to extract data from pages and into their own backend systems. This, coupled with the closer coordination with publishers, means that they’re much more likely to stick to a specification, assuming that the off-the-shelf tools do.

Publisher-Driven Data Sharing

The final type of data sharing is driven by publishers. At legislation.gov.uk, we’re motivated to make our data available for reuse for transparency/accountability reasons (to help citizens understand the law), efficiency reasons (to help parliament and government departments to publish new legislation better) and economic reasons (to foster innovation in legal publishing). We don’t have any individual consumers in mind when we publish our data, but have found that simply by publishing it well, we foster reuse.

In this case, we as publishers are highly motivated to ensure that the data we publish is easily parsed with something off-the-shelf, since that lowers the barrier for potential consumers. Publishers like us are very likely to have unique, specialised, content and need to use a vocabulary that fits closely to our internal data structures as this lowers implementation cost. Consumers can also trust publishers like us: we simply have no motivation to lie in the data that we provide for reuse.

Mixed Markup

As I’ve outlined above, publishers like legislation.gov.uk need to target several potential consumers at the same time:

large-scale consumers such as search engines
small-scale consumers that provide us with a useful service
specialist consumers that are interested specifically in our data

We cannot use a single vocabulary for all these different purposes. (Well, we could write our own vocabulary and describe mappings to other vocabularies using RDFS, but search engines wouldn’t read it.)

We must therefore use a mix of vocabularies:

generic vocabularies about things that search engines care about
specialised vocabularies for particular small consumers
site-specific vocabularies for sharing our unique data

It’s repetitive, but it’s manageable so long as we have a syntax that enables us to say that an item of legislation is a http://scheme.org/CreativeWork and a http://purl.org/dc/dcmitype/Text and a http://www.legislation.gov.uk/def/legislation/Legislation and allows us to give multiple properties the same value.

The way things are going at the moment, we might well end up having to use multiple syntaxes on the same page, as some consumers understand microdata, others consume RDFa, and still others will parse microformats. This leads to more repetition: adding itemprop for microdata, property for RDFa and specialised class attributes for microformats. But worse (much worse), each of the syntaxes uses a different parsing model to create an entity-property-value structure, so not only do we have to learn substantially different markup patterns but our pages quickly become some kind of hideous polyglot mess trying to balance between them.

Looking Forward

As I said at the start, I’m fairly sure that my experience at legislation.gov.uk isn’t representative of the wider web, but I don’t have a clear idea about just how unrepresentative it is, in terms of technology use or motivations around data sharing. When I read my twitter stream or blogs, there’s a massive sampling bias, both in terms of who I follow and what I read, but also about who talks about what they’re doing. (I’m reminded of Jeff Atwood’s post on the Two Types of Programmers: the vast majority of web developers don’t make a noise about what they do.)

Taking part in web standardisation today often feels like being part on an ongoing cold war between distinct camps rather than a community working towards common aims. The underlying question seems to be “who’s side are you on?” Every decision and activity is cast as a victory or defeat. Time is wasted on attack and defence, or on raking over past slights and stupidities, rather than on progress. Valid criticism from outside a group cannot be listened to for fear of giving ground, cannot be made within a group where it seems like betrayal.

It is the Robbers Cave Experiment played out in web standards. As a psychologist, I find it fascinating. As a developer, and particularly one who doesn’t self-identify with any single group, it is frustrating. As a TAG member, trying to work for the longer-term good of the web, it is worrying, because situations of intergroup conflict lead to groupthink and non-optimal solutions.

As I described above, a non-optimal outcome seems to be the most likely result of the particular microdata vs RDFa conflict for us at legislation.gov.uk. While I know we are not generally representative, I believe that it will be similarly bad for other developers: publishers, consumers and tool implementers.

This is a problem for all who want to foster data sharing on the web using open standards; it is not one that any one group can fix on their own. It’s my hope that a balanced task force of individuals with a variety of experience and backgrounds can provide a focus for us all to work together to solve it. If we can’t, then we have let our prejudice and bias overcome our judgement.

XML Summer School 2009

Jeni — Thu, 25 Jun 2009 21:03:14 +0000

Registration has just opened for this year’s XML Summer School, held in Oxford on 20-25th September. I’m teaching a couple of sessions and helping with a workshop on the “XSLT, XSL-FO and XQuery” track along with Bob DuCharme, Michael Kay and Priscilla Walmsley. It’s one of my favourite events, for three reasons:

I get to listen to experts talk about particular technologies in depth. The sessions are particularly good because they’re provided by people who don’t just spend all their time training, but actually practice what they’re talking about, and therefore positively relish the kinds of discussions that normal trainers might shy away from.
I get to meet a whole bunch of people who are using XML in different areas: publishing, healthcare, government, you name it. In that way it’s like a conference: many of the most useful conversations happen during the breaks or at the bar.
I get to go punting, visit Oxford’s best pubs and dress up for a formal dinner — more social engagements in a single week than I usually have in a year!

I know a lot of beginners go to the XML Summer School for the introduction course, but to me the real value is for people who are actually using XML on a day to day basis and want to keep on top of the latest tools and technologies that will actually help them do their jobs. I learn something new every year.

Anyway, I wanted to blog about it because there’s a discount on registration up until 30th June. Grab ‘em while you can!

My Perfect XML-Based Publishing Platform

Jeni — Fri, 29 May 2009 21:40:20 +0000

For the last several months, I’ve been working on a project at TSO for publishing UK legislation using a native XML database (eg eXist or MarkLogic Server) with some middleware (eg Orbeon or Cocoon). It’s a powerful and flexible approach that’s built on declarative languages like XQuery, XSLT, and XML pipelines; you can see it in action with the Command and House Papers demo.

But the killer platform isn’t quite here yet, partly because the specs aren’t quite done. Both Orbeon and Cocoon use XML pipelines, but they use different languages to define them; XProc is just around the corner. XML databases are all over the place in their conformance to XQuery, its optional features and the not-quite-finalised specs for free-text searching and updating.

People talk about how productive you can be using Ruby on Rails or Django, and they work great for publishing data you can store in a relational database. What we need is a similarly easy-to-use platform for document-oriented, XML-based content. This is my wish-list.

The killer platform would have a configuration mechanism for mapping HTTP requests that it receives onto XProc pipelines. The pipeline that would be used could be based on one or more of:

the HTTP method
the requested URI
any HTTP header

The pipelines would have a signature like:

a primary ‘source’ input that encodes the HTTP method, headers and body of the request; this would use the <c:request> element used within the p:http-request XProc step
a parameter input populated by parsing the URI against a simplified version of URI templates
a primary ‘result’ output that is an XML version of the response body
a ‘response’ output that encodes the HTTP status code and headers of the response; this would use the <c:response> element used within the p:http-request XProc step
a ‘serialize’ output that holds a <c:parameters> element containing parameters for serializing the result body; possible serialisations would include serialising XSL-FO as PDF and SVG as JPEG, for example.

The pipeline engine would of course include efficient implementations of all the required steps, most importantly XSLT 2.0.

The platform would have an easy mechanism for invoking queries on its XML store through an implementation-defined step that was similar to the p:xquery XProc step. The step might have the signature:

a primary ‘query’ input for the query itself (like the ‘query’ input for p:xquery)
a parameter input for specifying the values of external variables within the query
a ‘database’ option for specifying the database to query
a primary ‘result’ output for the result of the query, this being a sequence of documents resulting from the query

The XML store itself would support:

XQuery 1.0, with no extensions to the syntax except those permitted by that specification
XQuery and XPath Full Text 1.0
XQuery Update Facility 1.0

It would also support setting up indexes on any expression for a particular kind of context node (usually an element); these would work like keys in XSLT, except that the XQuery engine would automatically detect when the index could be applied. For example, it would be possible to set up a key on a document for the expression substring(//dc:identifier, 7, 2) and if the query used exactly this expression, the index would be used.

The platform would provide an extensible architecture such that it would be possible to set up replicated XML store(s) on separate servers from the main pipeline engine. It would cache the results of queries against the XML store. It would serve up static content such as images and scripts bypassing the pipeline. It would be configured using files, so that it was easy to transfer a configuration between development and production platforms and to version control configurations through normal means.

Have you used (or developed!) anything that comes close? What’s on your wish-list?

Your Website is Your API: Quick Wins for Government Data

Jeni — Sun, 01 Feb 2009 09:28:57 +0000

This is the talk I prepared for the UKGovWeb Barcamp, in blog form. It’s probably better this way. Most of what’s written here seems blindingly obvious to me, and probably to most readers of this blog, but maybe Google will direct someone here who finds it useful.

Working with public-sector information on the web, one of the things that I take an interest in is making government data freely available for anyone to re-present, mash-up, analyse and generally do whatever they want to do. This post is born out of a feeling that the people who control data don’t realise that the smallest changes can be beneficial: they don’t need to do everything right now, just something.

There are three fundamental things that you need to do:

identify the data that you control
represent that data in a way that people can use
expose the data to the wider world

but you can choose the degree to which you do each of these things.

Identify

Take a look at what data you have some kind of responsibility for or control over. You might be a PDF containing a table of schools in the local area and their intakes over the last couple of years. You might have a spreadsheet of the amount of money assigned to maintaining the playgrounds within the borough. You might have a database of company information. You might have a set of HTML agendas for court cases.

The first step is simply to identify what the information is about. Schools, playgrounds, companies, court cases — each row in your table or spreadsheet or database, or each section in your document will be about something. We call this a resource.

To play nicely with the web, every resource should have an identifier. A Uniform Resource Identifier. A URI. That URI tells us where we can find information about the resource (we’ll get to what those look like later). So your second step is to work out URIs for each of your resources.

Now, there are actually three levels of URIs that you can care about:

identifier URIs
document URIs
representation URIs

You probably already have document and representation URIs on your web server. Representation URIs are URIs for particular formats and languages and views of the information that you make available. Document URIs are typically the same URI without an extension; web servers use content negotiation to work out which representation to serve up when a web browser asks for the page at a particular document URI.

So you already have a URI for the PDF that contains the table of schools, for the Excel spreadsheet about the playgrounds. You already have URIs for the results of a particular query on your database, and of course the HTML pages that you deliver have URIs already. That’s all in place. You don’t want to change it.

But identifier URIs are what are really important when it comes to opening up your data. They shift the focus from the documents that you serve to the resources that they are about. By assigning URIs to resources, you enable other people to talk about them. Even if that’s all you do, you have done good.

For example, if Companies House stated that companies could be referred to using URIs of the form http://www.companieshouse.co.uk/id/company/{registeredNumber} then other people who needed to talk about companies (websites containing customer feedback, monitoring companies going into receivership, displaying stock price information, whatever) could use these URIs whenever they referred to a company. If all websites that make data available about companies point to the same identifier for a company, then it’s possible to pull that data together very easily.

Now the URIs that you use should be short, clean, readable, hackable, hierarchical and so on. If you can, you should use a natural identifier for the resource within the URI for that resource. So URIs for registered companies should use their registered number. URIs for schools should use the school’s unique reference number (URN). URIs for playgrounds could use the name of the playground (scoped within the council responsible for the playground). URIs for court cases should include the court, the year, and the case number. And so on.

Remember as you’re creating these identifier URIs that they are nothing to do with the structure of your website or the user’s experience of navigating through your website. For navigation, you might want to group schools into primary, secondary and sixth-form, but you shouldn’t do that in the identifier URIs. To help decide, imagine someone wanting to construct a URI and the information that they need to do so. If any of the information they need can be derived from other information (as a school’s type can be derived from its URN), leave it out.

When you’re doing this, you might realise that actually you shouldn’t be the one in control of these URIs. If you’re not the one assigning the registered number, URN or case number then there’s probably a higher authority that does assign those (real-world) identifiers. Don’t let that stop you creating URIs — you’ll still find them useful for identifying your information about that particular resource — but do look to see if there are existing URIs that you could point to and reuse whatever scheme they’re using if there are.

Represent

So I said in the last section that assigning URIs to resources was useful. And it is. But it’s even more useful if you provide some kind of response when someone requests those URIs. A request for a URI can be done by a web browser or one of those search-engine-spider-things that crawls the web looking for data. Requests are done on the web using HTTP (hypertext transfer protocol), specifically using a GET request, which means “get this resource”.

When a web server receives a request, it sends back a response. The first part of the response is a status code that tells the browser, spider, or whatever issued the request, generally what kind of response it is. Now when a browser says “get this company” or “get this school” a web server should either respond with a 404 Not Found response or a 303 See Other response.

If the company or school doesn’t exist, a web server should respond with a 404 Not Found response. It’s actually really useful to give appropriate 404 Not Found responses, because it tells whoever made the request that the resource (company/school/playground/court case) doesn’t exist. This can act as simple validation: if I’m building a site that parents can use to rate schools, and a parent enters a URN into a form, I can construct a URI based on that URN, try to GET the information about that school, and if I get a 404 Not Found response then I know that the parent has entered an invalid URN.

If the company or school exists, a web server should respond with a 303 See Other response that points the browser to a document URI that contains information about the company or school. After all, the web server can’t very well deliver the company or school itself into your lap; all it can do is give you information about it. 303 See Other means “if you want information about that, see that other thing over there instead”. The “other thing over there” will be a document of some kind. It might be the PDF that contains information about the school, or the spreadsheet that contains information about the playground.

Simply giving a yes-this-exists or no-this-doesn’t-exist response is useful. Even if that’s all you do, you have done good.

It’s even more useful, though, if you can make the information that you have about the school, playground, company, court case or whatever, available in a format that can be processed by a computer reasonably easily. PDFs are really really hard to extract information from, so do everything you can not to use PDFs. Word documents and Excel spreadsheets are next worse; if you have to use them, keep them really really simple and definitely don’t use Word Art or embed images to display your data.

You should always make your data available in HTML. Try to make it as clean and regular as you can; use microformats to indicate information about people, places and events. If you want to push the boat out, use RDFa to mark up the data in your page even more explicitly.

The great thing about HTML is that it’s human readable as well as (if you do it well) machine readable. You can also make your data available in explicitly machine-readable forms as well if you want: XML, JSON, RDF/XML, whatever floats your boat. If there are already standard formats or ontologies for the kind of data that you’re making available, then use them, certainly, but it’s very likely that there aren’t. And in comparison to the nightmare of extracting anything useful from a PDF, it’s easy to transform between different formats, so you only have to concern yourself with different formats if you want to.

If you do provide multiple formats for your data, you should use server-driven content negotiation to deliver the data in an appropriate format to whatever’s requesting it. So a web browser will request HTML; a semantic web crawler will request RDF/XML; a Javascript program will request JSON and so on. The 200 OK response that the web server sends with your data should include a Content-Location header that gives the representation URI of whichever format is being returned, and a Vary header that tells caches how it’s decided which representation to serve up.

Expose

All the good work identifying resources and representing them comes to naught if you don’t expose it. You can (and should!) tell other people about the URIs that you’ve developed, but the best way to give them exposure is to use them yourself, within your website. Simply using the URIs within your website gives them exposure. Even if that’s all you do, you have done good. People who are interested in linking to you will look at your site and they will learn about your URI scheme from your use of it.

The identifier URIs that you’ve created might not be particularly easy to generate. For example, with the URI scheme that I suggested above for Companies House, unless you happen to know that Tesco Plc’s registered company number is 00445790, you’re not going to be able to get to information about them. So you should have a way of searching based on something that people will know, such as the name of the company. Use an HTML search form that makes GET requests like

http://www.companieshouse.gov.uk/company?name=Tesco Plc

The response should be a 302 Found that redirects (using the Location header) to the true identifier URI for the company (http://www.companieshouse.gov.uk/id/company/00445790). If it’s not possible to identify a single resource from the search string (for example, there are lots of companies with ‘Tesco’ in their name), then the correct response is a 300 Multiple Choices that provides a list of links to the possible URIs (in HTML).

There are other ways to help people find your data. If there aren’t gazillions of resources, you can list the URIs within your sitemap, which will make them discoverable by search engines. You can also list them on web pages and, especially for data that’s constantly updating, in (Atom) feeds which you link to from your HTML pages. Use metadata within the pages and feeds to help the consumers of your data work out what’s relevant to them.

To help even more, slice your Atom feeds into portions that different consumers of your data are going to be interested in. Slice by type, by area, by subject. That way people can stay up to date with just the resources that they’re interested in, and not be bothered with information about those that are irrelevant to them.

That’s It

What I’ve tried to describe here is the minimum that you need to do to help people use the information you have, and some of the other things that you can do to make it even more useful. Here are some things that you shouldn’t do:

don’t wait for someone else to define a URI scheme for the things that you want to talk about
don’t wait for someone else to define an XML schema or RDF ontology for your data
don’t wait until you can find the time and money to do it all “properly”

Just do what you can, now.

Working With Fragmented Overlapping Markup

Jeni — Sun, 28 Dec 2008 20:15:56 +0000

In my last post I talked about different techniques for representing overlap within XML. One technique is fragmentation. In the work that I’ve been doing, I’ve been using milestone-based formats similar to ECLIX, but my eyes were opened at the GODDAG workshop: fragmentation would make overlap so much easier to process in XSLT, especially when dealing with localised overlap such as revision or comment markup.

But how could fragmentation be used with full-on overlap? I had a little play and came up with some XSLT to demonstrate.

Fragmentation Example

First, an example of how to represent overlap using fragments. Using fragments for overlap comes straight out of the support for aggregation within TEI where they’re used to not only represent overlapping structures but to construct completely new “virtual” elements that are not necessarily contiguous (and may even contain fragments in different orders from how they appear in the text). In TEI, they usually use the next and prev attributes to point from one fragment to another in order to reconstruct the element.

In the example here, I’ve done something slightly different, namely to use an ID in the http://www.jenitennison.com/xslt/fragmentation namespace to link the elements: all elements with the same ID are actually the same element. Here’s what it looks like.

<book xmlns:f="http://www.jenitennison.com/xslt/fragmentation">
    <page f:id="page199" n="199">
        ...
    </page>
    <poem>
        <page f:id="page199" n="199">
            <title>
                <pl>Recueillement</pl>
            </title>
            <stanza>
                <sl><s><pl>Sois sage, ô ma douleur, et tiens-toi plus </pl>
                                                          <pl>tranquille.</pl></s></sl>
                <s>
                    <sl><pl>Tu réclamais le Soir; il descend; le voici:</pl></sl>
                    <sl><pl>Une atmosphère obscure enveloppe la ville,</pl></sl>
                    <sl><pl>Aux uns portant la paix, aux autres le souci.</pl></sl>
                </s>
            </stanza>
            <stanza>
              <s f:id="s3">
                  <sl><pl>Pendant que des mortels la multitude vile,</pl></sl>
                  <sl><pl>Sous le fouet du Plaisir, ce bourreau sans merci,</pl></sl>
                  <sl><pl>Va cueillir des remords dans la fête servile,</pl></sl>
                  <sl><pl>Ma douleur, donne moi la main; viens par ici,</pl></sl>
        </s>
            </stanza>
            <stanza>
                <sl><pl><s f:id="s3">Loin d'eux. </s><s f:id="s4">Vois se pencher les défuntes </s></pl>
                                                                        <pl><s f:id="s4">Années, </s></pl></sl>
                <s f:id="s4">
                  <sl><pl>Sur les balcons du ciel, en robes surannées; </pl></sl>
                  <sl><pl>Surgir du fond des eaux le Regret souriant; </pl></sl>
        </s>
            </stanza>
        </page>
        <page f:id="page200">
            <stanza>
              <s f:id="s4">
                  <sl><pl>Le Soleil moribund s'endormir sous une arche, </pl></sl>
                <sl><pl>Et, comme un long linceul traînant à l'Orient, </pl></sl>
                <sl><pl>Entends, ma chère, entends la douce Nuit qui </pl>
                                                               <pl>marche.</pl></sl>
                </s>
            </stanza>
        </page>
    </poem>
</book>

You can see that I’ve played pretty fast and loose here with the markup language. The <s> elements can be children of <stanza> or <sl> or even <pl>, purely depending on what happens to most neatly contain them. This makes the XML inconsistent, but less verbose than it would otherwise be. Elements that are actually fragments have f:id attributes, and multiple elements may have the same f:id; this is precisely what’s used to work out that they’re the same element.

Desired Rendering

So what would we like to do when processing this? Say we wanted to create an HTML rendition of the poem, looking something like:

page 199

Recueillement

Sois sage, ô ma douleur, et tiens-toi plus

tranquille.

Tu réclamais le Soir; il descend; le voici:

Une atmosphère obscure enveloppe la ville,

Aux uns portant la paix, aux autres le souci.

Pendant que des mortels la multitude vile,

Sous le fouet du Plaisir, ce bourreau sans merci,

Va cueillir des remords dans la fête servile,

Ma douleur, donne moi la main; viens par ici,

Loin d’eux. Vois se pencher les défuntes

Années,

Sur les balcons du ciel, en robes surannées;

Surgir du fond des eaux le Regret souriant;

page 200

Le Soleil moribund s’endormir sous une arche,

Et, comme un long linceul traînant à l’Orient,

Entends, ma chère, entends la douce Nuit qui

marche.

The logic behind this rendition is:

process the pages; for each one, create a horizontal rule followed by a paragraph giving the page number
process the parts of the poem within each page; give the title if it has one in this fragment, followed by the stanzas
create an ordered list for each stanza, starting at the number for the stanza line within the (whole) poem, and process the stanza lines
create a list item for each stanza line; if the line contains parts of two sentences and the first of these sentences doesn’t begin in this line, highlight it as this indicates an interesting overlap between prosodic and syntactic structures
process the page lines within each stanza line; if there’s more than one, align the second to the right

It wouldn’t be easy to express that logic against the fragmented XML above, for two reasons.

First, the fragmented markup above is inconsistent: you can’t tell what kinds of children a particular element will have and which elements will be fragmented. You could fix this in the markup by deciding, for example, that the prosodic hierarchy of book/poem/stanza/sl would be primary and all other elements fragmented as necessary; you could further decide which of the hierarchies would be secondary within this: whether an <sl> element would hold <s> or <page> elements as children.

Second, though, the different logical steps require the markup to be structured in different ways: 1 requires a physical hierarchy where the markup is primarily divided into pages; 2 and 3 require a prosodic hierarchy within the page, dividing the poem into stanzas and stanza lines; 4 requires a syntactic hierarchy, where the stanza lines are split into sentences; 5 requires switching back to the physical hierarchy to see the page lines within the stanza line.

What you can do (and what I’ve done) is to write a function to help with this kind of processing by switching between the different hierarchies as and when necessary.

Labelling Elements

To prepare for switching, you must annotate the elements in the document with an indication of the trees that they belong to. The trees can be called anything you like; for the example above, I could use the labels “physical” (book, page, page line), “syntactic” (book, poem, sentence) and “prosodic” (book, poem, stanza, stanza line). The idea of labelling elements based on a tree that they belong to comes from the multi-coloured trees technique, but I think it’s more useful to use meaningful labels if you can.

You could imagine a built-in extension element that allowed you to describe the trees that different elements belonged to, and the annotation happening at the level of the XPath Data Model as its created. But to make things easier I’m using a f:trees attribute on each element. Adding the attribute can be done in XSLT with code like:

<xsl:template match="*" mode="annotate">
  <xsl:copy>
    <xsl:attribute name="f:trees">
      <xsl:apply-templates select="." mode="trees" />
    </xsl:attribute>
    <xsl:copy-of select="@*" />
    <xsl:apply-templates mode="annotate" />
  </xsl:copy>
</xsl:template>

<xsl:template match="book" mode="trees">prosodic syntactic physical</xsl:template>
<xsl:template match="poem" mode="trees">prosodic syntactic</xsl:template>
<xsl:template match="title | stanza | sl" mode="trees">prosodic</xsl:template>
<xsl:template match="s" mode="trees">syntactic</xsl:template>
<xsl:template match="page | pl" mode="trees">physical</xsl:template>

Once elements are labelled with the trees they belong to, it’s possible to work out dominance hierarchies. An element A is a descendant of an element B if the elements share a tree and A starts and ends within B. If A is within B but they don’t appear in the same tree, then the containment is happenstance and does not imply dominance.

Note: Trees should be defined so that all the elements within a given tree fit within each other without fragmenting. I haven’t considered how self-overlap should be handled here; the elements need to be part of the same tree, but they can still overlap and therefore be fragmented even when that particular tree is primary. In my experience, self-overlap usually occurs in situations like comments or revision markup, in which the self-overlapping markup is never primary anyway, so I’m not sure how serious this issue is.

Swapping Hierarchies

Once the elements are annotated, it’s possible to swap between hierarchies. The function I’ve written — f:swap() — takes two or three arguments. The first is an element, and the f:swap() function returns this same element (actually a copy) but with its children, and possibly its parents, rearranged based on the trees listed in the second argument. The third argument defaults to the element specified as the first argument and provides a starting point from which the rearrangement takes place; the two most useful values for this argument are the element itself (which means that its children are restructured) and the root of the tree (which means that the entire document is rearranged).

Some examples will help make this clearer. Starting with the poem above, to get the rendering I want, I need to swap to a “physical” view and process the pages:

<xsl:apply-templates select="$annotated/book/f:swap(., 'physical')/page" />

The f:swap() call here returns the <book> element but with its descendants rearranged so that the physical hierarchy is primary. The new version of the <book> element will have <page> children, which will themselves have <pl> children. The <pl> elements will contain fragments of <poem>, <s>, <stanza> and <sl> elements, nested purely based on their happenstance containment within a particular <pl>.

Here’s the code for processing the <page> elements:

<xsl:template match="page">
  <hr />
  <p style="text-align: right; ">page <xsl:value-of select="@n" /></p>
  <xsl:apply-templates select="f:swap(., ('prosodic', 'syntactic'))/poem" />
</xsl:template>

So each <page> generates a horizontal rule, a paragraph containing the page number and then processes… here the switch is from the physical hierarchy to a prosodic/syntactic hierarchy. The list of two items as the second argument of f:swap() means that the primary hierarchy is prosodic (poems, containing stanzas, containing stanza lines), but once you reach the bottom of the prosodic hierarchy (the stanza lines) you switch to a syntactic hierarchy (sentences) rather than a physical hierarchy (page lines).

The fact that the f:swap() call above only has two arguments means that the rearrangement starts from the <page> element that’s being processed. The ancestry of the <page> element itself stays the same, and only its content is rearranged according to the views specified in the second argument. So in this case the <poem> elements that a given <page> contains will be fragments.

Processing the poems can continue in the normal way:

<xsl:template match="poem">
  <xsl:apply-templates select="title" />
  <xsl:apply-templates select="stanza" />
</xsl:template>

<xsl:template match="title">
  <h3><xsl:value-of select="." /></h3>
</xsl:template>

The next difficulty appears when I want to start the numbering for a particular stanza based on the number of the first line within the stanza. I’m doing this by setting the start attribute like this:

<xsl:template match="stanza">
  <ol>
    <xsl:attribute name="start">
      <xsl:number select="f:swap(., 'prosodic', /)/sl[1]" 
        count="sl" from="poem" level="any" />
    </xsl:attribute>
    <xsl:apply-templates select="sl" />
  </ol>
</xsl:template>

This illustrates the three-argument version of the f:swap() function. To number the stanza line, I need to know the number of that stanza line within the poem that contains it. That would be easy to do with <xsl:number> (or in other ways), but for the fact that the <poem> element the <stanza> element appears in is currently fragmented between two <page> elements. To work out the number of the line, I really want an XML document in which the physical hierarchy is completely ignored, and the elements are arranged book/poem/stanza/sl.

The three-argument version of f:swap() allows me to swap to a prosodic hierarchy, starting from the very root of the document. It returns the element given as the first argument as it appears within the new hierarchy. Unlike the two-argument version, which only affects the descendants of the first argument, the three-argument version may also affect its ancestors, and even merge the element if it’s originally fragmented or split it if it doesn’t appear in the primary hierarchy. In this example, the returned <stanza> element’s parent <poem> is a child of the <book> element rather than being a fragmented child of the <page> element.

The rearrangement for the purposes of computing the start number for the list doesn’t affect the tree that’s being processed; the template for the <stanza> elements goes on to process the <sl> elements it contains, which use this template:

<xsl:template match="sl">
  <li>
    <xsl:if test="count(s) > 1 and not(f:first(s[1]))">
      <xsl:attribute name="style">background-color: yellow; </xsl:attribute>
    </xsl:if>
    <xsl:apply-templates select="f:swap(., 'physical')/pl" />
  </li>
</xsl:template>

Recall that the hierarchy currently being processed is a prosodic/syntactic hierarchy. The <sl> elements contain <s> elements, and it’s therefore possible to check whether the <sl> element being processed contains more than one <s>. The f:first() function checks whether a given fragment is the first fragment of that element, so the test in the <xsl:if> in this template checks whether the <sl> contains more than one <s> and the first <s> is not the first fragment of the sentence it represents.

To get the rendering I want, I need to generate an HTML paragraph for each page line within the stanza line. Currently the <sl> elements contain <s> elements, so to get the page lines I need to switch once more to the physical hierarchy and process the <pl> elements that are children of this <sl>. That processing is done by the template:

<xsl:template match="pl">
  <p>
    <xsl:if test="preceding-sibling::pl">
      <xsl:attribute name="style">text-align: right; </xsl:attribute>
    </xsl:if>
    <xsl:value-of select="." />
  </p>
</xsl:template>

and we’re done.

Final Thoughts

The one thing that concerns me about the approach I’m taking is the fact that because XSLT can’t actually amend an existing tree, the f:swap() function essentially makes a copy of the entire tree every time you use it, and I don’t know how well that will scale (both in terms of memory and in terms of work copying elements) when you get to documents that are larger than this toy example. Maybe processors are clever enough to discard trees they no longer need so it won’t be an issue; I just don’t know.

Other than that, I think this approach is promising because it enables users to mostly use familiar tree-processing approaches rather than having to learn new paradigms for transforming overlapping markup or introducing a raft of new axes.

Free Our Bills

Jeni — Mon, 31 Mar 2008 19:10:14 +0000

The Free Our Bills campaign was launched recently in the UK. Some of the comments I’ve seen about the campaign makes me think that it might be helpful if people understood more about how Bills and legislation get published in the UK. I thought I’d offer a bit of background based on my experience (though there are many people with more intimate knowledge of the processes involved; perhaps they’ll correct me when I get it wrong).

Bills are draft legislation that is under discussion within the House of Commons or House of Lords. A Bill becomes law (legislation) when it is enacted.
Bills are published by Parliament and are available on the Parliament website. Legislation is published by The Stationery Office (TSO) under contract to the Office of Public Sector Information (OPSI) on the OPSI website.
Bills are changed (amended) as they progress through the Houses of Parliament. People are mostly interested in the most recent version of a Bill. Legislation can be changed (amended) by other legislation; the version of a piece of legislation with all the changes applied to it is known as consolidated legislation. Consolidated legislation is published in the Statute Law Database as well as (too a more limited extent) on the OPSI website.
Bills are edited by a dedicated team of Parliament employees who must reflect the amendments that the MPs say they want to make. They use a WYSIWYG XML editor. As is usual in an environment that has only been concerned about printed copies for centuries, they tend to focus on appearance rather than semantics, even when the XML supports the semantics.
The Free Our Bills campaign is not about making Bills (or legislation) easier for humans to read and understand, it’s about making it easier to extract information from a Bill so that people can be notified when a new Bill comes along on a subject they care about, or an old Bill is redrafted, and so on.
Bills are already available for the public to view on the web, in PDF and HTML forms. The problem is that the HTML is Really Really Bad (View Source to see) and that makes it Really Really Hard to extract useful information from them.
There are reasons for the Bills HTML being Really Really Bad:
- The HTML must look exactly like it does in printed form, otherwise Members of Parliament (MPs) would get Really Really Confused.
- MPs refer to pieces of a Bill (which they might want to change) by page and line number, not by the semantic structure of the Bill, so the HTML must have page and line numbers in it or MPs would get Really Really Confused.
- Although the formatting of Bills is pretty consistent, there’s always the chance that a piece will need to be formatted specially. It might be safe to assume a particular presentation for a particular semantic 99% of the time, but if that 1% isn’t formatted in the different way, MPs would be Really Really Confused.
- The code that creates the Bill HTML was written several years ago, when browser support for CSS was Really Really Bad.
The picture for legislation is rather better because a strategic decision was made to focus on semantics rather than presentation. When a Bill is enacted, it gets converted into reasonably good semantic XML, which forms the basis of all the HTML views. It also helps that this HTML was designed fairly recently, for modern browsers; it makes heavy use of CSS so there’s relatively little obfuscation of the content.

I think there are interesting general lessons here:

Different user communities have different requirements. MPs have different requirements from Bills from the general public, who don’t care (as) much about line or page numbers. On the other hand, you need to actually consult with users about what they need rather than make assumptions about it: are MPs really likely to get Really Really Confused if the HTML presentation of a Bill looks slightly different from the PDF print version? I don’t know.
Authors don’t care about what they don’t use. When the only way of using a Bill is to print it, it’s natural that authors and publishers only care about how it looks when it’s printed. Training people to care about semantic markup is really hard, and it’s made harder by WYSIWYG tools that allow them to override the semantic style. If a difference isn’t visible, then in author’s eyes it doesn’t exist.
You have to positively decide to ignore appearance. When transforming from a WYSIWYG view, replicating appearance is the obvious thing to do. But it’s worthwhile in the long run to focus on extracting the semantics, because the resulting documents are so much more reusable.
HTML, XML and XSLT are not inherently good. Parliament wanted Bills in HTML so that they were more accessible on the web. But the HTML is dreadfully inaccessible because of the other requirements placed on it. Similarly, XML can be incredibly obfuscated, or entirely about presentation, as formats such as OOXML illustrate. And just because your code is written in XSLT does not make it inherently easier to maintain then (say) a SAX transformation. It’s easy to misuse a technology.
Developers who produce atrocious HTML aren’t necessarily ignorant. Unfortunately, there’s sometimes a limit to how much you can argue with your customers.

RDF and XML Q&A: Which should I use?

Jeni — Sun, 17 Feb 2008 20:10:12 +0000

Another question to answer:

I’ve been reading about RDF, and I’m not sure in what situations it is more appropriate to use RDF over straight XML. I usually see RDF expressed as XML, but sometimes I see it written as language-independent functions (or methods).

Part of me is wondering if RDF is more appropriate for this project. What might the benefits be? And if it is, how difficult it would be to refactor it.

(Note that the person asking the question is talking about a small data-oriented project.) There’s a huge amount that could be said about this, so I might well post about some of it again. Here, I’m going to cut to the chase. This is what I’d recommend:

Model your application in RDF terms: Create a description of what classes of resources your application needs to deal with, and which properties link those together. You can call this description a RDF schema or conceptual model or ontology, depending on how impressive you want to sound. This modelling activity is useful in itself, largely because it helps you understand what information you’re dealing with and how it fits together.
Create a markup language that can be mapped to RDF: An XML version of your data allows you to make your data more generally available and reusable than locking it away in a triple store. Do one of the following:
- Define a subset of RDF/XML for your application: The full flexibility of RDF/XML is complicated to handle for plain XML processors, so subset it to, for example, always used typed elements (such as <my:Course>) rather than rdf:type properties, and to use referencing or nesting in a consistent way.
- Design markup languages that use RDFa attributes to reflect the semantics of the data: This gives you a standard way of mapping your markup language into RDF triples without having to adopt the “striped” design of RDF/XML in your markup language. A lot of the attributes can be defaulted to leave the markup language fairly streamlined.
- Design markup languages exactly as you like, and define GRDDL mappings from them into RDF/XML: This gives you the most flexibility in your markup language design (though not complete flexibility — you still need to be able to identify the statements that you want to make from the XML), at the expense of having to write some XSLT.

The point of doing this is to put you in a position where you can just use XML if you want, but you also have the flexibility of using RDF either now or in the future.

The benefits of using RDF are partly to do with the ease with which you can do certain kinds of processing (specifically combining “facts” together to draw conclusions) and partly to do with the potential of reuse of your data. In the same way that XML gives people a common syntax and thus aids interchange of information, RDF allows others to draw some conclusions (more than they would with a random mess of elements and attributes) about what your data means.

I don’t think that using RDF triple stores, SPARQL and all that jazz gives you a great return for a small-scale, personal project — you’re better off sticking to flat files and some XSLT — but it doesn’t hurt to build in some of the formality of RDF anyway.

Posterity

Jeni — Wed, 17 Oct 2007 22:08:29 +0000

We just had photos taken of the children, and it’s put me in a reflective mood. Norm posted the other day about his experience with information/task management products:

Then it hit me.

None of them, with the notable exception of Tinderbox, seem to store the data in any open format. I was seriously considering one of these commercial black boxes for an important chunk of the data that drives my day-to-day life. The little voice in my head reacted viscerally when the observation was made: “What the hell you thinking, man! Stop that!”

I’ve been experimenting a bit with GTD applications recently, and have the same reaction as Norm. The two tools that I’ve tried, ThinkingRock and FreeMind, can both export to an XML format (and import too), which is great, no doubt about it, but you’ve got to remember to do the exporting to take advantage of it. I know me: I just won’t do it (even with my GTD tool to remind me to). What I really want is an application that natively stores its data as XML, preferably in some nicely structured, standard format. So even after I’ve wiped the original application off my computer, or moved the file from one computer to another, I can still read that file and (with a little XSLT magic) load it into something else.

This is possibly the biggest thing that bugs me about most of the Web 2.0 applications out there. Of course I’ve got to be connected to the ‘net to use them, and I’m not all the time (most particularly at tech conferences, it seems). But more important, they’ve got my data tucked away in their databases, out of reach. Some of them will let me export it, or get at it through an API, but that isn’t enough for me. I want it here, so that even if the company folds or I forget my login and password, or the key I used to encrypt my personal data from potentially prying eyes… even years later, I can still read that file. It’s part of my history, but I won’t remember to keep it until it’s gone.

I’m thinking, you see, about some of the things I did on computers years ago. A half-finished book I wrote when I was about 18. The code I wrote for my PhD. Letters from my university days. These aren’t from that long ago, but now here I am using radically different software, in a completely different world, and these pieces from my past are lost, irretrievable because of the formats used to save them (as well as the hardware on which they’re saved: it’s getting harder to read a floppy nowadays).

I read Glasshouse by Charles Stross a couple of months ago (well worth the read). It’s set in the far future, and contains the following passage:

“We know why the dark age happened,” Fiore continues. “Our ancestors allowed their storage and processing architectures to proliferate uncontrollably, and they tended to throw away old technologies instead of virtualizing them. For reasons of commercial advantage, some of their largest entities deliberately created incompatible information formats and locked up huge quantities of useful material in them, so that when new architectures replaced old, the data became inaccessible.

“This particularly affected our records of personal and household activities during the latter half of the dark age. Early on, for example, we have a lot of film data captured by amateurs and home enthusiasts. They used a thing called a cine camera, which captured images on a photochemical medium. You could actually decode it with your eyeball. But a third of the way into the dark age, they switched to using magnetic storage tape, which degrades rapidly, then to digital storage, which was even worse because for no obvious reason they encrypted everything. The same sort of things happened to their audio recordings, and to text. Ironically, we know a lot more about their culture around the beginning of the dark age, around old-style year 1950, than about the end of the dark age, around 2040.”

I’m looking forward to the end of the dark age. In the meantime, the photos of the children will be hardcopies in the shoebox at the bottom of the wardrobe. And I think I might try the index card version of GTD.

Web 2.0 Project: Using Atom and XML with Graph Data Structures

Jeni — Sun, 02 Sep 2007 19:57:36 +0000

A Ruby on Rails specialist friend and I are building a Web 2.0 application. I would say it’s “social networking for the dead” except that I doubt that description would be attractive to most people (my ex-Goth defacto being a rare exception), and it can be for the living too. It’s a bit like all those genalogy websites, except that our focus is on people’s social relationships as well as their familial ones.

(I should say that this is all very casual. We’re both fitting it in around our other responsibilities, and are mainly interested in working together, learning new things, and trying out all the best practices that everyone keeps talking about. So don’t think I’m becoming a dotcom entrepreneur or anything. Its got a very Web 2.0 name, and I’m only not telling you in case you start hitting our servers. We’re nowhere near ready for visitors.)

We’re using the Gentech data model as the basis for the application (though I expect that we’ll tweak it a bit). You don’t really need to know anything about it to follow what I’m talking about here. The Gentech data model is very much a relational model. They might call it a logical model, but for anyone who isn’t a database head, it’s a physical model. That’s fine; we’re storing our data in a database, so a relational model for that is great.

In the Rails world, the model that Rails is object-oriented rather than relational. So there’s a certain amount of mapping from the relational world into the OO world, in particular eliding the tables that are created simply for normalisation purposes. Making that mapping is one thing that Rails is very good at, of course.

Then we’re into the worlds that I’m particularly interested in. One of our goals is to use Atom as an API, on the basis that it’s a fairly generic way of packaging things (entries) and lists-of-things (feeds) with a bunch of metadata. Plus, the Atom Publication Protocol shows you how to do RESTful applications right.

The trouble, as others have found is that Atom is designed for a flattish structure, in which you have things, and a list of things. Like blog posts and feeds of posts, or pictures and feeds of pictures. But the model that we’re starting from is relational, or object-oriented, or anyway it’s a graph. And that makes things more complicated.

The first steps are pretty obvious. Objects are equivalent to entries, and lists of objects equivalent to feeds. So every object has its own URL, and every significant feed has its own URL too. There’s the obvious http://www.example.com/people/DarwinC01 for a person, and http://www.example.com/people/ for a feed of people, but also http://www.example.com/people/DarwinC01/events/ for events that are related to a particular person. An entry’s content is an XML document that describes the equivalent object. It has attributes and children to represent the properties from the OO model (columns in the database tables).

Atom defines a bunch of metadata that you can associate with the content in an entry. These are:

id
title
summary (optional, as long as there’s textual or XML content)
updated
published (optional)
category (multiple, optional)
source (optional)
author (multiple, optional as long as there’s a source that specifies one or the entry’s in a feed that specifies one)
contributor (multiple, optional)
link (multiple, optional as long as there’s some content)
rights (optional, defaults to the feed’s rights)
extension elements (optional)

The metadata properties need to be used to indicate who created/updated the object and when. This gets confusing because some of the information in our system is likely to be about content that has authors and publishing dates and so on: the Gentech data model is strong on documenting the sources of information about people you’re reasearching. Even when documenting the source of some information, the Atom metadata should still be metadata about that object in our data model.

The set of Atom metadata does indicate a place where we’re going to want to tweak the Gentech data model though: every object should have metadata associated with it, at the very least an updated date, to populate the Atom metadata fields. Also, we need to identify the property of each object that is used in the <atom:title>, though the title can be something generic if there isn’t an obvious one.

Now the question that’s vexing me: how should we represent relationships to other objects/entries? Let’s take the example of documenting Charles Darwin’s voyage on HMS Beagle. It goes something like this:

<atom:entry>
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      ...
    </passenger>
  </atom:content>
</atom:entry>

The <evr:passenger> element needs to reference a person and a voyage (event), to say that Darwin was a passenger on the voyage.

Here are the options, I think:

Use <atom:link> within the <atom:entry>, with a URL in rel that indicates the kind of relationship

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:link rel="/link-relationships/assertion-persona"
             href="/persona/DarwinC01" />
  <atom:link rel="/link-relationships/assertion-event"
             href="/events/BeagleVoyage" />
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
    </passenger>
  </atom:content>
</atom:entry>

Use extension elements within the <atom:entry>

<atom:entry xml:base="http://www.example.com">
  ...
  <persona href="/persona/DarwinC01" />
  <event href="/events/BeagleVoyage" />
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
    </passenger>
  </atom:content>
</atom:entry>

Use child elements in the object’s XML, referencing the URLs of the related objects

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      <persona href="/persona/DarwinC01" />
      <event href="/events/BeagleVoyage" />
    </passenger>
  </atom:content>
</atom:entry>

Use child elements in the object’s XML, embedding the related objects’ Atom entry or feed

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      <persona>
        <atom:entry>
          ...
          <atom:title>Charles Darwin</atom:title>
          <atom:content>
            <persona>
              <name>Charles Darwin</name>
              ...
            </persona>
          </atom:content>
        </atom:entry>
      </persona>
      <event>
        <atom:entry>
          ...
          <atom:title>Beagle Voyage</atom:title>
          <atom:content>
            <event>
              <name>Beagle Voyage</name>
              <date-range>
                <date>1831-12-27</date>
                <date>1836-10-02</date>
              </date-range>
              ...
            </event>
          </atom:content>
        </atom:entry>
      </event>
    </passenger>
  </atom:content>
</atom:entry>

Use child elements in the object’s XML, embedding the related objects’ XML content

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      <persona>
        <name>Charles Darwin</name>
        ...
      </persona>
      <event>
        <name>Beagle Voyage</name>
        <date-range>
          <date>1831-12-27</date>
          <date>1836-10-02</date>
        </date-range>
        ...
      </event>
    </passenger>
  </atom:content>
</atom:entry>

I don’t think that there’s any point in using an extension element (#2), given that using <atom:link> (#1) situates the information in the same place in a more standard way.

Embedding information (as in #4 and #5) is a good thing because it means fewer requests to the server in order to get some useful information. Providing access to Atom feeds (as in #1, #3 and #4) is a good thing because it means you can get metadata about who created the refenced objects, and additional information about them. So #4 is good, since it does both these things, but I don’t like embedding Atom in the XML because it’s a lot of extra weight in the XML (making it harder to read/process).

In fact, #1, #3 and #5 aren’t mutually exclusive. It’s possible to add relevant <atom:link>s to the metadata, reference the URLs of the other objects and embed their content at the same time:

    <atom:entry xml:base="http://www.example.com">
      ...
      <atom:link rel="/link-relationships/assertion-persona"
                 href="/persona/DarwinC01" />
      <atom:link rel="/link-relationships/assertion-event"
                 href="/events/BeagleVoyage" />
      <atom:content type="application/xml">
        <passenger>
          <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
          <persona src="/persona/DarwinC01">
            <name>Charles Darwin</name>
            ...
          </persona>
          <event src="/events/BeagleVoyage">
            <name>Beagle Voyage</name>
            <date-range>
              <date>1831-12-27</date>
              <date>1836-10-02</date>
            </date-range>
            ...
          </event>
        </passenger>
      </atom:content>
    </atom:entry>

We embed the core information for easy access (#5), reference its original URI for more details (#3), and then we may as well add the <atom:link>s (#1) so that run-of-the-mill Atom readers who have no knowledge about our content can do something useful. We don’t get the metadata embedded in the XML, but it’s retrievable: a client could use the entry as a kind of “low resolution” information set, which they can add to by retrieving the “high resolution” Atom for the referenced objects, via their URLs, as necessary.

The problem with using an embedding method rather than a referencing method is that the object model is a graph, not a hierarchy. So you can’t always embed an object’s XML: sometimes you have to only use a reference (#3 without #5) to avoid getting into an endless loop of repeated information. As a publisher, sometimes you might want to only use a reference, because the information is only tangential to the main subject of the original entry. I’m imagining that we might serve several different Atom entries for the same object, with different amounts of detail. Maybe.

As an author, creating this XML, you can’t include a reference if you’re constructing XML (either in code or by hand) for new objects because they won’t have URLs yet. Therefore, for the purpose of creating objects as defined by the Atom Publishing Protocol, you’ll use embedded XML (#5) with references to existing objects if necessary. The resource returned will include the references for all the created objects. When updating, you’ll want to include as little as possible aside from the updated information, I imagine (small updates being less prone to clashes than large ones).

By the way, I’m using src attributes when the information is embedded and href attributes when the information is purely referenced (or almost purely referenced; the referencing elements might still have some content equivalent to the <atom:title> element, in the interests of presenting a clickable link).

So that’s the plan at the moment, but we’re open to suggestions. Anybody?

The perils of default namespaces

Jeni — Sun, 01 Jul 2007 19:32:44 +0000

A lot of people run into problems with namespaces, and most of those arise from using default namespaces (ie not giving namespaces prefixes). The transformation technology you use can have a big effect on how confusing and irritating it gets.

Default namespaces make XML documents easier to read because they allow you to just give the local name of an element rather than using prefixes all over the place. For example, using:

<house status="For Sale" xmlns="http://www.example.com/ns/house">
  <askingPrice>...</askingPrice>
  <address>...</address>
  <layout>...</layout>
</house>

rather than:

<h:house status="For Sale" xmlns:h="http://www.example.com/ns/house">
  <h:askingPrice>...</h:askingPrice>
  <h:address>...</h:address>
  <h:layout>...</h:layout>
</h:house>

In some cases, specifically documents that are validated against a DTD or interpreted by non-namespace-aware applications, you might be forced to use the default namespace. The biggest example of this is (X)HTML.

In transformation technologies, such as XSLT, XQuery and XLinq in VB.NET, you have to deal with at least two documents: the source documents that you are processing and the result documents that you are creating. Often, the source and result documents will use default namespaces, or at any rate you’ll want to query and create the documents without using prefixes. Sometimes, the source and result documents all use the same namespace, but it’s far more common that they don’t.

So transformation technologies have to support at least two default namespaces: one for querying and one for construction.

In XPath 1.0, you must specify a prefix for each namespace you want to use. A path like /house/layout will only select <layout> elements in no namespace. In XSLT 1.0, the default namespace in the stylesheet (as declared by the xmlns attribute on <xsl:stylesheet>) is then free to be used for the result documents. For example, I might do

<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:h="http://www.example.com/ns/house"
  exclude-result-prefixes="h"
  xmlns="http://www.w3.org/1999/xhtml">

<xsl:template match="h:house">
  <div class="house">
    <h1><xsl:apply-templates select="h:askingPrice" /></h1>
    ...
  </div>
</xsl:template>

</xsl:stylesheet>

[The best way to deal with multiple result documents in different default namespaces is to simply have different stylesheet documents to handle their generation, all included or imported into your main stylesheet application.]

Users of XSLT 1.0 found it confusing that they couldn’t just copy the namespace declarations (including a default namespace declaration) from a sample source document and have the paths just work. So in XPath 2.0, rather than no prefix meaning no namespace, the default element/type namespace in the context is used for element names with no prefix. If the default element/type namespace is set to http://www.example.com/ns/house then the path /house/layout will select all <layout> elements in the http://www.example.com/ns/house namespace. You can set this default element/type namespace in XSLT 2.0 using the [xsl:]xpath-default-namespace attribute, which can go anywhere but will usually be situated on the <xsl:stylesheet> element (in which case it appears without the xsl: prefix). The default element/type namespace can be scoped to a particular area of your stylesheet in the same way as namespace declarations.

Otherwise, XSLT 2.0 works like XSLT 1.0 in that the default namespace in the stylesheet supplies the default namespace for created elements, so you can do:

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns="http://www.w3.org/1999/xhtml"
  xpath-default-namespace="http://www.example.com/ns/house">

<xsl:template match="house">
  <div class="house">
    <h1><xsl:apply-templates select="askingPrice" /></h1>
    ...
  </div>
</xsl:template>

</xsl:stylesheet>

By keeping the default query namespace and the default construction namespace separate, you’re able to use unprefixed names in both paths and element constructors, even if the default namespaces in the two cases are different.

XQuery and VB.NET, on the other hand, provide a single default namespace that is used for both queries and construction, and they work in slightly different ways.

In XQuery you can declare the default namespace for the query, with

declare default element namespace "http://www.example.com/ns/house";

which means that you can query the source document with paths like /house/askingPrice and create elements in the http://www.example.com/ns/house namespace with direct element constructors without prefixes, like

<house status="For Sale">
  <askingPrice>...</askingPrice>
  <address>...</address>
  <layout>...</layout>
</house>

If you then want to generate XHTML (or some other result for which you’d prefer to use the default namespace), you can use a default namespace declaration on the XHTML you generate:

<div class="house" xmlns="http://www.w3.org/1999/xhtml">
  <h1>...</h1>
  ...
</div>

But the default namespace declaration in the element constructor carries through into the embedded expressions, so

  <div class="house" xmlns="http://www.w3.org/1999/xhtml">
    <h1>{ /house/askingPrice }</h1>
    ...
  </div>

won’t work. As a result, you end up having to use the query’s default namespace declaration to set the default namespace to XHTML (or whatever the default namespace is in the result), and use prefixes in your queries (essentially the same situation as in XSLT 1.0).

In XLinq in VB.NET, there’s the same kind of pattern. The Imports statement allows you to declare a default namespace that’s used in both queries and construction, as in:

Imports <xmlns="http://www.example.com/ns/house">

and you can use a default namespace declaration on the XHTML you generate to provide the default namespace for the elements in the XML literal:

houseDiv =
  <div class="house" xmlns="http://www.w3.org/1999/xhtml">
    <h1>...</h1>
    ...
  </div>

But unlike in XQuery, the default XHTML namespace declaration in the <div> doesn’t have an effect on the default namespace used in embedded expressions, which means you can still use unprefixed element names in any paths used within the XML literal, like this:

houseDiv =
  <div class="house" xmlns="http://www.w3.org/1999/xhtml">
    <h1><%= doc.<house>.<askingPrice> %></h1>
    ...
  </div>

However, if you build up your XHTML gradually, perhaps by using separate variables or methods, then every time you create a snippet of XHTML you have to specify this default namespace. Again, people will end up using the Imports statement to set the default namespace to the default result namespace and using prefixes in their paths.

The other factor to consider is that sometimes no prefix really does mean no namespace. If you’re querying an XML document that contains elements in no namespace, you have to set the default query namespace to no namespace. In XSLT 1.0, that’s always the case anyway; in XSLT 2.0, the xpath-default-namespace shouldn’t be set (or should be unset for those places that need to query no-namespace elements). In XQuery you can’t use the query default namespace declaration and in XLinq in VB.NET you can’t use the Imports statement. In both these cases, you better hope your result is in no namespace too. If not, the best route (to make it work at all in XQuery, and to avoid repetitive xmlns attributes in VB.NET) is to create a no-namespace version of your result first, and have a standard function or method that will add the right default namespace to that result.