xml

Free Our Bills

Jeni — Mon, 31 Mar 2008 20:10:14 +0100

The Free Our Bills campaign was launched recently in the UK. Some of the comments I’ve seen about the campaign makes me think that it might be helpful if people understood more about how Bills and legislation get published in the UK. I thought I’d offer a bit of background based on my experience (though there are many people with more intimate knowledge of the processes involved; perhaps they’ll correct me when I get it wrong).

Bills are draft legislation that is under discussion within the House of Commons or House of Lords. A Bill becomes law (legislation) when it is enacted.
Bills are published by Parliament and are available on the Parliament website. Legislation is published by The Stationery Office (TSO) under contract to the Office of Public Sector Information (OPSI) on the OPSI website.
Bills are changed (amended) as they progress through the Houses of Parliament. People are mostly interested in the most recent version of a Bill. Legislation can be changed (amended) by other legislation; the version of a piece of legislation with all the changes applied to it is known as consolidated legislation. Consolidated legislation is published in the Statute Law Database as well as (too a more limited extent) on the OPSI website.
Bills are edited by a dedicated team of Parliament employees who must reflect the amendments that the MPs say they want to make. They use a WYSIWYG XML editor. As is usual in an environment that has only been concerned about printed copies for centuries, they tend to focus on appearance rather than semantics, even when the XML supports the semantics.
The Free Our Bills campaign is not about making Bills (or legislation) easier for humans to read and understand, it’s about making it easier to extract information from a Bill so that people can be notified when a new Bill comes along on a subject they care about, or an old Bill is redrafted, and so on.
Bills are already available for the public to view on the web, in PDF and HTML forms. The problem is that the HTML is Really Really Bad (View Source to see) and that makes it Really Really Hard to extract useful information from them.
There are reasons for the Bills HTML being Really Really Bad:
- The HTML must look exactly like it does in printed form, otherwise Members of Parliament (MPs) would get Really Really Confused.
- MPs refer to pieces of a Bill (which they might want to change) by page and line number, not by the semantic structure of the Bill, so the HTML must have page and line numbers in it or MPs would get Really Really Confused.
- Although the formatting of Bills is pretty consistent, there’s always the chance that a piece will need to be formatted specially. It might be safe to assume a particular presentation for a particular semantic 99% of the time, but if that 1% isn’t formatted in the different way, MPs would be Really Really Confused.
- The code that creates the Bill HTML was written several years ago, when browser support for CSS was Really Really Bad.
The picture for legislation is rather better because a strategic decision was made to focus on semantics rather than presentation. When a Bill is enacted, it gets converted into reasonably good semantic XML, which forms the basis of all the HTML views. It also helps that this HTML was designed fairly recently, for modern browsers; it makes heavy use of CSS so there’s relatively little obfuscation of the content.

I think there are interesting general lessons here:

Different user communities have different requirements. MPs have different requirements from Bills from the general public, who don’t care (as) much about line or page numbers. On the other hand, you need to actually consult with users about what they need rather than make assumptions about it: are MPs really likely to get Really Really Confused if the HTML presentation of a Bill looks slightly different from the PDF print version? I don’t know.
Authors don’t care about what they don’t use. When the only way of using a Bill is to print it, it’s natural that authors and publishers only care about how it looks when it’s printed. Training people to care about semantic markup is really hard, and it’s made harder by WYSIWYG tools that allow them to override the semantic style. If a difference isn’t visible, then in author’s eyes it doesn’t exist.
You have to positively decide to ignore appearance. When transforming from a WYSIWYG view, replicating appearance is the obvious thing to do. But it’s worthwhile in the long run to focus on extracting the semantics, because the resulting documents are so much more reusable.
HTML, XML and XSLT are not inherently good. Parliament wanted Bills in HTML so that they were more accessible on the web. But the HTML is dreadfully inaccessible because of the other requirements placed on it. Similarly, XML can be incredibly obfuscated, or entirely about presentation, as formats such as OOXML illustrate. And just because your code is written in XSLT does not make it inherently easier to maintain then (say) a SAX transformation. It’s easy to misuse a technology.
Developers who produce atrocious HTML aren’t necessarily ignorant. Unfortunately, there’s sometimes a limit to how much you can argue with your customers.

RDF and XML Q&A: Which should I use?

Jeni — Sun, 17 Feb 2008 20:10:12 +0000

Another question to answer:

I’ve been reading about RDF, and I’m not sure in what situations it is more appropriate to use RDF over straight XML. I usually see RDF expressed as XML, but sometimes I see it written as language-independent functions (or methods).

Part of me is wondering if RDF is more appropriate for this project. What might the benefits be? And if it is, how difficult it would be to refactor it.

(Note that the person asking the question is talking about a small data-oriented project.) There’s a huge amount that could be said about this, so I might well post about some of it again. Here, I’m going to cut to the chase. This is what I’d recommend:

Model your application in RDF terms: Create a description of what classes of resources your application needs to deal with, and which properties link those together. You can call this description a RDF schema or conceptual model or ontology, depending on how impressive you want to sound. This modelling activity is useful in itself, largely because it helps you understand what information you’re dealing with and how it fits together.
Create a markup language that can be mapped to RDF: An XML version of your data allows you to make your data more generally available and reusable than locking it away in a triple store. Do one of the following:
- Define a subset of RDF/XML for your application: The full flexibility of RDF/XML is complicated to handle for plain XML processors, so subset it to, for example, always used typed elements (such as <my:Course>) rather than rdf:type properties, and to use referencing or nesting in a consistent way.
- Design markup languages that use RDFa attributes to reflect the semantics of the data: This gives you a standard way of mapping your markup language into RDF triples without having to adopt the “striped” design of RDF/XML in your markup language. A lot of the attributes can be defaulted to leave the markup language fairly streamlined.
- Design markup languages exactly as you like, and define GRDDL mappings from them into RDF/XML: This gives you the most flexibility in your markup language design (though not complete flexibility — you still need to be able to identify the statements that you want to make from the XML), at the expense of having to write some XSLT.

The point of doing this is to put you in a position where you can just use XML if you want, but you also have the flexibility of using RDF either now or in the future.

The benefits of using RDF are partly to do with the ease with which you can do certain kinds of processing (specifically combining “facts” together to draw conclusions) and partly to do with the potential of reuse of your data. In the same way that XML gives people a common syntax and thus aids interchange of information, RDF allows others to draw some conclusions (more than they would with a random mess of elements and attributes) about what your data means.

I don’t think that using RDF triple stores, SPARQL and all that jazz gives you a great return for a small-scale, personal project — you’re better off sticking to flat files and some XSLT — but it doesn’t hurt to build in some of the formality of RDF anyway.

Posterity

Jeni — Wed, 17 Oct 2007 23:08:29 +0100

We just had photos taken of the children, and it’s put me in a reflective mood. Norm posted the other day about his experience with information/task management products:

Then it hit me.

None of them, with the notable exception of Tinderbox, seem to store the data in any open format. I was seriously considering one of these commercial black boxes for an important chunk of the data that drives my day-to-day life. The little voice in my head reacted viscerally when the observation was made: “What the hell you thinking, man! Stop that!”

I’ve been experimenting a bit with GTD applications recently, and have the same reaction as Norm. The two tools that I’ve tried, ThinkingRock and FreeMind, can both export to an XML format (and import too), which is great, no doubt about it, but you’ve got to remember to do the exporting to take advantage of it. I know me: I just won’t do it (even with my GTD tool to remind me to). What I really want is an application that natively stores its data as XML, preferably in some nicely structured, standard format. So even after I’ve wiped the original application off my computer, or moved the file from one computer to another, I can still read that file and (with a little XSLT magic) load it into something else.

This is possibly the biggest thing that bugs me about most of the Web 2.0 applications out there. Of course I’ve got to be connected to the ‘net to use them, and I’m not all the time (most particularly at tech conferences, it seems). But more important, they’ve got my data tucked away in their databases, out of reach. Some of them will let me export it, or get at it through an API, but that isn’t enough for me. I want it here, so that even if the company folds or I forget my login and password, or the key I used to encrypt my personal data from potentially prying eyes… even years later, I can still read that file. It’s part of my history, but I won’t remember to keep it until it’s gone.

I’m thinking, you see, about some of the things I did on computers years ago. A half-finished book I wrote when I was about 18. The code I wrote for my PhD. Letters from my university days. These aren’t from that long ago, but now here I am using radically different software, in a completely different world, and these pieces from my past are lost, irretrievable because of the formats used to save them (as well as the hardware on which they’re saved: it’s getting harder to read a floppy nowadays).

I read Glasshouse by Charles Stross a couple of months ago (well worth the read). It’s set in the far future, and contains the following passage:

“We know why the dark age happened,” Fiore continues. “Our ancestors allowed their storage and processing architectures to proliferate uncontrollably, and they tended to throw away old technologies instead of virtualizing them. For reasons of commercial advantage, some of their largest entities deliberately created incompatible information formats and locked up huge quantities of useful material in them, so that when new architectures replaced old, the data became inaccessible.

“This particularly affected our records of personal and household activities during the latter half of the dark age. Early on, for example, we have a lot of film data captured by amateurs and home enthusiasts. They used a thing called a cine camera, which captured images on a photochemical medium. You could actually decode it with your eyeball. But a third of the way into the dark age, they switched to using magnetic storage tape, which degrades rapidly, then to digital storage, which was even worse because for no obvious reason they encrypted everything. The same sort of things happened to their audio recordings, and to text. Ironically, we know a lot more about their culture around the beginning of the dark age, around old-style year 1950, than about the end of the dark age, around 2040.”

I’m looking forward to the end of the dark age. In the meantime, the photos of the children will be hardcopies in the shoebox at the bottom of the wardrobe. And I think I might try the index card version of GTD.

Web 2.0 Project: Using Atom and XML with Graph Data Structures

Jeni — Sun, 02 Sep 2007 20:57:36 +0100

A Ruby on Rails specialist friend and I are building a Web 2.0 application. I would say it’s “social networking for the dead” except that I doubt that description would be attractive to most people (my ex-Goth defacto being a rare exception), and it can be for the living too. It’s a bit like all those genalogy websites, except that our focus is on people’s social relationships as well as their familial ones.

(I should say that this is all very casual. We’re both fitting it in around our other responsibilities, and are mainly interested in working together, learning new things, and trying out all the best practices that everyone keeps talking about. So don’t think I’m becoming a dotcom entrepreneur or anything. Its got a very Web 2.0 name, and I’m only not telling you in case you start hitting our servers. We’re nowhere near ready for visitors.)

We’re using the Gentech data model as the basis for the application (though I expect that we’ll tweak it a bit). You don’t really need to know anything about it to follow what I’m talking about here. The Gentech data model is very much a relational model. They might call it a logical model, but for anyone who isn’t a database head, it’s a physical model. That’s fine; we’re storing our data in a database, so a relational model for that is great.

In the Rails world, the model that Rails is object-oriented rather than relational. So there’s a certain amount of mapping from the relational world into the OO world, in particular eliding the tables that are created simply for normalisation purposes. Making that mapping is one thing that Rails is very good at, of course.

Then we’re into the worlds that I’m particularly interested in. One of our goals is to use Atom as an API, on the basis that it’s a fairly generic way of packaging things (entries) and lists-of-things (feeds) with a bunch of metadata. Plus, the Atom Publication Protocol shows you how to do RESTful applications right.

The trouble, as others have found is that Atom is designed for a flattish structure, in which you have things, and a list of things. Like blog posts and feeds of posts, or pictures and feeds of pictures. But the model that we’re starting from is relational, or object-oriented, or anyway it’s a graph. And that makes things more complicated.

The first steps are pretty obvious. Objects are equivalent to entries, and lists of objects equivalent to feeds. So every object has its own URL, and every significant feed has its own URL too. There’s the obvious http://www.example.com/people/DarwinC01 for a person, and http://www.example.com/people/ for a feed of people, but also http://www.example.com/people/DarwinC01/events/ for events that are related to a particular person. An entry’s content is an XML document that describes the equivalent object. It has attributes and children to represent the properties from the OO model (columns in the database tables).

Atom defines a bunch of metadata that you can associate with the content in an entry. These are:

id
title
summary (optional, as long as there’s textual or XML content)
updated
published (optional)
category (multiple, optional)
source (optional)
author (multiple, optional as long as there’s a source that specifies one or the entry’s in a feed that specifies one)
contributor (multiple, optional)
link (multiple, optional as long as there’s some content)
rights (optional, defaults to the feed’s rights)
extension elements (optional)

The metadata properties need to be used to indicate who created/updated the object and when. This gets confusing because some of the information in our system is likely to be about content that has authors and publishing dates and so on: the Gentech data model is strong on documenting the sources of information about people you’re reasearching. Even when documenting the source of some information, the Atom metadata should still be metadata about that object in our data model.

The set of Atom metadata does indicate a place where we’re going to want to tweak the Gentech data model though: every object should have metadata associated with it, at the very least an updated date, to populate the Atom metadata fields. Also, we need to identify the property of each object that is used in the <atom:title>, though the title can be something generic if there isn’t an obvious one.

Now the question that’s vexing me: how should we represent relationships to other objects/entries? Let’s take the example of documenting Charles Darwin’s voyage on HMS Beagle. It goes something like this:

<atom:entry>
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      ...
    </passenger>
  </atom:content>
</atom:entry>

The <evr:passenger> element needs to reference a person and a voyage (event), to say that Darwin was a passenger on the voyage.

Here are the options, I think:

Use <atom:link> within the <atom:entry>, with a URL in rel that indicates the kind of relationship

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:link rel="/link-relationships/assertion-persona"
             href="/persona/DarwinC01" />
  <atom:link rel="/link-relationships/assertion-event"
             href="/events/BeagleVoyage" />
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
    </passenger>
  </atom:content>
</atom:entry>

Use extension elements within the <atom:entry>

<atom:entry xml:base="http://www.example.com">
  ...
  <persona href="/persona/DarwinC01" />
  <event href="/events/BeagleVoyage" />
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
    </passenger>
  </atom:content>
</atom:entry>

Use child elements in the object’s XML, referencing the URLs of the related objects

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      <persona href="/persona/DarwinC01" />
      <event href="/events/BeagleVoyage" />
    </passenger>
  </atom:content>
</atom:entry>

Use child elements in the object’s XML, embedding the related objects’ Atom entry or feed

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      <persona>
        <atom:entry>
          ...
          <atom:title>Charles Darwin</atom:title>
          <atom:content>
            <persona>
              <name>Charles Darwin</name>
              ...
            </persona>
          </atom:content>
        </atom:entry>
      </persona>
      <event>
        <atom:entry>
          ...
          <atom:title>Beagle Voyage</atom:title>
          <atom:content>
            <event>
              <name>Beagle Voyage</name>
              <date-range>
                <date>1831-12-27</date>
                <date>1836-10-02</date>
              </date-range>
              ...
            </event>
          </atom:content>
        </atom:entry>
      </event>
    </passenger>
  </atom:content>
</atom:entry>

Use child elements in the object’s XML, embedding the related objects’ XML content

<atom:entry xml:base="http://www.example.com">
  ...
  <atom:content type="application/xml">
    <passenger>
      <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
      <persona>
        <name>Charles Darwin</name>
        ...
      </persona>
      <event>
        <name>Beagle Voyage</name>
        <date-range>
          <date>1831-12-27</date>
          <date>1836-10-02</date>
        </date-range>
        ...
      </event>
    </passenger>
  </atom:content>
</atom:entry>

I don’t think that there’s any point in using an extension element (#2), given that using <atom:link> (#1) situates the information in the same place in a more standard way.

Embedding information (as in #4 and #5) is a good thing because it means fewer requests to the server in order to get some useful information. Providing access to Atom feeds (as in #1, #3 and #4) is a good thing because it means you can get metadata about who created the refenced objects, and additional information about them. So #4 is good, since it does both these things, but I don’t like embedding Atom in the XML because it’s a lot of extra weight in the XML (making it harder to read/process).

In fact, #1, #3 and #5 aren’t mutually exclusive. It’s possible to add relevant <atom:link>s to the metadata, reference the URLs of the other objects and embed their content at the same time:

    <atom:entry xml:base="http://www.example.com">
      ...
      <atom:link rel="/link-relationships/assertion-persona"
                 href="/persona/DarwinC01" />
      <atom:link rel="/link-relationships/assertion-event"
                 href="/events/BeagleVoyage" />
      <atom:content type="application/xml">
        <passenger>
          <source href="http://www.aboutdarwin.com/voyage/voyage01.html" />
          <persona src="/persona/DarwinC01">
            <name>Charles Darwin</name>
            ...
          </persona>
          <event src="/events/BeagleVoyage">
            <name>Beagle Voyage</name>
            <date-range>
              <date>1831-12-27</date>
              <date>1836-10-02</date>
            </date-range>
            ...
          </event>
        </passenger>
      </atom:content>
    </atom:entry>

We embed the core information for easy access (#5), reference its original URI for more details (#3), and then we may as well add the <atom:link>s (#1) so that run-of-the-mill Atom readers who have no knowledge about our content can do something useful. We don’t get the metadata embedded in the XML, but it’s retrievable: a client could use the entry as a kind of “low resolution” information set, which they can add to by retrieving the “high resolution” Atom for the referenced objects, via their URLs, as necessary.

The problem with using an embedding method rather than a referencing method is that the object model is a graph, not a hierarchy. So you can’t always embed an object’s XML: sometimes you have to only use a reference (#3 without #5) to avoid getting into an endless loop of repeated information. As a publisher, sometimes you might want to only use a reference, because the information is only tangential to the main subject of the original entry. I’m imagining that we might serve several different Atom entries for the same object, with different amounts of detail. Maybe.

As an author, creating this XML, you can’t include a reference if you’re constructing XML (either in code or by hand) for new objects because they won’t have URLs yet. Therefore, for the purpose of creating objects as defined by the Atom Publishing Protocol, you’ll use embedded XML (#5) with references to existing objects if necessary. The resource returned will include the references for all the created objects. When updating, you’ll want to include as little as possible aside from the updated information, I imagine (small updates being less prone to clashes than large ones).

By the way, I’m using src attributes when the information is embedded and href attributes when the information is purely referenced (or almost purely referenced; the referencing elements might still have some content equivalent to the <atom:title> element, in the interests of presenting a clickable link).

So that’s the plan at the moment, but we’re open to suggestions. Anybody?

The perils of default namespaces

Jeni — Sun, 01 Jul 2007 20:32:44 +0100

A lot of people run into problems with namespaces, and most of those arise from using default namespaces (ie not giving namespaces prefixes). The transformation technology you use can have a big effect on how confusing and irritating it gets.

Default namespaces make XML documents easier to read because they allow you to just give the local name of an element rather than using prefixes all over the place. For example, using:

<house status="For Sale" xmlns="http://www.example.com/ns/house">
  <askingPrice>...</askingPrice>
  <address>...</address>
  <layout>...</layout>
</house>

rather than:

<h:house status="For Sale" xmlns:h="http://www.example.com/ns/house">
  <h:askingPrice>...</h:askingPrice>
  <h:address>...</h:address>
  <h:layout>...</h:layout>
</h:house>

In some cases, specifically documents that are validated against a DTD or interpreted by non-namespace-aware applications, you might be forced to use the default namespace. The biggest example of this is (X)HTML.

In transformation technologies, such as XSLT, XQuery and XLinq in VB.NET, you have to deal with at least two documents: the source documents that you are processing and the result documents that you are creating. Often, the source and result documents will use default namespaces, or at any rate you’ll want to query and create the documents without using prefixes. Sometimes, the source and result documents all use the same namespace, but it’s far more common that they don’t.

So transformation technologies have to support at least two default namespaces: one for querying and one for construction.

In XPath 1.0, you must specify a prefix for each namespace you want to use. A path like /house/layout will only select <layout> elements in no namespace. In XSLT 1.0, the default namespace in the stylesheet (as declared by the xmlns attribute on <xsl:stylesheet>) is then free to be used for the result documents. For example, I might do

<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:h="http://www.example.com/ns/house"
  exclude-result-prefixes="h"
  xmlns="http://www.w3.org/1999/xhtml">

<xsl:template match="h:house">
  <div class="house">
    <h1><xsl:apply-templates select="h:askingPrice" /></h1>
    ...
  </div>
</xsl:template>

</xsl:stylesheet>

[The best way to deal with multiple result documents in different default namespaces is to simply have different stylesheet documents to handle their generation, all included or imported into your main stylesheet application.]

Users of XSLT 1.0 found it confusing that they couldn’t just copy the namespace declarations (including a default namespace declaration) from a sample source document and have the paths just work. So in XPath 2.0, rather than no prefix meaning no namespace, the default element/type namespace in the context is used for element names with no prefix. If the default element/type namespace is set to http://www.example.com/ns/house then the path /house/layout will select all <layout> elements in the http://www.example.com/ns/house namespace. You can set this default element/type namespace in XSLT 2.0 using the [xsl:]xpath-default-namespace attribute, which can go anywhere but will usually be situated on the <xsl:stylesheet> element (in which case it appears without the xsl: prefix). The default element/type namespace can be scoped to a particular area of your stylesheet in the same way as namespace declarations.

Otherwise, XSLT 2.0 works like XSLT 1.0 in that the default namespace in the stylesheet supplies the default namespace for created elements, so you can do:

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns="http://www.w3.org/1999/xhtml"
  xpath-default-namespace="http://www.example.com/ns/house">

<xsl:template match="house">
  <div class="house">
    <h1><xsl:apply-templates select="askingPrice" /></h1>
    ...
  </div>
</xsl:template>

</xsl:stylesheet>

By keeping the default query namespace and the default construction namespace separate, you’re able to use unprefixed names in both paths and element constructors, even if the default namespaces in the two cases are different.

XQuery and VB.NET, on the other hand, provide a single default namespace that is used for both queries and construction, and they work in slightly different ways.

In XQuery you can declare the default namespace for the query, with

declare default element namespace "http://www.example.com/ns/house";

which means that you can query the source document with paths like /house/askingPrice and create elements in the http://www.example.com/ns/house namespace with direct element constructors without prefixes, like

<house status="For Sale">
  <askingPrice>...</askingPrice>
  <address>...</address>
  <layout>...</layout>
</house>

If you then want to generate XHTML (or some other result for which you’d prefer to use the default namespace), you can use a default namespace declaration on the XHTML you generate:

<div class="house" xmlns="http://www.w3.org/1999/xhtml">
  <h1>...</h1>
  ...
</div>

But the default namespace declaration in the element constructor carries through into the embedded expressions, so

  <div class="house" xmlns="http://www.w3.org/1999/xhtml">
    <h1>{ /house/askingPrice }</h1>
    ...
  </div>

won’t work. As a result, you end up having to use the query’s default namespace declaration to set the default namespace to XHTML (or whatever the default namespace is in the result), and use prefixes in your queries (essentially the same situation as in XSLT 1.0).

In XLinq in VB.NET, there’s the same kind of pattern. The Imports statement allows you to declare a default namespace that’s used in both queries and construction, as in:

Imports <xmlns="http://www.example.com/ns/house">

and you can use a default namespace declaration on the XHTML you generate to provide the default namespace for the elements in the XML literal:

houseDiv =
  <div class="house" xmlns="http://www.w3.org/1999/xhtml">
    <h1>...</h1>
    ...
  </div>

But unlike in XQuery, the default XHTML namespace declaration in the <div> doesn’t have an effect on the default namespace used in embedded expressions, which means you can still use unprefixed element names in any paths used within the XML literal, like this:

houseDiv =
  <div class="house" xmlns="http://www.w3.org/1999/xhtml">
    <h1><%= doc.<house>.<askingPrice> %></h1>
    ...
  </div>

However, if you build up your XHTML gradually, perhaps by using separate variables or methods, then every time you create a snippet of XHTML you have to specify this default namespace. Again, people will end up using the Imports statement to set the default namespace to the default result namespace and using prefixes in their paths.

The other factor to consider is that sometimes no prefix really does mean no namespace. If you’re querying an XML document that contains elements in no namespace, you have to set the default query namespace to no namespace. In XSLT 1.0, that’s always the case anyway; in XSLT 2.0, the xpath-default-namespace shouldn’t be set (or should be unset for those places that need to query no-namespace elements). In XQuery you can’t use the query default namespace declaration and in XLinq in VB.NET you can’t use the Imports statement. In both these cases, you better hope your result is in no namespace too. If not, the best route (to make it work at all in XQuery, and to avoid repetitive xmlns attributes in VB.NET) is to create a no-namespace version of your result first, and have a standard function or method that will add the right default namespace to that result.

XML Paths in Programming Languages

Jeni — Tue, 05 Jun 2007 22:35:48 +0100

I’ve finally finished my “Progress in Processing” talk for this year’s XML Summer School. It’s been really interesting looking at the different APIs developed for different programming languages in the last few years, all so much easier to use than the DOM. One of the themes is the use of path-based syntax to query XML.

Even with the simpler XML APIs, accessing nodes in an XML tree can be pretty laborious. For example, get all the <room> elements in the first <floor> element of a house with (this is XLinq):

doc.Element("house").Element("floor").Elements("room")

Of course XPath does this pretty well:

/house/floor[1]/room

and many of the APIs that I looked at provided XPath access. For example (this is Ruby’s REXML):

doc.elements["/house/floor[1]/room"]

But using XPath is tricky for a couple of reasons:

A cognitive leap is required to switch from the usual object/method dot-notation syntax that you use in the surrounding language to the specialised XPath notation. In particular, it’s difficult mixing the one-based indexing in XPath with the zero-based indexing that’s used in most programming languages.
XPaths have to be passed as strings; there’s a temptation to construct the strings automatically, which leads to all sorts of headaches (such as remembering to put quotes around the strings that you concatenate into the XPath when you really want them to be interpreted as strings rather than element names). [A clean way of approaching this would be to use variables in the XPath and pass in a set of variable bindings when you use the XPath, but I don’t know any API that actually does this.]

Because of these issues, there’s been some effort to use the native dot-notation syntax to query XML within general-purpose programming languages. I knew about JAXB before I started looking, but didn’t know before about Uche Ogbuji’s Amara or the details of the VB.NET interface. Whereas with JAXB you have to compile a schema (an XML Schema schema, what’s more) into Java classes, with Amara and VB.NET there’s the kind of dynamic binding you get with XPath. In Amara, for example, you can do:

doc.house.floor.room

while in VB.NET you can use (I think):

doc.<house>.<floor>.First().<room>

(Don’t ask me how to get the rooms on the second floor in VB.NET; that, I couldn’t figure out. In Amara, it’s doc.house.floor[1].room.)

Path-based syntax in general-purpose programming languages is really neat: it exposes XML documents as if they were objects, which makes them “closer” to you as a programmer. They work particularly well for data-oriented XML in which elements contain either elements or text and not both.

There are two main areas where the path-based languages differ.

First, what they do with paths with intermediate steps that select more than one element. For example, in XPath, /house/floor/room gets you all the rooms in all the floors of the house, as does doc.<house>.<floor>.<room> in VB.NET: both provide an implicit iteration over the selected elements in the intermediate steps. In Amara,doc.house.floor.roomgets you all the rooms in the *first* floor of the house, so you have to explicitly iterate over the` elements if you want to collect all the rooms in the house.

Second, how they handle namespaces. In XPath, you have to provide a set of namespace bindings whenever you evaluate an XPath expression, and the prefixes you use on element names are resolved against those namespace bindings. In XPath 1.0, element names with no prefix only match elements in no namespace; in XPath 2.0, you can also provide a default namespace that’s used for names with no prefix.

That works well when XPath is embedded in some XML (such as in XSLT, XForms, XProc and so on), because the namespace bindings from the XML environment can provide the namespace bindings for the XPath expression. But that can’t generally happen when XPaths are used in a programming language.

All the APIs that use XPaths allow you to specify the namespace bindings explicitly, but some, such as REXML, do an automatic namespace binding based on the namespace bindings from the source document. So if I have:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  ...
</xs:schema>

in the document I’m querying then I can use the xs prefix to mean the XML Schema namespace in the path that I use to query the document, such as /xs:schema/xs:element/@name to get the names of the global element declarations.

This makes paths nice and simple… right until you have to use them process a document that uses different namespace bindings. For example, it’s not uncommon to find XML Schema documents that use the prefix xsd instead of xs, or even the default namespace; for those documents, the automatic binding won’t work and the path /xs:schema/xs:element/@name will give you an error. [REXML also provides XPath.match() and XPath.each(), to which you can provide an explicit set of namespace bindings; you’ll use these if you care about keeping the indirection between prefixes and namespaces.]

In Amara (when using Pythonic paths), you can just forget about namespaces: the elements and attributes are selected purely based on their local name. The only time you’ll run into problems is if you actually have, in the same context, two elements from different namespaces with the same local name, which is an event that’s rarer than people using different prefixes for a given namespace. In the XML Schema example, you can use doc.schema.element.name (yes, attributes are picked up with the same syntax as elements), and will only have a problem if there’s an <element> element in some other namespace. [Amara also provides XPath-based querying, and you can supply explicit namespace bindings for that.]

In VB.NET, the Imports directive is used to provide global namespace bindings, so it gets the benefits that you would have from using XPath in an XML context. What’s more, you can use a default namespace binding so that you don’t have to use prefixes in your paths. So you can do:

Imports <xmlns:xs="http://www.w3.org/2001/XMLSchema">

and then doc.<xs:schema>.<xs:element>.@name and it will work as planned, no matter what prefixes were actually used in the schema document. Or you can do:

Imports <xmlns="http://www.w3.org/2001/XMLSchema">

and doc.<schema>.<element>.@name. Overall, I think it’s pretty impressive that VB.NET is going to have support for querying XML documents built in at such a low level.

Using default namespaces in paths is a tricky issue, though. I’ll have to dedicate a different post to that; this one’s quite long enough already.

Unicode database in XML

Jeni — Mon, 14 May 2007 21:11:23 +0100

Whatever algorithm you use to calculate Levenshtein distance, one of its great features is that you can tweak the cost of letter substitutions. For example, you can do a case-insensitive comparison of two strings, or perhaps more interestingly a semi-case-sensitive comparison of two strings, where the cost of replacing a character for its upper or lower case equivalent is less than the cost of replacing a character with an unrelated character, but more than zero. But that requires knowledge of whether and how two characters are related.

Of course all that information is stored in the Unicode Database, which are a bunch of text files in a structured format. I looked for an XML version but couldn’t find one (well, Googling “Unicode database XML” isn’t much help). So I downloaded UnicodeData.txt and NamesList.txt and put together an XSLT 2.0 stylesheet to create an XML version of the Unicode database.

The XML contains practically everything that you can get from those two files, which means:

block and subblock structures
hexadecimal and decimal codepoints
names, aliases and comments
category and numeric information
uppercase, lowercase and titlecase equivalents
decomposition of various kinds
related characters
bidi information

It might prove easier to search than grepping the text files, if you’re used to using XPath. I might split it up and put together an AJAX browser, in my Copious Spare Time.