xslt

UK-based XML/XSLT job

Jeni — Thu, 17 Apr 2008 21:02:16 +0100

I’ve been asked if I could advertise the following vacancy. Any interested parties should contact Graham Fuller from Peopleworks (but say you saw it here; I’ll get a reward!).

Developer * XSLT * XML * Schemas * JavaScript * XHTML * CSS.

Global retail organisation and household name is looking for 2 (two) Front-End/User Interface Developers to work on a major consumer e-commerce portal.

MAIN TASKS/REQUIREMENTS:

Development of enterprise solutions

Development of Consumer driven applications

Adherence to Software Development Methodology

ESSENTIAL SKILLS:

XSLT & XML

XML schemas

JavaScript

XHTML

Cross browser and platform CSS positioning

Accessibility

DESIRABLE (not essential) SKILLS:

Understanding of web design

JavaScript, including AJAX & DHTML

OO JavaScript

These roles represent the opportunity to consult for a global multi billion global organisation.

The roles will pay £400 to £450 per day and they are 3 to 6 months contracts.

The role is based in Welwyn Garden City in Hertfordshire.

It is a 15 to 20 minute walk from the train station and there is a company service bus every 15 minutes at peak times from the station to the campus.

naturally there is loads of parking space for cars.

XSLT Q&A: Refactoring templates

Jeni — Sun, 06 Apr 2008 20:20:00 +0100

A question about how to refactor some repetitive templates.

The issue is in creating XHTML headings.

For a small docbook article, I have the following templates in one of my stylesheets:

<xsl:template match="article/title | article/info/title">
  <h1><xsl:apply-templates /></h1>
</xsl:template>

<xsl:template match="article/section/title">
  <h2><xsl:apply-templates /></h2>
</xsl:template>

<xsl:template match="article/section/section/title">
  <h3><xsl:apply-templates /></h3>
</xsl:template>

<xsl:template match="article/section/section/section/title">
  <h4><xsl:apply-templates /></h4>
</xsl:template>

<xsl:template match="article/section/section/section/section/title">
  <h5><xsl:apply-templates /></h5>
</xsl:template>

<xsl:template match="article/section/section/section/section/section/title">
  <h6><xsl:apply-templates /></h6>
</xsl:template>

Obviously this was a quick and (VERY) dirty way to achieve the output I wanted.

So, I know you can do something similar with an <xsl:choose> and some cases, but I have a feeling there’s a more automatic way.

Seek out the similarities. The last five of these templates all match <title> elements within a <section> element. They all create an XHTML heading element and apply templates to the content of the <title> to get the content of the heading.

Identify the differences. They’re different in the level of heading that they create and in the number of ancestor <section> elements the <title> has.

Find the algorithm. Here’s the mapping:

number of `<section>` ancestors	required heading
1	2
2	3
3	4
4	5
5	6

So the level of the heading is the number of ancestor <section> elements plus one.

Put it together. Get the number of ancestor <section> elements with count(ancestor::section). Create the name of the heading element to create using an attribute value template in the name attribute.

<xsl:template match="section/title">
  <xsl:variable name="nAncestorSections"
    select="count(ancestor::section)" />
  <xsl:variable name="headingLevel"
    select="$nAncestorSections + 1" />
  <xsl:element name="h{$headingLevel}">
    <xsl:apply-templates />
  </xsl:element>
</xsl:template>

Of course there are differences between this refactored code and the original. In particular, this template deals improperly with the case where there are more than five nested sections, because it creates an <h7> element, which isn’t legal. If you thought that was likely to occur, you could change how $headingLevel is calculated to:

<xsl:variable name="headingLevel">
  <xsl:choose>
    <xsl:when test="$nAncestorSections >= 5">6</xsl:when>
    <xsl:otherwise>
      <xsl:value-of select="$nAncestorSections + 1" />
    </xsl:otherwise>
  </xsl:choose>
</xsl:variable>

or:

<xsl:variable name="headingLevel"
  select="if ($nAncestorSections >= 5)
          then 6 else $nAncestorSections + 1" />

in XSLT 2.0.

The other problem is that the template deals differently with <title> elements that appear within a <section> whose parent is neither <article> nor another <section> (which aren’t matched by the original templates). There are other possible parents for <section> namely <appendix>, <chapter>, <partintro> and <preface>, so if these elements are likely to appear in the subset of DocBook you’re using and you want the code to behave differently you need to either add more templates or some extra conditions into this one.

Free Our Bills

Jeni — Mon, 31 Mar 2008 20:10:14 +0100

The Free Our Bills campaign was launched recently in the UK. Some of the comments I’ve seen about the campaign makes me think that it might be helpful if people understood more about how Bills and legislation get published in the UK. I thought I’d offer a bit of background based on my experience (though there are many people with more intimate knowledge of the processes involved; perhaps they’ll correct me when I get it wrong).

Bills are draft legislation that is under discussion within the House of Commons or House of Lords. A Bill becomes law (legislation) when it is enacted.
Bills are published by Parliament and are available on the Parliament website. Legislation is published by The Stationery Office (TSO) under contract to the Office of Public Sector Information (OPSI) on the OPSI website.
Bills are changed (amended) as they progress through the Houses of Parliament. People are mostly interested in the most recent version of a Bill. Legislation can be changed (amended) by other legislation; the version of a piece of legislation with all the changes applied to it is known as consolidated legislation. Consolidated legislation is published in the Statute Law Database as well as (too a more limited extent) on the OPSI website.
Bills are edited by a dedicated team of Parliament employees who must reflect the amendments that the MPs say they want to make. They use a WYSIWYG XML editor. As is usual in an environment that has only been concerned about printed copies for centuries, they tend to focus on appearance rather than semantics, even when the XML supports the semantics.
The Free Our Bills campaign is not about making Bills (or legislation) easier for humans to read and understand, it’s about making it easier to extract information from a Bill so that people can be notified when a new Bill comes along on a subject they care about, or an old Bill is redrafted, and so on.
Bills are already available for the public to view on the web, in PDF and HTML forms. The problem is that the HTML is Really Really Bad (View Source to see) and that makes it Really Really Hard to extract useful information from them.
There are reasons for the Bills HTML being Really Really Bad:
- The HTML must look exactly like it does in printed form, otherwise Members of Parliament (MPs) would get Really Really Confused.
- MPs refer to pieces of a Bill (which they might want to change) by page and line number, not by the semantic structure of the Bill, so the HTML must have page and line numbers in it or MPs would get Really Really Confused.
- Although the formatting of Bills is pretty consistent, there’s always the chance that a piece will need to be formatted specially. It might be safe to assume a particular presentation for a particular semantic 99% of the time, but if that 1% isn’t formatted in the different way, MPs would be Really Really Confused.
- The code that creates the Bill HTML was written several years ago, when browser support for CSS was Really Really Bad.
The picture for legislation is rather better because a strategic decision was made to focus on semantics rather than presentation. When a Bill is enacted, it gets converted into reasonably good semantic XML, which forms the basis of all the HTML views. It also helps that this HTML was designed fairly recently, for modern browsers; it makes heavy use of CSS so there’s relatively little obfuscation of the content.

I think there are interesting general lessons here:

Different user communities have different requirements. MPs have different requirements from Bills from the general public, who don’t care (as) much about line or page numbers. On the other hand, you need to actually consult with users about what they need rather than make assumptions about it: are MPs really likely to get Really Really Confused if the HTML presentation of a Bill looks slightly different from the PDF print version? I don’t know.
Authors don’t care about what they don’t use. When the only way of using a Bill is to print it, it’s natural that authors and publishers only care about how it looks when it’s printed. Training people to care about semantic markup is really hard, and it’s made harder by WYSIWYG tools that allow them to override the semantic style. If a difference isn’t visible, then in author’s eyes it doesn’t exist.
You have to positively decide to ignore appearance. When transforming from a WYSIWYG view, replicating appearance is the obvious thing to do. But it’s worthwhile in the long run to focus on extracting the semantics, because the resulting documents are so much more reusable.
HTML, XML and XSLT are not inherently good. Parliament wanted Bills in HTML so that they were more accessible on the web. But the HTML is dreadfully inaccessible because of the other requirements placed on it. Similarly, XML can be incredibly obfuscated, or entirely about presentation, as formats such as OOXML illustrate. And just because your code is written in XSLT does not make it inherently easier to maintain then (say) a SAX transformation. It’s easy to misuse a technology.
Developers who produce atrocious HTML aren’t necessarily ignorant. Unfortunately, there’s sometimes a limit to how much you can argue with your customers.

Extension primitives in XSDL

Jeni — Sat, 19 Jan 2008 23:00:20 +0000

Michael Sperberg McQueen (CMSMcQ) has written a couple of interesting posts about datatypes in W3C’s XML Schema (XSDL). (The second is a response to a comment from John Cowan, and attempts to justify some of the seemingly arbitrary decisions made in the set of datatypes present in XSDL 1.0.) The posts are a discussion of one of the issues against XSDL 1.1 raised by Michael Kay:

Michael proposes: just specify that implementations may provide additional implementation-defined primitive types. In the nature of things, an implementation can do this however it wants. Some implementors will code up email dates and CSS lengths the same way they code the other primitives. Fine. Some implementors will expose the API that their existing primitive types use, so they choose, at the appropriate moment, to link in a set of extension types, or not. Some will allow users to provide implementations of extension types, using that API, and link them at run time. Some may provide extension syntax to allow users to describe new types in some usable way (DTLL, anyone?) without having to write code in Java or C or [name of language here].

Since I’m principally responsible for the Datatype Library Language (DTLL) it’ll come as no surprise that I think that XSDL is currently deficient in not providing mechanisms for creating new primitive types (such as colours) or different lexical representations for the primitive types it has (such as UK-style dates). So yes, I do think XSDL would be a better schema language if it supported “extension primitives”.

In XSLT and XPath, providing extensibility hooks has proved very effective. It’s enabled implementers to innovate (and those innovations fed back into the design of XSLT 2.0 and XPath 2.0). It’s provided users with functionality (such as xxx:node-set()) that they would otherwise not have had for years, and therefore made the lives of thousands of users much easier.

On the other hand, it’s impossible to say how XSLT and XPath would have developed if those extensibility hooks hadn’t been there. Would implementers have extended the language anyway, leading to fragmentation? Would the WG have felt more pressure to get later versions of XSLT out the door if the only way the language could have been improved was through centralised changes?

I think the big thing that helped XSLT’s extensibility actually work was EXSLT (but then, I would say that, wouldn’t I?). The majority of XSLT processors implement EXSLT extensions, and even those processors that don’t implement all (or any) of EXSLT have other extensibility hooks (such as <msxsl:script> or <xsl:function>) and there are third-party implementations of EXSLT’s functions available so it’s possible to write interoperable stylesheets while still using those functions.

(EXSLT is by no means perfect: if we were doing it over again, we’d build in much better methods for receiving user contributions of various kinds. But I think the general principle is sound.)

If XSDL were to allow extension primitives, you’d hope for something similar to happen:

a repository for common extension primitives
implementations that respond to user demand for extensions in the repository
development of higher-level languages for defining extension primitives
implementations that provide hooks (in whatever way) for defining extension primitives

You can’t predict what implementers will do, but it seems likely that they’d provide hooks for users to create their own extension primitives (albeit most likely using Java or .NET or whatever rather than a higher-level language such as DTLL). And once they do that, it’s possible for the community to provide third-party implementations for extension primitives, thus retaining interoperability.

So I think it could work, if implementers do the right thing and the user community gets involved.

(Just in case you get the wrong impression: I still think RELAX NG is a vastly superior schema language to XSDL. If you need extension datatypes, you can have them in RELAX NG right now. Unfortunately, however, in the real world, you don’t always get to make the right technical choice.)

XSLT 2.0 Q&A: Linking elements in different documents

Jeni — Fri, 18 Jan 2008 17:33:14 +0000

The first of what will probably become a series of posts where I answer publicly questions that people post me privately (with permission, of course)…

How should I model (and store) data for Courses, while being able to pull info about a Course into a particular context (a Semester or Curriculum)?

I’m not quite sure how to do this in terms of writing the schema (and consequently the XML), and/or how to connect it with XSL (if that is appropriate). My experience with XSLT is limited to pretty much straight templates, and I’ve never cross-referenced two nodesets and used the result to provide a different output before.

The important thing is that within whatever XML you use, you have identifiers of some description that you can use to work out what a particular element means. The simplest and most general purpose identifiers are xml:id attributes, but they’re a bit limited because they have to be legal IDs.

If you have something like a course that’s identified by a number then it makes more sense to use the number of the course as the identifier; that can’t live in an xml:id attribute, so you have to use some other attribute (eg number for it). (You can use an element instead of an attribute, of course, but identifiers are usually metadata, and metadata should usually be an attribute unless it’s structured.)

You might have something that is actually uniquely identified by a combination of values. For example, a course might be identified by the department that offers the course plus the number identifying the course; it might be possible for two courses to have the same number, but be different courses, offered by different departments. Again, ultimately all that’s really important is that it’s possible to identify the department from the course, but the identifier itself might be on the element representing the course or on one of its ancestors, it doesn’t really matter.

In the XSLT, the first task is to pull in all the documents that hold the information you want to use and store them as global variables:

<xsl:variable name="curriculum" as="document-node()"
  select="document('curriculum.xml')" />
<xsl:variable name="transcript" as="document-node()"
  select="document('transcript.xml')" />
<xsl:variable name="database" as="document-node()"
  select="document('database.xml')" />

Then you should set up keys that will create indexes of the information in your documents based on their identifier(s). A simple key would look like:

<xsl:key name="departments" match="dept" use="@xml:id" />

The name attribute is a name for the key; you can call it anything you like, but for identifiers I usually just use the plural of a noun for the thing I’m identifying. The match attribute is a pattern that matches the elements that you’re indexing (don’t forget namespaces, if you have them). The use attribute holds an XPath that should return a value for a given node that you’re indexing, in the above case it’s the value of the xml:id attribute on the <dept> element.

For elements that use a combination of values as an identifier, you can use the concat() function to create a unique value that combines the identifying values. For example, if your XML looks like:

<courses dept="CMSC">
  <course number="131">...</course>
  <course number="198W">...</course>
  <course number="434">...</course>
</courses>

then you could index each course by its department and number with:

<xsl:key name="courses" match="course" use="concat(../@dept, ':', @number)" />

It’s not really necessary in this case, but I’ve put a separator in the concat() call out of habit as it helps prevent problems such as something identified as 'a' + 'bc' being given the same identifier as something identified through 'ab' + 'c'.

Note that the keys don’t indicate which document the information is held in. It’s only when you call the key that you say which document you want to use. For example:

key('courses', 'CMSC:198W', $curriculum)

would pull out the <course> element whose parent had a dept attribute equal to 'CMSC' and a number attribute with the value '198W' from the document held in the $curriculum variable. (This used to be harder to manage in XSLT 1.0, when there wasn’t a third argument; without the third argument, the XSLT processor looks in the document you’re currently in.)

You can just call the key() function directly, but if you’re using XSLT 2.0 then I’d suggest wrapping it in a function like this:

<xsl:function name="my:course" as="element(course)">
  <xsl:param name="dept" as="xs:string" />
  <xsl:param name="number" as="xs:string" />
  <xsl:variable name="identifier" select="concat($dept, ':', $number)" />
  <xsl:sequence select="key('courses', $identifier, $curriculum)" />
</xsl:function>

This means you can use

my:course('CMSC', '198W')

to locate the <course> element you’re after.

Of course, most of the time you won’t have fixed values for the arguments for that function: you’ll have some XML that refers to the course in its own way. For example, you might have:

<semester season="fall" year="2007">
  <course dept="ARTT" number="210" />
  <course dept="INFM" number="210" />
</semester>

and want to make a list of the titles of the courses. You could do this with:

<xsl:template match="course">
  <li>
    <xsl:apply-templates select="my:course(@dept, @number)/title" />
  </li>
</xsl:template>

Just a final thought: if you have control over it, it’s useful to make a clear distinction between elements that define information and elements that reference those definitions. One way of doing that is to name them differently (eg <course> and <courseRef>) or make sure that they have different sets of attributes (eg number and numberRef). Using the name of an element can be more useful because you can use that when you’re defining the types of parameters and return values. For example:

<xsl:function name="my:courses" as="element(course)+">
  <xsl:param name="semester" as="element(semester)" />
  <xsl:variable name="courseRefs" as="element(courseRef)+"
    select="$semester/courseRef" />
  <xsl:sequence select="$courseRefs/my:course(@dept, @number)" />
</xsl:function>

Partial implementations #2: XSLT in Google Search Appliance

Jeni — Fri, 23 Nov 2007 22:22:19 +0000

A Google Search Appliance (GSA) is a box that you plug into your network which crawls and indexes your data, and serves up the results of searches. Search results come in an XML format, and there’s a built in XSLT engine which means you can convert that XML into as many different views as you like. So you can have HTML-based search results, summaries, feeds, and so on.

My task recently was to debug some XSLT that transformed the GSA XML into an Atom feed. Easy enough, right? The GSA XML format is pretty hideous — most of the elements max out at three capital letters in length (whatever happened to human-readability) — but logical enough, and the mapping is hardly complex.

But all was not as it seemed. The GSA’s XSLT implementation is… how can I put this politely?… “non-standard”. This post describes some of the problems and workarounds.

To get the GSA to use your own XSLT, you have to go through its web interface. Basically there’s a form with a text field in which you can type your XSLT. Or you can upload a file that you develop offline. Naturally you’re going to do the latter because it means you can use your favourite editor with helpful things like syntax highlighting and validation-as-you-type, but of course that means switching between web browser windows and your IDE as you develop.

So I upload the transformation, point the browser at a relevant search page, and… oh…

When the GSA doesn’t like the XSLT that you use, you get a really helpful error message. It says:

Internal server error.

So you know that there’s been an error. With the server. Internally.

Back to basics, I thought. Let’s find out what processor the server’s using. Then we can develop on that processor and be pretty sure the resulting XSLT will work. So I load up the default XSLT (which is used to create an HTML result) and add the line

<xsl:value-of select="system-property('xsl:vendor')" />

Save the XSLT, reload the page, and…

Internal server error.

Okaaay… so this is an XSLT processor that doesn’t support the xsl:vendor system property. If it doesn’t support that, I’m going to have to tread carefully. So let’s start with something really simple:

<xsl:stylesheet version="1.0"
   xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:template match="/">
  <xsl:copy-of select="." />
</xsl:template>
</xsl:stylesheet>

Save the XSLT, reload the page, and…

Internal server error.

On a whim, I tried

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
   version="1.0">

<xsl:template match="/">
  <xsl:copy-of select="." />
</xsl:template>
</xsl:stylesheet>

instead. Save the XSLT, reload the page, and… Success!

Can you spot the difference? Yes, that’s right: it’s the order of the XSLT namespace declaration and the version attribute. Namespace declaration first, you’re OK, version first, you’re not.

Okaaay… so this is an XSLT processor that doesn’t support the XML Recommendation (which says that attribute order doesn’t matter). But heck, why split hairs? At least it’s working! Now to create some Atom instead:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
   version="1.0"
   xmlns="http://www.w3.org/2005/Atom">

<xsl:template match="/">
  <feed />
</xsl:template>
</xsl:stylesheet>

Save the XSLT, reload the page, and we’re back to

Internal server error

At least there’s some documentation about this one:

XSL stylesheets that include other files may not be used with the Google search engine. An XSL stylesheet that contains the following tags generates an error result:

<xsl:import>

<xsl:include>

xmlns:

document()

Read that again. Yes, the third bullet point. That’s right, it’s saying that an XSLT that contains a namespace declaration will generate an error result because it “includes other files”.

But, but, but, namespace declarations in XSLT stylesheets (or elsewhere for that matter) do not indicate file inclusion. Namespace URIs are identifiers, not locations. They are strings. They are not resolved. You do not need to be connected to the ‘net to use them.

And how am I supposed to serve an Atom feed, since Atom documents use a namespace? Or XHTML for that matter? Fortunately, the GSA only goes so far in banning namespace declarations: you’re OK as long as you don’t put them on the <xsl:stylesheet> element. Moving it to the <feed> element as in

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
   version="1.0">

<xsl:template match="/">
  <feed xmlns="http://www.w3.org/2005/Atom" />
</xsl:template>
</xsl:stylesheet>

and you’re OK. Of course you have to repeat the namespace declaration in every template so you don’t end up creating elements in no namespace. Tedious, oh so tedious, but workable.

(I have a vague suspicion that the idea behind banning namespace declarations is something to do with certain XSLT processors using namespace URIs to pull in Java classes. But addressing that problem by banning namespace declarations entirely isn’t just throwing the baby out with the bathwater, it’s throwing the whole bathroom suite out of the window. And if you then allow namespace declarations further down the stylesheet, you haven’t actually solved the problem.)

Amazingly enough, given the inauspicious beginning, everything else I tried actually worked. I suspect that it’s some standard XSLT processor underneath with a regex based filter that (among other things) limits what’s allowed in the <xsl:stylesheet> start tag. They probably disallow system-property('xsl:vendor') for security — knowledge is power, after all.

Anyway, my suggestions to others who might want to create a customised XSLT processor:

Use a custom URL resolver to restrict access to documents.
Restrict external function calls using something like the ALLOW_EXTERNAL_FUNCTIONS property in JAXP
Document the restrictions you’re placing on the stylesheets.
Produce meaningful error messages that explain the extra restrictions when they’re broken.

Detecting streamability in XPath expressions and patterns

Jeni — Tue, 06 Nov 2007 19:45:45 +0000

The XSL Working Group gave some comments recently on the Last Call Working Draft of XProc. One of the comments was about a bunch of standard steps that we’ve specified which do things you can do in XSLT, such as renaming certain nodes. These steps generally use XPath expressions or XSLT patterns to identify which nodes should be processed.

What bothers the XSL WG is that these steps aren’t guaranteed to be streamable. In a streamable process, an input document can be delivered to the processor as a stream of events (and an output similarly generated as a stream of events) rather than as an in-memory representation. Such processes will start producing results more quickly and require less memory than non-streamable ones. And, because they don’t need as much memory, they are able to work on larger documents.

If the processes we defined in XProc were streamable, there’d have a clear advantage over their XSLT equivalents, and therefore a purpose. However, since they’re not guaranteed streamable, it looks like we’re simply creating yet another transformation language.

My response was basically that we left it down to implementers to detect when a particular expression/pattern was streamable because defining a streamable subset of XPath would (a) take too long, (b) require people to learn a particular XPath subset, raising the adoption barrier, (c) require implementers to implement their own XPath engines, raising the implementation barrier.

But if you put those pragmatic reasons to one side, I think there are good abstract reasons not to specify a streamable XPath subset. First, there is no clear line that can be drawn between a streamable XPath and an unstreamable one, only a scale between “buffering nothing” and “buffering everything” (building an object model). Second, you can’t judge the streamability of an XPath expression on its own: there are multiple other factors that effect how streamable a given XPath expression is for a particular processor.

To illustrate, say that we’re renaming all elements that we select, and let’s start with an expression that’s obviously streamable:

//section

No problems here: as soon as we hit an start-tag (or end-tag) for a <section> element, we can change its name.

Now add a predicate:

//section[@type = 'summary']

This predicate tests the value of the type attribute on the <section> element. If we’re using SAX or StAX events, then this is as straightforwardly streamable as the previous example, because attribute values are reported at the same time as start-tags. But that’s purely down to the API: the underlying algorithm for streaming RELAX NG validation uses a different event model, for example, in which attributes are reported after the start tag begins (and before the start tag ends). So streamability depends on the event model.

Now a different predicate:

//section[title = 'Summary']

This predicate tests the value of the <title> child of the <section> element. In fact, it tests if the <section> element has any <title> child with the value 'Summary'. Normally, an XPath processor won’t be able to tell that a <section> doesn’t satisfy the predicate until it gets to the end-tag of the element. So it will have to buffer the events from each <section> start tag until its end tag until it can work out whether to do the renaming or not.

But say the <section> elements in this markup language can only contain a single <title>, and that’s the first child of the <section>. For an XPath processor that’s aware of the DTD or schema that the document adheres to, the situation is then very similar to the previous one, which tested the attribute. So streamability depends on how much the processor knows about the markup language.

Changing the XPath to

//section[title[1] = 'Summary']

similarly limits how much the processor will have to buffer if the <title> always appears, and always appears first, even without the processor being told that rule through a schema. So streamability depends on the markup language itself.

Anyway, I had a quick look at some academic work on streamability, such as XSQ, TurboXPath or the recent paper “Efficient Algorithms for Evaluating XPath over Streams”. These papers really surprised me. The things that prove difficult include backwards axes (which is surprising since information about the previous nodes should be easily available), the descendant axis, and the position function. On the other hand, predicates are absolutely fine (despite requiring a “look ahead”). [Weirdly enough, all the papers I looked at contained XPath errors; I guess when you’re considering abstract algorithms you don’t have to care about insignificant things like language syntax.]

The paper I mentioned above actually defines something called Univariate XPath which conforms to the syntax:

Path      := Step | Path Step
Step      := Axis NodeTest
             | Axis NodeTest '[' Predicate ']'
Axis      := '/' | '//'
NodeTest  := Name | '*'
Predicate := Path
             | Path CompOp Path
             | Predicate 'and' Predicate
             | Predicate 'or' Predicate
             | 'not' Predicate                            [sic]
CompOp    := '=' | '!=' | '>' | '>=' | '<' | '<='

This might be a useful starting point, but it omits useful things like attributes and functions which (as far as I can tell) wouldn’t effect the applicability of the algorithms. It’s also worth noting that it will allow paths such as /database[dummy]/record, which would involve buffering every <record> until the end tag of the <database> document element was reached. This illustrates that just because an XPath is theoretically streamable (can be evaluated based on a stream of events) doesn’t mean it can be evaluated efficiently.

Some final thoughts:

I wonder if there’s scope for an XPath subset that can be mapped to RELAX NG syntax and therefore evaluated using Brozozowski derivatives
what about an algorithm that evaluates XPaths using a pipeline process, whereby the stream of events is actually passed through several filters in order to provide the final evaluation
I’m sure there’s preprocessing that could be done on some XPath expressions that would increase their streamability

Teaching XSLT 1.0

Jeni — Fri, 05 Oct 2007 22:28:37 +0100

I’m exhausted after two days of teaching XSLT 1.0. Yes, there are still people out there who want to learn it. The exhaustion comes mainly because I’m an introvert (INFP, Myers-Briggs fans!) who finds it tiring just being in the same room as someone else.

In fact, I’ve been teaching XSLT 1.0 rather a lot in the past couple of months. The first lot was a bunch of C# programmers who had done some light XSLT work, the second a bunch of developers who’d been using XSLT for years, but wanted to improve. The majority of people on this most recent course weren’t even developers.

It’s interesting seeing who struggles and who sails through the course. Some observations:

You need to understand XML to use XSLT. At the very least, you need to know what elements and attributes look like in an XML document, so you just know that attributes need matching quotation marks and elements need end tags. Half the syntax problems learners have with XSLT are because they’re not using proper XML syntax. Without that fundamental terminology, you haven’t a hope, because you can’t even talk about XSLT, let alone getting information out of an XML document. I guess this is an advantage of XQuery: at least as a teacher you know you have to teach the basic syntax of the language, rather than taking it as a given. Actually, scratch that: you still have to teach XML syntax with XQuery, since that’s what people are processing and generating, and you have to teach the bastardised, almost-XML syntax that XQuery uses. I haven’t done an XQuery course yet, but surely that must be confusing?
I always always always have to spend a long time explaining namespaces. That’s not going to surprise anybody. In the end I just provide these rules of thumb:
1. Declare the XSLT namespace (usually xsl prefix)
2. Declare the namespaces you want to appear in the result, with the prefixes that you want to appear in the result. This can include a default namespace declaration.
3. Declare the namespaces that appear in the source, with prefixes for every one (even if they’re usually the default namespace in the source). Add an exclude-result-prefixes attribute that lists these prefixes (at least the ones that aren’t also used in the result).
4. Declare the namespaces that you use in the stylesheet. List the ones that are used for extensions (elements or functions) in extension-element-prefixes (technically, you don’t have to declare the ones you use for extension functions; I just think it’s clearer if you list all the prefixes used for extensions here); make sure the other ones are listed in exclude-result-prefixes.
Having a programming background is a blessing and a curse when learning XSLT. It’s a blessing because you understand basic principles like “instructions” and “expressions” and “operators” and “functions” and “code blocks”. But it’s a curse because most programmers use conventional programming languages like C# and Java, which are procedural, and XSLT’s way of doing things is completely different. The most recent course I taught was to people who dealt with completely data-oriented XML; trying to explain to them why applying templates is a useful thing to do was really hard.
Creating exercises that are based on the markup languages used by the people attending the course is always worthwhile. It takes time to prepare, but they get a better grasp on what’s going on in the transformations that they create (because they understand the domain), and they learn how to do things on the course that are going to be directly useful for them in their ongoing practice.
The students find the examples the most helpful things in the slides. Whenever I see them look at the slides as they’re doing the exercises, they’re looking at the examples, not the abstract descriptions of how something works. Perhaps I should try having slides with (almost) nothing but examples. (It’s interesting, because I’m always interested in the theoretical underpinnings of the things I learn about — if I don’t understand why, I can hardly understand how — but it seems I’m in the minority; it’s the N in INFP, I guess.)
Having a bit of enforced interaction in the middle of the presentation works well, such as a list of examples and going around the room asking the students to explain what each means. It wakes up the students who are falling asleep. It forces them to think about what they’ve been listening to. It highlights parts that they haven’t understood (so I know they need a fresh explanation). And it seems to encourage them to ask questions. Just asking a question to the room doesn’t work so well — it tends to be the same people that answer each time, or there’s an embarrassing silence.
I’ve been trying to use a Socratic Method when I’m asked for help during the exercises, at least when the problem isn’t “obvious” (like a mis-spelled element name). The difficulty for me is finding the right question to ask, but eventually I find one that they’re able to answer easily, and things flow well from there. More often than not, the student’s able to find the answer to their problem without me providing the solution, which has to be good for learning.
In the end, the main things that differentiate between those that learn a lot and those that learn a little are a willingness to try (those who don’t attempt the exercises don’t absorb what I’ve taught from the front of the class) and a willingness to ask questions (those who don’t end up getting stuck on one thing, and can’t move past it). I also think cut-and-paste coders learn less — cut-and-paste coding practices recognition (finding the right example to cut-and-paste) rather than generation (creating something new from scratch) — but maybe that’s just snobbishness on my part.

Anyway, I’m hoping I can get rid of my XSLT 1.0 course soon, and move on to teaching XSLT 2.0. It’ll be a longer course, but there’ll be less “no, there’s no built-in support for that in XSLT 1.0”.

View-Model-Template

Jeni — Fri, 27 Jul 2007 11:02:04 +0100

I don’t know anything about Struts 1, but Bill de hÓra’s recent post has got some interesting web-application-design tips. There were two particular bits that spoke to me:

struts-config.xml struts-config tries to capture primarily the flow of application state on the server, by being an awkward representation of a call graph. In doing it misses a key aspect of the web - hypertext. In web architecture, HTML hypertext on the client is the engine of application state, not an XML file on the server.

In other words (I think) in web applications your state in the page you’re on and taking action is about following the links (or submitting the forms) on the page. Your actions (and therefore the transitions between different states) are determined by what links and forms are on the page. But in fact, URLs should be hackable, and transitions unlimited. When you design the application what you really need to think about are the tasks the users want to achieve (and therefore the transitions that they might want to make) rather than the possible state transitions.

On the web, a suitable pattern is View, Model, Template [rather than Model, View, Controller (MVC)]. A request to a URL is dispatched to a View. This View calls into the Model, performs manipulations and prepares data for output. The data is passed to a Template that is rendered an [sic] emitted as a response. ideally [sic] in web frameworks, the controller is hidden from view. Note that this framework style is often called MVC anyway, confusing matters somewhat; The key differences are that Views and Templates are cohesive and Controllers are pushed down into the framework infrastructure.

I’ve been thinking recently about whether and how XSLT might fit into a Ruby on Rails set-up. In RoR, the controller usually either queries the database (via the model) to set up instance variables, and then renders a (template) view, or updates the database (via the model) and redirects to another view. The templates (for (X)HTML) use fairly standard <% ... %> placeholders to hold code and insert values.

I’ve spent most of my professional life cursing (X)HTML documents with <% ... %> in, because they use unescaped less-than-signs and therefore can’t be generated or processed by XML tools, particularly XSLT. There’s an advantage of having templates that are themselves well-formed, not least that you can easily process the templates themselves (for example to generate, update or document them). Plus if your templates are declarative, rather than containing embedded code, you aren’t tied to a particular framework: I could move templates from Ruby on Rails to Django and they wouldn’t need modification. When I think “declarative templates”, I think “XSLT”.

The other advantage of using XSLT is that it can be used on the client side as well as the server side. So there’s the possibility of moving that rendering from one server to client completely or using it on particular clients, perhaps in an AJAX set-up, while having the same stylesheets on the server for those browsers that don’t support client-side XSLT.

You still need a way of getting the data from the model into the stylesheet, which can be done through a combination of XML and parameters. The XML is itself a view of the model, of course, but if you’ve got any kind of intention to make your web application mashable, you’re going to want to generate XML, probably Atom, anyway (yeah, or JSON, but it’s easy enough to get from XML to JSON using XSLT too). If you add caching to the equation, this approach might help reduce database requests.

So I think that using XSLT as a templating language, even within a RoR framework, has at least something going for it. What I hope is that I’m not falling into the “when you’ve got a hammer everything looks like a nail” trap.

The perils of default namespaces

Jeni — Sun, 01 Jul 2007 20:32:44 +0100

A lot of people run into problems with namespaces, and most of those arise from using default namespaces (ie not giving namespaces prefixes). The transformation technology you use can have a big effect on how confusing and irritating it gets.

Default namespaces make XML documents easier to read because they allow you to just give the local name of an element rather than using prefixes all over the place. For example, using:

<house status="For Sale" xmlns="http://www.example.com/ns/house">
  <askingPrice>...</askingPrice>
  <address>...</address>
  <layout>...</layout>
</house>

rather than:

<h:house status="For Sale" xmlns:h="http://www.example.com/ns/house">
  <h:askingPrice>...</h:askingPrice>
  <h:address>...</h:address>
  <h:layout>...</h:layout>
</h:house>

In some cases, specifically documents that are validated against a DTD or interpreted by non-namespace-aware applications, you might be forced to use the default namespace. The biggest example of this is (X)HTML.

In transformation technologies, such as XSLT, XQuery and XLinq in VB.NET, you have to deal with at least two documents: the source documents that you are processing and the result documents that you are creating. Often, the source and result documents will use default namespaces, or at any rate you’ll want to query and create the documents without using prefixes. Sometimes, the source and result documents all use the same namespace, but it’s far more common that they don’t.

So transformation technologies have to support at least two default namespaces: one for querying and one for construction.

In XPath 1.0, you must specify a prefix for each namespace you want to use. A path like /house/layout will only select <layout> elements in no namespace. In XSLT 1.0, the default namespace in the stylesheet (as declared by the xmlns attribute on <xsl:stylesheet>) is then free to be used for the result documents. For example, I might do

<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:h="http://www.example.com/ns/house"
  exclude-result-prefixes="h"
  xmlns="http://www.w3.org/1999/xhtml">

<xsl:template match="h:house">
  <div class="house">
    <h1><xsl:apply-templates select="h:askingPrice" /></h1>
    ...
  </div>
</xsl:template>

</xsl:stylesheet>

[The best way to deal with multiple result documents in different default namespaces is to simply have different stylesheet documents to handle their generation, all included or imported into your main stylesheet application.]

Users of XSLT 1.0 found it confusing that they couldn’t just copy the namespace declarations (including a default namespace declaration) from a sample source document and have the paths just work. So in XPath 2.0, rather than no prefix meaning no namespace, the default element/type namespace in the context is used for element names with no prefix. If the default element/type namespace is set to http://www.example.com/ns/house then the path /house/layout will select all <layout> elements in the http://www.example.com/ns/house namespace. You can set this default element/type namespace in XSLT 2.0 using the [xsl:]xpath-default-namespace attribute, which can go anywhere but will usually be situated on the <xsl:stylesheet> element (in which case it appears without the xsl: prefix). The default element/type namespace can be scoped to a particular area of your stylesheet in the same way as namespace declarations.

Otherwise, XSLT 2.0 works like XSLT 1.0 in that the default namespace in the stylesheet supplies the default namespace for created elements, so you can do:

<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns="http://www.w3.org/1999/xhtml"
  xpath-default-namespace="http://www.example.com/ns/house">

<xsl:template match="house">
  <div class="house">
    <h1><xsl:apply-templates select="askingPrice" /></h1>
    ...
  </div>
</xsl:template>

</xsl:stylesheet>

By keeping the default query namespace and the default construction namespace separate, you’re able to use unprefixed names in both paths and element constructors, even if the default namespaces in the two cases are different.

XQuery and VB.NET, on the other hand, provide a single default namespace that is used for both queries and construction, and they work in slightly different ways.

In XQuery you can declare the default namespace for the query, with

declare default element namespace "http://www.example.com/ns/house";

which means that you can query the source document with paths like /house/askingPrice and create elements in the http://www.example.com/ns/house namespace with direct element constructors without prefixes, like

<house status="For Sale">
  <askingPrice>...</askingPrice>
  <address>...</address>
  <layout>...</layout>
</house>

If you then want to generate XHTML (or some other result for which you’d prefer to use the default namespace), you can use a default namespace declaration on the XHTML you generate:

<div class="house" xmlns="http://www.w3.org/1999/xhtml">
  <h1>...</h1>
  ...
</div>

But the default namespace declaration in the element constructor carries through into the embedded expressions, so

  <div class="house" xmlns="http://www.w3.org/1999/xhtml">
    <h1>{ /house/askingPrice }</h1>
    ...
  </div>

won’t work. As a result, you end up having to use the query’s default namespace declaration to set the default namespace to XHTML (or whatever the default namespace is in the result), and use prefixes in your queries (essentially the same situation as in XSLT 1.0).

In XLinq in VB.NET, there’s the same kind of pattern. The Imports statement allows you to declare a default namespace that’s used in both queries and construction, as in:

Imports <xmlns="http://www.example.com/ns/house">

and you can use a default namespace declaration on the XHTML you generate to provide the default namespace for the elements in the XML literal:

houseDiv =
  <div class="house" xmlns="http://www.w3.org/1999/xhtml">
    <h1>...</h1>
    ...
  </div>

But unlike in XQuery, the default XHTML namespace declaration in the <div> doesn’t have an effect on the default namespace used in embedded expressions, which means you can still use unprefixed element names in any paths used within the XML literal, like this:

houseDiv =
  <div class="house" xmlns="http://www.w3.org/1999/xhtml">
    <h1><%= doc.<house>.<askingPrice> %></h1>
    ...
  </div>

However, if you build up your XHTML gradually, perhaps by using separate variables or methods, then every time you create a snippet of XHTML you have to specify this default namespace. Again, people will end up using the Imports statement to set the default namespace to the default result namespace and using prefixes in their paths.

The other factor to consider is that sometimes no prefix really does mean no namespace. If you’re querying an XML document that contains elements in no namespace, you have to set the default query namespace to no namespace. In XSLT 1.0, that’s always the case anyway; in XSLT 2.0, the xpath-default-namespace shouldn’t be set (or should be unset for those places that need to query no-namespace elements). In XQuery you can’t use the query default namespace declaration and in XLinq in VB.NET you can’t use the Imports statement. In both these cases, you better hope your result is in no namespace too. If not, the best route (to make it work at all in XQuery, and to avoid repetitive xmlns attributes in VB.NET) is to create a no-namespace version of your result first, and have a standard function or method that will add the right default namespace to that result.