schema

RELAX NG for matching

Jeni — Thu, 06 Mar 2008 14:59:03 +0000

I’m still thinking about doing automatic markup with XML pipelines, and the kind of components that you might need in such a pipeline. These are the useful ones (list inspired by the components offered by GATE):

a tokeniser that uses regular expressions to add markup to plain text
a gazetteer that uses a lookup to add markup to plain text
an annotater that adds attributes to existing elements based on their context/content
a grouper that adds markup around sequences of existing markup
a stripper that removes markup
a general purpose transformer that uses XSLT to do just about everything else

The “grouper” is the most interesting and difficult of these. It needs to act like a tokeniser, except use regular expressions over markup rather than over text. For example, say I had:

<number>06</number><punc>/</punc><number>03</number><punc>/</punc><number>08</number>

I want to be able to create a rule that says “any sequence that looks like a number element that contains a two-digit number between 1 and 31, followed by a punc element that contains a slash, followed by another two-digit number between 1 and 12, followed by a punc element that contains a slash, followed by another two-digit number should be wrapped in a date element”.

Now this is something that XPath is really bad at. Try writing an expression that selects, from a sequence of elements that may contain other <number> and <punc> elements as well as other elements, only those sequences of elements that match the pattern I just described. It’s something like:

number[. >= 1 and . <= 31 and string-length(.) = 2]
      [following-sibling::*[1]/self::punc = '/']
      [following-sibling::*[2]/self::number[. >= 1 and . <= 12 and string-length(.) = 2]]
      [following-sibling::*[3]/self::punc = '/']
      [following-sibling::*[4]/self::number[string-length(.) = 2]]
  /(self::number, following-sibling::*[position() <= 4])

which is fiddly and messy and only works in this particular example because I know precisely how many elements there are supposed to be in the group.

In fact, it’s even difficult to do this kind of grouping using XSLT, even with <xsl:for-each-group> because the grouping is designed around elements either returning the same value or starting or ending with a particular kind of element, rather than grouping together a sequence that has a particular internal structure.

The language that is designed to describe sequences of elements is RELAX NG. Obviously RELAX NG is really useful as a schema language, but it’s really all to do with defining regular expressions over XML structures. We can use RELAX NG to describe the pattern of elements we want to match:

<group>
  <element name="number">
    <data type="integer">
      <param name="minInclusive">1</param>
      <param name="maxInclusive">31</param>
      <param name="pattern">[0-9]{2}</param>
    </data>
  </element>
  <element name="punc">
    <value>/</value>
  </element>
  <element name="number">
    <data type="integer">
      <param name="minInclusive">1</param>
      <param name="maxInclusive">12</param>
      <param name="pattern">[0-9]{2}</param>
    </data>
  </element>
  <element name="punc">
    <value>/</value>
  </element>
  <element name="number">
    <data type="integer">
      <param name="pattern">[0-9]{2}</param>
    </data>
  </element>
</group>

or, in compact syntax:

element number { 
  xs:integer { minInclusive = "1" maxInclusive = "31" pattern = "[0-9]{2}" }
},
element punc { "/" },
element number { 
  xs:integer { minInclusive = "1" maxInclusive = "12" pattern = "[0-9]{2}" }
},
element punc { "/" },
element number { 
  xs:integer { pattern = "[0-9]{2}" }
}

As a language, RELAX NG is really good at this. You could even imagine adding attributes to name subexpressions which you could then do things with (in the same way as you can get the substring matching a subexpression when you use a regular expression over text).

So I think a “grouper” component should use RELAX NG to identify sequences to be marked up. But I have no idea if there are RELAX NG libraries out there that can be used in this way: to identify and extract matching sequences rather than to validate entire documents.

Extension primitives in XSDL

Jeni — Sat, 19 Jan 2008 23:00:20 +0000

Michael Sperberg McQueen (CMSMcQ) has written a couple of interesting posts about datatypes in W3C’s XML Schema (XSDL). (The second is a response to a comment from John Cowan, and attempts to justify some of the seemingly arbitrary decisions made in the set of datatypes present in XSDL 1.0.) The posts are a discussion of one of the issues against XSDL 1.1 raised by Michael Kay:

Michael proposes: just specify that implementations may provide additional implementation-defined primitive types. In the nature of things, an implementation can do this however it wants. Some implementors will code up email dates and CSS lengths the same way they code the other primitives. Fine. Some implementors will expose the API that their existing primitive types use, so they choose, at the appropriate moment, to link in a set of extension types, or not. Some will allow users to provide implementations of extension types, using that API, and link them at run time. Some may provide extension syntax to allow users to describe new types in some usable way (DTLL, anyone?) without having to write code in Java or C or [name of language here].

Since I’m principally responsible for the Datatype Library Language (DTLL) it’ll come as no surprise that I think that XSDL is currently deficient in not providing mechanisms for creating new primitive types (such as colours) or different lexical representations for the primitive types it has (such as UK-style dates). So yes, I do think XSDL would be a better schema language if it supported “extension primitives”.

In XSLT and XPath, providing extensibility hooks has proved very effective. It’s enabled implementers to innovate (and those innovations fed back into the design of XSLT 2.0 and XPath 2.0). It’s provided users with functionality (such as xxx:node-set()) that they would otherwise not have had for years, and therefore made the lives of thousands of users much easier.

On the other hand, it’s impossible to say how XSLT and XPath would have developed if those extensibility hooks hadn’t been there. Would implementers have extended the language anyway, leading to fragmentation? Would the WG have felt more pressure to get later versions of XSLT out the door if the only way the language could have been improved was through centralised changes?

I think the big thing that helped XSLT’s extensibility actually work was EXSLT (but then, I would say that, wouldn’t I?). The majority of XSLT processors implement EXSLT extensions, and even those processors that don’t implement all (or any) of EXSLT have other extensibility hooks (such as <msxsl:script> or <xsl:function>) and there are third-party implementations of EXSLT’s functions available so it’s possible to write interoperable stylesheets while still using those functions.

(EXSLT is by no means perfect: if we were doing it over again, we’d build in much better methods for receiving user contributions of various kinds. But I think the general principle is sound.)

If XSDL were to allow extension primitives, you’d hope for something similar to happen:

a repository for common extension primitives
implementations that respond to user demand for extensions in the repository
development of higher-level languages for defining extension primitives
implementations that provide hooks (in whatever way) for defining extension primitives

You can’t predict what implementers will do, but it seems likely that they’d provide hooks for users to create their own extension primitives (albeit most likely using Java or .NET or whatever rather than a higher-level language such as DTLL). And once they do that, it’s possible for the community to provide third-party implementations for extension primitives, thus retaining interoperability.

So I think it could work, if implementers do the right thing and the user community gets involved.

(Just in case you get the wrong impression: I still think RELAX NG is a vastly superior schema language to XSDL. If you need extension datatypes, you can have them in RELAX NG right now. Unfortunately, however, in the real world, you don’t always get to make the right technical choice.)

XTech 2007: Wednesday 16th May Afternoon

Jeni — Sun, 20 May 2007 22:52:14 +0100

Yes, I’m determined to write up every talk I attended at XTech 2007, so that I have a record of it if nothing else. On Wednesday afternoon, I attended sessions on microformats, internationalisation and NVDL (as well as giving my own talk, of course).

Microformats: the nanotechnology of the semantic web

Jeremy Keith

This was a supremely well-put-together presentation on microformats: beautiful slides, drama and humour, and a reference to Neal Stephenson’s Diamond Age (was I really one of only three people in the packed room to have read it?). There was a lot about what microformats are, how they’re designed, what their niche is (Jeremy was very up-front about the fact they don’t solve every problem), and how they’re developed. But there weren’t any demonstrations of microformat-based applications, which I would have really liked to see. The other thing I thought was worth noting was that Jeremy talked about the dangers of “grey goo” (he was using a nanotechnology metaphor): the proliferation of microformats. He expressed the strong desire that the set of microformats be kept small, and even said (I paraphrase) “Do use semantic class names in your HTML, but don’t call them microformats [unless they’ve been through the microformats standardisation process]!”

Liam Quin gave a paper entitled Microformats: Contaminants or Ingredients at Extreme last year, asking what we, as traditional markup geeks, should do about them. Some were very sceptical, saying something along the lines of “They’re headed for a trainwreck; and we should sit back, watch it happen, and pick up the pieces.” Others wanted to celebrate: the fact that tagging has become understood is really good news for the semantic web, open data and all that jazz.

Both the traditional markup and the microformats community have the same goals: they want to make information easier to search for, to query, to integrate and so on. The microformats approach is to minimise the cost to those supplying information, and to target just a few, very common, kinds of data such as contact information, events and social networks. Traditional markup, on the other hand, aims to cover every single kind of information you might want to make available, and has to worry about issues like validating, styling, and distinguishing between tag sets.

It seems that a fundamental problem is that the benefits of including semantic markup aren’t immediately obvious to the supplier. Whether you use semantic class names in HTML or use elements in known namespaces, it’s purely a matter of faith that this will make your information easier to locate or use. You can’t know that search engines will include that information in their weighting algorithms, or that people reading your page will have the screen-scraping software necessary to pull anything out. With so little (obvious) benefit, authors will only supply semantic data if the cost is low. Adding class names to existing HTML elements is easy whether a web page is generated by hand or automatically. Adding namespaces and authoring special CSS might not be that much more costly to do, but it’s much more costly to grok.

So if we want authors to start putting elements in their own namespaces in their web pages, we need an application that immediately cranks up the benefit of doing so. I have no idea what that is.

Applying the Internationalization Tag Set

Yves Savourel

This was a good introduction to [a standard] I only knew about vaguely. It’s definitely worth knowing about the its:* attributes for defining i18n features such as indicating which content should be translated, which are terms, providing comments for localisation and so on, just in case you need to build those in to new markup languages.

I also have much admiration for how the ITS standard doesn’t expect people to completely rework their markup languages to incorporate ITS data. Instead of using the ITS attributes directly in a document, you can use global rules embedded in the document itself, referenced from the document, or embedded in the schema for the document. I think this approach will prove useful in the development of LIX, when we get around to formalising it.

NVDL - a breath of fresh air for compound document validation

Jirka Kosek & Petr Nálevka

NVDL is Part 4 of DSDL, specifically targeted at organising the validation of documents that incorporate multiple namespaces, such as XHTML documents containing islands of SVG, RDF and MathML. NVDL’s approach is to identify subtrees within the document that need to be validated against a particular schema. The subtrees don’t need to only hold one namespace, but often that will be the case.

The XML Schema wonks in the room (Henry Thompson and Michael Sperberg-McQueen) were a bit befuddled, I think, because with XML Schema you just supply a whole bunch of schema documents to the processor, for different namespaces, and as long as the schemas contain wildcards they’ll do the right thing. The concept of supplying multiple schemas to a validator isn’t part of RELAX NG’s validation approach, so you need something like NVDL if you don’t want to rework your schema for every combination of namespaces.

Henry and Michael were particularly concerned about the fact that it means you can override the original schema, allowing elements from foreign namespaces in situations where the original schema hasn’t allowed them. But as Henry said, it just means that the primary schema you use to define what’s allowed where is actually an NVDL schema: it’s not auxiliary validation like Schematron is, but a language for the primary schema you use.

Later, I wondered how much the XProc work would render NVDL irrelevant. After all, XProc can invoke validation of subtrees against multiple external schemas. On the other hand, NVDL’s syntax is going to be easier to use if that’s all you want to do. Perhaps someone will write a tool to convert NVDL schemas to XProc pipelines…

Actually, Jirka & Petr’s experience with JNVDL is interesting from the XProc viewpoint, in particular the problems that they had with reporting meaningful line numbers when validating subtrees. Something that XProc implementers might want to look at in regard to error reporting with <p:viewport>.

XTech 2007: Wednesday 16th May Morning

Jeni — Thu, 17 May 2007 21:50:59 +0100

Since there’s next to no ‘net connection at XTech 2007 (obviously the Web is not so ubiquitous as all that), I have nothing to do in the sessions but listen! Here are some thoughts about the sessions that I attended on the morning of Wednesday 16th. I haven’t included the keynotes not because they weren’t interesting but because I can’t think of anything to say about them at the moment.

XML and LINQ: What’s New in Orcas and Beyond

Erik Meijer (Microsoft)

I thought I’d better go to this one because I’m supposed to be talking about XML APIs at this year’s XML Summer School and LINQ, or XLINQ, is one of the hot topics. I’m not a .NET developer, so it’s all kinda passed me by thus far, and I’m not sure I really understand it now. (I’d welcome corrections and clarifications.) The three things that seemed to be important are:

You can get at information held in objects, databases or XML using the same syntax. (Erik showed accessing XML with faulty XQuery syntax, which made me and Priscilla Walmsley grimace at each other.) This means you can decide how you want to actually hold your data further down the line. A big distinction between previous attempts to work across paradigms is that the data doesn’t get converted, but the queries do. So you write your LINQ query in LINQ syntax and it gets mapped on to SQL to query your SQL database, or on to XQuery (I guess) to query your XML document. This all seemed to assume data-oriented information: I have no idea, yet, how or whether mixed content gets handled.
XML is a “first class datatype” in LINQ, so to create static XML you just write XML in your program (a bit like in XQuery). The example Erik showed included an XML declaration, which is just plain weird: dunno if that was an error or it’s a way of indicating what version of XML you’re using, or what. To create dynamic portions of the XML, you use <%=...%> “expression holes” which can contain .NET code, including calls to a new API for creating XML elements and attributes (a DOM replacement).
Erik talked about writing applications in .NET and then automatically refactoring them (with a click in a context menu) to work in client/server architectures, and refactoring again to work across several clients. Presumably this creates all the code necessary to make the application work with WS* messaging, so you don’t have to program it. This all sounded really dodgy to me: I don’t want to rely on a tool to make a language/approach/architecture usable.

There was an amusing digression into the art of rendering triangles, and thus three-dimensional models, with zero-width, zero-height, bordered <div>s in XHTML. And a mention of the “backbutton” problem that you get when you spawn tabs/windows in your web browser and then go back to your original tab/window and hit submit, which made me think that perhaps a RESTful architecture would make a whole lot of complexity go away.

Data Model Perspectives for XML Schema

Felix Michel (ETH Zurich), Erik Wilde (UC Berkeley)

Felix mentioned that I might be interested in his talk in a comment here, and sure enough I found it fascinating. He’s created a single-file representation of XML Schemas (consolidating schemas that, by virtue of using different namespaces, must be in different physical documents), and a set of XSLT 2.0 user-defined functions that provide access to and queries on the XML Schema information.

For example, you can go from an instance element in your document to its type, find out if it’s an extension or restriction, go to its base type, look at the annotations on it, and so on and so on. And all this in Basic XSLT 2.0 (the functions that work on instance elements traverse the instance document and schema in parallel to locate the element declaration that applies). You could use these functions to do everything you can do in Schema-Aware XSLT 2.0, with more flexibility, at the expense of performance.

He also mapped content models onto <occurrence> elements that encode the “follow set” for a particular occurrence, so you can easily answer the question “what elements could come next?”. I can’t immediately think of a way of using that information in a stylesheet, but perhaps he can describe one.

Anyway, I think Felix’s point was not to provide XSLT programmers with a set of useful functions, but to demonstrate the kind of standard, fairly light-weight, API that we might use to access XML Schema information. There was some discussion, in the development of XPath 2.0, of providing this kind of API, but getting agreement on XDM was hard enough!

However, my thoughts were veering off in different directions. To my mind, validation and annotation are separable processes, and the data types, element groups and linking behaviour that you might find useful on a data set are processing-specific. For example, it might make sense for one process to annotate the element <foo>2007-05-17</foo> as having the type date, while for another process (such as a transformation that deletes all <foo> elements) it’s unnecessary. I really don’t want to have to define an XSD schema for my entire schema just to indicate that the <foo> element is of type xs:date.

Just as it’s better to define the links between elements using keys, rather than relying on ID annotations made by a DTD, I think type annotations and node groups (why limit it to elements?) could be defined in the stylesheet. To give an idea:

<!-- all date attributes have the type named 'xs:date' -->
<ann:type name="xs:date" match="@date" />
<!-- h1, h2, h3, h4, h5, h6 elements are heading elements -->
<ann:group name="xhtml:heading" 
  match="xhtml:h1 | xhtml:h2 | xhtml:h3 | xhtml:h4 | xhtml:h5 | xhtml:h6" />
<!-- oh, and so's the h element -->
<ann:group name="xhtml:heading" match="xhtml:h" />

It’d be reasonably easy to give rudimentary support for ann:type($node) and ann:group($node) user-defined functions based on these, but they’d really have to be built into the XSLT processor to get full pattern support and to work with modularised stylesheets. This all requires more detail than I have time to write right now, but is it even worth pursuing?

A Creole by any other name...

Jeni — Wed, 25 Apr 2007 21:09:28 +0100

Argh. I’ve been contacted by the guys at WikiCreole who want me to change the name of Creole. What should I do? Not only is “Creole” a great name for a schema language that deals with concurrent markup, but it’s a great acronym too (Composable regular expressions for overlapping languages etc.)

I did Google when I first came up with the name in August 2006, but didn’t discover WikiCreole (unsurprisingly, since it was only coined in July 2006 itself). But now far more many people know, care about and use WikiCreole than Creole grammars. So any suggestions for alternative names?