markup

Automatic markup and XML pipelines

Jeni — Mon, 25 Feb 2008 22:02:57 +0000

The project I’m working on at the moment aims to use RDFa (in XHTML) to expose some of the semantics in some natural-language text. We’re aiming moderately low — marking up dates, addresses, people’s names, and various other more domain-specific things — at least at the moment.

The problem we’re getting into now is how to get that information marked up. Because the information comes from various pretty unregulated sources, there’s no way we can force the authors to do the mark up. And the scope for making it “worth their while” (in terms of making their authoring job easier or more effective or even offering financial rewards) is very low.

So we’re taking a look at the technologies we might use for automating the markup, specifically GATE and UIMA.

These technologies basically use pipelines of components which each add some (out of line) annotations to the text. The annotations are done out of line because they might overlap, but you can (usually) serialize them into XML, which is what we want to do.

I find these technologies frustrating for a number of reasons:

any configuration we do will be specific to that particular application; it’ll be hard to for us to change to another implementation later on, and reuse by others will be limited to those who use the same implementation
they involve a fair bit of proper coding (by which I mean Java or C++)
where components can be configured through declarative means (such as keyword lists), there’s no way to reuse (XML/RDF) resources that we already have; we’ll have to manage transformations from them into the accepted formats through some external means, and I just know they’ll get out of sync
their user documentation is dreadful; it seems like you need to have a good understanding of natural language processing to have a hope of even getting started

It strikes me that the really powerful part of each of these technologies is the pipelining. The pipelining allows you to string together relatively simple operations (tokenising text, extrapolating sentences, marking up keywords, resolving ambiguities based on context etc.) which together give you something reasonably sophisticated.

Using XProc to coordinate the pipeline would alleviate many of my frustrations. XProc can and will be implemented on many platforms, in many languages, so it’ll be possible to move the pipeline from place to place (assuming that the components of the pipelines are similarly generic). It’s declarative, so no “proper coding”. We’ll be able to incorporate any transformations from existing XML/RDF data to the required configuration formats right into the pipeline. And… OK, it won’t automatically give us great user documentation or GUIs, but they’ll come.

The big problem is that XProc is still a Working Draft and the XProc ecosystem isn’t well-developed. If we were one or two years down the line, XProc would be a Recommendation, there’d be a .NET implementation readily available, and even perhaps extension XProc step types for tokenising, grouping and the other things we’d need to do; anything that was missing we could pull together using XSLT.

As it is, we’re in that annoying in-between-time when the Right technology isn’t ready and it looks like we’re going to have to put effort into working with what feels like the Wrong technology just to get things done. But perhaps I’m overlooking something in GATE or UIMA, or have missed another technology that would help us. Anyone out there got some experience that could help guide us?

Whitespace in markup languages

Jeni — Sun, 22 Jul 2007 09:32:25 +0100

I wrote previously about the, to my mind, wrong-headed use of xml:space in WordML (and OOXML), and promised something a bit more positive about how whitespace should be handled in markup languages. So here it is.

A bit of a disclaimer up front: my attitude on this topic is highly skewed by the fact I use XSLT all the time, and it has particular ways of dealing with whitespace. I happen to think that the way XSLT deals with whitespace is pretty solid, but that might just be because it’s what I’m used to.

The aim of this post is to answer the following question “when designing a markup language, what should I say about whitespace processing?”

Markup languages are data formats, not applications. Different applications may add or remove whitespace from documents in a given markup language. For example:

automatic indentation algorithms may change whitespace to make the document easier to read
programs querying documents may normalise whitespace when doing (text-based) searches or comparisons
renderers may ignore, collapse, change and add whitespace when rendering the document; for example browsers generally collapse whitespace and wrap the text when they display XML with a default (CSS) stylesheet

As a markup language designer, you need to describe how whitespace should be handled in your markup language. You need to answer the questions of people generating documents in your markup language:

can/should/must I add whitespace here?
if I add whitespace here, will it make a difference to my target application?

and the questions of the people processing the documents:

can/should/must I ignore this whitespace?
does this whitespace change the meaning of this value?

Whitespace in Element-Only Content

Whitespace in element-only content can be ignored without changing the meaning of an XML document. “Element-only content” means that the element can only ever contain elements. That’s something like

element name { element given { text },
               element family { text } }

in compact RELAX NG. It does not mean element instances that happen to contain only elements and whitespace. If you declare <name> as:

element name { text &
               element given { text } &
               element family { text } }

then the whitespace in

<name><given>Jeni</given> <family>Tennison</family></name>

does not count as whitespace in element-only content. Parsers that come from the Microsoft stable have an annoying tendancy to think you can just get rid of this whitespace, which is why you should only ever use them with preserveWhiteSpace set. (This has the unfortunate side-effect of keeping all your whitespace, but it’s better to have whitespace that you don’t need than to not have whitespace you do need.)

The XML spec requires parsers to pass all characters on to the application, although validating parsers can indicate if a character is a whitespace character that appears in element-only content according to the DTD associated with the document. The element content whitespace infoset property of these characters has the value true.

In practice, since MSXML doesn’t do it right, since you can’t rely on DTDs being accessible, and since applications don’t tend to strip element content whitespace automatically anyway, people processing documents generally have to ignore this whitespace manually. In XSLT, for example, you should use <xsl:strip-space> to get rid of whitespace from the elements you list. This reduces the size of the tree you’re dealing with and prevents you from counting whitespace-only text nodes or outputting them and thus getting screwy indentation in the output.

As a markup language designer, it doesn’t hurt to clarify matters by making a global statement like “whitespace that appears in element-only content can be ignored by processing applications” and then listing which elements this applies to. But mostly people will assume this to be the case anyway.

Whitespace in Data Content

Next up is whitespace that appears in elements or attributes that contain data. Whitespace rules here usually come into play when testing values. For example, you’ll usually want date = '2007-07-12' to be true for both

<date>
  2007-07-12
</date>

and

<date>2007-07-12</date>

There are three standard kinds of whitespace normalisation. These are defined most formally in XML Schema, but actually arise from the whitespace normalisation done to attribute values in basic XML:

preserve: all whitespace is preserved
replace: every whitespace character is replaced with a space character
collapse: all runs of whitespace are replaced by a single space; leading and trailing whitespace is stripped

Attribute values are always either replaced (the default if there’s no DTD) or collapsed (if they are typed as something other than CDATA in a DTD) during the parsing of the document. Like other text, element values are never touched during normal parsing.

The types that you use in a DTD or schema should indicate to people writing or processing documents in your markup language which kind of whitespace processing is going to be done. Here’s how it goes for the XML Schema datatypes:

if you define the type as xs:string, it means the whitespace should be preserved (although this won’t happen for attribute values, since their whitespace gets replaced automatically)
if you define the type as xs:normalizedString, it means the whitespace should be replaced
otherwise, (including xs:token) it means the whitespace should be collapsed

It’s worth thinking carefully about which of xs:string, xs:normalizedString or xs:token should be used when defining enumerations. If you base an enumeration on xs:string as in

<xs:simpleType name="windowType">
  <xs:restriction base="xs:string">
    <xs:enumeration value="single glazed" />
    <xs:enumeration value="double glazed" />
  </xs:restriction>
</xs:simpleType>

then no whitespace processing is done before the value is assessed against the enumerated values; the only values that are allowed are "single glazed" and "double glazed". For example

<window>
  single
  glazed
</window>

would be invalid. On the other hand, if you base an enumeration on xs:token as in

<xs:simpleType name="windowType">
  <xs:restriction base="xs:token">
    <xs:enumeration value="single glazed" />
    <xs:enumeration value="double glazed" />
  </xs:restriction>
</xs:simpleType>

then whitespace is collapsed before the value is assessed against the enumerated values, so the example above would be valid.

Generally, when you enumerate values you do want to collapse whitespace, so you should base the type on xs:token. In RELAX NG, this is the default, and doing

<choice>
  <value>single glazed</value>
  <value>double glazed</value>
</choice>

will result in the same behaviour as basing a simple type on xs:token. If you don’t want to strip whitespace, then you can use the type attribute on the <value> element to specify that the values are strings, not tokens

<choice>
  <value type="string">single glazed</value>
  <value type="string">double glazed</value>
</choice>

Of course one time you’ll really want enumerated values to be based on strings is if they can consist purely of whitespace.

Whitespace in Document Content

Once you have document content (content that is targeted at human consumption, which is usually mixed content), the only applications that should touch whitespace are rendering applications. Markup languages should be designed so that processors don’t have to add (or remove) whitespace to get something human readable.

I’ve been dealing with a markup language recently where this rule is broken in two ways. First, within a <text> element, processing applications have to add a space before any processing instruction or element that it contains. For example, look at:

<text>The proviso to section 6(2) of the<?change id="797826" 
  type="commentary"?>Statutory Orders (Special Procedure) 
  Act 1945 (power to withdraw an order or submit it to 
  Parliament for further consideration by means of a Bill for its 
  confirmation) shall have effect in relation to compensation 
  orders as if for the words<quotation class="double">"may 
  by notice given in the prescribed manner, withdraw the 
  order or may"</quotation> there were substituted the 
  word<quotation class="double">"shall"</quotation>.</text>

See how there’s no space before the <?change?> PI or the <quotation> elements? In this case, spaces need to be added before them. On the other hand, if the <?change?> or <quotation> element happens to start after certain kinds of punctuation, such as quotation marks or brackets then whitespace shouldn’t be added.

Second, in this badly designed markup language, if a <text> element has element-only content, processing applications have to ignore the whitespace around them. For example:

<commentarycontent><text>S. 39(5)(</text>
<text>
<font class="italic">b</font>
</text>
<text>) repealed by Industry Act 1980 (c. 33, SIF 64), 
  Sch. 2</text></commentarycontent>

In this case, the whitespace in the second <text> element can be ignored without any problems, but of course if the first <text> didn’t end with a bracket, but instead with a comma or a letter, the whitespace would have to be preserved (or at least turned into a space).

Whitespace processing of the kind illustrated here is time-consuming, hard to specify and inaccurate. The only people who really know what whitespace is needed in document-oriented content are the authors who create it, so it’s really important that they have the right, and responsibility, to determine where whitespace appears.

Rendering applications are a special case, because they have to munge whitespace to make text more readable. That often includes normalising whitespace away and adding line breaks (and hyphens) in the rendered view of the document. Different presentation-oriented markup languages use different algorithms to do this normalisation:

In HTML, whitespace within a block gets collapsed, while whitespace at the beginning or end of a block gets stripped away, except in <pre> elements where all whitespace is preserved.
In XSL-FO, whitespace processing depends on the linefeed-treatment, white-space-treatment and white-space-collapse properties, which provide practically any kind of behaviour; the default is the HTML rules.
In WordML, whitespace is replaced (not collapsed); whitespace at the beginning or end of a <w:t> element is stripped away unless xml:space="preserve" for that <w:t> element.

Because every piece of human-readable text eventually makes its way into HTML, I think it’s best to try to make your markup language follow the HTML rules. That means defining what the blocks are (the equivalent of paragraphs) and which ones have significant whitespace in them.

On `xml:space`

As the above discussion has shown, there are at least three different kinds of whitespace normalisation that can be used on an XML document. But xml:space has just two values: default and preserve. So one problem with using xml:space is that it doesn’t have any predefined semantics: in one application is might identify the distinction between collapsing and preserving, in another the distinction between replacing and preserving, in another (OOXML) the distinction between replacing-with-leading-and-trailing-whitespace-stripped and replacing.

So my advice would be: don’t use it. Instead, take the time to define explicitly how whitespace should be handled in the schema and documentation for your markup language.

I could just about be argued into providing xml:space as an optional attribute on elements where the user is likely to want to change the way whitespace is processed on a case-by-case basis. But I can’t actually think of any example where that might happen.

The biggest problem with whitespace handling is not that it can’t be defined, but that so many applications do it wrong. I’m sure that the bad whitespace use I described above, where PIs and elements implicitly added whitespace, arose not because the markup language designer decided that it would be a good idea but because the applications the authors used either added whitespace in their WYSIWYG displays or stripped it out when it was saved. Likewise, the abysmal whitespace-stripping behaviour of MSXML has led to many a strange use of markup, like David Carlisle adding xml:space="preserve" to his <html> elements.

So it’s the responsibility of markup language designers to specify how whitespace should be used, but it’s equally the responsibility of processors to honour those specifications.

Things that make me scream: xml:space="preserve" in WordML

Jeni — Fri, 13 Jul 2007 19:56:00 +0100

I intend to do a series of “things that make me scream” posts. Many of them will be about WordML (as in the markup language used by Word 2003) because that’s what I’m struggling with at the moment and because it’s so goddam awful. I don’t want to get into the whole ODF vs OOXML open standard-or-not debate. My problems with WordML (and OOXML) are mainly about aesthetics rather than process: I look at it and… well, it makes me want to scream. Examining what it is about the language (or implementation thereof) that prompts this visceral reaction might help in designing better languages.

So: did you know that Word 2003 puts a xml:space="preserve" attribute on the <w:wordDocument> document element of the XML that it produces and doesn’t indent its output? This is a nightmare if you ever have to actually look at the documents: auto-indentation programs (like the one in <oXygen/>) quite rightly won’t add whitespace to elements that are in the scope of an xml:space="preserve" attribute, which means you can’t use these programs to indent XML automatically.

In fact, <oXygen/> has syntax-highlighting-related problems when you open a document that has very long lines (like over 5000 characters; it doesn’t actually crash, but it eats all your CPU until you kill it, which I suppose is a kind of assisted suicide). This is usually mitigated by the fact <oXygen/> now prompts you to auto-indent when it detects such a document. But that doesn’t help with these WordML documents, because the auto-indent can’t actually reduce the line size. (Bear in mind that even the shortest WordML document — one with no actual content — created in Word 2003 is 4kb in size; 3926 characters in one line.)

So my regular experience debugging these WordML stylesheets I’m working on is to edit something in Word, save as XML, open in WordPad, remove xml:space="preserve", hit Save, remember that I can’t save it in WordPad while it’s still open in Word, close it in Word, go back to WordPad, hit Save, open in <oXygen/>, auto-indent, look around and debug the code. And repeat. Argh.

I could write an XSLT output filter that removed the xml:space="preserve", but really I shouldn’t have to. What on earth is xml:space="preserve" doing on the document element? It’s meant to be used on elements that really do contain significant whitespace that really must be preserved. The examples in the XML 1.1 Recommendation are of <poem> and <pre> where you’d want to see the line breaks, tabs and spaces when you viewed the content of the element in some kind of default viewer. In other words, the examples are elements whose content should be displayed with white-space: pre in CSS, or white-space-treatment="preserve" in XSL-FO. That just isn’t the case for the <w:wordDocument> element. Far from it.

In fact, in Word 2003, whitespace is only significant in terms of the appearance of the document in a handful of elements, the most common being <w:t>, which holds text inside a run. I also observe the following:

line breaks are done with <w:br>, carriage returns with <w:cr> and tabs with <w:tab>
any non-space whitespace within a <w:t> always gets converted to a space, however xml:space is set
any runs of spaces between words within a <w:t> get preserved, however xml:space is set
runs of leading and trailing spaces get stripped if xml:space isn’t set to preserve, and are preserved otherwise

I don’t know if the same thing happens in Word 2007, because I haven’t got a copy, but I note that in the OOXML spec, all the examples have xml:space="preserve" on all <w:t> elements, and it says (in section 2.3.1, page 34):

It is also notable that since leading and trailing whitespace is not normally significant in XML; some runs require a designating [sic] specifying that their whitespace is significant via the xml:space element [sic].

This seems to be a… umm… misinterpretation of the XML spec. Whitespace is always reported to XML applications (by any conformant parser, anyway), and the application gets to decide what to do with it. The default whitespace handling is that the application should use its default whitespace handling whatever that means. So I reckon that OOXML could just specify that whitespace is generally ignored except for in <w:t> (and a few other) elements which are normalized strings in the XML Schema sense (xs:normalizedStrings have all whitespace characters replaced by a space). To be honest, I really don’t see the point of xml:space here at all.

Whitespace handling is one of the hardest things to get right in any markup language or application, and there’s no single right way to do it, but WordML’s nowhere near right in my opinion. I’m gonna have to put my thoughts about how it should be done in a separate post, or just let the markup design experts out there have their say in comments on this one.

XTech 2007: Wednesday 16th May Afternoon

Jeni — Sun, 20 May 2007 22:52:14 +0100

Yes, I’m determined to write up every talk I attended at XTech 2007, so that I have a record of it if nothing else. On Wednesday afternoon, I attended sessions on microformats, internationalisation and NVDL (as well as giving my own talk, of course).

Microformats: the nanotechnology of the semantic web

Jeremy Keith

This was a supremely well-put-together presentation on microformats: beautiful slides, drama and humour, and a reference to Neal Stephenson’s Diamond Age (was I really one of only three people in the packed room to have read it?). There was a lot about what microformats are, how they’re designed, what their niche is (Jeremy was very up-front about the fact they don’t solve every problem), and how they’re developed. But there weren’t any demonstrations of microformat-based applications, which I would have really liked to see. The other thing I thought was worth noting was that Jeremy talked about the dangers of “grey goo” (he was using a nanotechnology metaphor): the proliferation of microformats. He expressed the strong desire that the set of microformats be kept small, and even said (I paraphrase) “Do use semantic class names in your HTML, but don’t call them microformats [unless they’ve been through the microformats standardisation process]!”

Liam Quin gave a paper entitled Microformats: Contaminants or Ingredients at Extreme last year, asking what we, as traditional markup geeks, should do about them. Some were very sceptical, saying something along the lines of “They’re headed for a trainwreck; and we should sit back, watch it happen, and pick up the pieces.” Others wanted to celebrate: the fact that tagging has become understood is really good news for the semantic web, open data and all that jazz.

Both the traditional markup and the microformats community have the same goals: they want to make information easier to search for, to query, to integrate and so on. The microformats approach is to minimise the cost to those supplying information, and to target just a few, very common, kinds of data such as contact information, events and social networks. Traditional markup, on the other hand, aims to cover every single kind of information you might want to make available, and has to worry about issues like validating, styling, and distinguishing between tag sets.

It seems that a fundamental problem is that the benefits of including semantic markup aren’t immediately obvious to the supplier. Whether you use semantic class names in HTML or use elements in known namespaces, it’s purely a matter of faith that this will make your information easier to locate or use. You can’t know that search engines will include that information in their weighting algorithms, or that people reading your page will have the screen-scraping software necessary to pull anything out. With so little (obvious) benefit, authors will only supply semantic data if the cost is low. Adding class names to existing HTML elements is easy whether a web page is generated by hand or automatically. Adding namespaces and authoring special CSS might not be that much more costly to do, but it’s much more costly to grok.

So if we want authors to start putting elements in their own namespaces in their web pages, we need an application that immediately cranks up the benefit of doing so. I have no idea what that is.

Applying the Internationalization Tag Set

Yves Savourel

This was a good introduction to [a standard] I only knew about vaguely. It’s definitely worth knowing about the its:* attributes for defining i18n features such as indicating which content should be translated, which are terms, providing comments for localisation and so on, just in case you need to build those in to new markup languages.

I also have much admiration for how the ITS standard doesn’t expect people to completely rework their markup languages to incorporate ITS data. Instead of using the ITS attributes directly in a document, you can use global rules embedded in the document itself, referenced from the document, or embedded in the schema for the document. I think this approach will prove useful in the development of LIX, when we get around to formalising it.

NVDL - a breath of fresh air for compound document validation

Jirka Kosek & Petr Nálevka

NVDL is Part 4 of DSDL, specifically targeted at organising the validation of documents that incorporate multiple namespaces, such as XHTML documents containing islands of SVG, RDF and MathML. NVDL’s approach is to identify subtrees within the document that need to be validated against a particular schema. The subtrees don’t need to only hold one namespace, but often that will be the case.

The XML Schema wonks in the room (Henry Thompson and Michael Sperberg-McQueen) were a bit befuddled, I think, because with XML Schema you just supply a whole bunch of schema documents to the processor, for different namespaces, and as long as the schemas contain wildcards they’ll do the right thing. The concept of supplying multiple schemas to a validator isn’t part of RELAX NG’s validation approach, so you need something like NVDL if you don’t want to rework your schema for every combination of namespaces.

Henry and Michael were particularly concerned about the fact that it means you can override the original schema, allowing elements from foreign namespaces in situations where the original schema hasn’t allowed them. But as Henry said, it just means that the primary schema you use to define what’s allowed where is actually an NVDL schema: it’s not auxiliary validation like Schematron is, but a language for the primary schema you use.

Later, I wondered how much the XProc work would render NVDL irrelevant. After all, XProc can invoke validation of subtrees against multiple external schemas. On the other hand, NVDL’s syntax is going to be easier to use if that’s all you want to do. Perhaps someone will write a tool to convert NVDL schemas to XProc pipelines…

Actually, Jirka & Petr’s experience with JNVDL is interesting from the XProc viewpoint, in particular the problems that they had with reporting meaningful line numbers when validating subtrees. Something that XProc implementers might want to look at in regard to error reporting with <p:viewport>.