google

Partial implementations #2: XSLT in Google Search Appliance

Jeni — Fri, 23 Nov 2007 22:22:19 +0000

A Google Search Appliance (GSA) is a box that you plug into your network which crawls and indexes your data, and serves up the results of searches. Search results come in an XML format, and there’s a built in XSLT engine which means you can convert that XML into as many different views as you like. So you can have HTML-based search results, summaries, feeds, and so on.

My task recently was to debug some XSLT that transformed the GSA XML into an Atom feed. Easy enough, right? The GSA XML format is pretty hideous — most of the elements max out at three capital letters in length (whatever happened to human-readability) — but logical enough, and the mapping is hardly complex.

But all was not as it seemed. The GSA’s XSLT implementation is… how can I put this politely?… “non-standard”. This post describes some of the problems and workarounds.

To get the GSA to use your own XSLT, you have to go through its web interface. Basically there’s a form with a text field in which you can type your XSLT. Or you can upload a file that you develop offline. Naturally you’re going to do the latter because it means you can use your favourite editor with helpful things like syntax highlighting and validation-as-you-type, but of course that means switching between web browser windows and your IDE as you develop.

So I upload the transformation, point the browser at a relevant search page, and… oh…

When the GSA doesn’t like the XSLT that you use, you get a really helpful error message. It says:

Internal server error.

So you know that there’s been an error. With the server. Internally.

Back to basics, I thought. Let’s find out what processor the server’s using. Then we can develop on that processor and be pretty sure the resulting XSLT will work. So I load up the default XSLT (which is used to create an HTML result) and add the line

<xsl:value-of select="system-property('xsl:vendor')" />

Save the XSLT, reload the page, and…

Internal server error.

Okaaay… so this is an XSLT processor that doesn’t support the xsl:vendor system property. If it doesn’t support that, I’m going to have to tread carefully. So let’s start with something really simple:

<xsl:stylesheet version="1.0"
   xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:template match="/">
  <xsl:copy-of select="." />
</xsl:template>
</xsl:stylesheet>

Save the XSLT, reload the page, and…

Internal server error.

On a whim, I tried

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
   version="1.0">

<xsl:template match="/">
  <xsl:copy-of select="." />
</xsl:template>
</xsl:stylesheet>

instead. Save the XSLT, reload the page, and… Success!

Can you spot the difference? Yes, that’s right: it’s the order of the XSLT namespace declaration and the version attribute. Namespace declaration first, you’re OK, version first, you’re not.

Okaaay… so this is an XSLT processor that doesn’t support the XML Recommendation (which says that attribute order doesn’t matter). But heck, why split hairs? At least it’s working! Now to create some Atom instead:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
   version="1.0"
   xmlns="http://www.w3.org/2005/Atom">

<xsl:template match="/">
  <feed />
</xsl:template>
</xsl:stylesheet>

Save the XSLT, reload the page, and we’re back to

Internal server error

At least there’s some documentation about this one:

XSL stylesheets that include other files may not be used with the Google search engine. An XSL stylesheet that contains the following tags generates an error result:

<xsl:import>

<xsl:include>

xmlns:

document()

Read that again. Yes, the third bullet point. That’s right, it’s saying that an XSLT that contains a namespace declaration will generate an error result because it “includes other files”.

But, but, but, namespace declarations in XSLT stylesheets (or elsewhere for that matter) do not indicate file inclusion. Namespace URIs are identifiers, not locations. They are strings. They are not resolved. You do not need to be connected to the ‘net to use them.

And how am I supposed to serve an Atom feed, since Atom documents use a namespace? Or XHTML for that matter? Fortunately, the GSA only goes so far in banning namespace declarations: you’re OK as long as you don’t put them on the <xsl:stylesheet> element. Moving it to the <feed> element as in

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
   version="1.0">

<xsl:template match="/">
  <feed xmlns="http://www.w3.org/2005/Atom" />
</xsl:template>
</xsl:stylesheet>

and you’re OK. Of course you have to repeat the namespace declaration in every template so you don’t end up creating elements in no namespace. Tedious, oh so tedious, but workable.

(I have a vague suspicion that the idea behind banning namespace declarations is something to do with certain XSLT processors using namespace URIs to pull in Java classes. But addressing that problem by banning namespace declarations entirely isn’t just throwing the baby out with the bathwater, it’s throwing the whole bathroom suite out of the window. And if you then allow namespace declarations further down the stylesheet, you haven’t actually solved the problem.)

Amazingly enough, given the inauspicious beginning, everything else I tried actually worked. I suspect that it’s some standard XSLT processor underneath with a regex based filter that (among other things) limits what’s allowed in the <xsl:stylesheet> start tag. They probably disallow system-property('xsl:vendor') for security — knowledge is power, after all.

Anyway, my suggestions to others who might want to create a customised XSLT processor:

Use a custom URL resolver to restrict access to documents.
Restrict external function calls using something like the ALLOW_EXTERNAL_FUNCTIONS property in JAXP
Document the restrictions you’re placing on the stylesheets.
Produce meaningful error messages that explain the extra restrictions when they’re broken.

XTech 2007: Thursday 17th May Afternoon

Jeni — Sun, 27 May 2007 23:03:24 +0100

UPDATE: Dare Obasanjo has written an interesting critique on using the Atom Publishing Protocol as the basis for general purpose sharing of data in the way that the Google Data API does.

Thursday afternoon had a few really interesting talks. I learned about the Google Data API (no longer called gData); Oracle’s use of XLink to represent relationships between documents, and the requirements that entails; using XSLT to create JSON to use Exhibit widgets; and using XMPP to enhance instant messaging.

Google Data API

Frank Mantek

The Google Data API is the unified API that Google offers to all its services, such as Google Base, Blogger, Google Calendar, Google Spreadsheets and so on.

Frank talked about how awful SOAP/WSDL is, in particular how two services developed in different platforms can’t talk to each other (which one might imagine is rather the point of Web Services). (Later, when challenged by a Microsoft guy about this claim, he revealed that he’d been a major developer of the SOAP/WSDL stuff at Microsoft, so knew exactly what he was talking about from bitter experience.)

So the Google Data API is a RESTful API, using the Atom Publishing Protocol with a few additions:

extra data model
querying
concurrency control
extra authentication

What this basically means is that you can query any of the Google services using HTTP, and get back an Atom document. The URI can contain queries (the precise nature of which depend on the service; Google Base, for example, uses a single URI request parameter that has a complex internal query syntax), and you get back the feed with the items that you’d requested. The Atom items themselves have the basic Atom elements, but then a bunch of service-specific elements that provide the extra information you need.

Listening to this talk I finally got what Tim Bray was talking about at the XML Summer School a couple of years ago: REST gives us verbs and Atom gives us objects and lists of objects. I didn’t get it before, because, after all, aren’t all XML documents objects? But I think the point is that Atom has a lot of the mechanics that you need for talking about objects built into it, and the extensibility necessary for adding your own information to it (which is what each of Google’s services are doing).

The really interesting part of the talk was where Frank started talking about what the problems (still) are. The problems I noted were:

Atom’s verbose
Google have to use <category> to indicate the kind of thing they’re representing (as opposed to using the document element which is what you’d do with normal XML documents)
the rel attribute is too vague
they made up their own markup languages, rather than reusing existing standards
they should be using ETags for concurrency control
they haven’t got any versioning (eek)
incremental updates are a problem; they don’t want to serve the whole Atom feed (to a mobile device) when only a small amount has changed, so what they do is have several feeds, each of which reveals a different part of the information

From Trees to Graphs: Evolving XML for building enterprise applications

Ravi Murthy

Ravi Murthy talked about the provision for defining links between documents in Oracle’s database, and their consequent requirements. Information Oracle’s XML database has a file system abstraction (every XML ‘object’ has a file path) with access control, versioning, metadata and protocol access. Within an XML ‘object’ stored in the database, they use XLink to represent the relationships with other objects. When you export the XML, the XLinks get resolved to create the XML document.

Using XLink to represent relationships between documents brings a whole new set of constraints that you might want to express in a schema language, or annotations that you can use to describe the links (depending on how you look at it):

The type of the linked resource (eg the document element’s name, substitution group or XSD type)
The scope of a particular reference, similar to the scoping of XSD’s identity constraints
That a particular link is acyclic (eg, given an XPath expression, keep evaluating it and make sure you never get back to where you started)
The kind of a link, one of:
- hard: the target of the link must exist, and cannot be deleted while this resource exists (but can be renamed) — these are similar to links in normal databases
- symbolic: trust the file path specified by the link and only resolve it on demand
- weak: like a hard link, except the target can be deleted, in which case the link becomes symbolic
The versioning of a link, whether it points to the “current” version of a resource or a specific version

These extra constraints are expressed as annotations on the definitions of xlink:href attributes in XSD schemas for the documents held in the database.

Ravi also talked a bit about expressing decomposition rules: how an XML document should be shredded when it gets put into the database. They use XPath to specify rules that indicate that particular elements should be placed at a particular filepath.

I was really flattered in the tea break. Chatting with a guy called Phil working at the University of Bath, who politely asked about my presentation, and after I’d explained how it was all to do with overlapping markup and that kind of hard-core theory he said: “You don’t look like a markup geek”. Me: “What, because I’m a girl?”. Him: “No, no, that’s not what I meant. You just look more Web 2.0-ey.” Max was there at the time, and labelled me “the Geekess of XSLT”, which I think clarified things. (Actually most of the people at XTech this year were Web 2.0-ey rather than markup geeks, but I’m glad I looked as though I fitted in.)

XML-powered Exhibit: A Case Study of JSON & XML Coexistence

Chimezie Ogbuji

“What’s Exhibit?” I hear you ask. Or maybe you’re more with-it than I am, but that’s what I was asking. Chimezie never really explained, but I kinda gathered that it’s a funky AJAX toolset for creating views of data by importing scripts and using magical IDs and extension attributes within web pages. The other phrase that Chimezie dropped in was Rich Web Application Backplane, which again I hadn’t heard of. Even having read the W3C Note, I still don’t get it. Ho hum.

Anyway, Chimezie made the point that while entering data using XForms is great, it’s too heavy-weight for viewing that data. Exhibit gives a lot more flexibility (take a look at the US presidents example), which enables users to explore data more freely. In Exhibit pages, you provide a JSON schema for your data, a number of lenses/views/widgets that you can use to view the data, then you embed the widgets in the HTML page and point it at the data source. The JSON schema indicates the type of a particular property (eg “country”), and gives labels for it (including a plural label (“countries”) and a reverse label (“country of”)) that it uses in the widgets.

But that requires JSON, right? Chimezie showed how easy it is (and it’s really easy) to transform data-oriented XML into JSON using XSLT.

You know, there are all these cool ways out there for viewing information, I just wish I had some really meaty data to use them on! Timelines are one thing, but I’d also love to find some data to employ in Gapminder or even in an interface like the one for the music of Philip Glass. Perhaps I should just mine Google Base, but I’d like it to be something personally or collectively useful.

Real-time user-to-user web with Mozilla and XMPP

Massimiliano Mirra

This talk was strong on motivation — the requirement to enhance basic instant messaging functionality — and strong on demonstration, with Massimiliano chatting and playing with a pre-programmed bot, but really weak on the technical details. It was only through the post-talk questions that we learned that what we’d seen was based on XMPP (the Extensible Messaging and Presence Protocol), which allowed DOM events to be passed between clients. Have to read the paper if you want to learn more.

XTech 2007: Thursday 17th May Morning

Jeni — Fri, 25 May 2007 22:34:18 +0100

On Thursday morning, I was down to chair the first session in the “Core Technologies” track. Two interesting papers: one on XForms and one on Google Base. Then I snuck on to the “Applications” track to hear about scientific Wikis and the trials of managing schema repositories.

XForms, REST, XQuery… and skimming

Mark Birbeck

Mark Birbeck, one of the developers of formsPlayer (and an invited expert on the XForms and XHTML WGs), discussed the rationale behind using XForms. The only thing that really stood out for me was the fact that he used an XML document to provide the labels for the form controls (in just the same way as you can use XML documents to provide the data in the form controls). That was quite neat, and made me think of the different requirements of data entry and data presentation: a topic that returned in Chimezie Ogbuji’s talk later that afternoon.

Another theme here, for me, was the use of declarative programming: you write a form, which is just some XML and leave all the technical stuff about submitting a PUT HTTP request to the XForms player. Mark talked about using WebDAV and eXist on the server to store the XML documents, and demonstrated using <oXygen/> to load and save documents. Hmm… I wonder if I should experiment with XForms and that Unicode database browser I was thinking about…

Google Base, a mashups database for the REST of us

Jeffrey Scudder

A very popular, thought-provoking, and slightly disturbing, talk on Google Base. So Google are asking us to upload data on anything (jobs, personals, cars, etc.) into their huge databases. And then they’ll serve us back that information (and other people’s information) in formats such as Atom, RSS and JSON, as well as standard web pages.

The thought-provoking bit, for me, was the fact that they don’t have any particular schema for each of these kinds of items. Now, I come from a knowledge engineering background where we’re very into ontologies and creating conceptual models and all that stuff. But Google don’t bother: you create categories and structure your data the way you want to, and they’ll serve it back in that way. But they look at all the data they have their hands on in order to decide how to display and serve information. So, for example, if I define cars with the property ‘shade’ but a hundred other people define them with the property ‘colour’ then on a feed that includes all our items, we’ll see the ‘colour’ property.

This is a kind of bottom-up ontology design: the properties of an item are the properties that other people think are important about an item. One thing that surprised me was that it looks like it’s not very intelligent yet: simple differences in case (like ‘color’ vs. ‘Color’) don’t seem to be detected, so I guess nothing else is. Time to dig out my old research on automated comparison of ontologies…

The slightly disturbing part? Well, Google are trying to get us to upload our data to their servers. And they’re not putting any limit on how much we upload. One member of the audience asked “What’s in it for you?”; Jeffrey seemed to have a hard time understanding the question and said something like “Better indexed information means we can give you better information”, but that doesn’t really answer the question. Presumably it’s all about being able to advertise to us better: the more data we upload, the more They know about us, the better targeted Their adverts can be.

What I found strange was the idea of uploading data to a central server. Surely the whole point of the web is that I put my data on my machine. I don’t have a problem putting the data together in a nice Atom feed so that Google can index it easily and pointing them at it, but I want to own it, y’know?

By the way, one thing that was apparent to me during this talk was how important it is that web pages look good with large font sizes, not just for people with poor eyesight, but also for when you’re demoing your cool web applications! The Google Base drop-down menus were impossible to see with increased font size because their height is fixed in pixels.

An Augmented Wiki for Interactive Scientific Visualization and Evolutionary Collaboration

Frank Marchese

On to the less well-attended “Applications” track. This talk was about supporting scientists (specifically biochemists) in providing side-by-side visualisation (of complex molecules) and textual analysis. Frank talked about a Wiki in which Jmol Java applets for visualising molecules are arranged side-by-side with standard journal articles. The articles themselves have links in them that animate the Jmol visualisation: highlighting particular groups of atoms, moving it to show a particular view, and so on.

It was kind of neat, as pretty pictures of molecules often are, but I didn’t think the Wikiness of the whole enterprise was really explored: I got the impression that the textual articles were basically static: you could add comments, but not collaboratively create an article about the molecule. Also, the link between the text and the animation of the molecule was through Javascript, as far as I could tell: I’d expect a declarative method of defining animations would make it a lot more accessible.

Real-world metadata registries; sharing concepts, schemas and semantics

Emma Tonkin

This talk took me back to the trials of creation of top-down conceptual models, focusing on the definition of metadata schemas. Unfortunately, there was a lot of philosophy and not many practical guidelines in the talk, and I didn’t get a lot out of it. One thing that Emma touched on, though, was the way that the meaning of a term can change over time, through:

extension or generalisation
narrowing or specialisation
amelioration (when a term gains approval)
deterioration or perjoration (when a term gains disapproval)

The latter two are particularly demonstrated by political correctness, whereby terms like “Eskimo” fall out of favour and “Inuit” becomes more acceptable (all highly culture-specific; see the Wikipedia Eskimo page for more discussion on what term to use).

The advantage of a principled conceptual model is that the concept itself and the term(s) you use for that concept are loosely coupled, so if a given term falls out of favour or becomes inappropriate, you can always decouple it. On the other hand, bottom-up tagging tends (I think) to have a 1:1 relationship between term and concept, so if the use of terminology changes you might be left with inaccurate tagging of legacy data. Maybe.