rest

URL design for searches and queries

Jeni — Sun, 12 Aug 2007 23:00:00 +0100

Another fascinating post from Bill de hÓra, this time on URL design for resources:

Let’s take editing some resource, like a document, and let’s look at browsers and HTML forms in particular, which don’t a do a good job of allowing you to cleanly affect resource state. What you would like to do in this suboptimal environment is provide an “edit-uri” of some kind. There are basically 5 options for this; here they are going from most to least desirable

Uniform method. Alter the state by sending a PUT to the document’s URL. The edit-uri is the resource URL. URL format: http://example.org/document/xyz

Function passing. Allow the document resource to accept a function as an argument. URL format: http://example.org/document/xyz?f=edit

Surrogate. Create another resource that will accept edits on behalf of the document. URL format: http://example.org/document/xyz/edit

CGI/RPC explicit: send a POST to an “edit-document” script passing the id of the document as a argument. URL format: http://example.org/edit-document?id=xyz

CGI/RPC stateful: send a POST to an “edit-document” script and fetch the id of the document from server state, or a cookie. URL format: http://example.org/edit-document

My current task at work is to look at how to add RDFa to a website that is completely driven by “CGI/RPC explicit” URLs. That includes URLs for the resources themselves, by the way, we’re not even talking about edit URLs here. Take a look at the URL for this page, for example (this isn’t the actual website that I’m working on, but it’s more or less the same in terms of URL design). The URL is

http://www.statutelaw.gov.uk/content.aspx?LegType=All+Legislation&title=wine&Year=2007&searchEnacted=0&extentMatchOnly=0&confersPower=0&blanketAmendment=0&sortAlpha=0&TYPE=QS&PageNumber=1&NavFrom=0&parentActiveTextDocId=3032571&ActiveTextDocId=3032571&filesize=16218

So here I am trying to construct RDF examples, and all the URLs look like this mess. What URI am I supposed to use in RDF to talk about the resource itself, rather than a particular view (table of contents, actual content, etc) of that resource?

In this case, the thing that identifies the resource in the URL is the value of the ActiveTextDocId request parameter: you can do

http://www.statutelaw.gov.uk/content.aspx?ActiveTextDocId=3032571

and see the same legislation; this could be mapped to a resource-oriented URL such as

http://www.statutelaw.gov.uk/legislation/3032571

very easily. It isn’t, but it could.

However, doing that, you do lose some context about what the original search was that led you to this page. In the case of the above URI, the fact that I searched for all 2007 legislation with “wine” in the title gets lost. And this is important because the breadcrumb on the page has to take me back to that original search.

Now, you could argue that this is bad website design: after all, you can navigate back to a search page using *gasp* the Back button, and not doing so just adds unnecessary items to your history. But what about providing previous and next links for navigating through the items found in a search? There, surely, you do need some state information that indicates how we got to this particular item?

Well, no. When you’re navigating through the results of a search, the primary resource that you’re viewing is the collection of items that have been identified by the search. Even if you’re just viewing one of the items in that collection, if the collection still matters then that item should be viewed as just a subresource of the collection.

In this case, the search has three fields — title, year and (legislation) number — so the search URL has three parts after the initial one. The general scheme (using URI template syntax) is

http://www.statutelaw.gov.uk/search/{title}/{year}/{number}

So here, I could use

http://www.statutelaw.gov.uk/search/wine/2007/_

as a URL that would give me a list of all the 2007 legislation with any number whose title contained “wine”, and then

http://www.statutelaw.gov.uk/search/wine/2007/_/1

would show me the first piece of legislation found in the context of that search, with a next button taking me to

http://www.statutelaw.gov.uk/search/wine/2007/_/2

An individual resource itself could occur in many searches, and thus you would get many URLs for the resource in the context of those particular searches, but that’s OK so long as there is one URI that identifies the resource itself. You should link to the actual resource from the page, of course, both in a <link> in the header and a <a> in the body (it feels like there should be a “this is the real resource” value for the rel attribute of <link> or <a>, but I don’t know of one).

The kind of URL above works fine when you have a fixed number of fields for searches, but what if you’re doing more complicated searches: something that requires a proper query language? Well, you can stuff a query language into a URL. See Dare Obasanjo’s comparison of Google and Astoria APIs for queries to see what that looks like.

Alternatively, several years ago Paul Prescod introduced me to the notion that the query itself is a resource — it’s something that you’ll probably want to save and edit — and can be assigned a unique identifier in the same way as other resources. So you visit

http://www.example.com/queries/new

to create a new query, which gets assigned the URL

http://www.example.com/queries/4328

and you can then visit that page to see a list of the results of the query. Unlike with a simple search, the query parameters themselves don’t get used in the URL: they’re stored on the server. So you can’t hack the URL to change the query, but you do have a simple URL that you can easily share with other people if you want to.

View-Model-Template

Jeni — Fri, 27 Jul 2007 11:02:04 +0100

I don’t know anything about Struts 1, but Bill de hÓra’s recent post has got some interesting web-application-design tips. There were two particular bits that spoke to me:

struts-config.xml struts-config tries to capture primarily the flow of application state on the server, by being an awkward representation of a call graph. In doing it misses a key aspect of the web - hypertext. In web architecture, HTML hypertext on the client is the engine of application state, not an XML file on the server.

In other words (I think) in web applications your state in the page you’re on and taking action is about following the links (or submitting the forms) on the page. Your actions (and therefore the transitions between different states) are determined by what links and forms are on the page. But in fact, URLs should be hackable, and transitions unlimited. When you design the application what you really need to think about are the tasks the users want to achieve (and therefore the transitions that they might want to make) rather than the possible state transitions.

On the web, a suitable pattern is View, Model, Template [rather than Model, View, Controller (MVC)]. A request to a URL is dispatched to a View. This View calls into the Model, performs manipulations and prepares data for output. The data is passed to a Template that is rendered an [sic] emitted as a response. ideally [sic] in web frameworks, the controller is hidden from view. Note that this framework style is often called MVC anyway, confusing matters somewhat; The key differences are that Views and Templates are cohesive and Controllers are pushed down into the framework infrastructure.

I’ve been thinking recently about whether and how XSLT might fit into a Ruby on Rails set-up. In RoR, the controller usually either queries the database (via the model) to set up instance variables, and then renders a (template) view, or updates the database (via the model) and redirects to another view. The templates (for (X)HTML) use fairly standard <% ... %> placeholders to hold code and insert values.

I’ve spent most of my professional life cursing (X)HTML documents with <% ... %> in, because they use unescaped less-than-signs and therefore can’t be generated or processed by XML tools, particularly XSLT. There’s an advantage of having templates that are themselves well-formed, not least that you can easily process the templates themselves (for example to generate, update or document them). Plus if your templates are declarative, rather than containing embedded code, you aren’t tied to a particular framework: I could move templates from Ruby on Rails to Django and they wouldn’t need modification. When I think “declarative templates”, I think “XSLT”.

The other advantage of using XSLT is that it can be used on the client side as well as the server side. So there’s the possibility of moving that rendering from one server to client completely or using it on particular clients, perhaps in an AJAX set-up, while having the same stylesheets on the server for those browsers that don’t support client-side XSLT.

You still need a way of getting the data from the model into the stylesheet, which can be done through a combination of XML and parameters. The XML is itself a view of the model, of course, but if you’ve got any kind of intention to make your web application mashable, you’re going to want to generate XML, probably Atom, anyway (yeah, or JSON, but it’s easy enough to get from XML to JSON using XSLT too). If you add caching to the equation, this approach might help reduce database requests.

So I think that using XSLT as a templating language, even within a RoR framework, has at least something going for it. What I hope is that I’m not falling into the “when you’ve got a hammer everything looks like a nail” trap.

XTech 2007: Thursday 17th May Afternoon

Jeni — Sun, 27 May 2007 23:03:24 +0100

UPDATE: Dare Obasanjo has written an interesting critique on using the Atom Publishing Protocol as the basis for general purpose sharing of data in the way that the Google Data API does.

Thursday afternoon had a few really interesting talks. I learned about the Google Data API (no longer called gData); Oracle’s use of XLink to represent relationships between documents, and the requirements that entails; using XSLT to create JSON to use Exhibit widgets; and using XMPP to enhance instant messaging.

Google Data API

Frank Mantek

The Google Data API is the unified API that Google offers to all its services, such as Google Base, Blogger, Google Calendar, Google Spreadsheets and so on.

Frank talked about how awful SOAP/WSDL is, in particular how two services developed in different platforms can’t talk to each other (which one might imagine is rather the point of Web Services). (Later, when challenged by a Microsoft guy about this claim, he revealed that he’d been a major developer of the SOAP/WSDL stuff at Microsoft, so knew exactly what he was talking about from bitter experience.)

So the Google Data API is a RESTful API, using the Atom Publishing Protocol with a few additions:

extra data model
querying
concurrency control
extra authentication

What this basically means is that you can query any of the Google services using HTTP, and get back an Atom document. The URI can contain queries (the precise nature of which depend on the service; Google Base, for example, uses a single URI request parameter that has a complex internal query syntax), and you get back the feed with the items that you’d requested. The Atom items themselves have the basic Atom elements, but then a bunch of service-specific elements that provide the extra information you need.

Listening to this talk I finally got what Tim Bray was talking about at the XML Summer School a couple of years ago: REST gives us verbs and Atom gives us objects and lists of objects. I didn’t get it before, because, after all, aren’t all XML documents objects? But I think the point is that Atom has a lot of the mechanics that you need for talking about objects built into it, and the extensibility necessary for adding your own information to it (which is what each of Google’s services are doing).

The really interesting part of the talk was where Frank started talking about what the problems (still) are. The problems I noted were:

Atom’s verbose
Google have to use <category> to indicate the kind of thing they’re representing (as opposed to using the document element which is what you’d do with normal XML documents)
the rel attribute is too vague
they made up their own markup languages, rather than reusing existing standards
they should be using ETags for concurrency control
they haven’t got any versioning (eek)
incremental updates are a problem; they don’t want to serve the whole Atom feed (to a mobile device) when only a small amount has changed, so what they do is have several feeds, each of which reveals a different part of the information

From Trees to Graphs: Evolving XML for building enterprise applications

Ravi Murthy

Ravi Murthy talked about the provision for defining links between documents in Oracle’s database, and their consequent requirements. Information Oracle’s XML database has a file system abstraction (every XML ‘object’ has a file path) with access control, versioning, metadata and protocol access. Within an XML ‘object’ stored in the database, they use XLink to represent the relationships with other objects. When you export the XML, the XLinks get resolved to create the XML document.

Using XLink to represent relationships between documents brings a whole new set of constraints that you might want to express in a schema language, or annotations that you can use to describe the links (depending on how you look at it):

The type of the linked resource (eg the document element’s name, substitution group or XSD type)
The scope of a particular reference, similar to the scoping of XSD’s identity constraints
That a particular link is acyclic (eg, given an XPath expression, keep evaluating it and make sure you never get back to where you started)
The kind of a link, one of:
- hard: the target of the link must exist, and cannot be deleted while this resource exists (but can be renamed) — these are similar to links in normal databases
- symbolic: trust the file path specified by the link and only resolve it on demand
- weak: like a hard link, except the target can be deleted, in which case the link becomes symbolic
The versioning of a link, whether it points to the “current” version of a resource or a specific version

These extra constraints are expressed as annotations on the definitions of xlink:href attributes in XSD schemas for the documents held in the database.

Ravi also talked a bit about expressing decomposition rules: how an XML document should be shredded when it gets put into the database. They use XPath to specify rules that indicate that particular elements should be placed at a particular filepath.

I was really flattered in the tea break. Chatting with a guy called Phil working at the University of Bath, who politely asked about my presentation, and after I’d explained how it was all to do with overlapping markup and that kind of hard-core theory he said: “You don’t look like a markup geek”. Me: “What, because I’m a girl?”. Him: “No, no, that’s not what I meant. You just look more Web 2.0-ey.” Max was there at the time, and labelled me “the Geekess of XSLT”, which I think clarified things. (Actually most of the people at XTech this year were Web 2.0-ey rather than markup geeks, but I’m glad I looked as though I fitted in.)

XML-powered Exhibit: A Case Study of JSON & XML Coexistence

Chimezie Ogbuji

“What’s Exhibit?” I hear you ask. Or maybe you’re more with-it than I am, but that’s what I was asking. Chimezie never really explained, but I kinda gathered that it’s a funky AJAX toolset for creating views of data by importing scripts and using magical IDs and extension attributes within web pages. The other phrase that Chimezie dropped in was Rich Web Application Backplane, which again I hadn’t heard of. Even having read the W3C Note, I still don’t get it. Ho hum.

Anyway, Chimezie made the point that while entering data using XForms is great, it’s too heavy-weight for viewing that data. Exhibit gives a lot more flexibility (take a look at the US presidents example), which enables users to explore data more freely. In Exhibit pages, you provide a JSON schema for your data, a number of lenses/views/widgets that you can use to view the data, then you embed the widgets in the HTML page and point it at the data source. The JSON schema indicates the type of a particular property (eg “country”), and gives labels for it (including a plural label (“countries”) and a reverse label (“country of”)) that it uses in the widgets.

But that requires JSON, right? Chimezie showed how easy it is (and it’s really easy) to transform data-oriented XML into JSON using XSLT.

You know, there are all these cool ways out there for viewing information, I just wish I had some really meaty data to use them on! Timelines are one thing, but I’d also love to find some data to employ in Gapminder or even in an interface like the one for the music of Philip Glass. Perhaps I should just mine Google Base, but I’d like it to be something personally or collectively useful.

Real-time user-to-user web with Mozilla and XMPP

Massimiliano Mirra

This talk was strong on motivation — the requirement to enhance basic instant messaging functionality — and strong on demonstration, with Massimiliano chatting and playing with a pre-programmed bot, but really weak on the technical details. It was only through the post-talk questions that we learned that what we’d seen was based on XMPP (the Extensible Messaging and Presence Protocol), which allowed DOM events to be passed between clients. Have to read the paper if you want to learn more.