genealogy

rdfQuery: Progressive Enhancement with RDFa

Jeni — Sat, 15 Nov 2008 16:46:02 +0000

Earlier this week I presented at SWIG-UK about rdfQuery. rdfQuery is a set of plugins that I’ve developed for jQuery in order to support RDFa parsing, querying and generation. There are a bunch of other Javascript libraries for RDFa around, such as Mark Birbeck’s Ubiquity RDFa and Ben Adida’s RDFa library. What I’ve really tried to do with rdfQuery is tie it in with the “Write Less, Do More” philosophy of jQuery and provide a neat, elegant API. At least that’s the aim!

So what does it do? Well, I’ve just added the demo that I used on Tuesday into the repository, so if you grab hold of that you can take a look. Here’s a screenshot of the demo.

The demo shows the overall concept of rdfQuery, namely that semantic markup can be useful not only to the crawlers that extract data from your pages to pump into massive triplestores, but also for you as a developer. In this case, which is a simple genealogy-type application, I want to have the people and places that are relevant to this particular extract highlighted within the text. I also want them listed on the left, with their details summarised.

So the demo illustrates three things that rdfQuery does to help:

gleaning triples from a section of the page (not just the whole page); in this case the triples are marked up with RDFa
querying the data to construct objects that represent the results of those queries, then doing things with those results
automatically adding RDFa to elements within the page, to update the data that it holds

It also demonstrates rough versions of a couple of things that rdfQuery could and should do that I aim to work on soon:

reasoning based on the data that’s found in the page
using ontologies to decide how to handle the data you find

By the by, it shows a simple, natural-language interface for making statements based on some text, but don’t let that distract you. It’s just regular expression processing.

There are four parts of this page. The main part contains some text about Charles Darwin. If you look at the source, you’ll see that it’s been marked up with some RDFa like this:

<span about="#CharlesRobertDarwin" typeof="foaf:Person" 
      property="rdfs:label" datatype="">
  <span property="foaf:firstName">Charles</span> Robert 
  <span property="foaf:surname">Darwin</span>
</span> was <span about="#CharlesRobertDarwin" rel="biografr:hasBirthPlace">born in 
<span about="#Shrewsbury" typeof="vcard:Address" 
      property="rdfs:label" datatype="">
  <span property="vcard:locality">Shrewsbury</span>, 
  <span property="vcard:region">Shropshire</span>, 
  <span property="vcard:country">England</span>
</span></span> on <span about="#CharlesRobertDarwin" property="biografr:bornOn" 
  content="1809-02-12" datatype="xsd:date">12 February 1809</span> at his 
  family home, the Mount. ...

This source is also shown in the textarea at the bottom of the page. Obviously this wouldn’t be visible in a real application, but it enables you to see what’s going on in the demo. Changing it won’t do anything; you can imagine that string being POSTed back to the server. I’ve also added some CSS that gives a border to any elements with a typeof attribute. Elements that have a property change their colour when you mouse over them.

So the page contains some RDFa. But what it doesn’t contain (in the source) is any information in the menu on the left. This gets populated based on the RDFa. If you click on Charles Robert Darwin, you’ll see the data that’s been gleaned about him, including (at the bottom), the derived fact that Robert Darwin was Charles Darwin’s father. Anything in black is information pulled from the RDFa; anything in orange is derived.

Next, try typing “Susannah Darwin was a person” into the text input and hit return. You should get Susannah Darwin added to the list of people. More importantly, if you look at the new source of the page, you’ll see that the phrase “Susannah Darwin” has been marked up with some RDFa to indicate that she was, indeed, a person and that “Susannah Darwin” can be used as a label for her.

You can try typing a few more facts into the box if you like. I suggest:

“Charles Robert Darwin was also known as Darwin”
“Susannah Darwin was Darwin’s mother”
“Susannah Darwin’s surname was Darwin”
“Josiah Wedgwood was a person”
“Susannah Darwin’s father was Josiah Wedgwood”

So this is all very cool and all, but the point was to show the code. So here we go.

Gleaning and Querying

Let’s look at how the lists are populated:

457: populateLists = function () {
458:   var rdf = $('#content').rdf();
459:   people.empty();
460:   places.empty();
461:   rdf
462:     .prefix('rdfs', ns.rdfs)
463:     .prefix('foaf', ns.foaf)
464:     .where('?person a foaf:Person')
465:     .where('?person rdfs:label ?label')
466:     .each(function () {
467:       addIndividual(people, this.person, this.label.value);
468:     })
469:     .reset()
470:     .where('?place a vcard:Address')
471:     .where('?place rdfs:label ?label')
472:     .each(function () {
473:       addIndividual(places, this.place, this.label.value);
474:     });
475: },

Line 458 parses the RDFa within the element with the id content (this is a <div> that wraps around the paragraphs about Charles Darwin). The rdf variable holds an rdfQuery object, which is very similar to a jQuery object except that it queries over RDF triples rather than DOM nodes. The rdfQuery object holds a pointer to a databank, which holds the triples that have been collected.

Lines 459 and 460 empty out the existing lists of people and places if there are any. The people and places variables are set earlier in the script and are jQuery objects.

Now the fun begins. I first set some prefixes on the rdfQuery object so that I can use those prefixes in CURIEs within the queries. In fact, these prefixes will have been set up by default anyway, because they’re declared in the HTML page, but it doesn’t hurt.

Lines 464 and 465 locate triples in the databank based on simple SPARQL-based queries. The first where() call creates a new rdfQuery object that, like a jQuery object, looks a bit like an array. The array contains objects, one for each triple that matches the pattern ?person a foaf:Person. Each of the objects has a person property containing the resource that represents the person. So the rdfQuery that results from this where() call looks like:

{ length: 2,
  0: { person: $.rdf.resource('<#CharlesRobertDarwin>') },
  1: { person: $.rdf.resource('<#RobertDarwin>') },
  ... bunch of other properties and methods that aren't important here ... }

The second where() call creates another new rdfQuery object based on combining the previous query results with the any triples that match the pattern ?person rdfs:label ?label. This holds objects with person and label properties, one for each person and their label. So the result of this looks like:

{ length: 2,
  0: { person: $.rdf.resource('<#CharlesRobertDarwin>'),
       label:  $.rdf.literal('"Charles Robert Darwin"') },
  1: { person: $.rdf.resource('<#RobertDarwin>'),
       label:  $.rdf.literal('"Robert Darwin"') },
  ... bunch of other properties and methods that aren't important here ... }

The each() on lines 466-468 then iterates over the objects that it’s constructed and calls the addIndividuals() function (which is particular for this demo), passing in the list in the HTML, the person resource and the value of the label literal.

Line 469 uses the reset() method to go back to the original rdfQuery object. If I didn’t do this, any further queries would simply add to the objects that I already have, or remove them if nothing matched.

Lines 470-474 do the same thing for the places that are marked up within the text.

There are various other places within markup.js that glean and query RDF. For example, the addDescription() function, which populates the list items on the left with data about particualr people and places. That function demonstrates the use of the about() method, which gives you all the triples about a particular subject, and shows how to use arguments with the each() method when you want to use the index of the query result or the triples that were used to create the query result.

Updating RDFa

So how easy is it to update the RDFa on the web page? Well, if you know what you want to add, then it’s dead easy. Here’s the code that does it on lines 452 and 532:

span.rdfa(triple);

The variable span here is a jQuery object. The triple itself is a $.rdf.triple. I’d tell you more about them but I think I’ve gone on long enough.

Final Words

rdfQuery is a Google Code project, released under an MIT license. If you’re interested in contributing, send me an email and I’ll add you as a member, or an owner if you’re really keen. If you’re interested, I’ve set up a discussion group. You can post any questions there, although of course if you find bugs, do add an issue.

Metadata about RDF triples: reification and Linked Data

Jeni — Fri, 11 Apr 2008 20:59:07 +0000

Those of you who have been following this blog will know that I’ve been thinking recently about how to handle uncertainty related to RDF triples (specifically in the context of a genealogical web app). Certainty isn’t the only kind of metadata-about-triples that you’d want to keep in an app like this. We need to know things like:

who made the statement
when the statement was made
what evidence that led to the statement being made
licensing information about the reuse of the statement
(if we go with the rating idea) what ratings the statement has been given
(if we allow editing of statements) what changes have been made to the statement over time

and so on. In short, all the metadata that you’d want to associate with resources you’d also want to associate with statements.

I’d anticipated using reification to associate metadata with statements. Something like this

<rdf:Statement rdf:about="#statement1">
  <rdf:subject rdf:resource="/people/CharlesDarwin" />
  <rdf:predicate rdf:resource="/ontology/event-roles/passenger" />
  <rdf:object rdf:resource="/events/BeagleVoyage" />
  <dc:creator rdf:resource="/users/JeniT" />
  <dc:date rdf:datatype="xsd:date">2008-04-11</dc:date>
  <g:certainty rdf:datatype="xsd:decimal">1.0</g:certainty>
  ...
</rdf:Statement>

or using rdf:ID, although this does limit the URI of our statements to hash-URIs:

<rdf:Description about="/people/CharlesDarwin">
  <r:passenger rdf:ID="#statement1" rdf:resource="/events/BeagleVoyage" /> 
</rdf:Description>
<rdf:Description about="#statement1">
  <dc:creator rdf:resource="/users/JeniT" />
  <dc:date rdf:datatype="xsd:date">2008-04-11</dc:date>
  <g:certainty rdf:datatype="xsd:decimal">1.0</g:certainty>
</rdf:Description>

(Please feel free to correct my RDF, RDF-folks!)

We can embed this information into our web pages using RDFa:

<div about="#statement1" instanceof="rdf:Statement">
  <p class="statement">
    <a rel="rdf:subject" href="/people/CharlesDarwin">
      Charles Darwin
    </a>
    was a
    <a rel="rdf:predicate" href="/ontologies/event-roles/passenger">
      passenger
    </a>
    on the
    <a rel="rdf:object" href="/events/BeagleVoyage">
      <span about="/people/CharlesDarwin" 
            rel="r:passenger" 
            resource="/events/BeagleVoyage">
        Beagle Voyage
      </span>
    </a>
  </p>
  <dl class="metadata">
    <dt>Author:</dt>
    <dd>
      <a rel="dc:creator" href="/users/JeniT">
        Jeni Tennison
      </a>
    </dd>
    <dt>Date:</dt>
    <dd property="dc:date" datatype="xsd:date" 
        content="2008-04-11">
      11 Apr, 2008
    </dd>
    <dt>Certainty:</dt>
    <dd property="b:certainty" datatype="xsd:decimal"
        content="1.0">
      <img src="stars5.gif" alt="five stars" />
    </dd>
  </dl>
</div>

Note that I’ve incorporated both the reified statement and the statement itself into the RDFa. If I’m correct in my mental parsing of RDFa, I think this leads to the set of triples from the RDF/XML in the above examples plus the triple:

</people/CharlesDarwin> r:passenger </events/BeagleVoyage> .

But then the other day, I was reading the tutorial How to publish Linked Data on the Web, which says

We discourage the use of RDF reification as the semantics of reification are unclear and as reified statements are rather cumbersome to query with the SPARQL query language. Metadata can be attached to the information resource instead, as explained in Section 5.

Jumping to Section 5, I find

Metadata: The representation should contain any metadata you want to attach to your published data, such as a URI identifying the author and licensing information. These should be recorded as RDF descriptions of the information resource that describes a non-information resource; that is, the subject of the RDF triples should be the URI of the information resource. Attaching meta-information to that information resource, rather than attaching it to the described resource itself or to specific RDF statements about the resource (as with RDF reification) plays nicely together with using Named Graphs and the SPARQL query language in Linked Data client applications…

There are some examples of what this looks like within the tutorial. The first is an “authoritative description” found at http://dbpedia.org/data/Alec_Empire after a 303 redirection from http://dbpedia.org/resource/Alec_Empire.

# Metadata and Licensing Information
<http://dbpedia.org/data/Alec_Empire>
    rdfs:label "RDF description of Alec Empire" ;
    rdf:type foaf:Document ;
    dc:publisher <http://dbpedia.org/resource/DBpedia> ;
    dc:date "2007-07-13"^^xsd:date ;
    dc:rights <http://en.wikipedia.org/wiki/WP:GFDL> .

# The description
<http://dbpedia.org/resource/Alec_Empire> 
    foaf:name "Empire, Alec" ;
    rdf:type foaf:Person ;
    rdf:type <http://dbpedia.org/class/yago/musician> ;
    rdfs:comment
        "Alec Empire (born May 2, 1972) is a German musician who is ..."@en ;
    rdfs:comment
        "Alec Empire (eigentlich Alexander Wilke) ist ein deutscher Musiker. ..."@de ;
    dbpedia:genre <http://dbpedia.org/resource/Techno> ;
    dbpedia:associatedActs 
      <http://dbpedia.org/resource/Atari_Teenage_Riot> ;
    foaf:page <http://en.wikipedia.org/wiki/Alec_Empire> ;
    foaf:page <http://dbpedia.org/page/Alec_Empire> ; 
    rdfs:isDefinedBy <http://dbpedia.org/data/Alec_Empire> ;
    owl:sameAs <http://zitgist.com/music/artist/d71ba53b-23b0-4870-a429-cce6f345763b> .

The second is a non-authoritative description found at http://sites.wiwiss.fu-berlin.de/suhl/bizer/pub/LinkedDataTutorial/ChrisAboutRichard:

# Metadata and Licensing Information
<>
    rdf:type foaf:Document ;
    dc:author <http://www.bizer.de#chris> ;
    dc:date "2007-07-13"^^xsd:date ;
    cc:license <http://web.resource.org/cc/PublicDomain> .

# The description
<http://richard.cyganiak.de/foaf.rdf#cygri> 
    foaf:name "Richard Cyganiak" ;
    foaf:topic_interest <http://dbpedia.org/resource/Category:Databases> ;
    foaf:topic_interest <http://dbpedia.org/resource/MacBook_Pro> ;
    rdfs:isDefinedBy <http://richard.cyganiak.de/foaf.rdf> ;
    rdf:seeAlso <> .

Note that rdfs:isDefinedBy does not necessarily point to the data you get when you retrieve the resource, but to an (presumably there can be more than one) authoritative description of the resource. It’s also associated with a particular resource rather than a particular statement.

To know which metadata applies to a particular statement, an application must know where it got the statement from. In effect, a statement here has four parts: subject, property, object and location (with the possibility that multiple statements with the same subject, property and object might have different locations and therefore different metadata). This is similar to assigning an ID to a statement, as with rdf:ID, but restricts the statement’s identifier to being the location where it was found.

So what does that mean for the genealogical web app? Well, in the app we’re going to find any given statement by a particular user quoted on lots of pages. I was intending to RDFa them all but that would mean lots of duplicate statements from different locations, potentially bloating applications that were harvesting the data.

I can’t work out whether I like or loathe the Linked Data concept of associating metadata with the document in which you find triples. In some ways it seems very natural — look for information about a resource at the URI for the resouce — but the metadata mechanisms restrict where you can place statements on the web (or at least assign semantics to their location which aren’t necessarily intended), and that seems like a Bad Thing. On the other hand, perhaps I’m just being overly influenced by the desire to use RDFa, which does lead one to want to mark up data wherever it appears.

I’d welcome any advice.

Web 2.0 project: privacy in genealogy

Jeni — Thu, 17 Jan 2008 21:48:58 +0000

There were a couple of comments on my previous post about RDF and uncertainty in our Web 2.0 genealogy project concerning the problems of privacy in a genealogy app. My ideas about this aren’t fully thought-through, let alone implemented, but I thought I’d share them anyway.

First, the things we could restrict access to are:

sources of information (eg birth records)
personas (eg Charles Darwin) and assertions about them
events (eg the Beagle Voyage) and assertions about them
groups (eg the Royal Society) and assertions about them

There are different kinds of things you might do to the resources:

who can know it exists?
who can read it?
who can change it?
who can delete it?

and different levels of access for each of those things:

global (public)
group (restricted)
user (private)

I imagine that at any point, a user will have a default set of permissions in play. For example, they might be generating information that anyone can know exists and can read, but only a restricted set of people can change, and only the user themselves can delete.

Searchability

I’m going to side-track for a second here to explain why I’ve separated out “knowing something exists” from “readable”.

Our genealogy application is evidence-based, which means that to say that someone exists you must have a source for that information, be it a birth certificate or a picture or the transcription of an interview. The persona that you believe to exist on the basis of one source may or may not be the same persona that you believe to exist on the basis of a different source. A separate step is required to link the two personas together.

The intention is that our application will help you link personas (and events, groups and anything else that might be quoted in more than one piece of evidence) together through searching through other personas (etc) to find those that are similar. They might have similar names, but most of the evidence for similarity will come from similar assertions having been made about them.

Now say that I have some evidence about Charles Darwin and enter it into the system, and you similarly have some evidence about Charles Darwin that you enter into the system. We both create “Charles Darwin” personas based on our evidence. I’d like it to be the case that, even if we don’t want the information we’ve captured about Charles Darwin to be readable by others, we can still make it searchable so that others can find out we’re working on that person too so that (after some interpersonal negotiation) the two sets of information can be brought together.

So the “who can know it exists?” access is about making your information searchable, even if the details aren’t readable.

Default access

As I understand it, in genealogy circles it would be bad form to make any information about living people publicly available. Some would argue that any information that is currently publicly available (on the web, or even off it, in electoral rolls, phone books, wherever) is fair game: if it’s already “out there” then you can use it. But I don’t agree with that. There’s a big difference between there being multiple disparate, human-readable sources of information about an individual “out there somewhere” and providing a single aggregated public source for all this data in a computer-readable format. In short, I agree with Jeff “Coding Horror” Atwood’s recent post on privacy in which he characterises privacy as an inalienable right.

Also, some people make money from doing genealogical research, and it would be short-sighted to prevent people from using the application if they simply wanted to keep what they were doing private for their own reasons.

So there must be a setting such that information can be kept entirely private, to the extent that only the person who constructed it knows that it exists.

On the other hand, many of the benefits of Web 2.0 applications come from integrating your own material with other people’s. Genealogy is a good example of this, because there’s a distinct thrill in hooking into other people’s research and discovering other branches of the family.

So there must be a setting such that information can be made globally available to all and sundry, even to the extent of others changing and deleting it.

I think the default has to be that the information you enter is completely private. I’d like to encourage people to share their information, but I think that the way to do that is through social mechanisms (eg the more useful information you share, the more kudos you get) rather than technological ones.

User Groups

Public and private access aside, there’s a huge middle ground of access that’s somehow restricted to a subset of users. I don’t think I’m being particularly radical if I say that the current state of the art of social software access control isn’t great, that the “restricted” subset is usually to “my friends”, but people don’t have one social circle: they operate in different spheres, and have a different set of friends/colleagues in each.

The same is true in our genealogy app. A given user might be working (collaboratively) on several projects, with different teams. For example, I might work on a family tree with my mother for her side of the family and my father for his, with no need for either of them to be aware of the other (unless they overlap, in which case “searchability” comes into play).

So I think it makes sense to use the familiar notion from *nix of “user groups”, and assign each item (sources/people/events/groups) to one or more user groups. User groups will probably mostly arise from those working on particular projects, but could be more arbitrary. I’m thinking:

moderated (by-invitation) and open (by-subscription) user groups
private, public and restricted permissions on the user groups

The other thing I was considering here was having a notion of “degrees of separation”. If I entered details about our daughters into a family tree, I might be happy for immediate family, and their immediate family, to see them: a separation of two. I don’t know whether this kind of network-based user group would be useful or a pain to manage, or even implementable; it’s just an idea I had.

Web 2.0 project: RDF and uncertainty

Jeni — Fri, 21 Dec 2007 22:03:51 +0000

I’ve been thinking a bit recently about how to deal with certainty in our Genealogical Web 2.0 application. We’ve come round to using an RDF model to represent what the Gentech data model calls “assertions”; assertions such as “Charles Darwin was a passenger on the Beagle Voyage” are represented as an RDF Statement in which (a resource representing) “Charles Darwin” is the subject, (a resource representing) “Beagle Voyage” is the object, and “was a passenger on” is the predicate/property.

All the statements in the genealogical application should be based on some source of information, either an external piece of evidence (such as a marriage certificate) or by combining existing statements. Either way, there’s certain metadata that we want to store about it, such as

who created the statement
when it was made
the date(s) when the statement was true
the certainty in the statement

The certainty factor is interesting. For statements based directly on evidence, there are three factors that come into play:

the reliability of the evidence itself; for example, a marriage certificate is more reliable than a diary entry for a wedding
the certainty the user has in drawing their conclusion based on the evidence; for example, you would be more certain in the statement that the groom named on a marriage certificate is a man than in the statement that the witness named on a marriage certificate is a friend of the groom
the reliability of the user who has made the statement: an expert in family history is likely to draw more accurate conclusions than someone who has only just started

So now the question is how to assess these factors. The usual Web 2.0 method is to use ratings. We could get users to rate each other to provide the third score. We could then get users to rate the reliability of particular pieces of evidence, modify that score based on the users’ reliability, and aggregate those scores.

The final certainty of the statement would be a combination of this score for evidence reliability and ratings from multiple users, again weighted according to the users’ reliability.