Archive

Archive for the ‘linked data’ Category

Visualising the Location Graph – example with Gephi and Ordnance Survey linked data

March 28, 2014 2 comments

This is arguably a simpler follow up to my previous blog post, and here I want to look at visualising Ordnance Survey linked data in Gephi. Now Gephi isn’t really a GIS, but it can be used to visualise the adjacency graph where regions are represented as nodes in a graph, and links represent adjacency relationships.

The approach here will be very similar to the approach in my previous blog. The main difference is that you will need to use the Ordnance Survey SPARQL endpoint and not the DBpedia one. So this time in the Gephi semantic web importer enter the following endpoint URL:

http://data.ordnancesurvey.co.uk/datasets/os-linked-data/apis/sparql

The Ordnance Survey endpoint returns turtle by default, and Gephi does not seem to like this. I wanted to force the output as XML. I figured this could be done in the using a ‘REST parameter name’ (output) with value equal to xml. This did not seem to work, so instead I had to do a bit of a hack. In the ‘query tag…’ box you will need to change the value from ‘query’ to ‘output=xml&query’. You should see something like this in the Semantic Web Importer now:

Screen Shot 2014-03-28 at 11.28.28

Now click on the query tab. If we want to, for example, view the adjacent graph for consistuencies we can enter the following query:

prefix gephi:<http://gephi.org/>
construct {
?s gephi:label ?label .
?s gephi:lat ?lat .
?s gephi:long ?long .
?s <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/touches> ?o .}
where
{
?s a <http://data.ordnancesurvey.co.uk/ontology/admingeo/WestminsterConstituency> .
?o a <http://data.ordnancesurvey.co.uk/ontology/admingeo/WestminsterConstituency> .
?s <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/touches> ?o .
?s <http://www.w3.org/2000/01/rdf-schema#label> ?label .
?s <http://www.w3.org/2003/01/geo/wgs84_pos#lat> ?lat .
?s <http://www.w3.org/2003/01/geo/wgs84_pos#long> ?long .
}

and click ‘run’. To visualise the output you will need to follow the exact same steps mentioned here (remember to recast the lat and long variables to decimal).

If we want to view adjacency of London Boroughs then we can do this with a similar query:

prefix gephi:<http://gephi.org/>
construct {
?s gephi:label ?label .
?s gephi:lat ?lat .
?s gephi:long ?long .
?s <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/touches> ?o .}
where
{
?s a <http://data.ordnancesurvey.co.uk/ontology/admingeo/LondonBorough> .
?o a <http://data.ordnancesurvey.co.uk/ontology/admingeo/LondonBorough> .
?s <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/touches> ?o .
?s <http://www.w3.org/2000/01/rdf-schema#label> ?label .
?s <http://www.w3.org/2003/01/geo/wgs84_pos#lat> ?lat .
?s <http://www.w3.org/2003/01/geo/wgs84_pos#long> ?long .
}

When visualising you might want to change the scale parameter to 10000.0. You should see something like this:

Screen Shot 2014-03-28 at 11.40.18

So far so good. Now imagine we want to bring in some other data – recall my previous blog post here. We can use SPARQL federation to bring in data from other endpoints. Suppose we would like to make the size of the node represent the ‘IMD rank‘ of each London Borough…we can do with by bringing in data from the Open Data Communities site:

prefix gephi:<http://gephi.org/>
construct {
?s gephi:label ?label .
?s gephi:lat ?lat .
?s gephi:long ?long .
?s gephi:imd-rank ?imdrank .
?s <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/touches> ?o .}
where
{
?s a <http://data.ordnancesurvey.co.uk/ontology/admingeo/LondonBorough> .
?o a <http://data.ordnancesurvey.co.uk/ontology/admingeo/LondonBorough> .
?s <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/touches> ?o .
?s <http://www.w3.org/2000/01/rdf-schema#label> ?label .
?s <http://www.w3.org/2003/01/geo/wgs84_pos#lat> ?lat .
?s <http://www.w3.org/2003/01/geo/wgs84_pos#long> ?long .
SERVICE <http://opendatacommunities.org/sparql> {
?x <http://purl.org/linked-data/sdmx/2009/dimension#refArea> ?s .
?x <http://opendatacommunities.org/def/IMD#IMD-score> ?imdrank . }
}

You will need to recast the imdrank as an integer for what follows (do this using the same approach used to recast the lat/long variables). You can now use Gephi to resize the nodes according to IMD rank. We do this using the ranking tab:

Screen Shot 2014-03-28 at 11.50.43

You should now see you London Boroughs re-sized according to their IMD rank:

Screen Shot 2014-03-28 at 11.51.51

turning the lights off and adding some labels we get:

Screen Shot 2014-03-28 at 12.04.27

All roads lead to? Experiments with Gephi, Linked Data and Wikipedia

March 26, 2014 3 comments

Gephi is “an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs”. Tony Hirst did a great blog post a while back showing how you could use Gephi together with DBpedia (a linked data version of Wikipedia) to map an influence network in the world of philosophy. Gephi offers a semantic web plugin which allows you to work with the web of linked data. I recommend you read Tony’s blog to get started with using that plugin with Gephi. I was interested to experiment with this plugin, and to look at what sort of geospatial visualisations could be possible.

If you want to follow all the steps in this post you will need to:

Initially I was interested to see if there were any interesting networks we might visualise between places. In order to see how Wikipedia relates one place to another was a simple case of going to the DBpedia SPARQL endpoint and trying the following query:

select distinct ?p
where
{
?s a <http://schema.org/Place> .
?o a <http://schema.org/Place> .
?s ?p ?o .
}

- where s and o are places, find me what ‘p’ relates them. I noticed two properties ‘http://dbpedia.org/ontology/routeStart‘ and ‘http://dbpedia.org/ontology/routeEnd‘ so I thought I would try to visualise how places round the world were linked by transport connections.  To find places connected by a transport link you want to find pairs ‘start’ and ‘end’ that are the route start and route end, respectively, of some transport link. You can do this with the following query:

select ?start ?end
where
{
?start a <http://schema.org/Place> .
?end a <http://schema.org/Place> .
?link <http://dbpedia.org/ontology/routeStart> ?start .
?link <http://dbpedia.org/ontology/routeEnd> ?end .
}

This gives a lot of data so I thought I would restrict the links to be only road links:

select ?start ?end
where
{?start a <http://schema.org/Place> .
?end a <http://schema.org/Place> .
?link <http://dbpedia.org/ontology/routeStart> ?start .
?link <http://dbpedia.org/ontology/routeEnd> ?end .
?link a <http://dbpedia.org/ontology/Road> . }

We are now ready to visualise this transport network in Gephi. Follow the steps in Tony’s blog to bring up the Semantic Web Importer. In the ‘driver’ tab make sure ‘Remote – SOAP endpoint’ is selected, and the EndPoint URL is http://dbpedia.org/sparql. In an analogous way to Tony’s blog we need to construct our graph so we can visualise it. To simply view the connections between places it would be enough to just add this query to the ‘Query’ tab:

construct {?start <http://foo.com/connectedTo> ?end}
where
{
?start a <http://schema.org/Place> .
?end a <http://schema.org/Place> .
?link <http://dbpedia.org/ontology/routeStart> ?start .
?link <http://dbpedia.org/ontology/routeEnd> ?end .
?link a <http://dbpedia.org/ontology/Road> .
}

However, as we want to visualise this in a geospatial context we need the lat and long of the start and end points so our construct query becomes a bit more complicated:

prefix gephi:<http://gephi.org/>
construct {
?start gephi:label ?labelstart .
?end gephi:label ?labelend .
?start gephi:lat ?minlat .
?start gephi:long ?minlong .
?end gephi:lat ?minlat2 .
?end gephi:long ?minlong2 .
?start <http://foo.com/connectedTo> ?end}
where
{
?start a <http://schema.org/Place> .
?end a <http://schema.org/Place> .
?link <http://dbpedia.org/ontology/routeStart> ?start .
?link <http://dbpedia.org/ontology/routeEnd> ?end .
?link a <http://dbpedia.org/ontology/Road> .
{select ?start (MIN(?lat) AS ?minlat) (MIN(?long) AS ?minlong) where {?start <http://www.w3.org/2003/01/geo/wgs84_pos#lat> ?lat . ?start <http://www.w3.org/2003/01/geo/wgs84_pos#long> ?long .} }
{select ?end (MIN(?lat2) AS ?minlat2) (MIN(?long2) AS ?minlong2) where {?end <http://www.w3.org/2003/01/geo/wgs84_pos#lat> ?lat2 . ?end <http://www.w3.org/2003/01/geo/wgs84_pos#long> ?long2 .} }
?start <http://www.w3.org/2000/01/rdf-schema#label> ?labelstart .
?end <http://www.w3.org/2000/01/rdf-schema#label> ?labelend .
FILTER (lang(?labelstart) = ‘en’)
FILTER (lang(?labelend) = ‘en’)
}

Note that query for the lat and long is a bit more complicated that it might be. This is because DBpedia data is quite messy, and many entities will have more than one lat/long pair. I used a subquery in SPARQL to pull out the minimum lat/long for all the pairs retrieved. Additionally I also retrieved the English labels for each of the start/end points.

Now copy/paste this construct query into the ‘Query’ tab on the Semantic Web Importer:

Screen Shot 2014-03-26 at 15.54.34

Now hit the run button and watch the data load.

To visual the data we need to do a bit more work. In Gephi click on the ‘Data Laboratory’ and you should now see your data table. Unfortunately all of the lats and longs have been imported as strings and we need to recast them as decimals. To do this click on the ‘More actions’ pull down menu and look for ‘Recast column’ and click it. In the ‘Recast manipulator’ window go to ‘column’ and select ‘lat(Node Table)’ from the pull down menu. Under ‘Convert to’ select ‘Double’ and click recast. Do the same for ‘long’.

Screen Shot 2014-03-26 at 16.01.19

when you are done click ‘ok’ and return to the ‘overview’ tab in Gephi. To see this data geospatially go to the layout panel and select ‘Geo Layout’. Change the latitude and longitude to your new recast variable names, and unclick ‘center’ (my graph kept vanishing with it selected). Experiment with the scale value:

Screen Shot 2014-03-26 at 16.09.49

You should now see something like this:

Screen Shot 2014-03-26 at 16.11.13

in your display panel (click image to view in higher resolution).

Given that this is supposed to be a road network you will find some oddities. This it seems to down to ‘European routes’ like European route E15 that link from Scotland down to Spain.

Tell Me About Hampshire – Linking Government Data using SPARQL federation 2

March 23, 2014 3 comments

Yesterday I blogged about how to do some SPARQL federated queries across various government websites, and this blog is a continuation of this with a different example. In this blog I give an example query which basically say ‘tell me stuff about Hampshire‘. I do this by linking up data from Ordnance Survey, the Office of National Statistics, the Department of Communities and Local Government and Hampshire County Council. This query is really just for illustrative purposes, but I want to ask ‘for all districts in Hampshire find me the index of multiple deprivation rank, the change order and operative date for that district, the website for the local authority of that district along with the addresses of parcels of land where it is planned to build new dwellings. To achieve this I need to take data from several sources and use SPARQL federation. Here is the query that answers my question. First I query Ordnance Survey linked data to find districts in Hampshire, and I then pass these districts to three other linked data services to retrieve the relevant information. To try this example head over to the Ordnance Survey SPARQL endpoint and copy/paste the following:

select ?districtname ?imdrank ?changeorder ?opdate ?councilwebsite ?siteaddress
where
{?district <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/within>
   <http://data.ordnancesurvey.co.uk/id/7000000000017765> .
  ?district a <http://data.ordnancesurvey.co.uk/ontology/admingeo/District> .
  ?district <http://www.w3.org/2000/01/rdf-schema#label> ?districtname .
 SERVICE <http://opendatacommunities.org/sparql> {
 ?s <http://purl.org/linked-data/sdmx/2009/dimension#refArea> ?district .
?s <http://opendatacommunities.org/def/IMD#IMD-rank> ?imdrank .
 ?authority <http://opendatacommunities.org/def/local-government/governs> ?district .
 ?authority <http://xmlns.com/foaf/0.1/page> ?councilwebsite .
 }
 ?district <http://www.w3.org/2002/07/owl#sameAs> ?onsdist .
 SERVICE <http://statistics.data.gov.uk/sparql> {
 ?onsdist <http://statistics.data.gov.uk/def/boundary-change/originatingChangeOrder>
          ?changeorder .
 ?onsdist <http://statistics.data.gov.uk/def/boundary-change/operativedate>
          ?opdate .
 }
 SERVICE <http://linkeddata.hants.gov.uk/sparql> {
   ?landsupsite <http://data.ordnancesurvey.co.uk/ontology/admingeo/district> ?district .
   ?landsupsite a <http://linkeddata.hants.gov.uk/def/land-supply/LandSupplySite> .
   ?landsupsite
<http://www.ordnancesurvey.co.uk/ontology/BuildingsAndPlaces/v1.1/BuildingsAndPlaces.owl#hasAddress>
   ?siteaddress .
   }
}

Happy SPARQLing…

Federating SPARQL Queries Across Government Linked Data

March 22, 2014 2 comments

SPARQL 1.1 introduces the idea of federated SPARQL queries – this enables you to execute part of your SPARQL query against a remote SPARQL endpoint. I thought I’d provide some examples of using this feature in government linked open data.

The Environment Agency has published a number of its open data offerings as linked data which you can explore here. One of these datasets is the Bathing Water Quality Data, and you can explore this via their SPARQL endpoint. I won’t go into this data in too much detail as it is not my area of expertise. The Environment Agency has created 5-star open data by linking their data to both Ordnance Survey and Office of National Statistics linked data. Look at linked data for the Eastoke bathing water site and you’ll see it linked to Havant and Hampshire in the Ordnance Survey data. A relatively straight forward SPARQL query will get you  a list of bathing waters, their name and the district they are in:

select ?x ?name ?district
where {
?x a <http://environment.data.gov.uk/def/bathing-water/BathingWater> .
?x <http://www.w3.org/2000/01/rdf-schema#label> ?name .
?x <http://statistics.data.gov.uk/def/administrative-geography/district> ?district .}

Now suppose we just want a list of bathing water areas in South East England – how would we do that? This is where SPARQL federation comes in. The information about which European Regions districts are in is held in the Ordnance Survey linked data. If you hop over the the Ordnance Survey SPARQL endpoint explorer you can run the following query to find all districts in South East England along with their names (please see a previous blog post for information about simple spatial queries):

select ?district ?districtname
where
{?district <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/within>+
   <http://data.ordnancesurvey.co.uk/id/7000000000041421> .
  ?district <http://www.w3.org/2000/01/rdf-schema#label> ?districtname .}

Using the SERVICE keyword we can bring these two queries together to find all bathing waters in South East England, and the districts they are in:

select ?x ?name ?districtname
where {
?x a <http://environment.data.gov.uk/def/bathing-water/BathingWater> .
?x <http://www.w3.org/2000/01/rdf-schema#label> ?name .
?x <http://statistics.data.gov.uk/def/administrative-geography/district> ?district .
SERVICE <http://data.ordnancesurvey.co.uk/datasets/boundary-line/apis/sparql>
{ ?district <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/within>+
   <http://data.ordnancesurvey.co.uk/id/7000000000041421> .
   ?district <http://www.w3.org/2000/01/rdf-schema#label> ?districtname .}
}
order by ?districtname

Now supposed we want to know the sediment types of the bathing waters in Havant. We can find this with the following query:

select ?x ?name ?sediment
where {
?x a <http://environment.data.gov.uk/def/bathing-water/BathingWater> .
?x <http://www.w3.org/2000/01/rdf-schema#label> ?name .
?x <http://statistics.data.gov.uk/def/administrative-geography/district> <http://data.ordnancesurvey.co.uk/id/7000000000017297> .
?x <http://environment.data.gov.uk/def/bathing-water/sedimentTypesPresent> ?sediment .
}

We can again use the SPARQL federation to do something more interesting. The follow query returns both sediment types in bathing waters in Havant together with sediment types of bathing water in regions that touch Havant:

select ?x ?name ?sediment
where {
{
?x a <http://environment.data.gov.uk/def/bathing-water/BathingWater> .
?x <http://www.w3.org/2000/01/rdf-schema#label> ?name .
?x <http://statistics.data.gov.uk/def/administrative-geography/district> <http://data.ordnancesurvey.co.uk/id/7000000000017297> .
?x <http://environment.data.gov.uk/def/bathing-water/sedimentTypesPresent> ?sediment .
}
UNION
{
SERVICE <http://data.ordnancesurvey.co.uk/datasets/boundary-line/apis/sparql>
{ ?district <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/touches>
   <http://data.ordnancesurvey.co.uk/id/7000000000017297> .
}
?x a <http://environment.data.gov.uk/def/bathing-water/BathingWater> .
?x <http://www.w3.org/2000/01/rdf-schema#label> ?name .
?x <http://statistics.data.gov.uk/def/administrative-geography/district> ?district .
?x <http://environment.data.gov.uk/def/bathing-water/sedimentTypesPresent> ?sediment .
}
}

Another great government open data resource is the Open Data Communities site. They have a SPARQL endpoint here. This federated SPARQL query (analogous to those above) can be used, for example, to find the Index of Multiple Deprivation Environment rank for Havant and surrounding districts. This works are follows:

select ?s ?imdrank
where
{
{
?s <http://purl.org/linked-data/sdmx/2009/dimension#refArea> <http://data.ordnancesurvey.co.uk/id/7000000000017297> .
?s <http://opendatacommunities.org/def/IMD#IMD-environment-rank> ?imdrank .
}
UNION
{
SERVICE <http://data.ordnancesurvey.co.uk/datasets/boundary-line/apis/sparql>
{ ?district <http://data.ordnancesurvey.co.uk/ontology/spatialrelations/touches>
<http://data.ordnancesurvey.co.uk/id/7000000000017297> .
}
?s <http://purl.org/linked-data/sdmx/2009/dimension#refArea> ?district .
?s <http://opendatacommunities.org/def/IMD#IMD-environment-rank> ?imdrank .
}
}

I will now leave it as an exercise to the reader to figure out how these all combine so you can ask for ‘all bathing waters in Havant and surrounding areas, and the IMD environment ranks of the areas containing those bathing waters’ – it is possible!

Please note that federated SPARQL can be slow…happy SPARQLing.

Categories: linked data, Semantic Web

Ordnance Survey Linked Data: The Search API

September 24, 2013 Leave a comment

Please note in some of the examples below I have been having trouble with wordpress ‘correcting’ quote marks in my text. If you find the queries don’t work you may need to manually replace the copied quote marks from below with new ones via your keyboard. Hope that makes sense.

One of the biggest improvements to the new Ordnance Survey Linked Data site is the much improved search functionality. You can either search over a specific dataset (e.g. the Code-Point(R) Open linked data) or over all the combined datasets. I will first give some examples of using the Boundary-Line(TM) search API.

The Boundary-Line search API explorer can be found here. The simplest use of this search API is to enter some text for the name of an administrative area or the GSS code (the ONS identifier for a statistical region) into the search box. To get started enter Southampton into the query box. You will see that the search results are returned in JSON (RSS and Atom are additional options). Results contain the URI of the entities that match your queries along with a number of useful attributes.

Note that the Request box shows the actual GET request that is being done, and you can use this GET request in your applications. Now try searching for a GSS code, enter E06000045 into the query box. You should see results for the City of Southampton returned. So far so straight forward. The search function also allows for wildcards in search, for example in the Query box type:

label:Southa*

It is also possible to narrow search results by type. Recall that the search for Southampton returned both Westminster constituencies and a unitary authority with Southampton in their name. To just find the Westminster constituencies search for the following:

label:Southampton AND type:”http://data.ordnancesurvey.co.uk/ontology/admingeo/WestminsterConstituency

The search API also allows you to perform a number of simple spatial queries. The first of these are bounding box queries. For the Boundary-Line data you can specify a bounding box, and find all the administrative regions whose centroids lie within that bounding box. The bounding box can be expressed in eastings and northings. For example try the following:

easting:[371000 TO 374000] AND northing:[161000 TO 164500]

in the query box.

The answers can be narrowed down further by specifying the type of object that should be returned. For example to just get the civil parishes in this bounding box try the following:

easting:[371000 TO 374000] AND northing:[161000 TO 164500] AND type:”http://data.ordnancesurvey.co.uk/ontology/admingeo/CivilParish

Another type of simple spatial query we can do in the search API is ‘find me all feature of a kind type within a certain radius of a given point’. Here the point can be specified in either lat/long or easting/northing. To find all of the civil parishes in a 50 km radius of the point with easting 442339 and northing 112882 put:

type:”http://data.ordnancesurvey.co.uk/ontology/admingeo/CivilParish

into the query box and put the appropriate values in the easting and northing boxes, followed by a 50 in the radio search box. If, for example, you want to perform this query again but find civil parishes and districts enter the following into the query box:

type:”http://data.ordnancesurvey.co.uk/ontology/admingeo/CivilParish OR type:”http://data.ordnancesurvey.co.uk/ontology/admingeo/District

and try the query again.

These are just some simple examples of the search API. The full documentation is here.

Interview in ODBMS.org

September 1, 2013 Leave a comment

I did a recent interview with my friend Prof. Roberto V. Zicari over at odbms.org. You can read it here. (Please forgive the few typos that crept into some of my answers).

Categories: linked data, Semantic Web

Experiments with schema.org

August 26, 2013 7 comments

Directly quoting from schema.org:

This site provides a collection of schemas, i.e., html tags, that webmasters can use to markup their pages in ways recognized by major search providers. Search engines including Bing, Google, Yahoo! and Yandex rely on this markup to improve the display of search results, making it easier for people to find the right web pages.

Many sites are generated from structured data, which is often stored in databases. When this data is formatted into HTML, it becomes very difficult to recover the original structured data. Many applications, especially search engines, can benefit greatly from direct access to this structured data. On-page markup enables search engines to understand the information on web pages and provide richer search results in order to make it easier for users to find relevant information on the web. Markup can also enable new tools and applications that make use of the structure.

My favourite band, New Model Army, are touring later this year so I thought creating a website for their UK tour dates would be a good way to experiment with schema.org markup. First off we have a webpage for the UK leg of their winter tour:

http://www.johngoodwin.me.uk/event/newmodelarmy-uk-wintertour-2013

which is a kind of Music Event, and this is related to a number of sub events such as:

http://www.johngoodwin.me.uk/event/newmodelarmy-uk-Aberdeen-20131112

which is also a kind of Music Event. Each of these events are related to a venue, via the ‘location‘ predicate:

http://www.johngoodwin.me.uk/venue/The-Garage-Aberdeen

I have related each of the venues to locations in the Ordnance Survey Linked Data using the contained in predicate.

For each of the events I have included further markup such as the start date, end date, ticket sites (via the offers predicate) and also links to other pages about the event via sameAs. Similarly I have linked pages about each venue to other pages about that venue via sameAs.

All of the markup is done using RDFa. You can view the machine readable content using the Google Structured Data Testing tool or this RDFa Parser. Here are some examples:

New Model Army UK Winter Tour 2013: rich snippets, RDFa distiller

Aberdeen tour date: rich snippets, RDFa distiller

The Garage Aberdeen (venue): rich snippets, RDFa distiller

Mapping is provided by OS OpenSpace.

Extra: someone asked me what the experiment was. Maybe not much of an experiment really, but simply put I’m curious to find out what happens should these pages get picked up by Google et al., and curious to see what they do with them.

Follow

Get every new post delivered to your Inbox.

Join 2,189 other followers