Open Source Computing and GIS in the UK

There has been a lot of fairly excitable posting recently about the continuing rise of OpenStreetMap, and how it’s now being used in place of Google Maps,  in particular since Google started charging for data. People have been talking about how “authoritative” crowd-sourced spatial data can be, and to be honest, I’ve found that the discussions seem to have missed the point a little bit. For me at least. So- here’s a few of my personal thoughts about OpenStreetMap and why it will be a while before I will consider it authoritative at least.

Firstly, a caveat. I love OpenStreetMap data and the whole ecosystem of “stuff” that has evolved around it. I contribute occasionally, though not as often as I’d like to. I’m pleased that it has forced the “traditional” data providers to reconsider what they offer, and perhaps to raise their game. I think it’s one of the main reasons why the idea of “open data” is so well known that even my Gran knows about it.

However, in the process of trying out WalkingPapers (my favourite addition to the OpenStreetMap ecosphere), I printed out the area around my house- in the centre of Lancaster. Not London, I admit, but still in the centre of a city. My road wasn’t on there. Read on, before you give up in disgust, saying “pah, you should just correct any errors you find”. How did I know my road was missing? There wasn’t an unfinished stub, with a note saying “I went home for my tea and I’ll finish this section tomorrow”, it was as if my road did not exist. I only knew my road was missing because I live there, and I have no readily available tools to help me judge the accuracy of an area that I am not familiar with. So, the idea that the end-user should correct any errors they find doesn’t really hold because you’re not always going to know that there’s an error!

I am aware that the traditional data providers do not produce totally error-free data. However, I would expect Ordnance Survey (for example) to have a workflow for their surveyors that doesn’t allow them to simply give up when they get bored or have to go home for their tea, or perhaps not survey a street because they don’t go down it very often.  I can also go to the site and find published information on positional accuracy if I need it. I guess I’m equating “authority” with trust, or reliability, and the issue is less about what’s there, than what’s not there.

When OpenStreetMap is being used as a static base map, by which I mean a backdrop to the information your map is really about, this is less of an issue. But, if like me, you come from a discipline where maps are collections of data to be manipulated and analysed, then you need to be able to trust your data a little more, or at least have a quantitative understanding of what the error is likely to be.

I know that this is not an easy question to answer, and I know people are trying to figure out ways of answering it. I also know it’s not that important for a lot of people, but I would like to see a more nuanced debate- I think that would be better for OpenStreetMap in the end.

Caution- this post won’t make you a pgRouting guru, but it will allow you to get pgrouting up and running on Ubuntu 11.10 and have some data on a map in approx 20 minutes.

1. Follow the instructions here to install PostgreSQL, PostGIS and pgRouting. Note that the instructions are for PostgreSQL 8.3 but they work just as well for PostgreSQL 9.0, just go with whatever version you want and substitute the package name accordingly.
2. If you’re using PostgreSQL 9.0, you might hit an error when loading the driving distance functions, as seen here. There’s a fix, which  may or not have been committed by the time you try this- if not then you need to apply it by replacing two altered files and recompiling/making and installing pgrouting (repeat the “compile pgRouting” step in the link in step 1).
3. Go here and download a selection of osm binary (pbf) files.
4. Follow the instructions here to download and run osm2po. Note that the documentation is currently available only in german. However, the example given in the link will work to get you a working demo, and it’s quite easy to follow the examples. Note also that you need a java runtime environment to use this- so if you don’t have one then download the jre of your choice using your favourite software package manager.

If this whets your appetite and you want to learn more, then there’s an excellent workshop available on the pgRouting site, along with lots of useful howtos and tools.

This, of course, only scratches the surface of pgRouting, but it just goes to show that thanks to all of the great links I’ve listed above, it is possible to get a working demo going extremely quickly.

If you end up doing a lot of work on mapserver and tilecache behind corporate proxy servers, you’re likely to hit a couple of snags, when the proxy thinks you’re trying to do something evil, when really all you want to do is seed a tilecache or look at an external WMS server. Fortunately there are a couple of useful workarounds, which I record here for my own sanity as much as anything else!

Mapserver

Mapserver has some great documentation on how to use proxy authentication when requesting data from an external web server, but unfortunately if you’re still on Mapserver 5.6.x, and on windows, you are likely to find that some of the directives don’t work. When you test your mapserver layer in a browser, you get a blank screen, and the following error message in your logs:

HTTP request error. CURLOPT_PROXYAUTH not supported. Requires Curl 7.10.7 and up

Needless to say, the later versions of mapserver (5.6.6) are using Curl versions greater than 7.10.7 so we’ll ignore that part of the message and concentrate on the CURLOPT_PROXYAUTH part. A quick google will inform you that this must be explicitly compiled into mapserver, and as far as I can tell, none of pre-compiled windows binaries for 5.6.x have been compiled with this option.

Enter cntlm, which is designed to stand between your application and your corporate proxy server, providing proxy authentication on the fly. To use it with mapserver, install the latest binaries and follow the configuration hints on the home page. The basic process is to run cntlm from a command line to establish the type of authentication that your proxy server uses, then fill in the supplied cntlm.ini file with the appropriate values for your configuration. If you don’t know these, you can easily find them by opening firefox or similar and  looking at preferences->advanced->network->settings.  You’ll need the URL of the proxy, the port, and the type.

Then, assuming that cntlm is installed on the same server as mapserver, your map file needs only the following layer metadata parameters (in addition to anything else you have defined):

wms_proxy_host: localhost

wms_proxy_port: specified in cntlm.ini (default is 3128)

wms_proxy_type: http or socks5 etc

With luck, once you have cntlm configured correctly, your external WMS layer should display just fine!

Tilecache

If when seeding your tilecache you receive “bad gateway” errors, or inexplicably corrupted tiles, it’s possible that the tilecache_seed.py script is also being blocked by the proxy server on your network. Somewhat counter-intuitively (at least it seems that way at first), this can be easily circumvented by switching off the proxy settings in Internet Explorer on the machine you’re running the script on (and switching back on once the script completes successfully). Note that it has to be Internet Explorer- you’ll need another approach if that’s not installed.

The reason for this is that tilecache_seed.py uses the python module urllib2, which uses ProxyHandler to handle proxies (who would have guessed). This will attempt to autodetect proxy settings, and one of the options it checks is in Internet Explorer (I guess this equates to a registry setting somewhere but we don’t go digging in the registry without our vaccinations up to date and all our affairs in order, do we?). A longer term solution might be to extend the tilecache_seed.py script to explicitly set up a ProxyHandler with no proxies defined, as in this example.

September is clearly conference month in the geo world. FOSS4G in Denver, which I didn’t go to, was closely followed by the AGI GeoCommunity in Nottingham, which I did. I participated FOSS4G vicariously, following twitter and starring lots of posts in my google feed, and it’s only now that I’m having chance to catch up on them all and assimilate them. Seems like there was a lot of soul-searching going on, about the future of the organisation, the conference, and the geospatial industry in general. Seems also like there was a really good vibe to the event, perhaps related to a really good choice of venue, which kept people together for socialising, networking and whatever. Perhaps these two factors are related? Get a bunch of conference-goers together over some geo-beers, and no doubt a lot of discussion will take place.

One of the key questions that seems to have come up as a result of the success of this year’s event has been whether or not there should be an annual North American event alongside the global event. Personally, not being in North America, I can only see this from the international perspective, and to me it looks bad. Diluting the FOSS4G brand has to be a bad thing. How many of the big players will choose to come to the international event when they “know” everyone “important” is going to be at the North American event? However, from the perspective of raising cash for the foundation, perhaps it’s a good idea because it’s probably more commercially viable. I don’t know, but I kind of wish the idea hadn’t been brought up…

With my new directors hat on, I’m sure there’s going to be a lot more discussion about this, alongside the organisational changes necessitated by dissolving the Executive Director’s post. I’m not going to go into that at all, but I would just like to say that I think Tyler did a fantastic job for OSGeo, particularly on the international scene- here in the UK he “was” OSGeo and things won’t be the same without him.

Over to the AGI GeoCommunity event. Having been on the working group helping to organise the event this year, again I had a different perspective on things to normal. Being on the working group is quite easy in the run up to the event as the AGI team do most (nearly all) of the hard work. However, you’re expected to help with exciting things like bag packing, chairing streams, and generally being around and visible throughout the two days. No sleeping during not so interesting talks! Attendance was good in the end, after a rush of late registrations, the new venue (the East Midland Conference Centre at the University of Nottingham) was a big success, and in the end even the last minute substitution of two plenary speakers (for perfectly good reasons) didn’t upset things!

I chaired the session on Open Source and Open Data on day one, which included some really worthwhile papers. Standing out (a couple of weeks after the event) are Antony Tuffour’s paper on Open Source Software Stack and Standards- Integration and GI for Everyone- about introducing a FOSS web mapping system at the London Borough of Hackney, and Mark Iliffe’s talk on When Gov 2.0 doesn’t exist- Mapping Services in the Developing World. I’ve seen Mark’s talk a couple of times now and it’s always inspirational. All the papers are available on the AGI website, so go take a look.  The famous soapbox event in the evening was won by my new(ish) boss. His talk was in reaction to the great FUD debate from a few weeks ago, excellently shown up for the absurdity that it is at the time by Paul Ramsey, and again here. Catch the YouTube video while you can!

Other things of note include the relaunch of the Technical Special Interest Group again. It’s easy to be cynical about this, as even I have now been around the AGI long enough to have seen the previous attempt lose momentum and stall, but enthusiasm is everything, and there is a lot of interest in the group as a relatively neutral umbrella under which both proprietary and open source vendors can talk techy without the sales pitch. Look out for events towards the end of the year.

There was plenty of soul-searching at the AGI GeoComm as well as at FOSS4G. While the two organisations are quite different, the same basic questions come up around providing best value, what the organisation is actually for, and how best to retain sponsorship and income in difficult financial times. Let’s just hope everyone pulls through in the end…