Open Source Computing and GIS in the UK

So, it’s 5 years since OSGeo was formed- that’s pretty cool! Spurred on by this post, I thought I would say a little bit about my involvement with OSGeo, and also rather clumsily segue this into an announcement about my impending change of job. It’s true- after years of not really thinking of myself as an archaeologist any longer, but rather ‘someone who works in an archaeological unit”, I’ve finally gone and got myself a real job. Anyhow, more on that later.

5 years ago, I was playing around with mapserver, because I wanted a way of serving data and some shiny web maps and we had no money to do it with. I sucked at it big time, with no programming experience, and barely any command line experience. My only linux knowledge consisted of messing around with a Sharp Zaurus (the best pda I’ve ever had), and I only really equated open source with “free as in beer”. I remember the excitement on the mapserver mailing list when OSGeo was announced, along with the open sourcing of MapGuide, and I also remember quite fancying the idea of a whole conference about this cool new stuff when FOSS4G was announced.

Since then, there have been so many lightbulb moments for me, such as getting the real “point” of open source, getting more involved with OSGeo via starting the UK chapter, breakthrough moments when I have really started to understand a particular program or language, getting over my fear of public speaking by talking about open source at conferences, teaching colleagues and strangers how to use this software, and getting consultancy jobs designing applications that use it.

Through that time, I’ve seen (heck, we’ve all seen) OSGeo go from strength to strength, and become a truly global organisation. More importantly though, it’s a great global community.

I should also say that my new job, as Web GIS Specialist/Lead Consultant at Astun Technology, could only have happened through the experience I’ve gained over the last five years, so thanks to everyone that’s helped with that. I’m sad to be leaving Oxford Archaeology, but massively looking forward to my new role, which I hope will allow me to keep up my involvement with OSGeo, and in particular the UK chapter.

Will I keep blogging? Yes, and hopefully more often as I learn lots of new and exciting stuff (new to me- I’m a recent very enthusiastic convert to Python-years after everyone else). Will I still be doing open source stuff? Hell, yes! Will I change the name of the blog? Pehaps…

Over Christmas I came across a couple of extremely useful publications for those people needing to promote open source to business users.  The first is a report on the International Status of Open Source Software, from the Cenatic Foundation.  As well as providing an overview of the state of Open Source adoption around the world, the report looks at various factors that might explain differing “maturity levels” (their term) or levels of adoption. They highlight the importance of engagement from the private sector, public sector, universities and development communities. The UK, for instance, is held back by the lack of promotion at government level, compared to countries such as Germany, where greater EU involvement also lends a helping hand. None of the findings are rocket science, I grant you, but it’s one of those useful documents that provides and overview and sources for some nice shiny graphs in presentations.

The other useful publication is the January 2011 issue of Open Source Business Resource, a Canadian monthly (free) magazine that deals with Open Source from a business perspective. It has a different theme each month and is available in pdf or html format. The January 2011 issue is entitled “The Business of Open Source” and contains, amongst other things, an article on ‘cost optimisation through open source software’. What this basically does is ask “Yeah, but…” to all those annoying FUD and whitewash articles that claim it’s cheaper to go proprietary because open source is not free  and has lots of hidden costs, like training or administration (…).  For example, they examine the salary costs required to administer a windows-based setup versus unix, bearing in mind the more diverse skill set of the average windows versus unix sysadmin (their argument, not mine).  Again, there’s no rocket science here, but some very useful facts and figures. You might also like the March 2009 issue of OSBR, which is all about Open Source Geospatial- and there are others in the archives about support, licensing, business models and so on.

It seems like all I’ve posted about recently is very quick updates on conferences that I’ve been to, and not a lot about what I actually spend most of my time doing. The principal reason for this is being too busy, but that’s getting a bit lame. Once upon a time I used to post lots of tips and tricks about things I’d figured out- sometimes just as a aide memoire after days of trawling through mailing lists and forums. Once upon a time I used to have opinions (sometimes quite strong) on what was going on in the geo-world. It’s time I got back into that mindset, so here goes with a random selection of musings, in no particular order,  from the last few weeks…

• When using mapserver, the path to the log file for debugging must be an absolute path, relative won’t work. This is documented, but I missed it…
• Also when using new versions of mapserver, if making a WMSGetFeatureInfo request from OpenLayers (like this example), you will initially get an error saying that “FORMAT is a required parameter”. This does not occur with Geoserver. You can’t simply add it in the options, because it’s not sent through as part of the request. You have to add it as a vendorParam, but this isn’t well documented.
• Some of my colleagues/ex-colleagues have come up with a great workflow for producing publication-quality cartographic output from QuantumGIS and GvSIG. This is always one of the key excuses for needing expensive software, so it’s nice to give some alternative options. You can find it here, along with our guide for working with survey data in open source GIS. Anna and Christina have been doing some great stuff with open source GIS recently, in the rather more high-pressure world of field excavation rather than the rarified (read lower pressure) geospatial analysis work that I tend to do. What they have proven is that, with relatively small changes in workflow, it is possible to produce high-quality standards-compliant output, as part of a chain of contracters, using these open source packages, where we previously used to rely on proprietary software. Well done guys!
• The ThinkOpen event in Newcastle a few weeks ago (see, I couldn’t resist a quick conference mention) was great fun and very well attended despite falling at the beginning of snowmageddon here in the UK. The focus of the event was on case studies for open source GIS, along with a chance to see some of the new things Ordnance Survey has been doing with it’s OpenSpace and OpenData initiatives. This was the first time I had seen audience electronic voting in a conference- with live results (sometimes).
• The UK chapter of OSGeo is going from strength to strength, mainly thanks to the enthusiasm of the Centre for Geospatial Sciences at Nottingham, and a growing band of dedicated individuals. We’re getting ourselves on a stronger footing now for providing the kind of services that I have always wanted the chapter to provide, and I have high hopes for 2011 onwards!
• I got voted an OSGeo Charter Member a few weeks back, which was a lovely suprise and very much appreciated, given the high number of extremely busy, dedicated candidates. Of course I will continue to do what I can to promote the foundation, particularly since the business case for open source GIS is stronger than ever.
• I am more and more in love with QGIS as my primary GIS package. In the last few releases (1.5 onwards) it seems to have matured tremendously, and a whole series of enhancements have snuck in without me really noticing. If you haven’t tried it recently, download the most recent version- you will be pleasantly suprised! I also love the OSGeo4W setup method. This makes it incredibly easy to keep the software up to date, and to roll out unattended installs across a network.  I have an idea in the back of my mind about extending this to a USB stick, to merge PortableGIS with the OSGeo4W project. However, it’s just an idea at the moment and needs a bit of time to mature…
• I’m teaching a couple of workshops on an Introduction to GIS over the next few weeks (snow permitting), and have been benefiting immensely from the material put out by Linfiniti and others. Much appreciated guys- and I will of course be putting my notes up when they are done.

Phew! Hope that’s enough to keep you all going!

Once in a while, Oxford Archaeology get called upon to do some really big archaeological projects, like road schemes and airport expansion, that cover huge areas, go on for years, and generate loads of data. We love these, because it’s not very often that you get to look at whole landscapes- how multiple prehistoric villages interact, for example, rather than tantalising snapshots where you have to play “join the dots”.

Not only do we generate loads of data (thousands of artefacts, records, environmental samples, photos etc), but we ask difficult questions of them, based on statistical analysis of finds distribution, travelling salesman algorithms, best-path analysis. I think you can call this “real GIS”, though often people are surprised to hear that archaeologists use GIS at all.

As part of our ongoing “open approach“, and to prove that we put our money where our mouth is, we are now trying to do all of this in open source software rather than using the “standard” proprietary packages.  I’ve blogged previously about how pleased I was with the integration between QGIS and PostgreSQL, and how easy it was to manage large amounts of data without regress to proprietary packages, well  our current large project needs 3D analysis and large amounts of imagery manipulation, and again we’re finding that the open source tools out there do the job splendidly. Furthermore, we have a choice of tools, so if one approach doesn’t quite work the way we expect or want, then we can choose another. Now that can’t be bad, can it?

So, in brief, we’re using Quantum GIS and GvSIG pretty much interchangeably for our desktop GIS. All the vector data is in PostgreSQL. We use the QGIS Grass plugin to get data into a sensible format for 3D display and analysis in Paraview and Visit. We’re mosaicing up aerial photos using GDAL tools, and using Geoserver to publish everything to people who just need read-only access, and a direct connection to PostgreSQL for those that need to edit. We’ve developed a workflow for creating high-quality cartographic output by exporting to Inkscape, and the next step is a project website with links to our database and a nice openlayers map. Simples!