I imagine I’m not alone in having parents and grandparents who don’t really understand what I do for a living. “I work in computing and do stuff with maps” is the easy approach (in fact it’s easier now that I don’t have to tag on the bit about being an archaeologist but not actually digging, and no it’s not like Time Team or Indiana Jones). Sometimes people ask why we don’t just “do everything with google maps”, which is the cue for a sit down and a longer chat about how (deep breath) you can’t do *everything* with google. So far, so good…

This all changed a couple of weeks ago, when, to my surprise I got elected to the Board of Directors of OSGeo. Cue shock, and great rejoicing (and in my head at least, tearful Oscars-style acceptance speeches). Mr Archaeogeek thinks this is cool enough to tell parents, grandparents, family friends etc. I do too, don’t get me wrong, but…

How to explain what it means to people who don’t use computers all that much, let alone know about gis, or open source or OSGeo?

There have been a few good articles on how to explain why you work in open source (the “how do you make money?” argument), but I feel like I need to go further back and explain about software licensing. I don’t think that most people really understand the difference between the way software is sold/licensed, and most other products, so I’m working on a car analogy that explains why open source software needs to exist. It goes something like:

“Can you imagine, that if you brought a car, and something went wrong with it, you couldn’t lift up the bonnet and take a look? Or if you did need to take it to a garage you had to use the one the car salesman told you to use? Or after 2 years they told you that they wouldn’t support your old car any longer, and you had to buy a new one? Or you couldn’t insure more than one person to drive it, but had to buy a new car for each person? Well that’s sort of what closed-source software is like”

(Note this is an analogy-in-progress)

Once that’s out of the way, then I can get on to gis, and hence to OSGeo, and all is fine!

Woefully out of date now, here’s a quick run down on the OSGIS 2011 conference, 3rd in that series, held at the University of Nottingham Centre for Geospatial Sciences in Nottingham over the 21st and 22nd of June.

The 21st was a day of workshops, under the banner of Interoperability and the OGC. My new colleague, Matt, and I did a workshop on using Ordnance Survey Open Data and Mastermap with Mapserver and PostgreSQL, using the OSGeo Live DVD. You can see a slightly edited version of the workshop below, or on slideshare. I have to admit that most of the kudos must go to Matt for creating some super scripts to make the initial data processing *much* easier, and to some of my other colleagues for their efforts in styling the data once it’s in Mapserver. The scripts and a small subset of the open data are available here- you’ll have to supply your own Mastermap!

Day Two was all about the talks- and I was impressed by the standard. The focus of OSGIS has always leant slightly towards the academic, so the stand-out talks for me were the ones that demonstrated that you can do real spatial analysis with open source GIS. There were also some very good papers on mapping in the developing world. Two of my ex-colleagues from Oxford Archaeology also did a joint paper showing how the use of open source software has progressed there- that was really good to see- it was nice to know that the baton had been passed on when I left! I gave people an introduction to the OSGeo:UK local chapter, which is also available on Slideshare here, and we had the chapter AGM. It’s extremely gratifying to see the numbers of people willing to hear about, and get involved with, the local chapter. I was going back through the reports I’d given in previous years, and the numbers of people signed up to our mailing list steadily creep up, year on year- we’re now well over the 100 people mark! (BTW, if you’re interested, our website is here).

All in all it was a really good couple of days. Next year the conference will be running from the 4th to the 5th of September, so for anyone that can’t make FOSS4G and wants to give OSGIS a try, now’s your chance!

A short case study into flexibility, collaboration, and why open source software is so damned cool:

At my new place of employment, we’re doing a lot of work with Ordnance Survey Mastermap data, so one of my colleagues built a quick python wrapper around the ogr2ogr script to easily pop the data into postgresql, or shape file, or whatever support format you like. This is now available on Github (caveat- it doesn’t do change-only updates yet- we’ll keep you posted on that). After a chance comment on the OSGeo:UK mailing list, our friends at Faunalia, who do great things with Quantum GIS, ported the code into a QGIS plugin.

So- in the space of one week, QGIS gained itself a Mastermap loader, through a simple bit of collaboration. Nice work guys! (Nothing to do with me, I was on holiday). Can you get that sort of flexibility, rapidity and collaboration with (insert your favourite proprietary package here)? I doubt it…

So, I’m late to the game and only just learning about the coolness that is python. To be honest, for years the need to keep the indents in the code neat and tidy put me off, but I figured I’d better have a proper look at some point. I spent some time over Christmas going over some tutorials (more below) and more recently I’ve chosen a python-based approach to problems where previously I would have used a different method. So far so good though no doubt my efforts will make grown coders weep…

These are just a few links that I have found really useful- the first is a very good tutorial for new python users who are particularly interested in Geoprocessing, whilst the others deal with more specific problems.

• Geoprocessing with Python using Open Source GIS. This is a full online course, with examples and assignments to try (and solutions, thank goodness). There is a couple of week’s work here to complete the full course.
• Using Psycopg2 with PostgreSQL. This allows you to work with postgresql databases in python (disclosure- other drivers are available). I found this straightforward enough to pick up and use in a real-world situation pretty much immediately.
• Connecting to an OpenOffice spreadsheet with Python. There is plenty of documentation about OpenOffice and Python, but again I found I could get going immediately with this particular link.

If anyone has any other resources for python beginners, particularly around Open Source GIS and PostgreSQL, feel free to add them in the comments.

So, it’s 5 years since OSGeo was formed- that’s pretty cool! Spurred on by this post, I thought I would say a little bit about my involvement with OSGeo, and also rather clumsily segue this into an announcement about my impending change of job. It’s true- after years of not really thinking of myself as an archaeologist any longer, but rather ‘someone who works in an archaeological unit”, I’ve finally gone and got myself a real job. Anyhow, more on that later.

5 years ago, I was playing around with mapserver, because I wanted a way of serving data and some shiny web maps and we had no money to do it with. I sucked at it big time, with no programming experience, and barely any command line experience. My only linux knowledge consisted of messing around with a Sharp Zaurus (the best pda I’ve ever had), and I only really equated open source with “free as in beer”. I remember the excitement on the mapserver mailing list when OSGeo was announced, along with the open sourcing of MapGuide, and I also remember quite fancying the idea of a whole conference about this cool new stuff when FOSS4G was announced.

Since then, there have been so many lightbulb moments for me, such as getting the real “point” of open source, getting more involved with OSGeo via starting the UK chapter, breakthrough moments when I have really started to understand a particular program or language, getting over my fear of public speaking by talking about open source at conferences, teaching colleagues and strangers how to use this software, and getting consultancy jobs designing applications that use it.

Through that time, I’ve seen (heck, we’ve all seen) OSGeo go from strength to strength, and become a truly global organisation. More importantly though, it’s a great global community.

I should also say that my new job, as Web GIS Specialist/Lead Consultant at Astun Technology, could only have happened through the experience I’ve gained over the last five years, so thanks to everyone that’s helped with that. I’m sad to be leaving Oxford Archaeology, but massively looking forward to my new role, which I hope will allow me to keep up my involvement with OSGeo, and in particular the UK chapter.

Will I keep blogging? Yes, and hopefully more often as I learn lots of new and exciting stuff (new to me- I’m a recent very enthusiastic convert to Python-years after everyone else). Will I still be doing open source stuff? Hell, yes! Will I change the name of the blog? Pehaps…

Over Christmas I came across a couple of extremely useful publications for those people needing to promote open source to business users.  The first is a report on the International Status of Open Source Software, from the Cenatic Foundation.  As well as providing an overview of the state of Open Source adoption around the world, the report looks at various factors that might explain differing “maturity levels” (their term) or levels of adoption. They highlight the importance of engagement from the private sector, public sector, universities and development communities. The UK, for instance, is held back by the lack of promotion at government level, compared to countries such as Germany, where greater EU involvement also lends a helping hand. None of the findings are rocket science, I grant you, but it’s one of those useful documents that provides and overview and sources for some nice shiny graphs in presentations.

The other useful publication is the January 2011 issue of Open Source Business Resource, a Canadian monthly (free) magazine that deals with Open Source from a business perspective. It has a different theme each month and is available in pdf or html format. The January 2011 issue is entitled “The Business of Open Source” and contains, amongst other things, an article on ‘cost optimisation through open source software’. What this basically does is ask “Yeah, but…” to all those annoying FUD and whitewash articles that claim it’s cheaper to go proprietary because open source is not free  and has lots of hidden costs, like training or administration (…).  For example, they examine the salary costs required to administer a windows-based setup versus unix, bearing in mind the more diverse skill set of the average windows versus unix sysadmin (their argument, not mine).  Again, there’s no rocket science here, but some very useful facts and figures. You might also like the March 2009 issue of OSBR, which is all about Open Source Geospatial- and there are others in the archives about support, licensing, business models and so on.

It seems like all I’ve posted about recently is very quick updates on conferences that I’ve been to, and not a lot about what I actually spend most of my time doing. The principal reason for this is being too busy, but that’s getting a bit lame. Once upon a time I used to post lots of tips and tricks about things I’d figured out- sometimes just as a aide memoire after days of trawling through mailing lists and forums. Once upon a time I used to have opinions (sometimes quite strong) on what was going on in the geo-world. It’s time I got back into that mindset, so here goes with a random selection of musings, in no particular order,  from the last few weeks…

• When using mapserver, the path to the log file for debugging must be an absolute path, relative won’t work. This is documented, but I missed it…
• Also when using new versions of mapserver, if making a WMSGetFeatureInfo request from OpenLayers (like this example), you will initially get an error saying that “FORMAT is a required parameter”. This does not occur with Geoserver. You can’t simply add it in the options, because it’s not sent through as part of the request. You have to add it as a vendorParam, but this isn’t well documented.
• Some of my colleagues/ex-colleagues have come up with a great workflow for producing publication-quality cartographic output from QuantumGIS and GvSIG. This is always one of the key excuses for needing expensive software, so it’s nice to give some alternative options. You can find it here, along with our guide for working with survey data in open source GIS. Anna and Christina have been doing some great stuff with open source GIS recently, in the rather more high-pressure world of field excavation rather than the rarified (read lower pressure) geospatial analysis work that I tend to do. What they have proven is that, with relatively small changes in workflow, it is possible to produce high-quality standards-compliant output, as part of a chain of contracters, using these open source packages, where we previously used to rely on proprietary software. Well done guys!
• The ThinkOpen event in Newcastle a few weeks ago (see, I couldn’t resist a quick conference mention) was great fun and very well attended despite falling at the beginning of snowmageddon here in the UK. The focus of the event was on case studies for open source GIS, along with a chance to see some of the new things Ordnance Survey has been doing with it’s OpenSpace and OpenData initiatives. This was the first time I had seen audience electronic voting in a conference- with live results (sometimes).
• The UK chapter of OSGeo is going from strength to strength, mainly thanks to the enthusiasm of the Centre for Geospatial Sciences at Nottingham, and a growing band of dedicated individuals. We’re getting ourselves on a stronger footing now for providing the kind of services that I have always wanted the chapter to provide, and I have high hopes for 2011 onwards!
• I got voted an OSGeo Charter Member a few weeks back, which was a lovely suprise and very much appreciated, given the high number of extremely busy, dedicated candidates. Of course I will continue to do what I can to promote the foundation, particularly since the business case for open source GIS is stronger than ever.
• I am more and more in love with QGIS as my primary GIS package. In the last few releases (1.5 onwards) it seems to have matured tremendously, and a whole series of enhancements have snuck in without me really noticing. If you haven’t tried it recently, download the most recent version- you will be pleasantly suprised! I also love the OSGeo4W setup method. This makes it incredibly easy to keep the software up to date, and to roll out unattended installs across a network.  I have an idea in the back of my mind about extending this to a USB stick, to merge PortableGIS with the OSGeo4W project. However, it’s just an idea at the moment and needs a bit of time to mature…
• I’m teaching a couple of workshops on an Introduction to GIS over the next few weeks (snow permitting), and have been benefiting immensely from the material put out by Linfiniti and others. Much appreciated guys- and I will of course be putting my notes up when they are done.

Phew! Hope that’s enough to keep you all going!

Once in a while, Oxford Archaeology get called upon to do some really big archaeological projects, like road schemes and airport expansion, that cover huge areas, go on for years, and generate loads of data. We love these, because it’s not very often that you get to look at whole landscapes- how multiple prehistoric villages interact, for example, rather than tantalising snapshots where you have to play “join the dots”.

Not only do we generate loads of data (thousands of artefacts, records, environmental samples, photos etc), but we ask difficult questions of them, based on statistical analysis of finds distribution, travelling salesman algorithms, best-path analysis. I think you can call this “real GIS”, though often people are surprised to hear that archaeologists use GIS at all.

As part of our ongoing “open approach“, and to prove that we put our money where our mouth is, we are now trying to do all of this in open source software rather than using the “standard” proprietary packages.  I’ve blogged previously about how pleased I was with the integration between QGIS and PostgreSQL, and how easy it was to manage large amounts of data without regress to proprietary packages, well  our current large project needs 3D analysis and large amounts of imagery manipulation, and again we’re finding that the open source tools out there do the job splendidly. Furthermore, we have a choice of tools, so if one approach doesn’t quite work the way we expect or want, then we can choose another. Now that can’t be bad, can it?

So, in brief, we’re using Quantum GIS and GvSIG pretty much interchangeably for our desktop GIS. All the vector data is in PostgreSQL. We use the QGIS Grass plugin to get data into a sensible format for 3D display and analysis in Paraview and Visit. We’re mosaicing up aerial photos using GDAL tools, and using Geoserver to publish everything to people who just need read-only access, and a direct connection to PostgreSQL for those that need to edit. We’ve developed a workflow for creating high-quality cartographic output by exporting to Inkscape, and the next step is a project website with links to our database and a nice openlayers map. Simples!

I’m helping to teach at a workshop on open source GIS at the University of Leicester in a couple of week’s time. As usual, this means running around trying to get all the software that we need installed on the university computers. As usual, what the course organisers think will be OK, and what the IT department think will be OK, are two different things!

We’d rather not use a LiveDVD, as we want the students to work in the environment which they are used to- in this case windows. We don’t really want to saddle the IT department with lots of PostgreSQL and Apache configuration, so we’re going to try using Portable GIS (gulp).

The current iteration (as available here), has an out of date version of Quantum GIS, doesn’t contain the statistical package R, and also contains a bunch of other stuff that we won’t use on the course (MySQL, GvSIG etc). The beauty of open source cross platform software though, is that rolling a customised version, containing just what software we need, and the latest versions, was quite easy.  Adding in the data, and even the course notes, will be straightforward (once we’ve written them, of course!). Installation is simply a case of copying everything onto each pc, and the students can take it all home with them when they are done.

The open source license is not the big deal here, despite the title of the post, but the added benefits that it brings are pretty cool. Free software means we can run the course without having to worry about buying licenses, or sending students home with limited demo versions afterwards. The cross-platform nature of the software means most of the donkey work is done in human readable files that can easily be edited to work in a portable fashion, and finally, because it’s open source, we can do that legally.

I think that’s all pretty cool…

(Small Plug) If you like the idea of using Portable GIS for a course, but don’t fancy customising or preparing it yourself, then get in touch!

There’s a fantastic article in today’s Guardian (via Computer World Magazine) about the International Intellectual Property Alliance, who say that countries advocating the use of open source software should be put on a “Specialist 301 list” (ie a trading watch list) because open source “weakens the software industry” and “fails to build respect for intellectual property”.  The IIPA is an umbrella group for organisations like the RIAA and the MPAA, who are of course well-known for their open-mindedness and forward thinking.

Go read the articles, I’ll wait…

As well as the obvious lack of understanding of, well, anything, this is all quite bizarre. It implies disapproval of almost the entire internet infrastructure, as well as the big (US) players like Google. It shows a complete lack of understanding about companies (in the US) making a revenue from an open source business model, or even including open source software within their offerings. And how you can claim that the open source licenses somehow harm intellectual property is beyond me.

Oh well, I guess that puts the British Government on the watchlist, as well as the US government, and presumably the IIPA need to put themselves on if they use Apache at all…