Open Source Computing and GIS in the UK

This is an idea that seems to come up every so often- what GIS programmes can you run from a USB stick. Well it appears that the list has just got longer. I’m probably the last person to realise it’s possible to do this, but I was really pleased to see that both GRASS and PostgreSQL can now be run from a USB stick, along with QGIS, XAMPP (inc Mapserver, OpenLayers, Tilecache), and FWTools.

Before I go any further, I should state that my aims for creating a portable GIS are not so much having a production GIS setup on a stick, because obviously performance and storage are an issue, but it might be useful for demonstration purposes, and I do feel as if the take up of these opensource tools might be more if they were easier to install and came in one handy package. So what I really want to achieve is a single portable package, probably a zip-file, that can be downloaded and unzipped onto a Window pc (they can learn about the joys of linux once we’ve won them over to the basic concept of opensource). No separate downloads, long installation and configuration processes, just a zip file. I do also like having portable packages on my main computer, as if it breaks it’s a lot easier to reconstruct the setup!

The light-bulb moment for me was when Leif told me that he had seen GRASS working on a USB stick using Msys at the recent CAA conference in Berlin. This intrigued me, so I looked into it and it works quite nicely- as long as you use the same drive letter to mount your USB stick to all the time. I still have to figure out how to figure out the drive letter and change the paths, probably in a batch file. Put simply, you download Mingw and Msys onto the stick, follow the instructions for downloading GRASS, edit the install_grass.bat file to change the drive letters, and off it goes.

PostgreSQL was something I had looked into a while ago, been told it wasn’t possible, and given up on. However, the recent 8.2 release appears to have dropped the requirement for a dedicated non-administrative user to run it. If you download the windows binaries, without the installer, you can set it up whilst logged on as the standard user. When you next run PostgreSQL from a different machine, you just have to use the same username as on your installation pc, regardless of whether that user exists on the pc that you are currently using. Once you have PostgreSQL installed, you can install Postgis into the same directory.

Currently there are a couple of gotchas. Most of these programmes require you to set environment variables, which might not be allowed on the particular pc that you are using, and require some command line experience. I still haven’t quite figured out how to get GRASS to work if I change the drive letter. Also, PostgreSQL does indeed run very slowly on a USB stick. And, I haven’t fully explored this to make sure there isn’t a hidden trap somewhere, or security holes that might be exploited. Having said that, if you want a package that can be unzipped onto a pc with a minimum of fuss then I think it’s almost there.

What’s next? Documentation, as always, and the permission of the various programmers that it’s OK to do this. Oh, and comments please, on what else we might include, or words of wisdom if you’ve already tried it.

Well, at last it’s OK for me to tell people that Oxford Archaeology now has a WFS server that is accessible from the outside world. The address is:

http://mapdata.thehumanjourney.net/cgi-bin/mapservwfs.cgi

It’s a standard MapServer setup, and at the moment contains static data about the sites we have worked on over the last thirty years. This is still a work in progress and there are a whole bunch of things I would like to improve (but at least it’s up and out there):

1: As I said, it’s static data. The aim is to get our main databases into PostgreSQL (I’ve talked about this before, and it’s not an easy process to convert messy, historic, access databases into PostgreSQL). In some circles there is a question as to whether we should actually use live data. There are sometimes issues with people looting archaeological excavations, and we don’t want to make that any easier…
2: You may have noticed that I said databaseS (plural rather than singular). Oxford Archaeology only took over the Lancaster Office a few years ago and we are still working towards merging our core systems (along with upgrading them all to sensible, robust OpenSource platforms where possible). The problem with having two databases is that in many cases the fields are not directly compatible, so to get the data out in the shortest possible time I simply included the elements that were common to both, and I will work towards getting more information out.

3: At the moment, there is no fancy front-end to this. I have two candidates in mind for frontends, and I’ve talked about them both a fair bit. They will serve two different purposes, in other words for internal and external use. Externally I’m working towards using OpenLayers, although this might mean that I have to convert or re-project all of our data that is in British National Grid format into WGS84 so I can use something like Google Maps as a backend. Not a problem, I just haven’t done it yet. OpenLayers will give me a fairly basic, but nice looking interface that works in a way that people are familiar with from Google Maps and other sites. It is easy to install and can be built into any web page, so it can be embedded in our corporate site and not look out of place. Internally I want to use MapGuide OpenSource, as it has advanced functionality and a fairly slick style built in (I could use Mapserver and build a front-end myself but this seems like the best approach). However, as my last few posts have explained, I am having quite a lot of trouble compiling this on our platform of choice, so we’ll see.

4: There is only one layer so far. We’ll work on this, but often have licensing issues with our data, so we’ll have to check that out first.

Anyhow, enjoy.

I have made a little more progress with evaluating the various free options for accessing PostgreSQL/Postgis database tables from ArcMap. I have to confess that some of the problem was down to my own lack of experience with Postgis!

The issue that I had with PGarc was that it would fail with an error if you had deleted tables from a database. It turns out that this is because deleting tables using the PostgreSQL “DROP TABLE” syntax does not remove it’s reference in the “Geometry_Columns” table. This must be removed as well- either separately or by deleting the whole shebang in one go using the Postgis DropGeometryTable function. Since PGarc uses “Geometry_Columns” to populate its list of available tables, it fails at this point. RTFM next time, archaeogeek!

There were a number of people in the google group with similar problems to me regarding ZigGis- namely that it would not display some layers. This turned out to be a problem with projections, or the SRID. If a table had an SRID of -1, in other words no projection set, then ZigGis could not display it. This bug has been resolved in the latest release.

My opinions of the two packages haven’t really changed. ZigGis has nice configuration files, and works directly with the Postgis table, whereas PGarc relies on DSNs, and creates a temporary shape file on your hard drive to work with. However, the clincher for me is that using PGarc you can select data and query it, whereas with ZigGis, I could not do that.

In the comments to my previous post, JeffÂ� kindly pointed out the trial version of Safe Software’s FME (Feature Manipulation Engine), which does include support for PostgreSQL databases. I haven’t had chance to try it out yet, and we are trying to work towards open-source solutions to our problems but I will evaluate it and post on my experiences.

This post is a discussion of my experiences with a couple of ArcGIS connectors for Postgis. To me, a reliable Postgis connector would be so useful, as we simply don’t have the resources (or the inclination) to purchase the equivalent proprietary products. We are also looking for a solution that will provide a flexible backend for all sorts of interfaces, such as MapGuide Open Source, UDig, and so on, alongside ArcGIS.

The two connectors for ArcGIS that I have tried so far are Ziggis and PGarc. Neither of these are what I would call production-ready yet, and have some problems but they are being actively developed so there is hope!

Ziggis:

I spent a rather frustrating day yesterday trying to install the new release of Ziggis and thought it was worth blogging the steps I had to take to get it installed as I found information on this to be confusing and sparse. Digging around in the ESRI discussion forums and such like seems to suggest that my problems will be relevant to the subset of people who don’t have things like Visual Studio installed, and for all I know will not be relevant in ArcGIS 9.1 or later. The short version is as follows:

1. Ziggis requires the Microsoft .net Framework and Arcobjects .net Assemblies to work,or the install will fail.
2. To get the Arcobjects .net Assemblies, you must have Microsoft .net version 1.1 installed before you install ArcGIS and you must choose the custom install of ArcGIS Desktop (should be disk one).
3. .net version 2.0 won’t work for ArcGIS 9.0 (don’t know about later versions).
4. If you already have ArcGIS installed before you install .net then you will need to do a full uninstallation of ArcGIS, reboot, install .net, and then install ArcGIS. Trying to cut corners and running the repair/reinstall/modify option won’t work.

Once I had it installed, I had an extra button in ArcMap for adding layers from a postgis table. Ziggis using configuration files (.zig files) for providing the information about connecting to the database. There is an example file in the program folder that can be modified using a standard text editor. You add new layers by navigating to the appropriate zig file for the database you wish to connect to, then type the table name that you want to add. This is a slightly frustrating step because you have to know which table you want, ie by firing up some other Postgis/PostgreSQL administrator first. A method of browsing for available tables would be a bonus.

Unfortunately, so far I haven’t managed to get anything to display on the map! The layer appears in the table of contents, and I can open the attribute table, but the geometry column contains the value “no geometry” despite being able to display using a different connector (PGarc). The next step is to try and enable the debugging and logging options in Ziggis to establish the problem.

PGarc:

PGarc appears to be in a state of flux at the moment. Some development is ongoing as there are discussions on the postgis mailing list about it, but the website version has not been updated in some time. A few months ago I was given some updated code in the form of an ArcGIS project, and this version at least works for me- with an important proviso (more later).

PGarc installs itself as an extra toolbar with two buttons, one to add new data from a database, and one to save data back into a database. To connect to a database you need to create a system dsn with the connection details and provide the name of that file, along with your username and password to PGarc. PGarc does list the available tables, from which you can pick one and add it to your map.

Saving data back into a database seems to work, although it only saves the visible extent (so you have to zoom out to the extent of a layer before saving it).

My biggest problem with it is that if I have deleted a table in my PostgreSQL database, then PGarc cannot connect to it because it is still looking for the deleted table. I don’t yet know if this is related to the dsn, PGArc, or PostgreSQL.

Comparision:

Both projects are at an early stage and as you can see I have had problems with both of them. PGarc seems to have the best feature set at the moment (you can select features, which is not possible, afaik, with Ziggis) and save data back into the database. However, the idea of text-based configuration files is more convenient than a system dsn, as they can be easily transferred between computers. I need to investigate the geometry issue with Ziggis more thoroughly, and will keep following the discussions, and then I should be in a better position to comment.

There is at least one other connector available, but I didn’t get further than downloading the instructions! Furthermore, ESRI have intimated that Postgis connectivity will be available in a later release of ArcMap. This should be a good thing, but there are concerns as to how it will be implemented. Watch this space…