Computing, GIS and Archaeology in the UK

Well, at last it’s OK for me to tell people that Oxford Archaeology now has a WFS server that is accessible from the outside world. The address is:

http://mapdata.thehumanjourney.net/cgi-bin/mapservwfs.cgi

It’s a standard MapServer setup, and at the moment contains static data about the sites we have worked on over the last thirty years. This is still a work in progress and there are a whole bunch of things I would like to improve (but at least it’s up and out there):

1: As I said, it’s static data. The aim is to get our main databases into PostgreSQL (I’ve talked about this before, and it’s not an easy process to convert messy, historic, access databases into PostgreSQL). In some circles there is a question as to whether we should actually use live data. There are sometimes issues with people looting archaeological excavations, and we don’t want to make that any easier…
2: You may have noticed that I said databaseS (plural rather than singular). Oxford Archaeology only took over the Lancaster Office a few years ago and we are still working towards merging our core systems (along with upgrading them all to sensible, robust OpenSource platforms where possible). The problem with having two databases is that in many cases the fields are not directly compatible, so to get the data out in the shortest possible time I simply included the elements that were common to both, and I will work towards getting more information out.

3: At the moment, there is no fancy front-end to this. I have two candidates in mind for frontends, and I’ve talked about them both a fair bit. They will serve two different purposes, in other words for internal and external use. Externally I’m working towards using OpenLayers, although this might mean that I have to convert or re-project all of our data that is in British National Grid format into WGS84 so I can use something like Google Maps as a backend. Not a problem, I just haven’t done it yet. OpenLayers will give me a fairly basic, but nice looking interface that works in a way that people are familiar with from Google Maps and other sites. It is easy to install and can be built into any web page, so it can be embedded in our corporate site and not look out of place. Internally I want to use MapGuide OpenSource, as it has advanced functionality and a fairly slick style built in (I could use Mapserver and build a front-end myself but this seems like the best approach). However, as my last few posts have explained, I am having quite a lot of trouble compiling this on our platform of choice, so we’ll see.

4: There is only one layer so far. We’ll work on this, but often have licensing issues with our data, so we’ll have to check that out first.

Anyhow, enjoy.

I have made a little more progress with evaluating the various free options for accessing PostgreSQL/Postgis database tables from ArcMap. I have to confess that some of the problem was down to my own lack of experience with Postgis!

The issue that I had with PGarc was that it would fail with an error if you had deleted tables from a database. It turns out that this is because deleting tables using the PostgreSQL “DROP TABLE” syntax does not remove it’s reference in the “Geometry_Columns” table. This must be removed as well- either separately or by deleting the whole shebang in one go using the Postgis DropGeometryTable function. Since PGarc uses “Geometry_Columns” to populate its list of available tables, it fails at this point. RTFM next time, archaeogeek!

There were a number of people in the google group with similar problems to me regarding ZigGis- namely that it would not display some layers. This turned out to be a problem with projections, or the SRID. If a table had an SRID of -1, in other words no projection set, then ZigGis could not display it. This bug has been resolved in the latest release.

My opinions of the two packages haven’t really changed. ZigGis has nice configuration files, and works directly with the Postgis table, whereas PGarc relies on DSNs, and creates a temporary shape file on your hard drive to work with. However, the clincher for me is that using PGarc you can select data and query it, whereas with ZigGis, I could not do that.

In the comments to my previous post, JeffÂ� kindly pointed out the trial version of Safe Software’s FME (Feature Manipulation Engine), which does include support for PostgreSQL databases. I haven’t had chance to try it out yet, and we are trying to work towards open-source solutions to our problems but I will evaluate it and post on my experiences.

This post is a discussion of my experiences with a couple of ArcGIS connectors for Postgis. To me, a reliable Postgis connector would be so useful, as we simply don’t have the resources (or the inclination) to purchase the equivalent proprietary products. We are also looking for a solution that will provide a flexible backend for all sorts of interfaces, such as MapGuide Open Source, UDig, and so on, alongside ArcGIS.

The two connectors for ArcGIS that I have tried so far are Ziggis and PGarc. Neither of these are what I would call production-ready yet, and have some problems but they are being actively developed so there is hope!

Ziggis:

I spent a rather frustrating day yesterday trying to install the new release of Ziggis and thought it was worth blogging the steps I had to take to get it installed as I found information on this to be confusing and sparse. Digging around in the ESRI discussion forums and such like seems to suggest that my problems will be relevant to the subset of people who don’t have things like Visual Studio installed, and for all I know will not be relevant in ArcGIS 9.1 or later. The short version is as follows:

1. Ziggis requires the Microsoft .net Framework and Arcobjects .net Assemblies to work,or the install will fail.
2. To get the Arcobjects .net Assemblies, you must have Microsoft .net version 1.1 installed before you install ArcGIS and you must choose the custom install of ArcGIS Desktop (should be disk one).
3. .net version 2.0 won’t work for ArcGIS 9.0 (don’t know about later versions).
4. If you already have ArcGIS installed before you install .net then you will need to do a full uninstallation of ArcGIS, reboot, install .net, and then install ArcGIS. Trying to cut corners and running the repair/reinstall/modify option won’t work.

Once I had it installed, I had an extra button in ArcMap for adding layers from a postgis table. Ziggis using configuration files (.zig files) for providing the information about connecting to the database. There is an example file in the program folder that can be modified using a standard text editor. You add new layers by navigating to the appropriate zig file for the database you wish to connect to, then type the table name that you want to add. This is a slightly frustrating step because you have to know which table you want, ie by firing up some other Postgis/PostgreSQL administrator first. A method of browsing for available tables would be a bonus.

Unfortunately, so far I haven’t managed to get anything to display on the map! The layer appears in the table of contents, and I can open the attribute table, but the geometry column contains the value “no geometry” despite being able to display using a different connector (PGarc). The next step is to try and enable the debugging and logging options in Ziggis to establish the problem.

PGarc:

PGarc appears to be in a state of flux at the moment. Some development is ongoing as there are discussions on the postgis mailing list about it, but the website version has not been updated in some time. A few months ago I was given some updated code in the form of an ArcGIS project, and this version at least works for me- with an important proviso (more later).

PGarc installs itself as an extra toolbar with two buttons, one to add new data from a database, and one to save data back into a database. To connect to a database you need to create a system dsn with the connection details and provide the name of that file, along with your username and password to PGarc. PGarc does list the available tables, from which you can pick one and add it to your map.

Saving data back into a database seems to work, although it only saves the visible extent (so you have to zoom out to the extent of a layer before saving it).

My biggest problem with it is that if I have deleted a table in my PostgreSQL database, then PGarc cannot connect to it because it is still looking for the deleted table. I don’t yet know if this is related to the dsn, PGArc, or PostgreSQL.

Comparision:

Both projects are at an early stage and as you can see I have had problems with both of them. PGarc seems to have the best feature set at the moment (you can select features, which is not possible, afaik, with Ziggis) and save data back into the database. However, the idea of text-based configuration files is more convenient than a system dsn, as they can be easily transferred between computers. I need to investigate the geometry issue with Ziggis more thoroughly, and will keep following the discussions, and then I should be in a better position to comment.

There is at least one other connector available, but I didn’t get further than downloading the instructions! Furthermore, ESRI have intimated that Postgis connectivity will be available in a later release of ArcMap. This should be a good thing, but there are concerns as to how it will be implemented. Watch this space…

In my spare time/lunch times I’m in the middle of a major project at the moment, to update our site database. Without going into the gory details of how it ended up in three separate, totally unlinked databases, it is supposed to document the archaeological sites we’ve worked on since the 1970’s, and to help with the administration of project archives, the location of finds within our finds store and so on.

For a long time I’ve wanted to sort this out, pull everything into one place, display the site locations on a web-based GIS and so on. This is not rocket science, but the data had ended up in such a bad state that I couldn’t see past fixing that before getting to the good and fun stuff. Attempts a few years ago to use Mapserver as a mapping interface were abandoned because the positional data for the sites was wildly inaccurate (we seemed to do a lot of work in the Scilly Isles, which are as close to 0, 0 on the British National Grid as you can get), and connecting to Microsoft Access was quite difficult and unstable. If anyone using a Windows XP machine tried to open the database, my map would not display. None of those problems are Mapserver’s fault of course, but you can’t roll out a map that has inaccurate data on it, and might not alwats work.

So, after a few years of muttering about this, I have finally bitten the bullet. I am working, in stages, towards an integrated database for all of our finds and archive information, in a PostgreSQL database, with both an Open Office base front-end for querying, and a web-based map interface. I’ll blog about each stage in this process, starting with integrating and data-cleansing in Microsoft Access.

The title of this post refers to the incredible feeling of achievement that I had when I got all this data together, followed by the difficulty I’m having moving that data into PostgreSQL. More later…