A quick note on some really useful things I’ve picked up this week. They might only be new to me, but I thought I’d jot them down for the sake of future google searches…

Crosstab Queries: These are those clever queries that take a set of records, aggregate them up, and transpose the rows into columns. In Microsoft Access there is a wizard for doing this, but in PostgreSQL you have to do it the hard way.  Fortunately, it turns out it’s not *that* hard! The official documentation is here, and there’s a very good tutorial here, showing you how to circumvent some of the restrictions on the final table, such as adding a total column and row, or adding additional columns.  All I’d like to add to the tutorial is that you must be careful about how you create your basic aggregate query (returning the records that you wish to transpose), because it’s easy to return a crosstab that doesn’t group your results as you’d like them to. Check it carefully!

QGIS Table Manager Plugin: I don’t know how I managed to miss this one, but having found it, I’ve used it intensively this week. Something that’s not entirely clear when using QGIS is that many of the attribute table management options, such as adding, renaming, or deleting columns are only available when using PostGIS tables. This is frustrating, because the options are present when working with other file types, but are not active. Enter the table manager plugin by Borys Jurgiel, which you can install from the third-party repositories. It’s still quite basic, in that it only works on shapefiles (I think)- but it allows you to re-order and rename columns as well as adding and deleting them. For those people that fall back to the perilous route of editing the dbf in a spreadsheet package, it’s a life-saver (What, you’ve accidentally sorted your dbf in a different order to your shp? Oh dear, that’s your attributes attached to the wrong features then).

Open Office Presentation Minimizer: Not quite GIS, though tested on a GIS-related presentation that I’m writing! This takes all the pain out of optimising images in your presentations to keep the file-size down whilst also maintaining image quality. You install it by downloading the oxt and adding it using the Open Office Extensions Manager,  unless you are using Ubuntu where this will appear to install correctly but remain inactive regardless of how many times you open and close Open Office Impress. For Ubuntu, find the package in the standard repositories and install using apt-get or your installer of choice. Do note the american spelling of “minimiz(s)er”, for those of us in the UK. I assume this will also work in LibreOffice as well as Open Office.

Part of my fabulous new plans for portable gis (which will be revealed soon-ish) involve including a database replication option. This might or might not work on the USB drive, but it needs to be open source, portable, and connect to postgresql at the very least. I have been looking at a couple of options for this, without a vast amount of success, it has to be said.

The packages that I have tried are: Daffodil Replicator, dbreplicator (a fork of daffodil replicator), Symmetric-DS, and db -connector for Funambol. Of these, daffodil and dbreplicator seem like the best bet as they will attempt to do some conflict resolution (eg when the same record is added/altered in both databases). They are also java-based, so should be nicely configurable. Symmetric-DS works well, but doesn’t do conflict resolution, so that kind of rules it out for me. I don’t really know about db-connector for funambol as the documentation that I found was quite out of date and I didn’t get very far with it.

So, daffodil replicator and dbreplicator…

They are quite easy to set up, although the documentation in both cases assumes rather more end-user knowledge than perhaps they should do. Basically you find an appropriate jdbc jar file for your database, grab log4j.jar from somewhere, and tell daffodilreplicator/dbreplicator where to find it, by setting appropriate paths in batch files. Then you start the publication server and set up the details for your “master” database, and then do the same for the subscription server and the “client” database. Both are clever enough to translate between different database types, such as postgresql and mysql, which in some cases would be really handy.

My difficulty with both flavours of the package came when creating the “subscription”, ie telling the client database where to go find the master database. In neither case have I been able to sucessfully set this up, because of various errors. Weirdly, both flavours of the package give me an error in my log file about the first ever subscription that I tried to set up, despite having reinstalled, deleted everything I could find, and started from scratch several times.

The documentation in both cases hasn’t been good enough to resolve my issues, and neither have the forums. Actually, I’m still waiting on dbreplicator, which seems to be a little more active, so I should give them the benefit of the doubt for a couple of days.

I can see a great need for this kind of setup, assuming I can get around these initial teething troubles. So, my question is, has anyone successfully set up any of these packages on windows (sorry, but it needs to be windows at this stage), or does anyone know of any other packages I should try?

If I do manage to get things sorted I’ll post a detailed how-to…

PostgreSQL from 8.2 onwards can be run easily from the command line in windows, without setting up as a service.

Go to your postgresql/bin folder and at a command line type:

pg_ctl start -D location\of\your\data\folder (as specified in initdb) -l logfile

This should output a notice telling you whether the server has started up correctly. It also saves output to a logfile in the bin folder. If the server starts without incident, open another command window at the same location and type:

psql -d yourdatabase

This will allow you to use PostgreSQL as a command line tool.

To stop the server, type (in your second command window):

pg_ctl stop -D location\of\your\data\folder

A post from Daily Cup of Tech a while ago inspired me to add a “tumblog” to my site, for short-format posts on tips and tricks. I will be posting to this regularly to build up a library of code snippets, linux and windows tips and archaeological information.

It’s taking me a while to get it properly integrated into the site, but the feed is working and can be subscribed to separately from the main site rss.

My main incentive for doing this was to make it easier to post little snippets of useful information in a form that is useful to me and hopefully to others. Until now, I had been writing these down in a moleskine that normally resides on my desk at work, but of course when I’m at home that information is pretty useless!

The next plan is to set a shell script on my zaurus (yes I do still use it despite my move to analogue GTD) that downloads a copy of it for offline reference- inspired by a lifehacker article about using wget to download copies of del.icio.us and google notebook for offline use.

The daily cup of tech article is the best to read for instructions on how to do it yourself- the only tip I can add is that if you are on a web host that only allows one database, it is possible to edit your wordpress config file to change the table prefix for the new blog and include multiple blogs in the same database. This does of course lay you open to risks if you break something, so it might be better to pay the extra money for a new database (thus speaks the voice of experience…).

This is an idea that seems to come up every so often- what GIS programmes can you run from a USB stick. Well it appears that the list has just got longer. I’m probably the last person to realise it’s possible to do this, but I was really pleased to see that both GRASS and PostgreSQL can now be run from a USB stick, along with QGIS, XAMPP (inc Mapserver, OpenLayers, Tilecache), and FWTools.

Before I go any further, I should state that my aims for creating a portable GIS are not so much having a production GIS setup on a stick, because obviously performance and storage are an issue, but it might be useful for demonstration purposes, and I do feel as if the take up of these opensource tools might be more if they were easier to install and came in one handy package. So what I really want to achieve is a single portable package, probably a zip-file, that can be downloaded and unzipped onto a Window pc (they can learn about the joys of linux once we’ve won them over to the basic concept of opensource). No separate downloads, long installation and configuration processes, just a zip file. I do also like having portable packages on my main computer, as if it breaks it’s a lot easier to reconstruct the setup!

The light-bulb moment for me was when Leif told me that he had seen GRASS working on a USB stick using Msys at the recent CAA conference in Berlin. This intrigued me, so I looked into it and it works quite nicely- as long as you use the same drive letter to mount your USB stick to all the time. I still have to figure out how to figure out the drive letter and change the paths, probably in a batch file. Put simply, you download Mingw and Msys onto the stick, follow the instructions for downloading GRASS, edit the install_grass.bat file to change the drive letters, and off it goes.

PostgreSQL was something I had looked into a while ago, been told it wasn’t possible, and given up on. However, the recent 8.2 release appears to have dropped the requirement for a dedicated non-administrative user to run it. If you download the windows binaries, without the installer, you can set it up whilst logged on as the standard user. When you next run PostgreSQL from a different machine, you just have to use the same username as on your installation pc, regardless of whether that user exists on the pc that you are currently using. Once you have PostgreSQL installed, you can install Postgis into the same directory.

Currently there are a couple of gotchas. Most of these programmes require you to set environment variables, which might not be allowed on the particular pc that you are using, and require some command line experience. I still haven’t quite figured out how to get GRASS to work if I change the drive letter. Also, PostgreSQL does indeed run very slowly on a USB stick. And, I haven’t fully explored this to make sure there isn’t a hidden trap somewhere, or security holes that might be exploited. Having said that, if you want a package that can be unzipped onto a pc with a minimum of fuss then I think it’s almost there.

What’s next? Documentation, as always, and the permission of the various programmers that it’s OK to do this. Oh, and comments please, on what else we might include, or words of wisdom if you’ve already tried it.

Well, at last it’s OK for me to tell people that Oxford Archaeology now has a WFS server that is accessible from the outside world. The address is:

http://mapdata.thehumanjourney.net/cgi-bin/mapservwfs.cgi

It’s a standard MapServer setup, and at the moment contains static data about the sites we have worked on over the last thirty years. This is still a work in progress and there are a whole bunch of things I would like to improve (but at least it’s up and out there):

1: As I said, it’s static data. The aim is to get our main databases into PostgreSQL (I’ve talked about this before, and it’s not an easy process to convert messy, historic, access databases into PostgreSQL). In some circles there is a question as to whether we should actually use live data. There are sometimes issues with people looting archaeological excavations, and we don’t want to make that any easier…
2: You may have noticed that I said databaseS (plural rather than singular). Oxford Archaeology only took over the Lancaster Office a few years ago and we are still working towards merging our core systems (along with upgrading them all to sensible, robust OpenSource platforms where possible). The problem with having two databases is that in many cases the fields are not directly compatible, so to get the data out in the shortest possible time I simply included the elements that were common to both, and I will work towards getting more information out.

3: At the moment, there is no fancy front-end to this. I have two candidates in mind for frontends, and I’ve talked about them both a fair bit. They will serve two different purposes, in other words for internal and external use. Externally I’m working towards using OpenLayers, although this might mean that I have to convert or re-project all of our data that is in British National Grid format into WGS84 so I can use something like Google Maps as a backend. Not a problem, I just haven’t done it yet. OpenLayers will give me a fairly basic, but nice looking interface that works in a way that people are familiar with from Google Maps and other sites. It is easy to install and can be built into any web page, so it can be embedded in our corporate site and not look out of place. Internally I want to use MapGuide OpenSource, as it has advanced functionality and a fairly slick style built in (I could use Mapserver and build a front-end myself but this seems like the best approach). However, as my last few posts have explained, I am having quite a lot of trouble compiling this on our platform of choice, so we’ll see.

4: There is only one layer so far. We’ll work on this, but often have licensing issues with our data, so we’ll have to check that out first.

Anyhow, enjoy.

I have made a little more progress with evaluating the various free options for accessing PostgreSQL/Postgis database tables from ArcMap. I have to confess that some of the problem was down to my own lack of experience with Postgis!

The issue that I had with PGarc was that it would fail with an error if you had deleted tables from a database. It turns out that this is because deleting tables using the PostgreSQL “DROP TABLE” syntax does not remove it’s reference in the “Geometry_Columns” table. This must be removed as well- either separately or by deleting the whole shebang in one go using the Postgis DropGeometryTable function. Since PGarc uses “Geometry_Columns” to populate its list of available tables, it fails at this point. RTFM next time, archaeogeek!

There were a number of people in the google group with similar problems to me regarding ZigGis- namely that it would not display some layers. This turned out to be a problem with projections, or the SRID. If a table had an SRID of -1, in other words no projection set, then ZigGis could not display it. This bug has been resolved in the latest release.

My opinions of the two packages haven’t really changed. ZigGis has nice configuration files, and works directly with the Postgis table, whereas PGarc relies on DSNs, and creates a temporary shape file on your hard drive to work with. However, the clincher for me is that using PGarc you can select data and query it, whereas with ZigGis, I could not do that.

In the comments to my previous post, Jeff kindly pointed out the trial version of Safe Software’s FME (Feature Manipulation Engine), which does include support for PostgreSQL databases. I haven’t had chance to try it out yet, and we are trying to work towards open-source solutions to our problems but I will evaluate it and post on my experiences.

This post is a discussion of my experiences with a couple of ArcGIS connectors for Postgis. To me, a reliable Postgis connector would be so useful, as we simply don’t have the resources (or the inclination) to purchase the equivalent proprietary products. We are also looking for a solution that will provide a flexible backend for all sorts of interfaces, such as MapGuide Open Source, UDig, and so on, alongside ArcGIS.

The two connectors for ArcGIS that I have tried so far are Ziggis and PGarc. Neither of these are what I would call production-ready yet, and have some problems but they are being actively developed so there is hope!

Ziggis:

I spent a rather frustrating day yesterday trying to install the new release of Ziggis and thought it was worth blogging the steps I had to take to get it installed as I found information on this to be confusing and sparse. Digging around in the ESRI discussion forums and such like seems to suggest that my problems will be relevant to the subset of people who don’t have things like Visual Studio installed, and for all I know will not be relevant in ArcGIS 9.1 or later. The short version is as follows:

1. Ziggis requires the Microsoft .net Framework and Arcobjects .net Assemblies to work,or the install will fail.
2. To get the Arcobjects .net Assemblies, you must have Microsoft .net version 1.1 installed before you install ArcGIS and you must choose the custom install of ArcGIS Desktop (should be disk one).
3. .net version 2.0 won’t work for ArcGIS 9.0 (don’t know about later versions).
4. If you already have ArcGIS installed before you install .net then you will need to do a full uninstallation of ArcGIS, reboot, install .net, and then install ArcGIS. Trying to cut corners and running the repair/reinstall/modify option won’t work.

Once I had it installed, I had an extra button in ArcMap for adding layers from a postgis table. Ziggis using configuration files (.zig files) for providing the information about connecting to the database. There is an example file in the program folder that can be modified using a standard text editor. You add new layers by navigating to the appropriate zig file for the database you wish to connect to, then type the table name that you want to add. This is a slightly frustrating step because you have to know which table you want, ie by firing up some other Postgis/PostgreSQL administrator first. A method of browsing for available tables would be a bonus.

Unfortunately, so far I haven’t managed to get anything to display on the map! The layer appears in the table of contents, and I can open the attribute table, but the geometry column contains the value “no geometry” despite being able to display using a different connector (PGarc). The next step is to try and enable the debugging and logging options in Ziggis to establish the problem.

PGarc:

PGarc appears to be in a state of flux at the moment. Some development is ongoing as there are discussions on the postgis mailing list about it, but the website version has not been updated in some time. A few months ago I was given some updated code in the form of an ArcGIS project, and this version at least works for me- with an important proviso (more later).

PGarc installs itself as an extra toolbar with two buttons, one to add new data from a database, and one to save data back into a database. To connect to a database you need to create a system dsn with the connection details and provide the name of that file, along with your username and password to PGarc. PGarc does list the available tables, from which you can pick one and add it to your map.

Saving data back into a database seems to work, although it only saves the visible extent (so you have to zoom out to the extent of a layer before saving it).

My biggest problem with it is that if I have deleted a table in my PostgreSQL database, then PGarc cannot connect to it because it is still looking for the deleted table. I don’t yet know if this is related to the dsn, PGArc, or PostgreSQL.

Comparision:

Both projects are at an early stage and as you can see I have had problems with both of them. PGarc seems to have the best feature set at the moment (you can select features, which is not possible, afaik, with Ziggis) and save data back into the database. However, the idea of text-based configuration files is more convenient than a system dsn, as they can be easily transferred between computers. I need to investigate the geometry issue with Ziggis more thoroughly, and will keep following the discussions, and then I should be in a better position to comment.

There is at least one other connector available, but I didn’t get further than downloading the instructions! Furthermore, ESRI have intimated that Postgis connectivity will be available in a later release of ArcMap. This should be a good thing, but there are concerns as to how it will be implemented. Watch this space…

In my spare time/lunch times I’m in the middle of a major project at the moment, to update our site database. Without going into the gory details of how it ended up in three separate, totally unlinked databases, it is supposed to document the archaeological sites we’ve worked on since the 1970′s, and to help with the administration of project archives, the location of finds within our finds store and so on.

For a long time I’ve wanted to sort this out, pull everything into one place, display the site locations on a web-based GIS and so on. This is not rocket science, but the data had ended up in such a bad state that I couldn’t see past fixing that before getting to the good and fun stuff. Attempts a few years ago to use Mapserver as a mapping interface were abandoned because the positional data for the sites was wildly inaccurate (we seemed to do a lot of work in the Scilly Isles, which are as close to 0, 0 on the British National Grid as you can get), and connecting to Microsoft Access was quite difficult and unstable. If anyone using a Windows XP machine tried to open the database, my map would not display. None of those problems are Mapserver’s fault of course, but you can’t roll out a map that has inaccurate data on it, and might not alwats work.

So, after a few years of muttering about this, I have finally bitten the bullet. I am working, in stages, towards an integrated database for all of our finds and archive information, in a PostgreSQL database, with both an Open Office base front-end for querying, and a web-based map interface. I’ll blog about each stage in this process, starting with integrating and data-cleansing in Microsoft Access.

The title of this post refers to the incredible feeling of achievement that I had when I got all this data together, followed by the difficulty I’m having moving that data into PostgreSQL. More later…