I’ve been out at a couple of Association for Geographic Information (AGI) events over the last couple of weeks- organised by their Northern Group. Their main function is to organised events in the North of England (hence the name), but the outgoing chairman Rollo, has been really pushing for events with a national attendance and relevance. I spoke briefly at both events, and my talks can be found on slideshare and on my talks page here.

The first event, a couple of weeks ago now, was Where2Now- a lively scamper through some leading edge ideas, mainly about geographic location for the masses rather than technical GIS per se (yes, I’m trying to avoid using the word NeoGeography, but that’s what I’m talking about). There were speakers from Google, Yahoo, Microsoft, OpenStreetMap, Geovation, Ordnance Survey- in fact if you attended the Geoweb stream at the main AGI conference you’ll have a good idea who spoke! I did a short talk on the impact of “open” (access/source/data) as a disruptive technology (as far as I’m concerned this neo/mashup/open era wouldn’t be here without it), and attempted to demo a couple of deeply cool new toys that we’re working with at the moment- mapchat and gvsigmobile, both of which I think have the potential to be incredibly useful and really big. Unfortunately my laptop had stage fright and refused to speak to the monitor, so I couldn’t do a live demo of gvsigmobile as I wanted to.

Perhaps my favourite talk of the day was John McKerrell, talking about mapme.at, with his geo-clock (if you’ve read Harry Potter, remember the Weasley’s clock, with hands for each member of the family pointing to where they are at any given moment). Luckily John hasn’t found a need for a “mortal peril” setting yet! Mapme.at is great- it’s one of those ideas which can only work with the ubiquitous nature of geolocation these days, in phones, and with all the geo-location apis that you can use. Basically it’s a way of mapping where you are, and of plotting your location history, using feeds from twitter, email, google latitude, fire eagle etc.  Someone asked what it could be used for, but I think that’s missing the point somewhat- John has provided the basic idea,  and it’s up to the user to figure out what to do with it!

Again, the #geocom twitter stream going on in the background provided an interesting counter-point to the talks, although it’s increasingly worrying as a speaker not knowing if there’s a discussion about how rubbish you are going on while you’re giving the talk!

The second event I attended last week was a World GIS Day event at the Grammar School in Leeds. This followed on from the incredibly successful closing presentation at the main AGI conference, in which kids from the school, and a couple of their teachers discussed the way in which GIS is used throughout the whole school. The event last week was a chance for professional GIS users to talk a little about the way in which they use GIS, and also to see in more detail how GIS is used in the school. We also attended part of a sixth-form geography lesson, which was really interesting (not the least for the looks of abject trauma on all the attendee’s faces at sitting in a class room again after many years).

With my “open” hat on, I was quite uneasy about the way in which ESRI is synonymous with GIS in that environment, but to be fair it’s because they have worked extremely hard to provide the material for the teachers, which isn’t yet available anywhere else. My other concern was that teaching GIS seemed to be more about teaching which buttons to press to get a particular result, rather than teaching the theory and asking the kids how to figure it out *in that particular software package*. My “open source” side is frustrated that they are producing a generation of kids who will think ESRI is the only GIS to use, and when they are in  a position to influence the use of GIS themselves, within organisations,  or other schools, that’s the route they will choose. However, where in a school curriculum is the chance to give kids a choice, and how can open source provide them with that? Things to think about…

Just as we are settling down with the transition to open source, it would seem that a lot of other people are at least considering their options along the same lines, for whatever reason. This thread on the osgeo_discuss list, started off as a question about the value of open source to individual’s careers, but rapidly morphed into a discussion about replacing the ESRI packages. This seems to have lead to a general consensus about the areas where the open source packages do well and do badly.

The consensus seems to be that the open source packages are great for the behind-the-scenes and/or web-based work, assuming a relatively technically literate user, but that cartographic capabilities are sadly lacking at the moment. Furthermore, if you’re looking for one single replacement, then you’re out of luck- but you can assemble a toolkit that will get you all the capabilities you need, in a modular fashion. This does require a change in mindset- but it’s a lot more flexible and hopefully more powerful.

I hope that this recent upswell of interest and consensus about where the open source packages need improvement will help focus development in those areas. This does seem to be happening: this page is about plans for a cartographic library that can be shared amongst different packages. Sign up to show your support!

PS- if you’re in the UK you can still come along to our first UK OSGeo meetup at Stansted tomorrow (Thursday 1st) at 4.30. It’s free to attend, there will be some talks to start with (better finish writing mine), then we’ll sit down and try and figure out what we want a UK local chapter to do. There will be tea, coffee and biscuits too, so what are you waiting for!

It would appear that ESRI (UK) were as surprised as we were when they received notification of ESRI (US)’s decision to tighten up the license agreements, and they are negotiating on behalf of all of us educational charities/associate sites to get us a little more leeway. It has to be said that the email we received from them, which I based my blog post on, contained nothing about this whatsoever, and basically told us we had till the end of March to pack our metaphorical bags, but we are grateful to them for their efforts.

However, the point still stands. By using closed source software, we are all subject to the whims of software vendors, who can change the license agreement to stop people using their products whenever they feel like it.  So it has never happened to you? I’d have said the same thing before 4pm yesterday…

UPDATE: We’ve had some more feedback about this from ESRI UK- see my other post for details (though don’t get too excited because nothing *really* changes)

So, our move to open source gets a boost today, from an unexpected quarter. In what can only be described as a noble act of self-sacrifice, ESRI have told us that as an educational charity we are no longer allowed to have an educational discount for using their software and, not only that, our license codes will cease to work at the end of this month. So, we have 3 weeks, with the Easter holidays in the middle, to extricate ourselves and our ongoing projects from ArcGIS and into something else or find the many thousands of pounds to buy the full licenses for all of our staff.

I should explain something about the nature of “commercial” archaeological units in the UK, to those that don’t know much about them. We are usually not affiliated with universities, so we are not educational establishments as such, although we are an educational charity. We exist to fulfil a legal remit to study the archaeology of an area before it is developed. We also undertake educational projects on behalf of English Heritage, the government’s body in charge of cultural heritage. The whole reason we exist is to further the archaeological and historical understanding about the country we live in.

Previously, my biggest gripe has been that we can’t get academic discounts for data, unlike our colleagues in universities or local government. Despite the fact that our job is a legal requirement, we have to pay through the nose for the mapping that we need to do it. Getting hold of geological data, or anything that might allow us to look deeper into the area we’re studying is often out of the question. But at least we had the software to work with.

I can’t figure out what ESRI hope to achieve by this. It clearly affects many other educational charities apart from ourselves, across many different industries. Do they think that we all have secret pots of cash hidden under our desks and that we’re just going to throw up our hands and go “fair cop guv, we’ll pay the full cost now, here’s 7,000 pounds per user”? Do they seriously think that 3 weeks is enough time to get the money or rebuild all of our work in other packages?

Well ESRI, in case it’s not clear, we’re not going to buy your full versions. It’s unlikely that we will ever use your software again, and you’ve made it much easier for us to openly campaign for open source solutions throughout our own industry and other related sectors, and anywhere else where people are concerned about getting screwed over by software vendors.

So, we have to part company. It was nice knowing you for a while, but you’ve changed and I just don’t think we’re compatible any more. Now, where’s QGIS…

Firstly, if you’ve seen this post before, apologies. It got lost in a previous re-organisation of the blog and I thought it was worth re-posting…

You can get free contour data for the UK from the Scottish Mountaineering Club website. It’s based on public domain NASA data, cleaned up by the club and made freely available. It comes in garmin .img format, in several zip files, which roughly represent areas of the country. See their web page for an explanation of the areas.

Edit: Leszek has some great alternatives methods and options for data- check the comments.

1. To convert the img files to something (more) useful, the best programme I’ve found is gpsmapedit. You need to register it to get the full functionality. This costs 42 euros (umm, about 32 pounds at the moment) but allows you to use the programme on up to four computers.
2. In gpsmapedit, click “open” or “add” and load up all the img files you want to convert. Note that they will be merged into one big file so for the sake of your poor computer, choose only the areas that you need.
3. Go to file/export/mapinfo file, select level “0″, this includes the minor contours (down to 10m) as well as the major contours (100m)
4. When it has finished exporting, you can shut it down. If you’re using mapinfo, then skip to step 6.
5. In ArcGIS, crack open ArcCatalog, and load the “Arcview 8.x tools” toolbar. Move your mapinfo files to a location on your computer that has no spaces in it’s path name and a short path. No really, do. It will crash otherwise. Choose the “mif to shapefile” converter. Alternatively, you could do this with ogr2ogr. See the website for usage.
6. The resulting file (mapinfo or shape) is in wgs84 projection, with the height values in feet, so for use in a British National Grid environment you need to re-project it. The height values are in the attribute field “label”, so add another field for holding the converted height and use the tool of your choice to convert the values in the “label” column to metres.

Note that this data is good for broad-based terrain modelling only. If you need something more accurate you’re going to have to buy it…

If you have any alternatives to this method let me know in the comments!

There’s an article on the All Points Blog that goes into more detail about the potential support for PostgreSQL/PosGIS in ESRI products. The caveats to this are listed below, and I have to ask- what’s the point guys?

It will only be in Enterprise ArcSDE- so you’ll need ArcGIS Server Enterprise Basic as a bare minimum, and of the windows products it will only work on Server 2000/2003. So- if you have loads of money to spend on your GIS you’ll be able to use PostGIS- but not if you don’t have thousands to spend.

It will contain it’s only spatial implementation alongside the PostGIS “spatial type”. So that will keep things clear and easy to follow then.

We had this debate ages ago, but it seems as if ESRI really haven’t listened. I guess it doesn’t matter that much to low-end users as there are alternatives but it seems like they are missing the boat. However apparently they are really interested in getting feedback at their non-existent forum so that’s OK…

I have just stumbled upon the ESRI Mapping Center and, while I had been following their rss feed for a while, I didn’t know about the wealth of useful information that they have been putting together for creating attractive maps. A lot of the tips can be applied to other software packages too- not just ESRI.

I particularly like the article on historical maps, or how to create an authentic looking treasure map. The section on “Ask a Cartographer” also looks promising, although there aren’t many questions up there yet. ESRI are also reissuing Eduard Imhof’s book Cartographic Relief Presentation, which is a bit of a classic in the field of cartography, so watch that space carefully.

BTW, in a low-key way I’m celebrating because this is the 50th post on archaeogeek. For those that know me and my short attention span, this is quite an achievement!

I have made a little more progress with evaluating the various free options for accessing PostgreSQL/Postgis database tables from ArcMap. I have to confess that some of the problem was down to my own lack of experience with Postgis!

The issue that I had with PGarc was that it would fail with an error if you had deleted tables from a database. It turns out that this is because deleting tables using the PostgreSQL “DROP TABLE” syntax does not remove it’s reference in the “Geometry_Columns” table. This must be removed as well- either separately or by deleting the whole shebang in one go using the Postgis DropGeometryTable function. Since PGarc uses “Geometry_Columns” to populate its list of available tables, it fails at this point. RTFM next time, archaeogeek!

There were a number of people in the google group with similar problems to me regarding ZigGis- namely that it would not display some layers. This turned out to be a problem with projections, or the SRID. If a table had an SRID of -1, in other words no projection set, then ZigGis could not display it. This bug has been resolved in the latest release.

My opinions of the two packages haven’t really changed. ZigGis has nice configuration files, and works directly with the Postgis table, whereas PGarc relies on DSNs, and creates a temporary shape file on your hard drive to work with. However, the clincher for me is that using PGarc you can select data and query it, whereas with ZigGis, I could not do that.

In the comments to my previous post, Jeff kindly pointed out the trial version of Safe Software’s FME (Feature Manipulation Engine), which does include support for PostgreSQL databases. I haven’t had chance to try it out yet, and we are trying to work towards open-source solutions to our problems but I will evaluate it and post on my experiences.

This post is a discussion of my experiences with a couple of ArcGIS connectors for Postgis. To me, a reliable Postgis connector would be so useful, as we simply don’t have the resources (or the inclination) to purchase the equivalent proprietary products. We are also looking for a solution that will provide a flexible backend for all sorts of interfaces, such as MapGuide Open Source, UDig, and so on, alongside ArcGIS.

The two connectors for ArcGIS that I have tried so far are Ziggis and PGarc. Neither of these are what I would call production-ready yet, and have some problems but they are being actively developed so there is hope!

Ziggis:

I spent a rather frustrating day yesterday trying to install the new release of Ziggis and thought it was worth blogging the steps I had to take to get it installed as I found information on this to be confusing and sparse. Digging around in the ESRI discussion forums and such like seems to suggest that my problems will be relevant to the subset of people who don’t have things like Visual Studio installed, and for all I know will not be relevant in ArcGIS 9.1 or later. The short version is as follows:

1. Ziggis requires the Microsoft .net Framework and Arcobjects .net Assemblies to work,or the install will fail.
2. To get the Arcobjects .net Assemblies, you must have Microsoft .net version 1.1 installed before you install ArcGIS and you must choose the custom install of ArcGIS Desktop (should be disk one).
3. .net version 2.0 won’t work for ArcGIS 9.0 (don’t know about later versions).
4. If you already have ArcGIS installed before you install .net then you will need to do a full uninstallation of ArcGIS, reboot, install .net, and then install ArcGIS. Trying to cut corners and running the repair/reinstall/modify option won’t work.

Once I had it installed, I had an extra button in ArcMap for adding layers from a postgis table. Ziggis using configuration files (.zig files) for providing the information about connecting to the database. There is an example file in the program folder that can be modified using a standard text editor. You add new layers by navigating to the appropriate zig file for the database you wish to connect to, then type the table name that you want to add. This is a slightly frustrating step because you have to know which table you want, ie by firing up some other Postgis/PostgreSQL administrator first. A method of browsing for available tables would be a bonus.

Unfortunately, so far I haven’t managed to get anything to display on the map! The layer appears in the table of contents, and I can open the attribute table, but the geometry column contains the value “no geometry” despite being able to display using a different connector (PGarc). The next step is to try and enable the debugging and logging options in Ziggis to establish the problem.

PGarc:

PGarc appears to be in a state of flux at the moment. Some development is ongoing as there are discussions on the postgis mailing list about it, but the website version has not been updated in some time. A few months ago I was given some updated code in the form of an ArcGIS project, and this version at least works for me- with an important proviso (more later).

PGarc installs itself as an extra toolbar with two buttons, one to add new data from a database, and one to save data back into a database. To connect to a database you need to create a system dsn with the connection details and provide the name of that file, along with your username and password to PGarc. PGarc does list the available tables, from which you can pick one and add it to your map.

Saving data back into a database seems to work, although it only saves the visible extent (so you have to zoom out to the extent of a layer before saving it).

My biggest problem with it is that if I have deleted a table in my PostgreSQL database, then PGarc cannot connect to it because it is still looking for the deleted table. I don’t yet know if this is related to the dsn, PGArc, or PostgreSQL.

Comparision:

Both projects are at an early stage and as you can see I have had problems with both of them. PGarc seems to have the best feature set at the moment (you can select features, which is not possible, afaik, with Ziggis) and save data back into the database. However, the idea of text-based configuration files is more convenient than a system dsn, as they can be easily transferred between computers. I need to investigate the geometry issue with Ziggis more thoroughly, and will keep following the discussions, and then I should be in a better position to comment.

There is at least one other connector available, but I didn’t get further than downloading the instructions! Furthermore, ESRI have intimated that Postgis connectivity will be available in a later release of ArcMap. This should be a good thing, but there are concerns as to how it will be implemented. Watch this space…

James Fee’s Spatially Adjusted Blog has a mighty useful post on it today- an updated list of his feeds, including GIS Blogs, ESRI-specific Blogs and other GIS Sites. Most of these are aggregated in Planet Geospatial (probably the most useful GIS Link you’ll ever need) but some are not. Best of all, he’s included the opml files for easy rss integration.