Thursday Tip Day: Importing UK Mastermap data into postgres or shape files
Moving to open source for our GIS packages continues to throw up some interesting challenges- and solutions. In the UK, the most common way for Ordnance Survey vector data to be delivered is in mastermap format, which without going into the technical details is gml, usually served in gz files. Lots of them. There is an additonal add-on for ArcGIS that converts this into a personal geodatabase, but initially there didn’t seem to be a way of doing this using open source.
Along comes the mastermap importer. This does exactly what it says on the box- imports mastermap data into ogr formats such as shapefiles, or into a postgresql database. It’s free for personal or non-commercial use and is a really great programme. I haven’t begun to explore any of it’s more esoteric options, but for simply converting 78 gzipped mastermap files into shapefiles it can’t be beat, even by the commercial importer add-on for ArcGIS.
The only problem I have is that 1 gzipped mastermap file translates to 1 folder containing 8 shapefiles, one for each component of the mastermap data (annotation, topographic area, point, line, etc). So if you have 78 gzipped files then you have 78*8 shapefiles to deal with. I would like to see some way of merging the matching shapefiles together automatically if possible. The only option I have so far tried is geomerge, which works well, but you still have to go through the folders picking the shapefiles you want in the first place. Surely there must be some way of iterating through the folders finding all the shapefiles with the same initial part of their name ( area shapefiles all start with MM_TopographicArea_, for example) and merging them together?
Note that I haven’t tried the postgresql options yet. If they allow you to append all the data of each type into one table, then we’re sorted as I can import them back out into shapefiles if I need to.
The next thing that I have to figure out is how to apply standard Ordnance Survey symbology to the result. That’s a tricky one…
Merging Multiple Shapefiles - For Free.
If have ArcGIS 9.2/9.3
Open ArcMap and ArcCatalog side by side if you can
ArcCatalog>Search on ‘MM_TopographicArea_*.*’
Then select [Shift select]
(This can be done in Windows Explorer as well as ArcCatalog can be slow on many 000’s of files)
drag into ArcMap and they will be order in annotation,point,polyine and polygon. Then use the merge function in the Toolbox to create one file for each data type.
Or use the Geowizards (9.7) [recommended]
http://www.ian-ko.com/
batch merge
Mapperz
http://mapperz.blogspot.com/
Hi Mapperz,
Thanks for the heads up- a useful tip IF one is still using ArcGIS! As we are moving towards open source we are looking for alternatives to the things you can do with ESRI products.
Thanks for popping by,
Jo
Jo,
You can use ogr2ogr small Shell or Python script that iterates through folders and shapefiles, recognizes similar name prefixes, and calles something like this for every matched shapefiles:
1. Frist, initialize merged shapefile:
ogr2ogr abc_merged.shp subdir1\ abc.shp
2. Append other files with the same name prefix (abc in this example):
ogr2ogr -update -append abc_merged.shp subdir2\abc.shp -nln abc_merged
and so on for subrid3, subdir4, …
Jo,
If you want to do anything with some group of files then find’s exec option is (usually) your friend.
eg to find all the files named ‘MM_TopographicArea_*’ in the current directory (and its subdirectories) and append it to foo.shp you’d do something like:
find . -name MM_TopographicArea_* -exec ogr2ogr -update -append foo.shp {} -nln foo \;
the command in the -exec option is executed for each file matching the -name parameter. The -exec option replaces any instances of {} with the matched filename and is terminated by \;
Hope that helps,
Andrew
@Andrew
…or xargs command:
find . -name ABC | xargs ogr2ogr …
xargs is usually faster than -exec, what may be important while working with huge sets of files.
@Mateusz
Yes - agreed, though I suspect the bottleneck in this case will be reading in the data, building spatial indexes rather than in the pausing and spawning of new processes.
Cheers,
Andrew
@Andrew
Yes, I think you are right.
Hi All,
Thanks for the suggestions- they are fab! (I sense a little batch file coming up…)
Jo
Jo,
Funny, I should known what’s in the GDAL FAQ but seems I didn’t
Anyway, we have answer related to your question in there:
How can I merge hundreds of shapefiles?
[...] last week I wrote about a mastermap importer that I had found. I said I would like to be able to merge the [...]