ArcGIS workflows for Big Data

ArcGIS workflows for Big Data

As many of you may know, recently I have been trying to work (unsuccessfully) with ArcGIS and offshore data (Bathymetry, benthic, marine mammals etc) and mostly this data comes under the Patrick Pickles “If its bigger than a spreadsheet, it is big data”.

All this has changed, thanks to a chat I had with an ESRI rep at Offshore Business 2013. There I was complaining to this guy at an Offshore exhibition about how unsuitable ArcGIS was for the offshore work I was doing and he points out that he was an ESRI sales rep…….Oops! What came after was great. Although he admitted that ArcGIS is unsuitable for working with big offshore data (like bathymetry) in vector format, he had some advice which has since saved me a lot of time and I haven’t seen written down anywhere.

  • Rasterize – Although ArcGIS will struggle with larger raster data, it is much more efficient with raster data than vector data.
  • LAS – If you have ArcGIS 10.1, you have the new LAS Toolbox (I didn’t know it was there either!). Using a point cloud is far more efficient and means you can handle terrabytes of data.
  • Try working with the ArcGIS Terrain format.

So, how do I recommend working with the data? Obviously, this is my opinion and should there be any better method, I would like to hear it, but this is what I have been doing and it has saved me hours. Below I mention use of LAS Tools, which can be downloaded here.

  • Singular large (1GB+) xyz, txt, asc, csv file format data – Convert to LAS (Lidar format & Use LAS Tools), not only does this store a smaller file but can be converted to many other formats. This is much more efficient and quicker within ArcGIS and ArcScene.
  • Multiple (1GB+) xyz, txt, asc, csv file format data – Convert each data to LAS format, then load them into a singular LAS Dataset (.lasd). This, I have found, is the most efficient way of viewing or analysing the data. For display of this data as a surface, I have found the best way is to merge the LAS data (note NOT LAS dataset) into one huge LAS file which can be converted to raster. Another method I have tried is to create a TIN of the largest LAS data and then “edit TIN” to add the rest of the LAS data in ESRI shp format, though this latter method is pretty hit & miss.

Airborne lidar data in ArcMap

One thing to note, although ArcGIS 10.1 works with LAS data, it doesn’t work with LAS data…to be able to use LAS data within ArcGIS and ArcCatalog, you need to create a “LAS Dataset” (Described here) within ArcCatalog, you can then load your LAS data into in either as a single data or as multiples.

Although it is early days, it is great to see ESRI starting to think about using big data and how it can be manipulated, a few months ago I would have told you to steer clear of the software for this kind of work but now I am starting to become friends with it again.

Nick D

Ordnance Survey Linked Data Portal

Ordnance Survey Open Linked Open Data Portal

Courtesy of Ordnance Survey  26th April 2013

Ordnance Survey is Great Britain’s national mapping agency, providing the most accurate and up-to-date geographic data, relied on by government, business and individuals.

OS OpenData is the opening up of Ordnance Survey data as part of the drive to increase innovation and support the “Making Public Data Public” initiative. As part of this initiative Ordnance Survey has published a number of its products as Linked Data. Linked Data is a growing part of the Web where data is published on the Web and then linked to other published data in much the same way that web pages are interlinked using hypertext.

The term Linked Data is used to describe a method of exposing, sharing, and connecting data via URIs on the Web. To find more Linked Data published as part of this initiative please go to data.gov.uk.

If you are not familiar with Linked Data, OS OpenData products are also available in alternative formats from the OS OpenData website. Ordnance Survey can provide support for the Ordnance Survey OpenData products, but cannot give advice or support on using RDF, SPARQL or SPARQL Endpoints.

Ordnance Survey has published three OS Open Data products as Linked Data: the 1:50 000 Scale Gazetteer, Code-Point Open and the administrative geography for Great Britain taken from Boundary Line. A combined OS Linked Data dataset combines these products into one database to support more flexible data access.

Each of the datasets is accessible as Linked Data and via a range of APIs.

Having spent many hours reading through all the UK OpenGov data, this site is a god-send, lets give it our full support and help progress this!

Nick D

GIS Tips – Styling Ordnance Survey Open Data

Image

How many of us have sat there for hours adjusting the RGB values of the OS Vectormap District or wasted a day setting the line weights of Boundary Line?….Only me? Thought as much….

Well, the frustration is over, it appears that Ordnance Survey now supply style files so that the Ordnance Survey style can be applied, using your GIS software of choice, to the OS OpenData vector products, they are available here:

Link to OS Open Data styles are here: SLD Format

Okay, no biggy, did you know that Ordnance Survey have also released the sea tiles (up to the UK Median) as well?

Link to OS Sea Tiles are here: GIS Format

All in all, Ordnance Survey have turned the tide and are now providing some quality data which, for the casual user or basic GIS technician, is superb, especially for a free open licence product! I must admit, 5yrs ago I never thought I would be using a FREE vector UK national coverage data.

OS Open Data can be downloaded here: Ordnance Survey Open Data download site

Nick D

ArcGIS Critical Information regarding Microsoft hotfix

from ESRI April 25th 2013

Critical information regarding Microsoft hotfix rollup and ArcGIS

If you have applied Microsoft hotfix roll-up KB 2775511 and created file geodatabases on a shared network location from a Windows 7 SP1 or a Windows 2008 R2 SP1 machine, your file geodatabases are unusable. You will see an error message referencing the item type when you create new datasets or items in a file geodatabase.

For example, the following message will appear when creating a new feature class:

“Failed to create feature class.
The item was not found. [ItemType:
{70737809-852C-4A03-9E22-2CECEA5B9BFA}]”

Cause
This issue is still under investigation.

Workaround
Esri is working on a patch to mitigate this issue. There are currently several workarounds.

See Esri Knowledge Base article 41119 for further information and updates

Old Maps Online

Old Maps Online

I may have said this one or a hundred times already but I am a bit of a cartophile and love the tactile feel and the amazing detail in old maps, in my search I found a great website called “Old Maps Online” – I know, who would have guessed!!

For those of you who love an old map, this is a great resource….best part? A lot of the maps are downloadable!!!

Old Maps Online website

Nick D

GIS Tips – Be more efficient with QGIS too!

Yesterday I wrote “GIS Tips – Be more efficient without trying” and realised my mistake when I woke this morning, I wrote “GIS” and spoke entirely about ArcGIS. Truth is, you cannot work in the field of GIS with just one software, until recently I was mostly working with consultancies and agencies who use ArcGIS. Recently though, I have found more requests coming from some industry leaders for QGIS templates or QGIS format files….is this the turn of the tide? It will be interesting to see when QGIS 2.0 comes out on June 7th 2013.

How does this all relate to the title? Well, if you weren’t aware, QGIS has shortcuts too, in fact it has a plethora of shortcuts and where shortcuts don’t exist, you can define your own!

Here is a list of the QGIS shortcuts: webpage

What is really useful is that both ArcGIS and QGIS share a lot of the same shortcuts, for example – undo last edit, copy feature, paste feature, zoom in, zoom out, new project and save all use the same shortcuts. The really great thing about QGIS is that if you go to SETTINGS, you can select CONFIGURE KEYBOARD SHORTCUTS to then use other keyboard shortcuts you may be using within ArcGIS.

So, what’s stopping you? configure those shortcuts like “refresh” and “zoom to layer” and you will make enough time to set up that GIS web page you said you’d get around to…

Nick D