ArcGIS workflows for Big Data
As many of you may know, recently I have been trying to work (unsuccessfully) with ArcGIS and offshore data (Bathymetry, benthic, marine mammals etc) and mostly this data comes under the Patrick Pickles “If its bigger than a spreadsheet, it is big data”.
All this has changed, thanks to a chat I had with an ESRI rep at Offshore Business 2013. There I was complaining to this guy at an Offshore exhibition about how unsuitable ArcGIS was for the offshore work I was doing and he points out that he was an ESRI sales rep…….Oops! What came after was great. Although he admitted that ArcGIS is unsuitable for working with big offshore data (like bathymetry) in vector format, he had some advice which has since saved me a lot of time and I haven’t seen written down anywhere.
- Rasterize – Although ArcGIS will struggle with larger raster data, it is much more efficient with raster data than vector data.
- LAS – If you have ArcGIS 10.1, you have the new LAS Toolbox (I didn’t know it was there either!). Using a point cloud is far more efficient and means you can handle terrabytes of data.
- Try working with the ArcGIS Terrain format.
So, how do I recommend working with the data? Obviously, this is my opinion and should there be any better method, I would like to hear it, but this is what I have been doing and it has saved me hours. Below I mention use of LAS Tools, which can be downloaded here.
- Singular large (1GB+) xyz, txt, asc, csv file format data – Convert to LAS (Lidar format & Use LAS Tools), not only does this store a smaller file but can be converted to many other formats. This is much more efficient and quicker within ArcGIS and ArcScene.
- Multiple (1GB+) xyz, txt, asc, csv file format data – Convert each data to LAS format, then load them into a singular LAS Dataset (.lasd). This, I have found, is the most efficient way of viewing or analysing the data. For display of this data as a surface, I have found the best way is to merge the LAS data (note NOT LAS dataset) into one huge LAS file which can be converted to raster. Another method I have tried is to create a TIN of the largest LAS data and then “edit TIN” to add the rest of the LAS data in ESRI shp format, though this latter method is pretty hit & miss.
One thing to note, although ArcGIS 10.1 works with LAS data, it doesn’t work with LAS data…to be able to use LAS data within ArcGIS and ArcCatalog, you need to create a “LAS Dataset” (Described here) within ArcCatalog, you can then load your LAS data into in either as a single data or as multiples.
Although it is early days, it is great to see ESRI starting to think about using big data and how it can be manipulated, a few months ago I would have told you to steer clear of the software for this kind of work but now I am starting to become friends with it again.