GIS Tips – ArcGIS, how big is big data?


Don’t know about you, but all this talk of “Big data” is scaring me. Why? Because I am using ArcGIS 10.1 on a local machine, don’t get me wrong, ArcGIS 10.1 is great and it fits many hats, along with QGIS and some other add-ins, there is not much you can’t do…well, almost….big data is a BIG problem.

I am relatively new to the offshore industry, until 5yrs ago I worked for Ordnance Survey and Councils, nice easy data, the biggest data might be a Lidar survey and this was a struggle but not too painful. Now that I am working on offshore sites and dealing with survey data I realise that ArcGIS is not very good at vector data processing.

Recently my client asked me to take some 1m spaced xyz (csv) data and find any gaps in the data and no matter how I cut up the data or reduced the data, I couldn’t even get ArcGIS to draw the data without falling over. The data itself was only 3GB and would draw up in QGIS absolutely fine, only QGIS doesn’t have a 3D engine so was unable to provide the end result I was after.

Anyway, I gave up hope and went along to an exhibition yesterday with the intention of looking at alternative software and went over to the QPS stand to look at Fledermaus and to have a little whinge about ArcGIS…Only to find out the guy I was opening my heart to was from…ESRI!! And after hearing me almost in tears and obviously feeling my pain, he had some great advice which I share with you here:

  • ArcGIS is no good for vector data processing of big data, this is a given, it is limited by RAM and still stuck in the realms of 32bits. So USE RASTER when you can!
  • Use the new LAS Tools!! Yup, I didn’t believe this either but converting the 3D data to LAS format means that my data no longer falls over and works in ArcSCENE!!
  • Consider Fledermaus for surface/terrain data processing – Yes, straight from ESRI! Fledermaus is designed for working with raw topo survey data and now integrates with ArcGIS,

Having taken all this on board the ESRI rep then went on to show me some 40cm spaced bathymetry covering the WHOLE WORLD, zooming in and out without a single jitter. I have since tested some of my survey bathymetry and using the LAS data toolset in ArcGIS 10.1 seems to be doing the trick.

To convert data to LAS format, I recommend trying these (FREE) tools LASTools

To read more on ESRIs partnership with QPS (makers of QINsy and Fledermaus) read here

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.