Before 3D GIS we were able to measure miles below the sea and miles beyond our own atmosphere to a great level of accuracy, so how did we do it?
There are a few methods for working between vertical coordinate systems but the one I want to discuss is possibly the simplest, not always the MOST accurate but good enough for many of todays simple 3D applications. It is one I was taught by a master mariner and offshore surveyor that is still used widely in the offshore industry today, commonly called measurement by Tidal survey opposed to Geodetic datum used onshore.
The method works by calculating the differences between Mean Sea Level (MSL), Tide gauge or local vertical vertical coordinate system as they are all related and constantly measured for both consistency and accuracy.
What is a tide gauge?
If you have ever been to the beach and seen a big ruler attached to the wall, this is a rudimentary tidal gauge.
Rudimentary tide gauge found on the beach
Tidal gauges are used to measure the tides (obvs) and MSL. By observing the sea over long periods of time we can not only get an accurate measure of datum but also make predictions.
How does a tide gauge dictate datum?
For decades, the only way to express topographic or bathymetric elevation was to relate it to the seas’ level. Geodesists believed that the sea was in balance with the earth’s gravity. MSL is the mathematical mean of hourly water elevations observed over a described 19-year cycle, by measuring over this specific period the highs and lows caused by gravitational effects can be averaged.
If you are reading this, I am assuming you already know a little about datum and geoids, therefore I’ll just say that this sea level creates an equipotential surface (geoid) which is commonly used to define WGS84.
Local coordinate systems are commonly based on local tide gauges, take for example the Ordnance Surveys’ Newlyn vertical datum. This is based on tide gauge measurements taken at the [Newlyn] observation centre in Cornwall.
A seemingly ordinary lighthouse….until you spot the GPS antenna….this is the OS Newlyn tidal observatory.
To keep this more accurate for mariners, there are many tide gauges located all around the UK, as this makes for a more accurate vertical reference. You may have heard of the term “Chart Datum”, this refers to the datum (recorded at the Lowest Astronomical Tide (LAT) see image above) for the tide gauge which is referenced in the navigational chart.
The UK tide gauge network courtesy of BODC http://www.ntslf.org
Working between datum
With tide gauges all around the world, there is also a plethora of data. Places like NOAA & NOC provide accurate calculations to work between these gauges.
So, if you have your data in Newlyn, you can quite easily recalculate all your data to another datum. This can be done for most, if not all tidal datum and then converted to geodetic datum.
PSMSML is a great website for working out the differences, see this screenshot of the Liverpool data:
Screenshot of Liverpool references from the PSMSL website
NOTE
Please keep in mind that this method of working with vertical data will work as a BASIC method of working between vertical coordinate systems. Keep in mind what accuracy your data currently has and also the accuracy of the conversion method (note above image that you may need to convert between 1994 MSL and 2016 MSL).
3D GIS is game changing, it can change the way you view your analysis, it can provide insights which you may have overlooked….but there is a little problem which I may had not shared in my deluge of blogs about how great 3D GIS is…..
…Sometimes it can be hard work. What I mean by this, is that it can sometimes need a lot more consideration than standard methods. Let’s look at some of the major culprits when using Esri ArcGIS Pro 1.4 & Desktop 10.5, keeping in mind that there isn’t many (or any) other 3D GIS you can do this kind of work in.
One of the 3D models built of Kendal, UK
Editing
Once you have converted or built your 3D model into a multipatch polygon you may find yourself struggling to edit or adjust your model.
Split
Having built entire cities, one issue that I’ve come across a fair bit is removing parts of multipatch polygons. For example, when adding multiple models, you may find that two features might overlap and need to remove part of one. In a 2D, standard GIS, you would simply turn the editing on and split the offending polygon.
This doesn’t work in a 3D GIS…Think about it, how does the GIS know where all the planes you require breaking lie with a 3D plane? 3D GIS can do some pretty clever assumptions but I am yet to find a way to remove part of a complex multipatch polygon.
True, within ArcGIS Pro 1.4 there are some editing tools for 3D multipatch features but it is early days at the moment. For simple cubes it is quite easy to adjust and manipulate the model a little but you don’t stand a chance if you have a curved edge.
Don’t despair, don’t give up on the 3D just yet, as you know, 3D isn’t new and there are many “workarounds”. One of my favourites is using the ArcGIS Pro “Replace Multipatch” tool. If you want to make multiple edits to a model (feature) you can export the multipatch to a collada format or keyhole markup language (kml) format, edit it in Sketchup, Blender, Meshlab or any of your favourite modelling suites and then import it again without affecting the other features in that layer.
If you are extruding simple 2D polygons with the intention of creating 3D mulitpatch polygons, it is a good idea to keep the conversion to multipatch until you explicitly need to. This way, you can edit, split and reshape your 2D polygon within normal ArcGIS Desktop without any issues at all and then draw up and extrude when you are 100% sure it all fits and works okay.
In my experience, pre-planning and clarity of the end goal means that you can be prepared for these slight niggles in advance.
Using 3D multipatches in ArcGIS Desktop
Overlay analysis
So, I’ve built out an entire city of 3D buildings, it looks amazing, last thing to do is clip them by the city boundary….oh, I forgot, the 2D polygon doesn’t truly intersect the multipatch polygon in 3D space, the “clip” tool doesn’t work, the “intersect tool” nor the “merge” tool…in fact you can ignore using modelbuilder.
Clip features
I learned the hard way that 3D features work best with 3D features. Unless the boundary polygon is a 3D feature, then you won’t be able to use spatial analysis to use any kind of overlay querying.
But, there is always a way to trick the system…
Although the multipatch polygon feature is a 3D object, you can still open it in ArcGIS Desktop, meaning that you can use the good ol’ fashioned “select by location” tool. No, it shouldn’t work but it does, furthermore, you can then export your selected data to a new multipatch feature. It begs the question why you can do this and not the “clip” as, in geoprocessing terms, they are the same thing (the clip tool is just the separate bits combined into a single script).
Let’s not get too hung up on it, as it works,
Volumetric Analysis
If you thought that the “volume” tool was just put there to look clever, think again. It can be a great tool for easily representing floor space or calculating the tonnage of aggregate that needs extracting from the ground to lay a new pipeline.
There is a slight problem though…it can be a little hit and miss, especially when using sourced models.
Calculating Volume
Try adding a model you built in Sketchup, Blender or Meshlab to ArcGIS Pro and then calculate volume (using the add z information tool), Nothing, right? But why? The reason is that it doesn’t like “open” multipatch 3D features, it isn’t fully enclosed. Even if there is a slither of a gap in the polygon and it isn’t 100% enclosed, the tool cannot calculate the volume.
There are other methods whereby you can use a raster surface, like the “surface volume” tool but this isn’t quite as accurate as using your super detailed vector multipatch.
You could try the “Enclose Multipatch” tool, as this closes the multipatch and then running the volume tool BUT you need to consider that unless the multipatch is cut to the surface, for example, where a building sits on a hill and the base isn’t perfectly flat, the volume will not be ideal. So please consider using the data as a high resolution TIN which is merged with the terrain to provide a more accurate volume result.
Oh, last point on this – make sure you use a projected coordinate system that uses metric for your data, a geographic coordinate system will leave you with your volume in degrees….is that even possible?!
Which brings me nicely to –
Issues occur when you don’t specify the datum
Vertical referencing
I distinctly remember my first adventures into 3D through Google Earth, trying to create some of Romsey, UK as 3D buildings using Sketchup. The first hurdle was always figuring out whether the building was “Absolute height”, “On the Ground” or “Relative to the ground”…I mean, what does that mean anyway? I just drew it to sit on the floor, why does it need to ask me more questions about it?
Correct Height Placement
Right now, if you are a regular reader of xyHt or a 3D Geoninja, you will be calling me a muppet. In reality though, you wouldn’t believe how often I am asked about this, especially now that Digital Surface Models (DSMs), Digital Terrain Models (DTMs), bathymetry and other elevation data are so readily available.
Elevation is never easy, worst still, there are either too many or too few options. With the Esri ArcGIS Pro, I am 100% confident that I know where my data sits within 3D space but it is only because I’ve worked with this day in and day out and understand the limitations and data sources.
Let’s consider the Esri “scene” – it’s a cool 3D map and as you zoom into that lovely globe you can see lovely mountains and valleys all popping out of the surface, my question to you is, what elevation data is it using? what is the resolution of that data? You see, I love that Esri provide a detailed and complete coverage elevation surface for the entire globe but the flip side of it is that you cannot know the exact limitations of that surface easily (the information is provided by Esri but it is not a simple “point & click” exercise).
My words of advice here are to use your own terrain when placing 3D multipatch features. Therefore you are in control of both the vertical datum and the resolution of the height.
While I’m here, I want to also point out that there isn’t a “snap to ground” feature in the editing tools within ArcGIS Pro either. This becomes an issue when you bring a model in which isn’t vertically reference, has no vertical datum because you then need to sit it on the surface. Even when your model is a captured point cloud and accurate to 0.5cm, you have no way to accurately place it on the ground. You can adjust it up and down and sit it by sight, though you cannot “snap” it.
The big takeaway here is that firstly, you need to ensure you are confident and know your elevation data if you plan to work in the 3D scene views and secondly that you need to set up your x,y & z coordinate systems correctly from the start to ensure that all the work you do is as precise as possible.
…and yes, I now know the difference between “absolute”, “relative to the ground” & “on the ground”….maybe an interesting blog for another day, though feel free to contact me if you need quicker answers!
And everything else
There are still many things I have not had a chance to mention, for example the complexities of cartographic representation using 3D models in a GIS, or ways of minimising the clashing of overlapping data plus other 3D centric issues such as shadow and light. Maybe a blog for another day?….
Last week, April 4th – 6th, one of the largest gatherings for the offshore industry took place, Ocean Business, in Southampton, UK. The bi-annual event is a large deal for the maritime industry. When I say large, I actually mean huge; there are over 350 exhibitors from 26 countries and 166 training and demonstration workshops in just those three days!
What is amazing about Ocean Business is that the whole maritime industry is represented. Of course there are the surveyors, the geologists, and the equipment hire companies, but also there are representatives from the GIS, CAD and engineering sectors all sitting side by side.
At the last event talk was about the latest Reson multibeam device. Whereas before conversation would have been around capturing scour and wreck details, this year was talk of analysing pipelines for cracks or whether bolts were where they should be. It was hard at times to remember that I was at an offshore event as the technology has developed so quickly.
Of course, in two years, technology has shrunk. With the onshore industry, this is not overly exciting but offshore this is BIG news. Take, for example, the Planet Ocean micro AUV, developed with the National Oceanographic Centrre. This micro-UAV is only 50cm long and shaped like the traditional “torpedo”. Being fully autonomous allows it to capture data in ways never previously considered.
Big improvements to Doppler Velocity Log (DVL) technology was demonstrated by most of the equipment exhibitors like Sonardyne, Nortek and Kongsberg. If you don’t know what a DVL is, it how you get location when you are underwater and have no GPS/GNSS. The systems on show were capable of capturing the location at better than 0.01m at depths up to 6000m. Just think that over and relate that to your super cool iPhone!
The data providers and geospatial companies who were present were murmering about BIM and how it is going to start to become an issue. How does BIM integrate with the information captured and provided offshore? Can the maritime industry build a common spatial data infrastructure?
One company I met that were really making strides on this was OceanWise. Dr Mike Osborne, a respected hydrographer, has taken it upon himself to get the maritime industry thinking about these issues and working towards a solution. OceanWise provides nautical geospatial data but also trains the offshore and onshore industries in the use of GIS for nautical charting, including maritime standards. Mike has set up maritime spatial data infrastructure groups across the world with 33 countries already involved, something which is unheard of in the onshore industries. Mike has agreed to be interviewed by xyHt, so look out for that in a later issue!
My final comment is about how much Lego was on display. I mean, do these guys have nothing better to do with their time? As much as there was UAVs, transponders, ROVs and new vessels, there were Lego survey vessels and tech remodelled in Lego. Maybe it is because I wasn’t allowed to take one home that I am so curious about it.
You know those moments where you are sat in a pub, the office or at a friends house and you say something, then suddenly wonder why you haven’t thought of it before?….I had that very issue in the Brew Dog pub near Spitalfields in London, UK a couple of weeks ago. You see, I was discussing the use of web mapping in 3D and writing a specification for the data being hosted and then it came out….
“I wonder if 3D GIS systems use 3D transformations”…...
Before I even finished the sentence I was starting to get palpitations at the thought of throwing all my hard work over the last few months away. Okay, so maybe that is a bit of an exaggeration. I do take into consideration the vertical transformation when working between data and am a bit of a tide datum nerd, though the question was now out there….
3D GIS…a quick reminder
First, let me address the “3D GIS” elephant in the room. Is there such a thing? My current stance is yes. Having used many different GIS over the last few years, I would say that many would tell you that they are, but they would almost all be wrong. Take for example CesiumJS, it is a fantastic 3D webgl viewer but can you edit the data on screen? Can you transform data between coordinate systems? (The answer you are looking for is no) Other systems which I have discounted are QGIS, MapInfo, CADCorp, Google Earth, ArcScene, GRASS, OpenGIS and also AutoCAD Map¹, purely due to them not having the capability of showing multiple coordinate system 3D data on a globe. Yes, I understand that you can pre-process the data so that it is all in the same coordinate system and then process the datum so that it is all consistent, but NOT on the fly.
There is only one “3D GIS” at the moment, the much overlooked ESRI ArcGIS Pro.
Compared to the 3D capabilities of the other GIS, asking it to handle vertical transformations on the globe as well as the most up to date horizontal transformations would be a bit of an ask, but you know what?…..It is pretty good…..
Disclaimer time, I am currently running the Beta version of ArcGIS Pro 1.4, though I am told it shouldn’t matter, also I have been talking with a few of the developers on functionality but this is a 3D GIS which is out on the shelves now and works. For the OCD spatial nerd, it ticks the boxes, I can use the provided international transformations or I can create my own, though more importantly, I can work with my survey (lidar) data alongside my topographic data, which is alongside my hydrographic data- so I can easily QA the data and share the results quickly.
For the n00bs, why do you need vertical transformations?
Simply put, a horizontal datum is a reference system for specifying positions on the Earth’s surface, so that the coordinate system is of uniform measure, likewise, a vertical datum is a reference to the height of the surface. All simple at the moment….a datum is based on an ellipsoid/spheroid (these geo people can’t decide what shape round they want to call it, so interchange all the time) Each ellipsoid is the best approximation of the Earths surface at any point on the Earth, you may have seen “Clarke 1866, GRS80 or WGS84” in the coordinate information of your data
So you can start to see, just transforming your 3D data from one coordinate system to another isn’t quite so simple as the 2D Helmert you’ve been using. Relax, I’m not going to now go into a geoids and equipotential surfaces (unless you want me to) I am merely pointing out that we need vertical transformations in out lives, otherwise, when you transform your beautiful 3D village model from WGS84 to British National Grid, you will be wondering why it is floating in the air.
This might not matter much to the casual 3D GIS dabbler, as most of the work may be small and the effects won’t be large enough to be noticeable, though when it gets to city size, you will certainly start seeing issues with the buildings at the periphery not sitting right (when you edit the centre to sit right).
In summary
Since joining Garsdale Design in January, I have been doing a LOT of 3D GIS and putting my geodetic knowledge to ensuring things are correct. In the last 6 months I have been having to use ArcGIS Pro more and more, partly due to functionality and partly due to data formats which clients use. My opinion is that this software is becoming the one stop shop for 3D GIS and it is capable of supporting the whole project lifecycle of the project….it could turn out to be the BIM solution that everyone has overlooked. More importantly though, is that ESRI have done their homework and are supporting vertical datum transformations….okay, there are a few missing here and there BUT the list is growing and they are putting these vertical transformations into the geoprocessing tools of ArcGIS Desktop too.
Screenshot of (a small portion of) the vertical transformation list
If you know of another 3D GIS, please let me know, as I’d be eager to try it. Please feel free to either contact me direct or through comment below.
So, the situation occurred yesterday where I was giving a workshop and sent out a load of QGIS styles, layer definition files and also a project file (.qgs)….Smugly, I told everyone to open the project file, then realised, as hands raised across the room that QGIS doesn’t work with relative paths and it also doesn’t do a “map package”. Working with so many different GIS, it’s hard to keep track of which ones do different things but I really should have remembered this one.
Surprisingly, the solution to repairing all the links and getting it all up and running is relatively easy if you are working with disconnected databases or vector files (shapefiles etc). Just make sure you have a text editor and away you go….
First of all, open the rogue .qgs file in your text editor, in the example above, I am using sublime text editor but during the workshop I found Windows notepad was just as capable. Upon opening, you will see that the project file is just a standard xml file with references to several processes.
Use your “Find” option in the text editor to find one of the <datasource> tags (as shown above)
It is simply a case of then changing the folders within that datasource tag to locate the correct location (most people store their data in a single location).
As you can see above, I want the project to read all the data from C:OS Southampton rather than the G:Work_Admin_Backup_Nov15GIS Core DataOS Southampton location, so using the REPLACE function (sometimes called the find/replace in some text editors) we can simply change ALL the locations in one go.
Pretty easy huh? A lot easier that using the interface which is provided by QGIS for updating each file link individually, after all, most times we just change folders, we don’t scatter our data around a drive location.
I am sure that this sort of functionality (changing the folder to reference all the links) could be done in bash or as an extra option within QGIS, if you know how, I look forward to hearing from you!
How often do you get someone ask “can I have a map”?….I have nothing against making maps, in fact I like it so much that I have made a career out of it. Not just any map either, I’ve made maps for the Nuclear, Environmental, Government, Renewable Energy, Offshore and Survey industries, to name a few, I’ve even made maps for a Prime Minister (You’re welcome Mr Blair).
My issue is the question whereby you are just asked for a map. This is not only time consuming for you, the creator but also for the poor client who you are charging 4 or 5 times over for updates to a map which should have been right the first time. This isn’t the clients’ fault, the client didn’t realise that there are multiple templates for the work or that positioning the label they wanted would take half an hour due to it causing a script in the GIS to crash. No, it is safe to assume that the client doesn’t know anything about GIS otherwise they would be doing it themselves….OR (eeek!) you could have the ultimate in intimidation that is the GIS Manager – someone who used GIS up until 10yrs ago and has forgotten how long things REALLy take *shudder*.
There are 2 was to charge people for making maps, either a fixed rate or the consultants favourite, by the hour. No matter which one you choose, it will be wrong. Charge by the hour and you spend 6hrs on 5 different versions of the map which looked fine the first time, OR you take the hit but as you make all those changes, that time you had earmarked has gone and you have to apologise to other clients who were never part of the issue. Lose, lose….
Manage Expectations
The most difficult time I had with this issue was working for local government here in the UK. One day I would be asked for a “PPL map”, the next I would be asked to map all the doctors surgeries near to elderly communities, then the next I would be asked for a map of otter holts. Every map was diverse and each required a different format, logo, information, size, paper density, north arrow…..you get the idea, no matter what, each map was unique and with little experience with making public facing maps I was taking my work home with me just to get all the work done near to deadline.
So, I created the “GIS Request Form” and I have used this in many different guises at every workplace I’ve worked and never looked back. Simply put, it is a list of basic requirements for making a map, this has sometimes be a list of 20 items but at present is a list of 7 or 8 which form the creation. If I know that the client is managing onshore assets, it is safe to assume they will require British National Grid, once things go coastal it is a completely different set of goalposts.
The basics I will always ask for are
Project Name
Time Allocated for the job
Presentation Size (is it for a report and to be inset or A3 folded?!)
Orientation of the map
Allow for ring-binding?
Description of the work (useful for context)
Other information which I ask of new clients may be:
The project level of accuracy
The stated project coordinate system
Preferred data provision format
Is this based on a previous map? If so, can a sample be porvided?
Is the map for internal or external use?
By having all this information at hand, you can manage the expectation a lot better and therefore saving A LOT of time. For one client, I can remember being told to run off about 25 maps for a client and I did it in the company standard A3 landscape, only for the Director to come back and tell me that the client had changed their mind and the figures had to be put into the text of the report……how do you tell a client that you are going to charge them for another 30-40hrs of work?! At least if you have it in print from the start you have a safety net!
Below is an example of the current (quick) request form…..yes, it is in the form of an email. Why? Because I ask my clients to have the message on their desktops, then if they need a map, all they have to do is fill in the details and click “send”.
First Map Perfection
Okay, so I still have to make re-iterations of the map but now I usually only make one…yes ONE. It is a personal goal of mine to ensure that a client only has to have one iteration of the map. In fact, I always tell the client that the first map is a draft and won’t be perfect as no-one has EVER made a perfect map first time (disclaimer: maybe they have, but I doubt it).
I hope this helps to smooth the workflow, feel free to comment and add your stories of client map expectations or when it has got to a crazy number of iterations!!
