So, the situation occurred yesterday where I was giving a workshop and sent out a load of QGIS styles, layer definition files and also a project file (.qgs)….Smugly, I told everyone to open the project file, then realised, as hands raised across the room that QGIS doesn’t work with relative paths and it also doesn’t do a “map package”. Working with so many different GIS, it’s hard to keep track of which ones do different things but I really should have remembered this one.
Surprisingly, the solution to repairing all the links and getting it all up and running is relatively easy if you are working with disconnected databases or vector files (shapefiles etc). Just make sure you have a text editor and away you go….
First of all, open the rogue .qgs file in your text editor, in the example above, I am using sublime text editor but during the workshop I found Windows notepad was just as capable. Upon opening, you will see that the project file is just a standard xml file with references to several processes.
Use your “Find” option in the text editor to find one of the <datasource> tags (as shown above)
It is simply a case of then changing the folders within that datasource tag to locate the correct location (most people store their data in a single location).
As you can see above, I want the project to read all the data from C:OS Southampton rather than the G:Work_Admin_Backup_Nov15GIS Core DataOS Southampton location, so using the REPLACE function (sometimes called the find/replace in some text editors) we can simply change ALL the locations in one go.
Pretty easy huh? A lot easier that using the interface which is provided by QGIS for updating each file link individually, after all, most times we just change folders, we don’t scatter our data around a drive location.
I am sure that this sort of functionality (changing the folder to reference all the links) could be done in bash or as an extra option within QGIS, if you know how, I look forward to hearing from you!
As you may be aware, the United Kingdom has a new transformation model that is OSTN15…..But why? What does it mean to the geospatial community?
Without being too nerdy, tectonic plate movement means that the “model” surface (the geoid) is slowly moving from best fit for the coordinate system. It has been 13yrs since Ordnance Survey implemented OSTN02 so the shift since then is enormous…..a whole 1cm and vertically it is 2.5cm. See this article here from Ordnance Survey.
The whole story is that sensors and our ability to calculate our positi0n relative to both the mathematical models and our relative position to those is constantly evolving too. So, just as OSTN02 revolutionised the accuracy of projecting GPS (WGS84) coordinates using a grid transformation (250 points over the 7 parameters used until 2002), OSTN15 both uses the OS Net of 250 points but has also been improved further with 12 zero order stations with accuracy of 2mm horizontal and 6mm vertical.
So how will this change the way you use your GIS?
If you are already using OSTN02 for your transformations between EPSG 27700 and EPSG 4326 – then you will only see a 5cm improvement over a 1m area at best and this is based on the worst places in the UK, on average you will only see a 2cm improvement anywhere in the UK. To put this into context, when you are zoomed in to an A3 map to about 1:100, you are talking about a few pixels on the screen….it won’t be groundbreaking [at the moment].
Currently, as this goes to press, the OSTN15 transformation has only been available for a few weeks and it is still being tested on different software to ensure it works, I am told that ESRI UK have been testing it with their software as this is being written.
As with OSTN02, I’ve created a fix for QGIS and OSTN15, I will describe how to implement this further in this.
There are many ways to use proj, without a GIS you can use it through a command line by defining parameters. QGIS uses the proj library by accessing a spatialite database called srs.db. This is held at .appsqgisresourcessrs.db in Windows and Linux.
The proj spatialite database is a relational database which, when analysed, holds tables for coordinate systems, epsg codes & transformations. What is really clever is that it recognises direction of transformation.
Why is direction important?
Most coordinate transformations go from the projected coordinate system to the geographic coordinate system, for example epsg 4277 to epsg 4326, OSTN15 bucks the trend and is the reverse direction, from 4326 to 4277.
As I found when I first tested OSTN15 with QGIS, I was getting a uniform 200m shift in the data which was being translated and I was really confused. After talking with the gridfile creator, I discovered that the file was created from ETRS89 to OSGB36, therefore the 200m shift I was getting.
QGIS is awesome, you’ve probably overlooked just how clever it is and so did I. Next time you run a transformation, or when you try this one, you may notice that there are 2 fields noted in the columns SRC (source) and DST (destination)…and this is a godsend for solving this issue, as QGIS can read the coordinate in both directions.
Show us the magic
So, I talked with Ordnance Survey and found that OSTN15 has been given the epsg of 7709 and created a new record with the srs.db which is distributed with Windows, Linux & Mac releases. To utilise this, all you need to do is to download the OSTN15 file from Ordnance Survey (here) and then place the OSTN15_NTv2.gsb file in the shared projections folder .shareprojOSTN15_NTv2.gsb this has been found to be correct in Mac and Windows (there should be similar in Linux). You know it is the right folder as there should be other .gsb files in there!
You can download the updated srs.db from here, this should be placed in the resources folder which can be found at .appsqgisresourcessrs.db – I highly recommend changing the name of the srs.db file in this folder to something like srs.db.old before adding the new version, just in case it doesn’t work for your particular set up BUT it has been checked on Mac and Windows distributions of QGIS from version 2.12 through to QGIS 2.17.
Last month I was at the Maptime in Southampton (UK), helping QGIS new users how to join tables and map EU referendum maps when I came across an issue with something on QGIS I hadn’t spotted in the last *ahem* years of using it.
When you drag and drop txt, csv or other delimited files into QGIS the fields automatically get converted to text format. No, I’m not making it up and it caused a lot of embarrassment when I was giving my demonstration.
This isn’t written to complain about QGIS but to notify others who are wondering why their joins aren’t working or why their interpolation can’t pick up the value field….You QGIS guys are going to say “why haven’t I raised this as an issue?”, well, firstly read Nyall Dawsons blog post on QGIS issues , secondly I tried to…..it turns out that trying to get access to submit issues has changed and even though I’ve asked for help to get access I’ve been waiting 1 month for response to request.
So…why does it happen?
If you add the file through the “add delimited file” button, none of this is an issue, this is due to the was that the software is written When the file is “dragged & dropped”, the software relies on OGR to add it as a comprehensible layer and this just renders all the fields as text (at present August 2016).
Why is it an issue?
If you are joining tables and aren’t aware of the issue, you drag and drop a table with a list of numerical values in, and then can’t join it to a spatial data with values in as you can’t join text to numbers. This could also cause issues with interpolation (reading of a value field) and also generation of points which need classification based on numbers.
Getting it fixed…
This is where things get a little tricky, as I don’t think it is entirely a QGIS issue and more related to the code which QGIS uses to parse the information, so until OGR update their code, it might be a bit of a wait.
Posted in xyht magazine 21st March 2016
This post almost didn’t happen, for over 7yrs I’ve been comparing GIS software in the hope that someone would be reading and take it upon themselves to fix a few of the issues which we all run into on a daily basis. With the new releases someone told me something and it almost stopped it all from happening…..
“ESRI is the Apple Inc and QGIS is the Google…”
As painful as it is to hear, it’s true. Even though I’ve been a hardcore user of QGIS for over 5yrs, I find myself falling back into using the ArcMap tools just to save myself rummaging through lists of plugins and I especially like that I can just drag and drop my data into the data frame and it just works.
As powerful as ArcMap/Arcscene/ArcGIS Pro/ArcGlobe/ArcGIS Explorer/ArcServer/Arc*add your own term here* is, it feels old and a little clunky, like using Windows XP. For a decade it has had the same comfy interface, the same basic functions and the same symbology. This is by no means a bad thing but the cartographic capabilities of QGIS are far superior and let’s not mention the true 64bit processing…
This year, there is a new contender from ESRI which I feel I need to discuss, ArcGIS Pro 1.2. Although it is in beta, it is already proving itself a worthy successor to ArcMap. Unfortunately, until it is out of beta, I won’t be testing it as I feel it is unfair to compare an unfinished software/
Before we get to the crux of things, I’d just like to add that there have been no major interface changes to either software and I am GIS neutral. I just want a GIS that works and is reliable, I have no bias as to one or the other.
So what is new in QGIS 2.14?
The most talked about is the inclusion of a 2.5D renderer (covered in earlier blogs) but what you may not have noticed are the subtle changes, changes to the processing, caching, labelling and legend. Here is a full list which links to the QGIS changelog
There are quite a few new raster types available for the software, some of which I have to admit that I had never heard of but interesting to see that you can now export topology errors in the ArcMap Basic license. For those wondering why I haven’t listed all the changes, please remember that this is a comparison of the basic GIS software.
When the tool is run against a geodatabase in Oracle or PostgreSQL, it now creates shared log file tables for the user. With the log file tables already created, the database administrator can remove privileges to create tables from the user, if needed, without negatively affecting the user’s ability to create large selection sets, edit data, or reconcile versioned data.
When you run the tool on an Oracle 12c database, the sde user is granted privileges required to import data using Oracle Data Pump. This allows you to complete your Oracle Data Pump export and import workflows without having to separately grant these privileges.
ArcGIS 10.4 has been upgraded to include Python 2.7.10. Additional third-party libraries including SciPy, pandas, Sympy, and nose have been included, and existing third-party libraries including NumPy and matplotlib have been upgraded to more current releases.
The GetPackageInfo function has been added to return a dictionary about packages.
The topology edit tool now supports additional commands and keyboard shortcuts for selecting and moving nodes and edges.
Press the N key to select topology nodes.
Press the E key to select topology edges.
Move and Move To commands are now available on the topology edit tool context menu.
The parcel Merge dialog box now has additional options for managing parent parcels. Parent parcels can be deleted, flagged as historic, or remain as current, active parcels.
When dividing parcels into equal areas, the number of parts on the Parcel Division dialog box can be set to 1. Set the number of equal area parts to 1 when a deed calls for a specific area to be divided from the parcel, leaving a remainder parcel.
Performance improvements have been made to the Mean Points tool. Furthermore, the tool now always stays active and can be used repeatedly without the need to specify a tolerance.
Collinear line sequences in the same plan are straightened during adjustment postprocessing if they lie within the specifiedStraighten Collinear Line Sequences tolerances. These tolerances are available on the Adjust Coordinates dialog box.
Geodatabases and databases
A new advanced option—All records for Tables—has been added to the Create Replica wizard. Checking this option allows you to designate the inclusion of all data from all tables in your replica. In earlier releases, you had to set the option to include data for each table individually.
There are two new geoprocessing environment settings added: LERC compression and parallel processing server.
The Georeferencing toolbar now has three new first-order transformations available: Only rotation and shift, Only shift and scale, Only rotation and scale.
A lot has changed since I last ran this, I no longer have the 8 core 32GB RAM “beast” anymore for starters. I expect that times will be changed but by how much?
For those who want the spec of the machine I am using:
Although it is more home computer spec, I am reliably told that this is should be more than adequate for running ArcMap & QGIS.
1. Load Time
As this is a comparison of the 2 new updates, rather that witter on about my personal views, I thought it better to provide some test results. To make the test equal, the method I use was to start up the software being tested, leave it for 10 minutes (to ensure all elements have loaded) and the add ALL 2010 AIS from a folder in the root – C:GIS. If you wish to test the data yourself, it can be downloaded from here. The timer was started from the moment that the data is added to the data frame (through the add data button). The timer is stopped when all data has finished loaded, indicated in ArcGIS by the globe in the bottom right not spinning and in QGIS by watching the windows processes & seeing when the CPU demand dropped to zero again.
For those who wish to go further down the rabbit hole, I categorised the data for both QGIS and ArcGIS and saved the styles. The AIS was categorised using standard deviation (n0) with 8 categories. Data was loaded directly from the qml/lyr.
The time given is the average of the 5 runs which were made.
ArcGIS 10.2 = 27.27 seconds
ArcGIS 10.3 = 38.5 seconds
ArcGIS 10.4 = 24.04 seconds
QGIS 2.6 = 9.103 seconds
QGIS 2.8 = 8.201 seconds
QGIS 2.14 = 5.08 seconds
On average QGIS was 20 seconds faster to load the data, furthermore, there was a slight speed increase for QGIS from 2.8 to the newer 2.14
2. Testing – Analysis
What use is your software if you spend your days waiting for it to finish a process? The amount of time I must have lost due to running processes is immense, so what if I could save some time? Because of this, I run a a viewshed test. I put a single point down, Then, using standard settings (and OS Terrain50), I run a viewshed (ensuring the output raster resolutions are identical)
ArcGIS 10.2 = 42 minutes
ArcGIS 10.3 = 4 failed attempts & gave up*
ArcGIS 10.4 = 12m 55s
QGIS 2.6 = 58 minutes
QGIS 2.8 = 1hr 16min
QGIS 2.14 = 26.86s
It looks like either this machine is far superior to the old desktop OR both software has made significant improvements in its calculation algorithms. To confirm, both outputs were identical and can be obtained through the link at the end of the article.
*To eradicate any issues with the data input or user error, the ArcGIS 10.2 map was saved and then opened in 10.3, the exact same parameters were used with no success.
5. Testing – Map Export
I seem to lose days exporting PDF files from GIS, so for me, this test is one of the most relevant, so that I can compare the amount of time consumed by the simple “PDF Export”.
For this test, the same EFH shapefiles used previously were loaded, no styles applied, then the bathymetric contours (also from the Marine Cadastre website) and the ESRI world countries shapefile. I then opened the layout/composer & set the orientation to landscape & the size to A3. Scale was set to 1:50000,000 then centered on the EFH data.
Once loaded and left to settle of 10mins, I set the PDF export to 500dpi and then made sure that both the QGIS & ArcGIS settings were the same* (no layers, no georeferencing etc). This is the average times for 5 runs.
The results were as follows:
ArcGIS 10.2 = 3mins 18s – File size 795MB
ArcGIS 10.3 = 3mins 30s – File size 903MB
ArcGIS 10.4 = 7.04s – File size 1.9MB
QGIS 2.6 = 37.4s – File size 72MB
QGIS 2.8 = 35.5s – File size 69MB
QGIS 2.14 = 10.3s – File size 18.16MB
*Just to be clear, ArcGIS and QGIS were run separately
This is a new category, as a few comments I have had asked how many crashes I have had whilst testing. I am please to say on this occasion NONE.
As I worked through, I must say that both software were pretty much as expected, nothing has really changed. The ONLY thing I would pick up on is that QGIS needs to add layers to their browser. When looking for data, QML & SLD are not visible in their relative folders.
As well as comparing the times above, I also run through both software by opening a list of tools which are used within the office on a regular basis to ensure that the software is safe to be used.
As stated previously, it would be unfair to compare the tools as some tools are unique or paid for plugins. Needless to say, I found no issues in my testing.
In this comparison, I think that, although ArcGIS 10.3 has released a HUGE number of new tools and updates, they are, in large for specialists. These tools are unlikely to get used on a regular basis like you & I.
QGIS has shown some muscle and has, in this testing, shown itsef much faster. Furthermore one of my “niggles” from the last article has been dealt with, the shadows for points and fonts. Although there are only a handful of updates & new features, the tools (for me) are more likely to be used.
Again, as before, there is no clear winner as they are both great systems and suited to different professionals in different way. For basic use or an introduction to the world of GIS, you wouldn’t go wrong whichever you choose.
You can now export your QGIS 2.5D maps straight to a web map thanks to the genius(es) Tom Chadwin & Luca Casagrande who developed the QGIS2Web plugin. Before I run away with myself and start talking 2.5D and cool effects, I think it’s best I clarify a few things.
No, you didn’t misread that, you can now export your QGIS 2.5D maps straight to a web map thanks to the genius(es) Tom Chadwin & Luca Casagrande who developed the QGIS2Web plugin. Before I run away with myself and start talking 2.5D and cool effects, I think it’s best I clarify a few things.
Firstly, I realise that I told a bit of a fib when I wrote the xyHt article on web mapping where I said that there wasn’t an easier way to build a web map. There is, it is QGIS2Web, the only thing you need is your own website and QGIS. You simply make your map and hit the plugin button….voila! Not only a preview of the web map but also options for measure, popups, scalebar and even basemaps – It is truly a thing of beauty. The only technical knowledge you need is how to copy and paste your folder onto your web host.
It sounds too good to be true, which is why I feel guilty for ignoring it for so long….Having been beaten by the geospatial industry for well over a decade, I naturally assumed it was some scam whereby I would have to buy into something or pay for a subscription but no, this is the real deal. This is what ArcGIS online should have been, you front the cost (or not – have another look at Github pages *ahem*) for your website and the rest is free. You can host as many maps as you want with whatever style and data….of course, with a little know how and you can even link them to other sources using hyperlinks in your fields.
So, let’s clarify what 2.5D is
2D is the everyday “flat” maps which you would generate in QGIS/ArcGIS/MapInfo/CadCorp (add your own here) although there is relation of how features lie in comparison to others, there is no depth. Buildings and trees appear as “top-down” flat objects.
In true Dragons8mycat style, the best way to describe this is using a 2D image of Super Mario Bros:
2.5D Adds depth, though it is not full 3D, it cannot be explored like Google Earth, rather it is an illusion of depth.
This can be seen in the image below of Mario in 2.5D
3D provides true depth and the ability to explore the depth, this would be the Google Earth buildings or the CityEngine webscenes. Unlike a fixed view of the 3D features, they allow full movement around the real-world feature. Here is Super Mario once again to show what 3D is:
Is this the right time to discuss 4D?….No?….Briefly then – 4D is the medium of time, if we were to add a timeslider to the 3D Super Mario above, and we could move around not only the features but also the time, then it would be 4D (temporal).
Out of interest I have had many discussions over what 3.5D is and so far the general consensus is that it would be a 2.5D map with temporal capability, so think of a 2.5D map which showed change over time….could this (below) be 3.5D?
Back to the future (qgis2web)
The QGIS2Web plugin provides the ability to create 2.5D features on a web map, there is the option to adjust & change colour and you can add functionality. What really surprised me was how easy it was.
Of course there are some minor things which I found as I worked through but I am sure that these will be fixed before this post goes live as the developers of this plugin are right on the ball.
Tip 1 – Use geojson files.
Although the Plugin uses any QGIS data (WYSIWYG) I found the plugin quicker and gave a more accurate representation with geojson files
Tip 2 – Forget your shadows
I was told that the plugin DOES honour the shadow effects but I found in my experience that the shapeburst fills and shadow effects didn’t work and in some cases caused the map to hang when switched on
Tip 3 – You need to remove the OSM maps in the code.
There are a plethora of basmaps available, I’m not going to knock it, BUT there is no option to turn them all off or to have them off by default. If you want to remove them, find the layers.js file and then remove the baselayers reference. var layersList = [baselayers,lyr_LandParcelsSedbergh,lyr_Trees]; should be changed to var layersList = [lyr_LandParcelsSedbergh,lyr_Trees];
I had, when I went into the office today, to do an amazing blog about the QGIS 2.5D functionality and how it was the beginning of QGIS’ journey into the 3rd dimension.
Only I can’t, Anita Graser, the author of “Learning QGIS 2.0” and owner of the “Free and open source ramblings” blog beat me to it. I was not only beaten but she went and added a temporal element to it all….I can’t be upset, I have a lot of time and respect for Anita. In this case it worked in my favour, my competitiveness came out and I sat and thought of just how I was going to do something better.
First let’s discuss the “2.5” functionality, why haven’t you seen it & why is it gaining publicity?
The 2.5D rendering is a new function added to the styling options in QGIS 2.14 which will be released [approx] 25th March. This is available now, primarily for testing purposes through the QGIS 2.13 development release on OSGEO, so what you are reading might change further before release.
Last year Matthias Kuhn set up a crowdfunding campaign to provide oblique 3D rendering as an option to the QGIS output but most importantly enable a way to print/output the result as many of the plugins available in QGIS are just visualisation tools. Little did Matthias realise how popular this idea was and it was funded by November 30th 2015.
“This project aims to improve the internal possibilities of QGIS to give an oblique view 3D effect based on a height attribute and an angle while fully preserving all the possibilities which the QGIS styling offers. But it doesn’t stop there, the whole rendering is built in a modular way so you can use all of its parts for countless other possibilities.”
As stated earlier, it is still in a development test phase but being QGIS it is still very stable….oh yes, it is a fully functional death star. Let’s look at the process for using this function:
As you can see, there is a new interface and options when you select the 2.5D option. Height of the “effect” is based on either a field in the attribution or a z (height) geometry. In this case  it is based on a field which I had to create based on the geometry called “z_rel” (short for relative z).
Now here is something interesting – I was using 3D data, in fact I was using 3D geospatial shapefiles in ArcGIS filegdb format with roof detail and even windows BUT it wouldn’t work. Not because of anything major but because the plugin only supports simple extrusion, or simple z geometry (meaning no multipatch!). Once the z values for the buildings were extracted to a field and the footprints extracted from the multipatch data, the result was good, as you may see below:
We can add a little more detail by using trees. I first tried using the 2.5D render to display the trees and found that it would provide an “extrude effect” down the side of the tree:
Above you can see in the 2 images side by side that the trees look too heavy, whereas by taking the tree canopy and applying a simple shadow, it is easy to make the trees come to life. Note that there is no option to remove the extrusion effect or side of the isometric option (referred to as the wall color) – if this was possible I believe a fairly realistic effect could be created as you could base the 2.5D on the height of the canopy and then add another vector data to simulate a trunk (or not if it looks bad).
From a cartographic point of view, I think this effect works well and with a little adjustment and refinement we could create some magical maps..my only question is that when we are trying to simulate 3D, why don’t we just draw 3D? After all, using exactly the same data I created the below using QGIS2Threejs, you can play with it yourself here (be warned that due to the amount of data it is a little slow)
One thing is for sure, I will be watching this closely and seeing what develops, I suggest you do the same!
In case you aren’t on Twitter, Facebook, LinkedIn, Slack, YouTube, Instagram or Geonet (don’t laugh I hear there are users) I decided that I needed a new challenge and to get behind a winning team.
There are many areas of the geospatial world that interest me, I am a huge fan hydrography and positional accuracy, though there is one area which I feel is getting under-represented…..3D. As I have stated before, we see, live and interact within a 3D world, in fact this year is set to be the year of the VR (Virtual Reality Headset) with sales estimated to be over $1bn. Having seen some of their work and hearing some of their plans, there was only one choice – Garsdale Design Ltd.
Garsdale Design struck a chord with me, they were REALLY eager to point out that a visualisation wasn’t just a simple thing to look at any more. The conversations we always had were around the concept of 3D GIS, where the visualisation ends and the real analysis begins, more on this in another blog, but think to yourself….what defines GIS?
Although it is early days, one of my drivers for Garsdale Design is to not only put 3D GIS on the map (apologies for the pun) but to be able to provide it in both proprietary and open source formats, providing platforms for all users to build and develop, create tools and have a product which we all have wanted.
Let’s be honest, ArcScene came close, though was too quirky, QGIS2ThreeJs is good but misses options for modelling larger areas….I could go on, but the future is ESRI CityEngine, Cesium & QGIS, no I’m not mad and no, I haven’t sold out either. CityEngine is by far the best tool for 3DGIS at the moment but Cesium is gaining ground with an open platform which could be developed to serve measuring and analysis tools. Then we get to QGIS who are developing things which most GIS users could have only dreamed of 5yrs ago….and are now talking 3D, can you imagine the possibilities?
As with all things, a week into the job and there are many, many more plans which are super exciting which I am forced to keep silent about, I am really glad I made the move.
So, here we are, the dream team, I am the one with the Parka, note that the fells are covered in snow, I am not yet acclimatised and able to go sans jacket like these crazy northern UK people.
Parka or not, Garsdale Design now has a full suite of GIS services as well as 3D GIS services such as;
Geospatial Data Management & Processing Asset Management Cartographic Representation Web Mapping – Mapserver/Open Layers/PostGIS Constraints Mapping Site Selection through Multi-criteria Evaluation GIS Model Production/Training Spatial Analysis Least Cost Path Analysis Site Design & Layout Design Refinement/Micrositing Zone of Theoretical Visibility (ZTV) 3D Urban & Site Modelling Geospatial Data Standards and implimentation Metadata Standards and implimentation Geodatabase modelling Geoprocess modelling Ordnance Survey data structure and implimentation Data distribution and implimentation, providing map sharing solutions Interactive geospatial solutions Automatic Identification Systems (AIS)
Feel free to contact me if you need help or training on any of the above!
Be rest assured that my blogs will continue as they always have, though from time to time I may well discuss 3D GIS a little more, watch this space….