QGIS – What do you do when you move your .qgs file?

What do you do when you move your file location in QGIS and lose all your links? Maybe try this….

So, the situation occurred yesterday where I was giving a workshop and sent out a load of QGIS styles, layer definition files and also a project file (.qgs)….Smugly, I told everyone to open the project file, then realised, as hands raised across the room that QGIS doesn’t work with relative paths and it also doesn’t do a “map package”. Working with so many different GIS, it’s hard to keep track of which ones do different things but I really should have remembered this one.

Surprisingly, the solution to repairing all the links and getting it all up and running is relatively easy if you are working with disconnected databases or vector files (shapefiles etc). Just make sure you have a text editor and away you go….

1

First of all, open the rogue .qgs file in your text editor, in the example above, I am using sublime text editor but during the workshop I found Windows notepad was just as capable. Upon opening, you will see that the project file is just a standard xml file with references to several processes.

Use your “Find” option in the text editor to find one of the <datasource> tags (as shown above)

It is simply a case of then changing the folders within that datasource tag to locate the correct location (most people store their data in a single location).

2

As you can see above, I want the project to read all the data from C:OS Southampton rather than the G:Work_Admin_Backup_Nov15GIS Core DataOS Southampton location, so using the REPLACE function (sometimes called the find/replace in some text editors) we can simply change ALL the locations in one go.

Pretty easy huh? A lot easier that using the interface which is provided by QGIS for updating each file link individually, after all, most times we just change folders, we don’t scatter our data around a drive location.

I am sure that this sort of functionality (changing the folder to reference all the links) could be done in bash or as an extra option within QGIS, if you know how, I look forward to hearing from you!

Dragons8mycat

Mapmaking – "Can I have a map?"

Originally posted on xyHt.com

“Can I have a map?”

How often do you get someone ask “can I have a map”?….I have nothing against making maps, in fact I like it so much that I have made a career out of it. Not just any map either, I’ve made maps for the Nuclear, Environmental, Government, Renewable Energy, Offshore and Survey industries, to name a few, I’ve even made maps for a Prime Minister (You’re welcome Mr Blair).

My issue is the question whereby you are just asked for a map. This is not only time consuming for you, the creator but also for the poor client who you are charging 4 or 5 times over for updates to a map which should have been right the first time. This isn’t the clients’ fault, the client didn’t realise that there are multiple templates for the work or that positioning the label they wanted would take half an hour due to it causing a script in the GIS to crash. No, it is safe to assume that the client doesn’t know anything about GIS otherwise they would be doing it themselves….OR (eeek!) you could have the ultimate in intimidation that is the GIS Manager – someone who used GIS up until 10yrs ago and has forgotten how long things REALLy take *shudder*.

There are 2 was to charge people for making maps, either a fixed rate or the consultants favourite, by the hour. No matter which one you choose, it will be wrong. Charge by the hour and you spend 6hrs on 5 different versions of the map which looked fine the first time, OR you take the hit but as you make all those changes, that time you had earmarked has gone and you have to apologise to other clients who were never part of the issue. Lose, lose….

arcgispro_toner_image

Manage Expectations

The most difficult time I had with this issue was working for local government here in the UK. One day I would be asked for a “PPL map”, the next I would be asked to map all the doctors surgeries near to elderly communities, then the next I would be asked for a map of otter holts. Every map was diverse and each required a different format, logo, information, size, paper density, north arrow…..you get the idea, no matter what, each map was unique and with little experience with making public facing maps I was taking my work home with me just to get all the work done near to deadline.

So, I created the “GIS Request Form” and I have used this in many different guises at every workplace I’ve worked and never looked back. Simply put, it is a list of basic requirements for making a map, this has sometimes be a list of 20 items but at present is a list of 7 or 8 which form the creation. If I know that the client is managing onshore assets, it is safe to assume they will require British National Grid, once things go coastal it is a completely different set of goalposts.

The basics I will always ask for are

  • Project Name
  • Time Allocated for the job
  • Presentation Size (is it for a report and to be inset or A3 folded?!)
  • Orientation of the map
  • Allow for ring-binding?
  • Description of the work (useful for context)

Other information which I ask of new clients may be:

  • The project level of accuracy
  • The stated project coordinate system
  • Preferred data provision format
  • Is this based on a previous map? If so, can a sample be porvided?
  • Is the map for internal or external use?

By having all this information at hand, you can manage the expectation a lot better and therefore saving A LOT of time. For one client, I can remember being told to run off about 25 maps for a client and I did it in the company standard A3 landscape, only for the Director to come back and tell me that the client had changed their mind and the figures had to be put into the text of the report……how do you tell a client that you are going to charge them for another 30-40hrs of work?! At least if you have it in print from the start you have a safety net!

Below is an example of the current (quick) request form…..yes, it is in the form of an email. Why? Because I ask my clients to have the message on their desktops, then if they need a map, all they have to do is fill in the details and click “send”.

work-request

 

First Map Perfection

Okay, so I still have to make re-iterations of the map but now I usually only make one…yes ONE. It is a personal goal of mine to ensure that a client only has to have one iteration of the map. In fact, I always tell the client that the first map is a draft and won’t be perfect as no-one has EVER made a perfect map first time (disclaimer: maybe they have, but I doubt it).

I hope this helps to smooth the workflow, feel free to comment and add your stories of client map expectations or when it has got to a crazy number of iterations!!

Dragons8mycat

 

 

 

 

Add OSTN15 to QGIS 2.16

As you may be aware, the United Kingdom has a new transformation model that is OSTN15…..But why? What does it mean to the geospatial community?

Without being too nerdy, tectonic plate movement means that the “model” surface (the geoid) is slowly moving from best fit for the coordinate system. It has been 13yrs since Ordnance Survey implemented OSTN02 so the shift since then is enormous…..a whole 1cm and vertically it is 2.5cm. See this article here from Ordnance Survey.

The whole story is that sensors and our ability to calculate our positi0n relative to both the mathematical models and our relative position to those is constantly evolving too. So, just as OSTN02 revolutionised the accuracy of projecting GPS (WGS84) coordinates using a grid transformation (250 points over the 7 parameters used until 2002), OSTN15 both uses the OS Net of 250 points but has also been improved further with 12 zero order stations with accuracy of 2mm horizontal and 6mm vertical.

So how will this change the way you use your GIS?

If you are already using OSTN02 for your transformations between EPSG 27700 and EPSG 4326 – then you will only see a 5cm improvement over a 1m area at best and this is based on the worst places in the UK, on average you will only see a 2cm improvement anywhere in the UK. To put this into context, when you are zoomed in to an A3 map to about 1:100, you are talking about a few pixels on the screen….it won’t be groundbreaking [at the moment].

Currently, as this goes to press, the OSTN15 transformation has only been available for a few weeks and it is still being tested on different software to ensure it works, I am told that ESRI UK have been testing it with their software as this is being written.

As with OSTN02, I’ve created a fix for QGIS and OSTN15, I will describe how to implement this further in this.

It’s all about the Proj

Proj (Proj.4) is a cartographic library which is based on the work of Gerald Evenden of USGS back circa 1980. Over time it has evolved to consume grid transformations and is used by GRASS GIS, MapServer, PostGIS, Thuban, OGDI, Mapnik, TopoCad,GDAL/OGR as well as QGIS.

There are many ways to use proj, without a GIS you can use it through a command line by defining parameters. QGIS uses the proj library by accessing a spatialite database called srs.db. This is held at .appsqgisresourcessrs.db in Windows and Linux.

The proj spatialite database is a relational database which, when analysed, holds tables for coordinate systems, epsg codes & transformations. What is really clever is that it recognises direction of transformation.

Why is direction important?

Most coordinate transformations go from the projected coordinate system to the geographic coordinate system, for example epsg 4277 to epsg 4326, OSTN15 bucks the trend and is the reverse direction, from 4326 to 4277.

As I found when I first tested OSTN15 with QGIS, I was getting a uniform 200m shift in the data which was being translated and I was really confused. After talking with the gridfile creator, I discovered that the file was created from ETRS89 to OSGB36, therefore the 200m shift I was getting.

QGIS is awesome, you’ve probably overlooked just how clever it is and so did I. Next time you run a transformation, or when you try this one, you may notice that there are 2 fields noted in the columns SRC (source) and DST (destination)…and this is a godsend for solving this issue, as QGIS can read the coordinate in both directions.

transformation-in-qgis

Show us the magic

So, I talked with Ordnance Survey and found that OSTN15 has been given the epsg of 7709 and created a new record with the srs.db which is distributed with Windows, Linux & Mac releases. To utilise this, all you need to do is to download the OSTN15 file from Ordnance Survey (here) and then place the OSTN15_NTv2.gsb file in the shared projections folder .shareprojOSTN15_NTv2.gsb this has been found to be correct in Mac and Windows (there should be similar in Linux). You know it is the right folder as there should be other .gsb files in there!

qgis_folder_location

You can download the updated srs.db from here, this should be placed in the resources folder which can be found at  .appsqgisresourcessrs.db – I highly recommend changing the name of the srs.db file in this folder to something like srs.db.old before adding the new version, just in case it doesn’t work for your particular set up BUT it has been checked on Mac and Windows distributions of QGIS from version 2.12 through to QGIS 2.17.

Enjoy

Dragons8mycat

 

Many thanks to Ordnance Survey for their help

Further reading about the model for Great Britain and OSTN15, I recommend this paper: A guide to Coordinate systems in Great Britain

QGIS CSV & Delimited text Issues

Originally posted on xyHt Magazine 10th August 2016

Last month I was at the Maptime in Southampton (UK), helping QGIS new users how to join tables and map EU referendum maps when I came across an issue with something on QGIS I hadn’t spotted in the last *ahem* years of using it.

When you drag and drop txt, csv or other delimited files into QGIS the fields automatically get converted to text format. No, I’m not making it up and it caused a lot of embarrassment when I was giving my demonstration.

 

draganddrop fields
By dragging and dropping the csv file, you can see that the field type is solely “String”

 

This isn’t written to complain about QGIS but to notify others who are wondering why their joins aren’t working or why their interpolation can’t pick up the value field….You QGIS guys are going to say “why haven’t I raised this as an issue?”, well, firstly read Nyall Dawsons blog post on QGIS issues , secondly I tried to…..it turns out that trying to get access to submit issues has changed and even though I’ve asked for help to get access I’ve been waiting 1 month for response to request.

So…why does it happen?

If you add the file through the “add delimited file” button, none of this is an issue, this is due to the was that the software is written When the file is “dragged & dropped”, the software relies on OGR to add it as a comprehensible layer and this just renders all the fields as text (at present August 2016).

Add layer fields e2e
By adding the csv file using the add layer method, you can see the fields are brought in correctly

Why is it an issue?

If you are joining tables and aren’t aware of the issue, you drag and drop a table with a list of numerical values in, and then can’t join it to a spatial data with values in as you can’t join text to numbers. This could also cause issues with interpolation (reading of a value field) and also generation of points which need classification based on numbers.

Getting it fixed…

This is where things get a little tricky, as I don’t think it is entirely a QGIS issue and more related to the code which QGIS uses to parse the information, so until OGR update their code, it might be a bit of a wait.

 

Dragons8mycat

 

Spatial Blog App now available

Like the blog?….always want to be kept up to date and be notified of new posts?

There is now an Android app available and you can download here

As well as getting the super awesome Spatial blog on your android device, you can also store your favourite posts to read offline. How cool is that?

Unfortunately, the only downside is that I am forced to charge a minimum fee of around 50p (£0.50) to cover the cost of publication and joining the Google Developer Community (Yes, they charge you!!)

Have fun and let me know how you get on with it!

hires

Using 3D Web Mapping to Model Offshore Archaeology

Ever since I started working in the renewables industry on offshore wind farms over 8yrs ago and had to analyse shipwrecks, I thought about how much more interactive and informative shipwreck analysis would be in 3D. There are many companies out there at the moment who produce the most amazing visualisations, where is the ability to move along a fixed track to view a 2.5D wreck but there is no ability to relate it to anything, no context and normally the cost is extremely high when the data captured is normally geospatial and used within a GIS such as QGIS, ArcGIS or Fledemaus.

Here is an example of the amazing model of the James Eagen Layne created by Fourth Element and the model of the Markgraf Shipwreck by Scapa Flow Wrecks

Please don’t get me wrong, I admire these models and they provide detail and information that would be almost impossible to render in a GIS web map without some serious development and a lot of modelling but technology has progressed. Five years ago I would have said that creating an offshore 3D web map was the thing of dreams, whereas today it is a few clicks of the mouse. Using ESRI software, I was able to combine both terrain and bathymetry, adjust for tide datum differences, import a 3D model and then add links and images to the web map (called a ‘scene’).

The most exciting thing we found in developing this, was the cost and time in implementing such a solution. With the ability to consume data from Sketchup, ESRI 3D Models and even Google Earth models, we can reduce the time which a scene takes to build from weeks to mere hours, the most time consuming part is adding the links & getting the colours nice!! Have a look below at what we created:

[iframe src=”https://cloudciti.es/scenes/SJaI7ZBO/embed&#8221; width=”836″ height=”470″ frameborder=”0″ style=”border: 1px solid whitesmoke” webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe><p><a href=”https://cloudciti.es/SJaI7ZBO”>Wreck of the James Eagen Layne</a> from <a href=”https://cloudciti.es/users/54930bf17b2842080022f175″>Garsdale Design Limited</a> on <a href=”https://cloudcities.io”>CloudCities</a&gt;. ]

The model can be navigated in a similar manner to Google Earth, the model should also be interactive, with the ability to click on areas of the wreck with information returned on the right of the screen. If you look at the bottom left there are a set of icons which I will explain.

Overview of the buttons

Camera Button

 

The camera button, highlighted in green, provides access to the scene bookmarks, click on any of these and the scene will move to the view relating to the text. It will also alter the layers shown to provide the best view (according to the creator)

Animation button

The animation button, highlighted green above, animates the scene by cycling through the bookmarks

Layers buttonThe layers button allows access to the information relayed on the scene. By default, the tidal water is turned off and only one model is shown.

Light Simulation

The light simulation button provides ability to cast shadow and simulate specific times of day. Although not really relevant for an underwater feature, it provides a method for viewing internal features better.

Mobile User Bonus Feature!

For those of you using a mobile device, you will notice one further button:

Cardboard button

Yes, the scene is fully 3D and the viewer fully supports Google Cardboard, so go ahead and have a go!

Future development

This is just the beginning, as you can see this viewer is extremely lightweight and responsive Moving forward, we (Garsdale Design Ltd) are looking to adding further information such as nearby wrecks, more detailed bathymetry, objects which may cause risk such as anchorages and vessel movement in the area. The potential is immense and where this is geographic (hit the map button on the right) you can relate this to a real world location….in future versions we are looking to implementing Admiralty charts and bathymetry maps to view side by side with the site.

Disclaimer

I am not an archaeologist or diver! – Data is sourced from open data sources (Inspire, EA Lidar, Wikipedia) with the exception of the model(s) which were built by myself from images and multibeam data. Photos were obtained from Promare, on the Liberty 70 project – Contains public sector information licensed under the Open Government Licence v3.0, This data is not to be used for navigation or diving.

For further information or to ask how Garsdale Design can assist you, please do not hesitate to contact me.

Pokémon Go leads the AR Revolution

Originally posted on xyHt Magazine 12/7/2016

 

Six months ago when I claimed that augmented reality was the future of GIS and geospatial services and it was met with a few sniggers. This week has seen the arrival of Pokémon Go, one of the most popular games hit the mobile phone market….and yes, it is augmented reality and yes, it is geospatial. It could well be the turning point for many geospatial companies.

pokemongo

What is Pokémon Go?

In case you have been hidden in a cave fro the last 25yrs, Pokémon is a game that first appeared on the Nintendo Gameboy (circa 1995) in which players openly walk around the world and capture mystical creatures (in a white and red ball) which they then train to fight (other creatures) in battle arenas called “gyms”. If their creature (called a Pokémon but there are hundreds of breeds) wins their battle, then they earn a badge. The aim is to collect all the badges. Simple right?

Since the mid 90’s there have been 18 manga books, 19 films and about 17 games….yes, this thing is HUGE. Pokémon Go is the new generation and it was inevitable that it would become a geocaching game.

“Geocaching /ˈˌkæʃɪŋ/ is an outdoor recreational activity, in which participants use a Global Positioning System (GPS) receiver or mobile device and other navigational techniques to hide and seek containers, called “geocaches” or “caches”, anywhere in the world.” – Wikipedia

Pokemon Encounter

Why Pokémonn Go is perfect for AR

Lets look at the concept of the game again, the user walks around the globe looking for Pokémon to capture and then train…..if this isn’t the definition of geocaching then I will eat my shorts (thanks Bart). Though any thoughts that this game was designed as a geocache promotion are quickly quelled when you realise that the first geocache was made around the millennium (by Dave Ulmer, Oregon).

For years we have struggled to get our children interested in mapping and geography not realising that this was sat under our nose the whole time! I ironically, it is all due to an Aprils Fools Day joke in 2014 in which it was claimed that Google had developed an augmented reality app…..Niantic saw that this could be a reality and worked with Google, the rest is history.

How does it work?

After logging into the app for the first time, the player creates their avatar. The player can choose the avatar’s style, hair, skin, and eye color, and can choose from a limited number of outfits. Once the avatar is created, it is displayed at the player’s current location along with a map of the player’s immediate surroundings. Features on the map may include a number of landmarks where Pokémon may be and Pokémon gyms (places where you battle your Pokémon).

As players travel the real world, the avatar moves along the game’s map. Different Pokémon live in different areas of the world; for example, water-type Pokémon are generally found near water. When a player encounters a Pokémon, they may view it either in augmented reality mode or with a pre-rendered background. AR mode uses the camera and gyroscope on the player’s mobile device to display an image of a Pokémon as though it were in the real world. Players can also take pictures, using an in-game camera, of the Pokémon that they encounter both with and without the AR mode activated.

There is a fantastic article on how Google chose the locations for the Pokémon/geocaches here (by Mashable) whereby it explains how safety was the primary concern. Caches were chosen based on open places that had some significance so that players wouldn’t be chasing a Pikachu (a type of Pokémon) across a train track.

Pokemon Batlle

Why will it change geo things?

Already there is a wave of companies looking at how they can use marketing to get a piece of the pie, see this & this, it is evident that people are aware that this is, excuse the pun, a game changer. It really isn’t hard to see that this is going to be popular….so this might well be the turning point for geospatial, AR & VR. If people are comfortable using AR through this game then they will start expecting it for their mapping, bringing back apps like LAYAR which augmented real world information.

AR on Mobile

The applications are immense and exciting for the geospatial industry, the ability to overlay real world issues and information to what the user sees through the camera would be the definitive mapping system. Even if the accuracy isn’t that amazing (mobile GPS – think about it) there is the potential to use clever imagery and presentation to overcome most issues. Imagine sending the worst member of your team to site, with an AR map you could be 90% sure that they would be able to find the correct building over using a 2D map, or think about how easy it could be to identify potential points of weakness or contamination around a site by just looking through your device, all set by someone sat at a desk on the other side of the world.

Of course this is speculation but consider the rise of VR which is now around us, soon we will all be fully immersed watching TV, playing games and riding roller coasters, 3D GIS has seen a rise over the last few years too, with many geospatial providers offering 3D add-ons or 3D alternatives. Furthermore the conferences were rife with talk and demonstration of 3D and VR. Of course all this innovation is led by CEOs and Project Managers who have seen their kid playing with some game and asking that all important question….

“Why can’t our company do that?”

You’d be a fool to think that this is all going to just disappear, the future of geospatial is now, we are seeing the evolution occur in front of our eyes – Just like the late Roger Tomlinson evolved the paper map to digital GIS in the 60’s, we are seeing 2D moving to 3D real world. I am all for it, it will bring new challenges, better accuracy and more interaction with the user…..though I draw the line at Pokemon Go myself.

Dragons8mycat