Using Live Shipping Data in GIS

Originally posted in xyHt 21st Feb 2018:

We’ve all been dazzled by these awesome maps which animate ships or aircraft as they move in near real time, but how do they work? What magic sits beneath the hood? As luck would have it, whilst working with the offshore wind farm industry I had to do a LOT of vessel analysis, not only assessing areas of issue but predicting travel paths and growth…It all seems complicated until you know how it is done, read on…

AIS (Automatic Identification System) is a requirement of vessels across the globe, it is like a tracker which is fitted to all vessels which meet certain requirements. This AIS is a VHF signal which is constantly sent out by the vessel and contains information about where the vessel is from, where it is going, what type of vessel, the current time and position amongst a small part of what is sent, I will go into this further later on. The signal which is sent out can be picked up within the line of sight of the VHF receiver (this is where your ZTV analysis comes in), in practice, when put high up, I got a range of about 30km. The data must then go through a transducer so that the NMEA (National Marine Electronics Association) data can be obtained from the VHF signal.

1302514295 !AIVDM,2,1,9,A,53aEOe4000010KGK7D0dtu0P4p@Dh0000000001?70>72t0Ht7kS1lUL,0*39
1302514295 !AIVDM,2,2,9,A,<<<dmlu<lt<<tMh,2*64</dmlu<lt<<
1302514297 !AIVDM,1,1,,A,H3P8VGlpwwwwwwwj0001p5230,0*6B

Example of Raw NMEA data

This data is relatively easy to break down, let’s look at the following line:


!AIVDM:        The NMEA message type
1              Number of Sentences (some messages need more then one)
1              Sentence Number (1 unless it´s a multi-sentence message)
The blank is the Sequential Message ID (for multi-sentence messages)
A              The AIS Channel (A or B)
14eG;…       The Encoded AIS Data
0*             End of Data
7D             NMEA Checksum

Using another software like Shipplotter, MarineTraffic or other, you can parse the AIS as a comma delimited textfile format and into a SQL database. All in, it cost less than £2000 for a receiver, the software and everything else needed to collect the data at 1 second intervals, using something like MarineTraffic you can even obtain everything you need for free. But I’m not here to tell you how to set up the system, you can read this and do it yourself with relative ease. I’m here to discuss how to use that AIS data and create some cool maps.

Once you have the NMEA decoded, you will see that there are 24 message types which you can receive, it can be useful to record a few of them but with experience you’ll learn to ignore most;

                      Ship Message Types

Now, you would think that message 1 & 5 would be the perfect route for this….and it took me a day of data capture to realise that you are best to capture messages 1,3 AND 5 as there is important information contained in each of them. This is where having the data stored in a database comes in useful – You are recording information for multiple vessels for which information are being received with 3 messages, therefore having the ability to store each message type in its’ own table so that they can be related based on the vessel is a HUGE bonus.

                                                                             Message Types

The one common field in all of the messages is the MMSI (Maritime Mobile Service Identity) an ID of 9 digits, similar to a registration number, of the vessel, with this you can identify vessels. For example, if you do an online search for “MMSI 235006680” you will come up with the information for a British vessel called “Red Osprey”.

Messages 1 & 3 provide very useful information on the vessel travel and more importantly location but not other useful information which would require a Lloyds register lookup table (or other vessel lookup table), but why would you? You have the data!! As long as you append message 5 to all the message 1 & 3 messages you can get everything you need.

So, if you’re keeping up with all this and have fully read the message types, you will realise that there will be further joins to perform on the tables as information like “Ship type” (message 5) and “Position fix” (message 5) are related to further tables which can be easily found on the internet on pages such as this (table 11) and (table 10). It all sounds rather complicated but, if you have ever used MS Access, QGIS, PostGres or ArcGIS to join some tables, it really is that easy! What you need is a way to visualise those joins, which I have thoughtfully put together here:

How the tables are joined

Now you have the data structured, you just need to use it! This is the easy bit, if you put in the work you will have a nice structured data which, depending on your storage format, you can snapshot and put into your GIS. The reason I say snapshot is that the data is coming in every second, within a day you have a crazy amount of data, in a month it is almost unusable, futhermore these systems aren’t the most reliable, so I would highly recommend a daily backup as I have been let down a few times with “glitches” due to weather or power shortages.

Screenshot of some data captured

Now that you have coordinates and time, you can create a map! There are many different ways to display the data;

  • Density by vessel type – for analysing trends in data…or for looking and those heavy polluting tankers…
  • Time based – Useful for seeing how vessels are moving and their patterns
  • Heat analysis – Similar to density but useful for identifying busy areas
  • Planning – If you put an oil rig on a shipping lane, where are the vessels going to go
  • Predictive analysis Using destination and other vessels’ route to that destination, what is the best possible route
  • Least cost analysis – use weighted to add input into cable route analysis

My personal favourite is the time based analysis, as it gives a visual representation of the daily/weekly/monthly movements – furthermore you can single out a single MMSI and watch a vessel plod around….sometimes you find that they have identified themselves as the wrong type of vessel, other times you realise you are looking at a diving club vessel as it travels out and circles a wreck, it can be fun sometimes.

Let’s have a look at how to do the time based analysis in QGIS – First of all make sure you have QGIS 2.18 if you want to follow along.

Open QGIS and add the Time Manager plugin, this can be obtained from the “plugin” tab at the top of the screen

                                                                      Time Manager Plugin


Once the plugin is installed you can activate it under the “plugin” tab, you will see the time manager drop down as one of the options, but don’t activate it straight away as we need to add the data first. Add the data and then I find it best to categorise the data by the vessel type. You can try to categorise by the vessel name or MMSI but in a single day you could have up to 200 vessels which would make for one hell of a lot of vessels to symbolise. You can find the option to categorise by right-clicking on the data, and selecting “properties”, if you then choose the “style” option on the left and then select “categorised” in the drop-down at the top of the screen, you can then select which field and colours you wish to use.

Symbolising by Vessel Type

Next, open the time manager window (you should have this in the bottom of your map frame, if not, activate using method mentioned earlier). Populate the required fields, ensuring that you read the time format and select the time field which corresponds (If using the data straight from ShipPlotter you will be fine).

Choose how you wish the data to be drawn, if you want to see the the vessel ONLY at its current position, then leave the “end time” box empty, if you wish to see the previous positions stay on the screen, then choose the “show accumulatively” for the time.

It really is that simple! Sit back and enjoy! My recommendation is that you save your map before playing, I’ve found if you have large data the map can crash and also ensure all your fields in time manager are set correctly as I’ve not found a way to change specific fields but instead find myself deleting the layer (in time manager) and then re-applying.

Cumulative vessels exported from the time manager with time (bottom right)


For bonus points and to see how to create some stunning time based visualisations with the QGIS Time Manager, created by xyHts’ friend, Anita Graser, see her blog here

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s