Saturday, January 30, 2016

How to Verify Computer File Integrity

Is this really navigation?  I would have to say no, unless we are talking about really modern times where the navigator gets more and more involved with computer programs and computer files.

We have been working on converting weather maps to echarts, which has led us to using several open source utilities. Often free programs we download online can get corrupted or polluted by accident or some malware.  Or maybe we want to send someone a file of data or a gpx file of a long list of way points or a detailed GPS track, and we worry that loading or unloading of it from an echart program might change the content of the file in some way. Or maybe you just want to send a long text document to someone and be sure it has not changed when it gets there or when it goes somewhere else.

It turns out that the tech world has a solution for this, which I suspect is well known to those who know it well.  It is called the MD5 sum. It is a unique number created by an encryption program that is a fingerprint of that file.

To use it, you need a program that reads the MD5 value of a file (any file: executable, image, text, etc) and then you just use that program to read the MD5 sum of that file and post it, or tell the recipient what it is.  Then when they get the file, they run it through the same program to check the values.  If anything at all has changed in that file, just one dot or one space, then the MD5 sum will have changed and you will know it has changed.

This test does not tell you what has changed, but I believe there are numerous programs that will also compare files to detect the change. But for images or encrypted executables, you still might not know what has changed.

This very easy to use test is just to confirm that it has not changed. That is all.

One convenient version I have found is called WinMD5.exe.  An example of the use of this is in this video below confirming the authenticity of the open source graphics program called Gimp... which we are suggesting could be useful for making BSB files from images.






Converting Weather Maps to BSB echarts


Our goal is to convert weather maps to echarts for use in forecast evaluation (overlaying model forecasts) and general route planning. We started with the easy ones, namely the color maps from OPC that have a nice lat-lon grid on them that lets us scale the margins of the charts. This way we can make a prescription for doing this that does not involve trimming the charts for each conversion.

Underway, however, we are most likely to have access to the black and white tif maps that we can get from saildocs in reduced file size.  The problem that comes up immediately is these maps do not have a nice lat-lon grid; it is shown only every 10º with large blank margins. Thus are simple approach of making the header files by hand does not work for these maps.  I have tried numerous tricks that I thought would work around this, but none succeeded. We are standing by for an easier solution if someone has it. Just post a comment.

One approach that does work is to call up two more custom programs and built the map in several steps.

Step 1. This always has to be done: download the map and convert to a png, and read the dimensions of the chart viewed upright—we typically get the maps sideways as tifs.

Step 2. Georeference the chart and define the borders using MapCal_2, and save what it calls chartcal.dir, which is a text file you can open in notepad.  This step is illustrated here:

Video Note: watching carefully you will see a blunder being made on the bottom right cal point, but it got caught and corrected during a simple check at the end.... nice to have such things to see how this works!  There is also an error early on calling a png file a gif file. Unfortunately, nothing to learn from that if we rule out author evaluations. 



Step 3. Use the program mc2bsbh to build the header from the chartcal.dir. The extension of the header made this way is .hdr.  We need to change that to .kap before using imgkap (you could also create one with the right name, but this is just as easy).

Step 4. Proceed to convert the png to a kap file with the header just created using imgkap.




In an earlier video-illustrated post we did this making the header file by hand, which worked for easy to scale charts that have a full lat-lon grid. That does not work for these BW maps with large vacant borders and few reference lines, which is why we move to this procedure.... at least till we learn a better way. On the other hand, once you have these utility programs in a folder, it is probably fastest to use this present method for all charts.

( I should mention for completeness, that if one has the high-end navigation program called Expedition, you don't need any of this. You just import the image, georeference two opposite corners with a slick, fast interface they have, and you are done, and ready to view the map as an interactive chart.  It is then saved for later use or for overlaying model forecasts, etc. One minute would be a long estimate of the time it takes from beginning to end! )

In principle, there is a package of utilities called kapgen that will do all of what we describe above at once, but I have not had any luck yet with using that.

It took long enough to learn to do what we do here that I want to document this process before I forget the details; later we might return to figuring out what we are doing wrong with kapgen.

All of these methods are described in the OpenCPN manual or FAQ, or links they provide. It seems easy enough when reading about it, but there are nuances that I hope we point out here.

Program sources:

imgkap.exe from http://www.dacust.com/inlandwaters/imgkap/

mc2bsbh.exe from http://www.dacust.com/inlandwaters/mapcal/

MapCal_2.exe comes from installing the program SeaClear. Install the program, then look in the Programs Files directory, and from their you can make a shortcut to MapCal_2.exe for your desktop or other folder you are usin for the conversions.

-----------------

For reference, here is the header file we created for the epac surface analysis, a map that is 1446 x 1728. With this, all you need is imgkap to make a chart out of any epac surface analysis. This is for the full size maps. We need to make another one for the saildocs versions that are half this size.

! Created by mc2bsbh beta09 - Use at your own risk!
VER/2.0
BSB/NA=pyba90
    NU=pyba90,RA=1446,1728,DU=628
KNP/SC=50000000,GD=WGS84,PR=MERCATOR,PP=UNKNOWN
    PI=UNKNOWN,SP=UNKNOWN,SK=0.0,TA=90.0
    UN=UNKNOWN,SD=none
    DX=2021.6,DY=2021.6
OST/1
REF/1,126,262,60,-170
REF/2,126,1582,20,-170
REF/3,1325,262,60,-120
REF/4,1325,1582,20,-120
CPH/0.0
PLY/1,64.940016,-175.045872
PLY/2,14.3864206,-175.045872
PLY/3,14.3864206,-115.162636
PLY/4,64.940016,-115.162636
DTM/0,0







Tuesday, January 26, 2016

Weather Maps—Where To Get Them and What We Get?

Updated May31, to account for NOAA change in links.

When we are underway paying $1.50 per minute for satellite phone time, we clearly care if we can get the same weather map from one place that is much smaller in size than from another place, not to mention that some services might not even allow larger files to be transmitted. The latest surface analysis of the Eastern Pacific, for example, can vary from 13 to 312 kb, with multiple options in between. Again, these are the same maps.

But not only download air time matters. Storage space on our computers or tablets underway, as well as the time it takes to process the files when we look at them in different applications are also factors that make us aware of file size.

Our present project at Starpath is studying ways to get these graphic maps into an echart program (i.e. convert them to nautical charts) so we can see our precise position on the map, read precise ranges and bearings, and also to compare these professionally produced graphic products with the vectorized grib format of single numerical model predictions, such as the GFS. With the weather map made into chart we can overlay the grib forecasts right on top of the graphic forecasts.

We want to use the digital grib forecasts as much as we can because route analysis is much easier with them, We can also read digital values of wind and pressure at any point on the map—not to mention that if we are using an automated weather routing tool we must use the grib files.

But any single numerical model forecast might not be right. We need to compare them with the forecasts made by the NWS, who are looking at several models in addition to the GFS as well as other factors when making their forecasts. The pure GFS will be close to the truth much more often than far from the truth, but actual differences can be crucial in some circumstances, especially with newly developed or fast moving systems. It is simply prudent navigation to compare them on some level.

To make a chart out of a map, we first need the graphic file of the map, which leads to this topic at hand.

Our own Starpath Custom Briefings pdfs discussed below (one for the Atlantic and one for the Pacific) are very convenient ways to request any of the maps, both onland and underway, so we will look at this option after looking at the more standard sources.

We will use the 12z Eastern Pacific Surface Analysis for the example, but other products have similar results. And we list file size as downloaded—which has to be said when the options include tifs, which increase in size notably when you do anything at all to them! Some maps are presented sideways, so they have to be rotated to be read or manipulated, which if tiffs will notably increase their size, but this all takes place after they are on your computer.

Note too that capitalization of name and extension matters in the file names. NOAA now has some in color and others in black and white (BW). Below we give dimensions as w x h as viewed in normal portrait mode. Sideways means we get the map in landscape mode and have to rotate it.

Where to download the maps
The main source of weather maps is the FTP site at NOAA, all are there with a specific file name for each map along with (for some, not all) a file name that represents the latest version of that map. For example, PYBA05.tif is the 12z surface analysis of the East Pacific. It updates once a day at about 1430z to the current day’s map for 12z, but the file name remains the same.  Likewise, PYBA07.tif is the map for 18z. But there is also a map called PYBA90.tif which is the latest version of this 12z analysis. It could be the 00z, 06z, 12z, or 18z map, which ever has updated the most recently.  Usually we want the latest version, but there could be reasons for looking at a specific map that is outdated.

If we know the file names, we could get them directly from the FTP link. But there are several sites that index the maps so we do not need to know specific file names... but what is at the other end of these links is what we look at now. In passing, the system of keeping the same file name and updating the contents daily is a real blessing when it comes to working with these maps.  Some other sources, and most European maps, have the date and time built into the file name, which really complicates any automated acquisition of them.

File names for NWS surface analysis maps. From Modern Marine Weather

Starting with the neat shortcut link weather.gov/marine, which is a nice portal to all weather products, we can find the graphic map links. They link to map indexes in three different formats, but as we shall see, it is only the index layout that changes, not the maps.

Since the latest NOAA change of links, some of these notes are not valid. We need to check when we can. One path to the new map locations can be now be found from the fax schedules themselves.

Atlantic Ocean fax schedule
Pacific Ocean fax schedule
Alaska region fax schedule
Hi region fax schedule

http://tgftp.nws.noaa.gov/fax/PYBA05.TIF       37 kb    1446 x 1728    BW    sideways
http://tgftp.nws.noaa.gov/fax/PYBA05.gif         90 kb    1446 x 1728    BW    sideways
http://tgftp.nws.noaa.gov/fax/pyba05.gif    90 kb    1446 x 1728    color    upright   

Condensed versions (same as main links but only latest) http://tgftp.nws.noaa.gov/fax/ptreyeslatest.shtml
http://tgftp.nws.noaa.gov/fax/PYBA90.TIF     37 kb     1446 x 1728    BW    sideways       
http://tgftp.nws.noaa.gov/fax/PYBA90.gif      90 kb     1446 x 1728     BW    sideways

 
http://tgftp.nws.noaa.gov/fax/PYBA05.gif     90 kb    1446 x 1728    BW    sideways
http://tgftp.nws.noaa.gov/fax/PYBA90.TIF    37 kb    1446 x 1728    BW    sideways

Mobile version (latest only) at http://www.nws.noaa.gov/om/marine/hfreyes_mobile.htm

http://tgftp.nws.noaa.gov/fax/PYBA90.gif    90 kb    1446 x 1728    BW    sideways

So knowing the maps are the same you can choose the index presentation that best meets your needs.

These links (except for the color maps) are direct to the FTP site: http://tgftp.nws.noaa.gov/fax.  The color maps seem to be a product of the Ocean Prediction Center, which I believe is where they are produced. They also provide a side link to another interactive radiofax schedule, but the maps we get there are in slightly different sizes than on the FTP site, which makes them less attractive for our purpose of making echarts out of them. Notice too that they use a different format for the file names—extensions and names all lower case.

http://tgftp.nws.noaa.gov/fax/pyba05.tif        312 kb      1440 x 1728    BW     upright
http://tgftp.nws.noaa.gov/fax/pyba05.gif        105 kb     1332 x 1600    color   upright
http://tgftp.nws.noaa.gov/fax/pyba05bw.gif     86 kb     1440 x 1728    BW     upright       

A very convenient way to request maps on land or underway is our own Starpath Custom Briefings. These are pdf graphic indexes to each of the maps, organized by map type and forecast period, including notes on each of file size and update times. The link above explains the use of the briefings. These briefings make use of the wonderful service of saildocs.com, which reduces the file sizes by about a factor of two. This is a blessing when underway on limited bandwidth.

In the Starpath Briefings we link to the saildocs latest version of each map, which you request by email from saildocs with a single click and send. In the example we are using here, they will send back to you in return email the PYBA90.TIF that is normally 37 kb at 1446 x 1728 reduced to 18 kb at 864 x 723, also sideways. This tif format will double in size when you rotate it and save it, but it was small when you downloaded it. The reduced file size is still plenty legible, but it does not get better when the size doubles after rotating.

Download  Starpath Atlantic Maps Briefing pdf index

Download  Starpath Pacific Maps Briefing pdf index.

For Reference (not needed when using the Starpath Briefings):
Alaska map file names
Hi map file names
Gulf of Mexico and Carib file names
Selected International map file names
Selected UK map file names

§§§

Saturday, January 23, 2016

Weather Maps to eCharts—Making the Header File

In a recent note I outlined how to make a BSB compatible echart from a graphic image of a weather map using a utility called imgkap.  The key to that process is the header file that tells imgkap how to process the image into a .kap file that reads in an echart program.

Our goal is to provide a set of these header files covering each type of weather map so the process can be quick and painless.  On the other hand, if you do want to get a little pain into the process you can see below how these header files are being made. There might even be other reasons, as something like this is required to georeference any image you  might want to use for live navigation. My main goal for writing this up is to document the process if we need it later. Some of the steps took a while to figure out, and would be easy to forget.

Several details of this process discussed here are illustrated in the video below. Later we will add the most important article and video, namely what to do with these georeferenced weather maps once you have them.



We will use a Surface Analysis map for this example.  Here are the steps (you might skip to the bottom for a moment to see the header file we are creating).

Step 1.  Download the image. The maps we are working with now are the new (as of last 2105) color format from OPC. Some are in the png format, others are gifs. The gifs must be converted to pngs, which can be done with most graphics programs, including the stock Paint program in Windows or Preview on a Mac. Just load the gif and save or export as png.

(Later we address the black and white tiff files of the same maps. These have a few special issues to address, including rectangular pixels! )

Step 2.  Read image dimensions (w x h), i.e. 1332 x 1600
In the header file, this is then RA=1332,1600

Step 3. Figure DX, DY = meters per pixel (m/px):
One approach is choose a latitude near the center of the chart, and measure the height of a rectangle some degrees on either side. For example,  if 40N is in the middle, measure the pixels between 30N and 50N, i.e. 587 px. We are going to call the Lat where this is valid PP=40.

Then figure m/px from: 587px = 20º = 1200 nmi x 1852m/nmi = 2222400 m, so there are 3786.0 m/px.   So DX=3786.0,DY=3786.0.

Aside: We have the option for different DX and DY values in the header. All NOAA Mercator charts have square pixels and these are the same for wx maps in png format. (Not sure yet what we will need to do here for the tifs).

Step 4.  We next figure a chart scale, which is needed for the header. The concept of chart scale is transparent for RNC, which are copies of printed charts that do indeed have a scale, i.e. 1” on a 1:80,000 printed paper chart is 80,000 inches on the land. This is not so clear a concept for these weather map files that do not have a standard print size. But we do have to input a dpi for the image called DU, so once we choose that along with the DX,DY values we are committed to a scale.

The png or gif files when downloaded are likely to be at 72 dpi when downloaded on a Mac or 96 dpi when downloaded to a PC. So the 587 px we measured for 1200 nmi, would (using a Mac) correspond to 587/72 = 8.15 inches, which is = 1200 nmi x 1852m/nmi x 100cm/m x 1 inch/2.54cm  = 87496063 inches, so the scale is 1: 87496063 / 8.15 = 10735713 = SC in the header.

You can do the above for whatever dpi the image comes with. It will not matter  so long as we are consistent as outlined above. The dpi value is not a property of the image. It is only related to printing, which we are not doing. What we are doing is fudging a format that was intended to represent a paper chart that had known and fixed dimensions. The announcement a few years ago that NOAA was expanding all of their RNC from 250 dpi to 400 dpi was just a way to say the file dimensions (w x h) are going to be bigger.

Step 5.  Now we have to figure the Lat, Lon of the corners, which involves extrapolation of the scales to project them out to the edges of the paper. (We could crop the images to the scales shown, but that would involve an extra graphics step on each conversion. Doing it this way, we  have no graphics work at all once the header is in hand describing the full image.)

In this surface analysis image the Lon meridians line up nicely with the edges of the images at -175.0 to the west and -115.00 to the east. Note West Lon and South Lat are negative in this system, which is common.  The top of the chart is 9 px above 65 N, and the width of 64N to 65N is 52 px, so the top edge Lat is 65 + 9/52 = 0.173, so top Lat = 65.173.

The bottom edge of the image is 37 px below Lat 16N, and the width of 16N to 17N is 23px, so the bottom of the image is at a lat of 37/23 = 1.609º below 16N, which is 14.391N.

Measuring the pixel width of the bottom margin (37 px) with the selection tool in Mac Preview (left) and Windows Paint (right). Paint does not show the bottom most line of pixels, so it adjusted up one. Step 1 in both cases is be sure the view is set to "Actual Pixels" or "Normal View." This measurement is more precise with a more sophisticated program such as Photoshop or the open source Gimp, with which you can zoom in to close view and still read the actual pixel count.
Step 6.  Now we can specify both the georeference points and the chart corners, which we have simplified here by making them the same 4 points.  The reference points (called REF/1, REF/2,…) can go in any order and be any number larger than 3, but the 4 corners (PLY/1, PLY/2,…) must go in order, starting at the SW corner and proceeding clockwise.  (Actual NOAA RNC include a reference point for all major lights on the chart!)

Step 7.  Now we can construct the header for this chart, using the standard minimum information as determined above. We will call it EPacSurfaceAnalysis. Pixel locations are referenced to the top left, with X increasing to the right and Y increasing toward the bottom.

! East Pacific surface analysis from OPC.
VER/2.0
BSB/NA=EPacSurfaceAnalysis
    NU=EPacSurfaceAnalysis,RA=1332,1600,DU=72
KNP/SC=10735713,GD=WGS84,PR=MERCATOR,PP=40
    PI=UNKNOWN,SP=UNKNOWN,SK=0.0,TA=90.0
    UN=NONE,SD=NONE
    DX=3786.0,DY=3786.0
OST/1
REF/1, 1,1600, 14.391,-175.0
REF/2, 1,1,65.173,-175.0
REF/3, 1332,1,65.173,-115.0
REF/4, 1332,1600,14.391,-115.0
CPH/0.0
PLY/1,14.391,-175.0
PLY/2,173,-175.0
PLY/3,65.173,-115.0
PLY/4,14.391,-115.0
DTM/0,0

If we name the image EPacSurfaceAnalysis.png, then we can copy this text and paste into a txt file and save it as EPacSurfaceAnalysis_header.kap, and now we are ready to make our map.

This is clearly a bit tedious, but we do not have to do this again. Once this header is made and tested we can use it as is for all East Pacific Surface Analysis maps in the future. The earlier note presented one that works with the EPac 24h surface forecast.  Then getting a map into the echart program will be just download the file to the right place, run imgkap, and open the map.  In fact, you can just copy and paste this text into a cmd line to get your map:  imgkap EPacSurfaceAnalysis.png EPacSurfaceAnalysis_header.kap EPacSurfaceAnalysis.kap

_______

To learn more about the kap header file see this short description:
http://opencpn.org/ocpn/kap_format

To get the primary information about the BSB format go to this link at the International Hydrographic Organization.  http://www.iho.int/mtg_docs/com_wg/HSSC/HSSC3/S-64_Edition_2.0.0/RNC_Test_Data_Sets/BSB_TDS/BSB_TDS.htm


Thursday, January 21, 2016

How to Load Weather Maps into OpenCPN — The Hard Way!

Since posting this note (and spending much time on figuring out how to do this!) we have learned from a comment that there is indeed a very much easier way. There will shortly be here a link to a new article.  The "new" method is a remarkable plugin to OpenCPN called Weatherfax, which has apparently been available for some time and I just did not know about it.  It not only allows image display with convenient georeferencing, but also includes a direct link to many maps online that show up automatically georeferenced in OpenCPN.  

So best bet is go the new article unless you have specific interest in the use of georeferenced images in the kap format as used with NOAA echarts.

-----------------------

With the latest surface analysis and forecast maps in an echart viewer, we can see our GPS position right on the map and plan routes across the weather patterns. It also facilitates reading wind and pressure from the map at places where none are shown. Another important application is overlaying the GFS grib forecasts on actual NWS forecasts to evaluate the GFS data—in areas where we do not have digitized NWS forecasts in the form of NDFD gribs, in which case this could be done more easily by just comparing gribs.

We discuss this in Modern Marine Weather in Section 7.2 called Georeferencing Weather Maps. In that discussion we had recommended the program Memory-Map Navigator as a way to do this as it was possible with their free, demo version. That program still has this functionality, but the demo version no longer includes this option.  Other commercial products also offer this, including Ocens MetMapper, and Polar View. In principle the free program Sea Clear can do this, but I have not yet succeeded with that. All of these solutions can create a weather map echart, but they would each be unique to their own viewers. The procedure below makes a file that will open in any echart viewer.

Our electronic charting and weather training now use OpenCPN for the in class exercises, so the goal at hand was to develop a method for loading weather maps into OpenCPN. The final product should in principle open in any echart viewer. The explanation of the process is outlined below, followed by a video showing each step in action.

We will use a free utility called imgkap.exe for the PC. It is used to convert images to the .kap file format that is used in the RNC echart format, also called BSB.  Once the file is made, it will open in Mac or PC versions of OpenCPN. I do not know of a Mac equivalent, but there could be something.

Below is the step by step procedure, and here is a video that shows the steps in action using a later 24h surface analysis.



Step 1. Download a copy of imgkap.exe (Windows binary, v 1.11, 3.5 MB) from dacust.com/inlandwaters/imgkap. This program is discussed at that link as well as in the OpenCPN Manual at this link opencpn.org/ocpn/imgkap.

Step 2. Make a folder on your desktop called imgkap and put imgkap.exe in it. Needless to say, you can call things what you want and store them where you want; we just need one place to put all the parts. We will need three parts: this program, the map image itself, and a header file that tells the program how to make the kap. With the nice tools made available to us, the work reduces to making the header file.... and I hope to reduce that part of the work by providing a header file for each of the standard wx maps we get from NOAA.

Step 3. Download the weather map image and save it to the imgkap folder. We use here the example of a 24h surface forecast for the Pacific, which you can get from www.opc.ncep.noaa.gov/shtml/P_24hrsfc.gif.  This gif image should be 907 x 1200 pixels. The main index to all of their maps is at www.opc.ncep.noaa.gov — click the ocean tabs to get to the maps. Below is the top part of the map we are working with.


Step 4. Convert that gif to a png using a graphics program. This can be done in any version of Windows by right clicking it, choose Open With, select Paint, then after it opens Save As, choose png, save it, and close Paint. Then you will have two images in the folder a gif and an png. You no longer need the gif and could delete it.

Step 5. Now create the header file: Open Notepad and copy this text into it:

! 24h pacific surface forecast from OPC.
VER/2.0
BSB/NA=24h-forecast
    NU=24h-forecast,RA=907,1200,DU=300
KNP/SC=53569020,GD=WGS84,PR=MERCATOR,PP=40
    PI=UNKNOWN,SP=UNKNOWN,SK=0.0,TA=90.0
    UN=NONE,SD=NONE
    DX=4535.5,DY=4535.5
OST/1
REF/1, 1,1200, 15.7,-157.0
REF/2, 1,1,62.5,-157.0
REF/3, 907,1,62.5,-108.0
REF/4, 907,1200,15.7,-108.0
CPH/0.0
PLY/1,15.7,-157.0
PLY/2,62.5,-157.0
PLY/3,62.5,-108.0
PLY/4,15.7,-108.0
DTM/0,0

Note that the indents matter; they signify that the line above it was more than 80 characters, which is the limit. No blank lines before this text and none after it.

Then save this notepad file in the imgkap folder and name it P_24hrsfc_header.kap.

Step 6. Now we are ready to make the kap file. Now open a cmd window, ie from the Start button or equivalent type cmd and enter and the window will popup.  Then type cd desktop/imgkap and enter and you should see something like this:




Then type   imgkap P_24hrsfc.png P_24hrsfc_header.kap filename.kap   and then enter. Filename is anything without spaces you want to name the map, ie 24h_valid_12z_jan16.kap.  You can copy and paste this into the cmd window if you like with right click and paste—assuming you are using the same file names. Put another way, you can make your own generic file name convention and just use that each time as long as you have the right extensions and list them in the right order. You will get a clear error msg saying what is wrong if you make an error.

Or you can just type imgkap and enter to see the inputs and the other things you can to with this utility, ie you get a help file.

Then if you do not get error messages you will have another file in the folder called filename.kap.  This is the file that will open in OpenCPN as a georeferenced wx map. Here is what you see after a successful map.



At this point, we need to just move the newly created file into a place the echart program can read it as shown in the video.

Here is what the map looks like in OpenCPN.


A final note on use of this type of home made kap file in other programs. Some echart viewers require a second descriptive file be created as well called (in analogy to above) filename.bsb.  OpenCPN does not require this; it just flashes a note that there is no information on the chart. The .bsb files are relatively easy to construct from the information in the header file. I will leave that till it comes up in a specific need.

For example, one of our favorite echart programs is Coastal Explorer from Rose Point Navigation. This program does require the bsb file.  These bsb files are most important for programs that are monitoring your charts to be sure they are up to date. They are also used in ECS functionality that links hotspots on the chart to announcements or corresponding Coast Pilot data and so on.

There is a note on how to make the header files here, but once you have one you can use it without knowing the details of how they are made.


§§§


Friday, January 8, 2016

How to Check the Accuracy of Airport Weather Reports

Barometers and the role of barometric pressure in marine navigation has been an ongoing focus of interest at Starpath since the mid 1980s. We make books on the subject, offer extensive online training in the optimum use of barometers, and we have even designed a state of the art barometer ourselves, which is now used by the NWS, Navies, and individual mariners around the world.

Thus it is no surprise that we appreciate related valuable contributions to this field. One that we have known about, praised, and used for many years is the subject at hand. It came up today in class when a student was following one of our course exercises on calibrating his own barometer using online pressure resources, in particular an airport pressure reported in the form of a metar.

Essentially all airports around the world offer pressure data in this format, which is available hourly  in plain language, giving sea level pressure in the form of QNH (called "altimeter" in these reports) or QFF (called "sea level pressure").  Both are pressures relative to sea level; the latter is closer to what the weather maps use, although these two pressures do not differ much unless the airport is high or the ambient temperature is extreme, or changed a lot in the past 12h. We have notes on this topic (and related one on temperature dependence of the conversions).

When we use one of these reports, or in fact any of the "official" reports, we tacitly assume the data are accurate, which is frankly—on the level we care about—not always the case.  Luckily, we have a very easy way to check this, which brings us to the Gladstone Family project run by Philip Gladstone.

The Gladstone Family website is a wealth of airport weather related information, but the part we address now is his monitoring or compiling of daily accuracy records of all the pressure sensors used in the metar reports around the world. Samples below are for an airport having some troubles at the moment and one that is doing just fine.


Two reports on pressure sensor accuracy from the Gladstone Family


It is always valuable to check his report when using one of the airport stations for barometer calibration data. I wish we had such a convenient service for the buoy and lighthouse data we get from NDBC.

For completeness I should also mention our own free service for doing something similar, using all official sources, including buoys and lighthouses, as well as airport metars.

You can read about how the Gladstone Family reported accuracy is determined. As I read this, they use  the same methods the NWS uses to evaluate ship and other station reports before incorporating the observations into the model forecasts.

Our free online service works differently.  It is located at starpath.com/barometers.  There is a detailed help file with an explanation of the process there.  Briefly, we do this:

You enter the Lat-Lon of your barometer, and we find the nearest 10 stations to you that have accurate pressure online. Then you look at a plot of their location relative to you, so you can decide which 4 or 5 (or more) offer a uniform distribution of references in all directions.

At this point you could just choose the closest one and use that, but you can do very much better.  Print out our form for entering  pressure data for this application. Then select at least 4 of them, and go to each one and record two times and two pressures that span a time you have a pressure recorded from your own barometer.  Once you have these 4 or 5 sets of data, enter them into the online version of the form and submit.

Our program will then normalize them all to a common time, and then do a least squares fit to a 3d pressure surface that spans this area, and then it drops a point at your Lat-Lon, which is equivalent to an average of all nearby sensors. This is an accurate sea level pressure value you can use for your calibration at a specific time.

But this is not yet analogous to the Gladstone Family service. So far we have only found an accurate pressure, but do not know its uncertainty.  The figure below is the final output from out method, and the numbers on the right are related to this question.


The bottom line is the best estimate of your SLP at the time given.  Pavg is the reported pressure at each station, normalized to the one common time. Pcomputed is the pressure we get from our computed surface at each station at this time. Delta is the difference between these two, which is a measure of how consistent that station is with its neighboring stations. If all stations agreed precisely, these deltas would all be zero.

In the example shown, we see that #1, 3, 4, and 5 are all consistent to within ± 0.5 mb, which is about all we can hope for.  #2 is questionable, but OK. The one that stands out is #6, off by more than 2 mb relative to the others.  When something like this is observed, we should just remove #6 from the list and do the pressure surface computation again, and generally you will see all deltas get smaller and the resulting pressure for your location will be better.... maybe not much, but better.

Thus we also have this way to check the accuracy of a pressure report,  but I will be the first to agree that the neat presentation of the Gladstone Family data is extremely valuable and we are grateful for the time he has spent on developing and maintaining that fine service.

To use the Gladstone Family service, look up the 4 letter metar code for the airport in question, ie  google "seatac metar" to learn it is KSEA, and then google "gladstone family ksea."

§§§

Friday, January 1, 2016

Article Categories

Weather
Using Buoy Data and BuoyCAMs to Study Passing Stor... (September 2018)
Vessel Icing: Resources and References (September 2018)
Near-live Ship Reports by Email (August 2018)
Free Barometer App Designed for Mariners (June 2018)
Hurricanes on the Route to Hawaii — Weather vs. Climate (May 2018)
Effect of Leeway on Knotmeter Speed (April 18, 2018)
OpenCPN, Quick Start, One Chart (April 15, 2018)
Introduction to Charts in OpenCPN (April 15, 2018)
Sadler Tropical Atlases (January 2018)
Decision Making in Weather Routing (January 1, 2018)
MSLP vs. MSLP (November 2017)
Global Warming and Tropical Cyclone Statistics (November 2017)
100-foot Waves Expected near Aleutian Islands (October 2017)
Getting Archived NDFD Data (September 2017)
Wind Speed from Beaufort Force Number (August 2017) 
BC Canada Marine Weather Guide Back in Circulation... (August 2017) 
Comparing RTOFS and NCOM Model Forecasts to HF Rad... (August 2017)
Fog and Smoke Epilogue... the next day (August 2017)
Local Weather Office Forecasts Smoke, but not Fog! (August 2017)
Forecasting Local Winds with Meteograms (August 2017) 
How to Add a Draw Style to LuckGrib (August 2017) 
How to Combine Grib Files (August 2017) 
Zone Forecasts by Email (July 2017)
NDFD: Oceanic, CONUS, and Regional (May 2017)
International List of Weather Services (April 2017)
ASTORIA TO POINTS NORTH — WEATHER PLANNING AND... (March 2017)
Buys Ballot Law to find wind direction from isobar... (January 2017)
Short Survey of Ocean Waves (January 2017)
Short Survey of Ocean Currents (January 2017)
Short Survey of Marine Weather (January 2017)
Wonderful Website on Barometers (December 2016)
GRIB Formatted Regional Wind Forecasts – Review an... (December 2016)
Wed Oct 12, 2016: Time to Start Your Barometer Cal... (October 2016)
Tropical Storm and Hurricane Advisories using FTPm... (July 2016)
Convert True Wind to Apparent Wind (July 2016)
Tropical Cyclone Advisories by Email Request (July 2016)
Wind Model Comparisons for Start of Pacific Cup (July 2016)
Vic-Maui Race Day 1, 2, (3) Winds (July 2016)
Inside Passage and R2AK Weather (June 3, 2016)
An Overview of Ocean Weather (May 18, 2016)
Great Lakes Wind Forecasts from HRRR (February 21, 2016)
Sailboat Racing on the Salish Sea—Part 1. Northern Waters (February 17, 2016)
Converting Weather Maps to BSB echarts (January 30, 2016)
Weather Maps—Where To Get Them and What We Get? (January 26, 2016)
Weather Maps to eCharts—Making the Header File (January 23, 2016)
How to load NOAA Weather Maps into OpenCPN (January 21, 2016)
How to Check the Accuracy of Airport Weather Reports (January 8, 2016)
How to Find Archived Weather Maps (December 19, 2015)
First pass at cell phone barometer calibration. (November 19, 2015)
Precision Barometers on Inland Commercial Vessels by Robert Reeder (November 6, 2015)
Weather Routing Options for Seattle to Victoria by Robert Reeder (November 6, 2015)
Personal Beaufort Scale (October 10, 2015)
Using a Portable Barometer to Set Ship's Barometer to SLP (October 8, 2015)
Manual of Barometry (WBAN) Now Online (September 29, 2015)
Grib Viewers — New Developments and Special Features (September 26, 2015)
Sides of a Tropical Cyclone, Part 2 Meteograms. (August 31, 2015)
Mariner’s Weather Checklist Before Departure (August 17, 2015)
High Seas Forecasts and Tropical Cyclone Alerts by Email Request (August 13, 2015)
Sides of a Tropical Storm (August 13, 2015)
New GRIB Viewer for Macs (August 2, 2015)
Direction of the Winds Aloft from Mare’s Tails — Time to Practice What we Preach (July 3, 2015)
Atlantic and Pacific Weather Briefings (June 19, 2015)
The National Digital Forecast Database (May 26, 2015)
Transpac Weather and Tactics by Stan Honey (May 17, 2015)
Weather Models for Transpac Sailors by Angeline Pendergrass (May 4, 2015)
TransPac Live Weather Resources (April 10, 2015)
Marine Weather Services Chart — How to Make Your Own. (March 19, 2015)
Reading and writing on weather maps (February 22, 2015)
Sydney-Hobart Climatic Winds (December 10, 2014)
Scatterometer Winds (December 2, 2014)
Taylor 1328PJ Sling Psychrometer — a Review (November 30, 2014)
Barometer for Wind Warning (November 7, 2014)
Barometric Pressure During Squall Passage (September 3, 2014)
Wind Near Clouds (March 5, 2014)
Model Predictions of World Winds (December 21, 2013)
Starpath Ship Reports Revised (December 17, 2013)
Local Pressure as a Squall Goes By (December 16, 2013)
Air Temperature Dependence of Sea Level Pressure Conversions (October 11, 2013)
"Point Four Four per Floor" –– QFE to QNH to QFF (August 3, 2013)
Mean Sea Level, Tides, and Barometers (July 29, 2013)
Weather by Satellite Phone (May 5, 2013)
Weather Information for Voyages in the Australian Region by Kenn Batt, BOM (April 7, 2013)
New way to look at ASCAT data (April 4, 2013)
A Comparision of Ocean Current Models (March 22, 2013)
Final Word on Relative Humidity and Dew Point (February 23, 2013)
A Real OMG Omega Block over the E. Pacific (January 12, 2013)
It is Raining – What does that mean? Part 3 (January 3, 2013)
It's Raining, What Does That Mean – Part 2 (December 25, 2012)
First Pressure Check on Lacrosse C86234 (December 25, 2012)
Tactical Use of Scatterometer Data (December 6, 2012)
Gill Pressure Ports (October 21, 2012)
Dew Point and Temperature vs altitude (September 25, 2012)
Relative Humidity and Dew Point as a Function of Altitude -- A Way to Estimate Cloud Ceilings (September 25, 2012)
The Buys Ballot Law (July 31, 2012)
Predicting ASCAT Satellite Pass Times (July 23, 2012)
Bermuda Race Satellite Winds (June 13, 2012)
How to overlay ASCAT Winds and Weather Maps on Google Earth (May 28, 2012)
ASCAT Notes and References (May 26, 2012)
It's Raining… What does that mean? (May 22, 2012)
How to learn the present winds offshore — West Coast of Vancouver Island, BC (April 19, 2012)
Str. of Georgia Wind Part 2 revised (April 5, 2012)
Southern Straits of Georgia Wind Forecast (April 4, 2012)



Celestial 
Finding Longitude From the Time of Sunset (September 2018)
USCG Deck Exam Questions at Starpath.com (July 2018)
Note on USCG Exam Questions (July 2018)
Finding Watch Rate (July 2018)
Dip Short, Distance Off, and Running Fix (June 2018)
Davis Mark 3 Sextant Part 1 — How to read the Angl... (June 2018)
Ways to Get Accurate GMT (UTC) (May 2018)
Pub. 249 Vol. 1, USNO, and OpenCPN (December 2017)
Why the Book "Hawaii by Sextant" is Unique (December 2017)
Analysis of a Celestial Navigation Sight Session (October 2017)
Celestial Navigators: Save Your Eclipse Viewing Glasses (August 2017) 
Refraction in Celestial Navigation–still an issue,... (February 2017)
Most Likely Position from 3 LOPs (July 2016)
Is the Moon Waning or Waxing? (May 5, 2016)
USCG Chronometer Time — Bless Their Hearts (December 9, 2015)
Latitude by Meridian Transit, Ex-meridians, and the USCG Cel Nav Exam (November 29, 2015)
The Navigation Triangle (November 29, 2015)
Ways to Get Accurate GMT (UTC) (April 1, 2015)
Telling Time by the Stars (March 2, 2015)
Nuts and Bolts of Successful Ocean Navigation (February 22, 2015)
Finding Longitude by Lunar Distance using the Stark Tables (June 3, 2014)
Hawaii by Sextant — An Overview of Celestial Navigation (May 9, 2014)
Fit-Slope Method — A Shortcut to the Manual Solution from 1856 (April 25, 2014)
The Navigator's Sun (March 25, 2014)
Molasses – The Eatable Artificial Horizon (February 7, 2014)
The Celestial Sphere. May It Rest in Peace. (February 6, 2014)
Checking your Compass with the Sun (October 14, 2013)
Why Study Celestial Navigation in the Age of GPS? (October 3, 2013)
Navigation Calculators Online (September 4, 2013)
How to spell FitzRoy... or is it Fitzroy? (June 11, 2013)
Finding Longitude from Sunset (April 27, 2013)
True Wind from Apparent Wind –– Revisited. (April 21, 2013)
Modern Barometry and its Important Role in Marine Navigation (April 19, 2013)
North Pacific Wind Statistcs (April 16, 2013)
Revisiting the Secret Sources (April 14, 2013)
Finding the North Star (March 2, 2013)
UTC by Lunar Altitudes (Celestial Navigation) (January 20, 2013)
Great Circle Sailing by Sight Reduction (December 13, 2012)
The direction of sunrise and sunset—the old fashioned way (October 31, 2012)



Electronic Chart Navigation
Chart Symbols: Rock or Coral on RNCs and ENCs (August 2018)
Paper Charts vs. Electronic Charts — Some Thoughts... (May 2018)
OpenCPN, Quick Start, One Chart (April 15, 2018)
Introduction to Charts in OpenCPN (April, 2018)
Minor Light Symbols on ENC in OpenCPN (September 2017)
Forecasting Local Winds with Meteograms (August 2017) 
How to Add a Draw Style to LuckGrib (August 2017) 
How to Combine Grib Files (August 2017) 
Simulate Instrument Data with OpenCPN's NMEA ... (August 2017)
Converting a Google Earth Path to a GPX Route (April 2017)
Rock Talk 2: RNC to ENC (November 2016)
Rock Talk — Is it all awash, or not? (November 2016)
History of Electronic Charting (November 2016)
How to Master Electronic Chart Navigation with the... (October 2016)
Network Connections to Navigation Software (October 2016)
Great App Idea from UKHO (October 2016)
How to Buy, Register, and Download Canadian Echarts (March 12, 2016)
Naming and Boundary Conventions on ENC eCharts (March 6, 2016)
The Strange World of Electronic Chart Boundaries (February 26, 2016)
Viewing and Downloading Nautical Chart with Google Earth (December 14, 2015)
Commercial Echart Ad is Food for Thought (August 26, 2015)
Don't Blame eCharts for Anything (December 8, 2014)
ECS Without GPS (January 20, 2014)
Free or Demo eChart Viewers (September 1, 2013)
Canadian eCharts for Electronic Chart Systems (August 4, 2012)


Coastal
Race to Alaska (R2AK) Navigation (June 2018)
Weems & Plath Expanding Square SAR Course Identifi... (June 2018)
Paper Charts vs. Electronic Charts — Some Thoughts... (May 2018)
OpenCPN, Quick Start, One Chart (April 15, 2018)
Introduction to Charts in OpenCPN (April, 2018)
Compass Bearing Fix — An Overview (November 2017)
UW Soundings Generator (August 2017)
Light Lists—A Modern Look (July 206)
Navigation Lights—The R2AK Short Course (July 2016)
Chart Plotting Tutorials (May 28, 2016)
Introduction to AIS (February 20, 2016)
Geographic Range — Time to Get Trivial (October 1, 2015)
Boxing the Compass (February 27, 2015)
Piloting Practice with Geocaching — You Can Do It Online (March 24, 2014)
Google Earth as an Aid to Chart Reading (December 27, 2013)
Print-on-Demand Charts and pdf Charts (October 26, 2013)
Checking your Compass with the Sun (October 14, 2013)
Tides in Puget Sound (August 8, 2013)
Small Craft Wave Danger as a Function of Wave Steepness (April 9, 2013)
How to make current stations into waypoints in a batch process (April 6, 2012)
Natural Ranges—How accurate are they? (April 4, 2012)



Nav Rules
USCG Deck Exam Questions at Starpath.com (July 2018)
Note on USCG Exam Questions (July 2018)
Showing the Right Day Shape (August 2017)
Can exceptional behavior be considered ordinary? (June 13, 2016)
Right of Way in the Traffic Lanes (March 14, 2016)
Quick Way to Know if you are in the Traffic Lanes (February 12, 2016)
The New Inland Navigation Rules (July 28, 2015)
Lobster Pot...my foot. (July 29, 2014)



Miscellaneous
AP Release on Tragic Loss of El Faro is Misleading. (October 5, 2015)
Ways to Get Accurate GMT (UTC) (April 1, 2015)
Telling Time by the Stars (March 2, 2015)
How a big, well-run, high-tech race boat, can go aground... almost (December 4, 2014)
Rotary Currents (September 1, 2014)
Tricky Terms in Navigation (May 20, 2014)
Worldwide Shipping Lanes (December 6, 2013)
One Rombe Wrong, and What Do You Get? (November 22, 2013)
Notes on Loran (November 10, 2013)
Open Letter to the America's Cup Rules Committee (August 27, 2013)
Answers to Geocaching Exercise (July 14, 2013)
New Starpath Kindle and iBooks now online (July 3, 2013)
Ebooks Onboard (June 8, 2013)
A Few Nuts and Bolts of Iridium SatPhone Usage (May 4, 2013)
How to Plot with Triangles (April 25, 2013)
High Definition (HD) and Broadband Radar by Larry Brandt (April 8, 2013)
Two New Books from Starpath (April 2, 2013)
Course Boxes - Still Valuable in Modern Sailing (March 12, 2013)
More mainstream media on Viking sunstones (March 7, 2013)
Obtaining Ocean Current Data Underway (February 10, 2013)
State of the Art Ocean Current Presentation (January 31, 2013)
Atlantic Ocean Current Data and 6-day Forecasts (January 12, 2013)
Ocean Currents (January 8, 2013)
A Local Mini Surge (December 17, 2012)
NOAA Chief Misspeaks on Sandy Surge (December 14, 2012)
OPC maps for Google Earth (November 2, 2012)
Track of HMS Bounty and the NHC Forecasts (November 2, 2012)
Leif Karlsen's work on Viking Sunstones (August 23, 2012)
Timekeeping in Navigation and Weather (July 21, 2012)
How to Print a Degree Symbol (June 11, 2012)
A Nice Thermometer Dial—tiptoeing onto the metric scale (May 31, 2012)
Refraction in a sink (May 21, 2012)
GPS Accuracy in Tight Quarters (April 12, 2012)
Route Planning and Sharing with gpx Files (March 31, 2012)
Robin Hood Complex in Book Publishing (March 22, 2012)