Monday, July 8, 2019

FTPmail No Longer Includes Observations—Plus Alternatives

For decades, mariners had a convenient way to request near-live wind, pressure, and sea state observations from buoys, as well as data from land based stations. This was achieved via the NWS FTPmail service, accessing data in the folder


Using standard FTPmail protocol, we could then send an email request to ask for a file such as 51001.txt (a HI buoy), wpow1.txt (West Point Lighthouse, near us), or any other buoy or station operated by NDBC.  This would provide a short text report of the latest data from that station. These were updated every 10 minutes.

The process was explained along with a list of the stations in a file called buoydata.txt. The file is still there (at least for now), but the first part that explains the process is no longer valid.  The list of stations, remains useful, but the notes on "present conditions" is long out of date.

We learned today, however, that the NDBC is no longer updating the data files in the FTPmail folder. It seems they stopped doing this, more or less without notice, at the end of 2018.  This source was still listed at the Ocean Prediction Center until today, when they removed the information at our request because this source is permanently gone.

We are trying to learn why NDBC discontinued transfering the data to the FTPmail folder, ending access via FTPmail, and now post it only on their own server, without any direct reference to it.

Going forward... 

We can still get these data by email request, but we have to go directly to the NDBC for it, and we have to rely on the good help of saildocs.com, a non-governmental organization of sailors.

We have several options, all of which are tied to the new folder the NDBC is using to store the files, namely:

One method is similar to what we historically got from FTPmail, namely latest report for just one station. These would be the same file names we used in FTPmail, namely, for example, wpow1.txt or 51001.txt, which include the following.
Station WPOW1
47° 39.7' N 122° 26.1' W 5:00 pm PDT 07/10/19
0000 GMT 07/11/19
Wind: S (170°), 14 kt
Gust: 15 kt
Pres: 30.05 steady
Air Temp: 65.1 °F
Dew Point: 61.2 °F 
and 
Station 51001
24° 27.2' N 162° 0.0' W 2:20 pm HAST 07/10/19
0020 GMT 07/11/19
Wind: E (80°), 9.7 kt
Gust: 13.6 kt
Pres: 30.07
Air Temp: 81.0 °F
Water Temp: 82.0 °F
Dew Point: 76.8 °F 
Wave Summary
1:40 pm HAST
2300 GMT 07/10/19
Swell: 5.6 ft
Period: 12.1 sec
Direction: NNW
Wind Wave: 1.3 ft
Period: 3.7 sec
Direction: ESE

This is the same as what we got before, but now there is no longer any government email source for observations—a giant step backwards in that observations are crucial to planning and email is the most common source of weather data offshore for a vast number of vessels. To get these observations by email we need to send a mail to query@saildocs.com with the following message

send https://www.ndbc.noaa.gov/data/latest_obs/wpow1.txt
send https://www.ndbc.noaa.gov/data/latest_obs/51001.txt

Then we get back two emails with the above text reports. You can ask for just one or multiple ones. It seems that the update times are not consistent from one station to another. The wpow1 lighthouse updates only every hour at this link, whereas the 51001 buoy in HI updates every 10 minutes. 

You can also put this link in your phone


and you have access to something very similar, if you have cell phone connections.

Or...  just use Dial-a-Buoy on your phone and find the station by Lat-Lon: call 888-701-8992‬ and follow instructions. Every navigator should have that phone number in their address book!


Alternative Formats of Observations data

Most recent observations from all stations updated every 5 minutes are available at this link


A sample of this file is below


This file is about 95 kB; it gives only the latest report, but lists all NDBC stations.

We can also get all data from a specific station updated every 10 min at this link


where we attach the station name at the end, such as 5100.txt. A sample is shown below. Some stations only have data every hour; others every 10 min with wave data maybe every hour.


The data go back 45 days.  File names are in caps (if letters are included) for this folder, but lower case in the other folder, and that matters for this download or email request.  A typical file size is 104 kB, which is clearly larger than we want by satphone.

Alternatively, we can ask for  a continuous-wind file from the same folder, such as

https://www.ndbc.noaa.gov/data/realtime2/WPOW1.cwind

#YY  MM DD hh mm WDIR WSPD GDR GST GTIME
#yr  mo dy hr mn degT m/s degT m/s hhmm
2019 07 08 22 00 160  2.6 150  3.6 2104
2019 07 08 21 50 160  2.6 999 99.0 9999
2019 07 08 21 40 157  2.1 999 99.0 9999
2019 07 08 21 30 158  2.6 999 99.0 9999
2019 07 08 21 20 164  2.6 999 99.0 9999
2019 07 08 21 10 157  3.1 999 99.0 9999
2019 07 08 21 00 156  3.1 160  3.6 2047
2019 07 08 20 50 162  3.1 999 99.0 9999
2019 07 08 20 40 165  3.1 999 99.0 9999
2019 07 08 20 30 158  2.6 999 99.0 9999
2019 07 08 20 20 166  2.6 999 99.0 9999
2019 07 08 20 10 171  2.6 999 99.0 9999
2019 07 08 20 00 165  2.6 170  3.1 1958
2019 07 08 19 50 167  2.6 999 99.0 9999
2019 07 08 19 40 163  2.6 999 99.0 9999
2019 07 08 19 30 157  2.6 999 99.0 9999
2019 07 08 19 20 158  2.1 999 99.0 9999
2019 07 08 19 10 161  2.1 999 99.0 9999
2019 07 08 19 00 163  1.5 160  2.1 1859
2019 07 08 18 50 178  0.5 999 99.0 9999
2019 07 08 18 40 268  0.0 999 99.0 9999
2019 07 08 18 30 308  0.5 999 99.0 9999
2019 07 08 18 20 000  0.0 999 99.0 9999
2019 07 08 18 10 319  0.5 999 99.0 9999
2019 07 08 18 00 326  0.5  40  1.5 1705
2019 07 08 17 50 328  0.5 999 99.0 9999

Here we get only wind data from only this station. This file lists the wind every 10 minutes at the station, updated every 5 min.  The file size is about 260 kB.  This is a text file, but without the standard .txt extension.  Once downloaded, highlight the file, and say open with Notepad (OC) or TextEdit (Mac).

Remember we can get any of these reports by email request to saildocs, i.e., send a mail to query@saildocs.com with one or all of these messages

send https://www.ndbc.noaa.gov/data/latest_obs/wpow1.txt
send https://www.ndbc.noaa.gov/data/latest_obs/latest_obs.txt
send https://www.ndbc.noaa.gov/data/realtime2/WPOW1.cwind

______________

The parameters for all of these files are explained at this link https://www.ndbc.noaa.gov/measdes.shtml

You can find  the best stations for your application most easily with their graphic interface at the the home page: https://www.ndbc.noaa.gov/   then make a list of the ones you care about.

Alternatively, If you do not know the name of a station, you can find available ones based on Lat and Lon at https://www.ndbc.noaa.gov/radial_search.php?

A list of stations and what products they have is here: https://www.ndbc.noaa.gov/data_availability/data_avail.php

Don't forget our powerful free email service to get live ship reports.  See starpath.com/shipreports. This provides all reports within 300 nmi of a specified Lat, Lon over the past 6 hr.

The navigation program Expedition has a unique feature of downloading and displaying all the latest buoy and ship reports, but they include only the latest data. Seeing the sequence of these is often instructive. The GPX file of reports we create in Starpath Ship Reports can be imported to any nav program.





Lat-Lon from Street Address

There is an online service that finds your Lat Lon based on a conventional address.   That service is found at


We have tested this for our own location, which was informative.  Here is the input

We then compared this to our actual location on Google Earth, which is shown below


We see that 3 of them are off quite a bit (143 yards east of our true position), but the other two are very close.  The Open Street Map solution is right on top of our front door at Starpath HQ.

We leave it as an exercise* to show than an equivalent international address for our front door is

///from.sticky.clip

My actual desk is located at ///agrees.pumps.libraries.  So much better than Lat-Lon—especially if your address is something like Wiggley Farm, Batchley.

________________________

* Hint.


Thursday, June 27, 2019

Expanding Scales on Universal Plotting Sheets

Universal plotting sheets (UPS) are intended for plotting sextant fixes along with DR track plotting (we cover their use in Celestial Navigation), but they are also useful for marine weather routing as we show in Modern Marine Weather.  These sheets could also be used for solving standard maneuvering board problems if needed, but there are sheets better suited for those.

A typical plot of an ocean track is shown below using the default scale of 60 nmi per printed latitude parallel. The longitude scale is set up using the inserted diagram, after marking the central Lat on the Lon scale of the diagram.



An ocean voyage proceeds as a sequence of fixes with DR between them. These real sextant sights are from our book Hawaii by Sextant. We will be looking specifically at Problem 22, which is zoomed into below.




This is a good fix, but we can't tell how good using the standard scale for the plots shown here. We can expand the standard scale by redefining the Lon diagram. Below is how it would change if we go from 60 nmi per parallel to 30 nmi per parallel.


Or we can expand the plots to 6 nmi per parallel as shown below.

This is the type of expansion we used in practice during the 1982 voyage shown above. A typical procedure might be to do a quick fix with just two or three sights to find an approximate center point, and then create the new plotting sheet centered about that point using 6 nmi per parallel.

At the time of the voyage, we chose 24º 39'N, 151º 16' W to plot from, and below is an image of the actual plotting sheet from that voyage, 37 year ago.

You can see that we had a good fix.  Refer to Hawaii by Sextant to review our Fit Slope Method in action along with our automated LOP advancement technique, which is required for best accuracy. This makes a set of 15 sights of four bodies an extended running fix over a 30 minute sight session to get equivalent sights at one time.  Both yield notable improvements to the multiple sights analysis. In this sight session, Capella was the weakest with just one sight; the other three had at least 4 each.

Here is a short video discussion of the above process:






Friday, June 14, 2019

How to Obtain Custom GRIB Files

We usually try to get all of the GRIB files we need from within the navigation program or GRIB viewer we are using.  This solves a key issue of being sure that the ones they let us request will indeed be viewable in their viewer.  All such sources of GRIB data offer a filter for Lat-Lon region and desired parameters, as well as resolution, and time intervals and duration (ie every 3h out to h72).

Some navigation programs, such as Expedition, have many of the top model products coded into their system so there is less reason for getting external data, others have limited products, and it is not uncommon that popular charting software might only offer GFS parameters internally, maybe with the addition of RTOFS for currents and WW3 for waves.

The popular OpenCPN can create email requests to Saildocs, but the packaged interface options do not include many important models Saildocs offers. OpenCPN can view some grib2 files, but not all. Some of the new high-res data come in a Lambert projection which takes new programming.  There are also new parameters to be included. The forthcoming version of OpenCPN (newer than present 5.0.0+9065270) will include simulated radar (composite reflectivity, REFC), which is available in several WRF models, as wells as the NBM, and now globally in the new GFS version that came into effect on June 12, 2019.

REFC is not yet available as part of a GFS request to Saildocs, but we have other ways to get the data underway from Saildocs, as shown below.  I understand that REFC will be added as a request parameter in XyGrib in July or August. XyGrib will show the files if you get them elsewhere, i.e., as outlined below.  It is already available in the European WRF models from https://openskiron.org/en/.  We can also get this parameter from several models using LuckGrib, and then export the file to be loaded into a navigation program. LuckGrib has the high-res regional versions of REFC as well as the global.

Sample of REFC over SW Pacific viewed in LuckGrib

The notes below go beyond the new REFC parameter and show how to request GRIB files in general directly from their primary source, NCEP.  The link we start from is:


The procedure is outlined below, followed by a video link demonstrating the process. 

Step 1. The main link takes you to a list of many models, with various ways to access the data.  We want to use here the "grib filter." The header name is also a link to the process we are outlining here. The other data sources are huge files of global coverage and hundreds of parameters.

Step 2. Choose a model.  For now we will use GFS 0.25 degree.  This is now the new GFS, and it will as of yesterday include the RECF.  Click "grib filter" for that one.

Step 3. This takes us to an index of data files, and we note (at least for now) that there is a format change yesterday when the new GFS went online.  We have to choose one newer than 20190613.  This is yyyymmdd. We choose 20190614.

Step 4. The next page shows the synoptic time computations: 00, 06, 12, 18. Normally we would want the latest model run. Let's say the latest is 12z. That will be the top one on the list.  Some hours  after 18z, the 18z file will appear at the top of the list.  (More on time delays and latency later on.)

Step 5. Now we are on the request page, where we select, valid forecast time, parameters, levels, and Lat-Lon region. At every synoptic time, there will be 364 forecasts made. Let's assume we want every 6h starting at the 12z run for the next 24h.  Thus we want 12zf000, 12zf006, 12zf0012, 12zf0018, and 12zf0024. This is 5 files and we have to ask for these individually.



Parameters

Let's say we want wind, pressure, and simulated radar. We could get others, as discussed below.

For the wind we ask for UGRD, VGRD, which are the E-W and N-S components of the wind; the grib viewer will make these into a wind vector.

For the Simulated radar we ask for REFC

For the pressure we could ask for PRMSL, which is the common one used, or we could ask for MSLET, which is actually a more precise pressure parameter, but some grib viewers might not be set up to recognize this.  We do not want PRES, which is a parameter used for weather on land.

Other useful parameters are discussed below.



Levels

For wind we need "10 m above the ground."  For PRMSL we need "mean sea level."  For REFC we need plain "entire atmosphere" and not "entire atmosphere (considered as a single layer)."



Step 6. After selecting a file, parameters, and levels—you will see online that they present this in a different order—put a check in the box "Make a sub region" and then enter the bounds. Lon West is negative (-);  Lat South is (-). We will get Cape Verde region for a small sample, -30 to -14, and 20 to 10.

Step 7. Put a check in the Box "Show URL only for web programming," then press enter and you will get this link:

URL=
https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t12z.pgrb2.0p25.f000&lev_10_m_above_ground=on&lev_entire_atmosphere=on&lev_mean_sea_level=on&var_PRMSL=on&var_REFC=on&var_UGRD=on&var_VGRD=on&subregion=&leftlon=-41&rightlon=-16&toplat=40&bottomlat=26&dir=%2Fgfs.20190614%2F12

If you are on land with an internet connection, you can then take away the "Show URL..." check mark, then when you press Download you will get the file called: 

gfs.t12z.pgrb2.0p25.f000

At this stage you might want to add a .grb to the file to get  gfs.t12z.pgrb2.0p25.f000.grb, then you know what you have, but this is not necessary. We have shown that both OpenCPN and XyGrib will open the files as downloaded without the .grb added, or with it.

This file will then load into OpenCPN, but you won't see the REFC until the next version of OpenCPN. The latest version at the moment  (5.0.0+9065270) does not have the REFC implemented in the GRIB plugin, but the developers have it working in the next version, as shown in the video.

To get these files from Saildocs when underway you would send a mail to query@saildocs.com with the message being

send https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t12z.pgrb2.0p25.f000&lev_10_m_above_ground=on&lev_entire_atmosphere=on&lev_mean_sea_level=on&var_PRMSL=on&var_REFC=on&var_UGRD=on&var_VGRD=on&subregion=&leftlon=-41&rightlon=-16&toplat=40&bottomlat=26&dir=%2Fgfs.20190614%2F12

send https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t12z.pgrb2.0p25.f006&lev_10_m_above_ground=on&lev_entire_atmosphere=on&lev_mean_sea_level=on&var_PRMSL=on&var_REFC=on&var_UGRD=on&var_VGRD=on&subregion=&leftlon=-41&rightlon=-16&toplat=40&bottomlat=26&dir=%2Fgfs.20190614%2F12

send https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t12z.pgrb2.0p25.f012&lev_10_m_above_ground=on&lev_entire_atmosphere=on&lev_mean_sea_level=on&var_PRMSL=on&var_REFC=on&var_UGRD=on&var_VGRD=on&subregion=&leftlon=-41&rightlon=-16&toplat=40&bottomlat=26&dir=%2Fgfs.20190614%2F12

You could request as many as you like. These are about 56 kB each as returned by regular email (~twice actual size).  Using special satcom email they could get reduced quite a bit. They will come back as individual emails with the files attached.

The video below shows how to get these by email, and then view them in XyGrib and in a forthcoming edition of OpenCPN, along with some discussion of this new REFC parameter.  

With this method of requesting specific parameters and levels, navigators may also find it useful to look into the 850 mb level for both wind and relative humidity (RH). This is the boundary layer level at about 1000 m, where the winds become less, if at all, affected by surface friction. About the altitude of low clouds. These winds are a good indication of the direction of squall motion (compare with REFC forecasts), and in the tropics, the streamline view at this elevation can give indication of wind flow on the surface that does not show up in the actual wind data. We watched a prominent example of this off the coast of Queensland, AU earlier this month. The RH at this level can also show a good outline frontal systems, which are not always easy to discern from wind and pressure.  

____________________

To obtain a HRRR CONUS  wind and pressure (surface analysis, h0 forecast = f00) on Feb 4, 2020 at 18z from saildocs, it would be:

send https://nomads.ncep.noaa.gov/cgi-bin/filter_hrrr_2d.pl?file=hrrr.t18z.wrfsfcf00.grib2&lev_10_m_above_ground=on&lev_mean_sea_level=on&var_MSLMA=on&var_UGRD=on&var_VGRD=on&subregion=&leftlon=-131&rightlon=-120&toplat=42&bottomlat=33&dir=%2Fhrrr.20200204%2Fconus

Adjust Lat, Lon and times as needed

____________________





Video demo of getting the files.




Video demo of viewing REFC in XyGrib.




Video demo of viewing REFC in OpenCPN.


On the list is to add a video devoted to the practical use of REFC and data we have to confirm it... which is easy to test because we have archived real weather radar online.... and  you can go to squally waters in real radar view, forecast some pattern and then see if it pans out.

We also want to add a short article and video note on use of the 850 mb data, as outlined above, which you can obtain with the methods outlined here.