Getting Archived NDFD Data

Noted added July, 2023. The NDFD oceanic data have been discontinued for some time now. The counterpart would be the National Blend of Models, Oceanic, which should incorporate the information of the previous data set.  NDFD CONUS is still available.

Usually we want to look forward, not backward, but to analyze a past event we may need to look into historic data. We come across this more often than most mariners would as we prepare training materials, looking for good examples. 

Fortunately, these datasets are all stored and we can get to them with normal tools. We look here first at the NDFD data as it is more complex than other data types—weather maps, GFS, RTOFS etc, are much easier. I record this here because I had learned this once, and then forgot it, and had to start over, and I do not want to repeat that.

First, the question is how far back do we need to go.  The real time source provides only the latest run, so this data expires every hour and we need special procedures to look back in time, even if just hours. There may be locations for intermediate data that I do not know about, but for now it seems either get the latest within the hour or wait till it moves to the archives. With maps, on the other hand, we have a full day.  We can get, say, the 06z map anytime until about 09z the next day, when it gets overwritten with the next run. 

Procedure for older NDFD data. 

(1) https://www.ncdc.noaa.gov  This was national climatic data center (NCDC), but is now National Centers for Environmental Information (NCEI).

(2) Click: Access data / Models / Datasets / NDFD / FTP / (There are other access points, but this one seems the most direct:  ftp://nomads.ncdc.noaa.gov/NDFD/

[ If you choose TDS you get to a catalog page https://nomads.ncdc.noaa.gov/thredds/catalog/ndfd/catalog.html  that will load into Panoply as a catalog, which is then easier if you use that wonderful free program. ]

(3) Then select the year and month (ie 201708) / then select the day, ie 20170829, which happens to the newest in the list going back to 2008. Today is 0902, so we note it takes 4 days for data to find its way into the archive.

This is another catalog page, that shows the individual files... all of them, CONUS, Oceanic, Regional etc, and as noted earlier the wind speed and wind direction are in two different files.  There is no pressure, but a lot of other parameters helpful with convection forecasting.

(4) Now we need to know which file to take, and we learn that from the file name structure, which all makes sense (in a sense) based on this page of conventions... except for the 99, which could be 98 or 97; I have yet to decode that part of the file name.

For conus we want the letter U, and for wind direction we want a B. These file names start  YBU

      ybuz9xkwbn   corresponds to  ST.expr/DF.gr2/DC.ndfd/AR.conus/ds.wdir in the live data.

For wind speed we want a C, or files starting with YCU.

     ycuz9xkwbn   corresponds to  ST.expr/DF.gr2/DC.ndfd/AR.conus/ds.wspd in the live data.

Sample files would be:

            File name                                         Start Time

YCUZ97_KWBN_201708292331        2017-08-30T00:00:00Z

YBUZ98_KWBN_201708290016        2017-08-29T01:00:00Z

YCUZ98_KWBN_201708291847        2017-08-29T20:00:00Z

YCUZ98_KWBN_201708291315        2017-08-29T14:00:00Z

In other words, the last 4 digits in the file name is a UTC, and the first forecast of the dataset will be the next whole hour. The only way I presume that is from having opened several of these. 

In files checked so far, 9x=97 on wind direction is about 10 MB with 16 forecasts at 6-hr steps starting at the date, whereas a 98 is about 32 MB with 48 forecasts in 1-hr steps. For wind speed, the difference is 16 MB vs. 47 MB. I have to sort out the 97, 98, 99 significance, and will add it when learned.  I think we always want the 98.

Note the files have no extension, but they will open in LuckGrib as is, or you can add an extension .bin to load them into Expedition. For the latter you can just add both, and then select show both in Weather Set-up to get wind vectors... or just drag them onto the chart and they load automatically. For LuckGrib we need to add the files externally as explained elsewhere and then load them.

I will come back shortly to illustrate this process by recreating the wind, sea, and current conditions at some recent maritime event of interest.