NDFD: Oceanic, CONUS, and Regional

Noted added July, 2023. The NDFD oceanic data have been discontinued for some time now. The counterpart would be the National Blend of Models, Oceanic, which should incorporate the information of the previous data set.  NDFD CONUS is still available.

In an earlier article (NDFD – The National Digital Forecast Database) we reviewed the use of numerical weather prediction model forecasts in GRIB format and how the NDFD forecasts differs from those. Namely, the NDFD is the digitized version of the NWS forecast, the combined work of local forecast offices, OPC and NHC, and as such should in principle be better than the results of any one unvetted model prediction. Here we add details of the various formats available for this data, some of which are just now becoming available to mariners in a convenient manner.

And this early note: this is wind (and waves) only data, no pressure—"waves" in NDFD means significant wave height of the combined seas. When pressure is needed, the NAM data will likely be best source for that, or HRRR where available. Also the present discussion is a pursuit of the best possible data, which might not always differ significantly from other good sources for practical routing applications, but we will not know that till we start using it, or at least start comparing it with other sources. 

When we get the NDFD data, it loads into our GRIB viewer or navigation program just as any model prediction might, but it pays to remember that this is not a model prediction, and it is a very complex dataset.  To say, for example, "I am using the NDFD forecast for my routing" does not adequately specify what exactly you are using. There are several options, which differ notably. Likewise,  having a source of weather data that simply specifies "NDFD" does not answer that question. We would need to look at that wind data specifically to see what it actually is, keeping in mind what the options are.  The goal here is to outline options available and how they differ.

There are three basic categories or sectors of NDFD data: Oceanic, CONUS (continental US), and Regional. The coverage areas of these categories are shown in Figure 1, adapted from the NDFD  website. 

Figure 1. NDFD sectors. Purple areas are the NDFD Oceanic coverage. Red inserts mark the CONUS, and three regional areas. There is also a small regional area around Guam. Where sectors  overlap, there are two sets of data available, with the CONUS or regional being the higher res and fresher forecasts.

Table 1 shows the specs for the data of these sectors, with important notes to follow.

All of these data sets are available in GRIB2 format directly from NOAA's Meteorological Development Laboratory, but it is a two-step process to get them that way. Various parts of the full set are available from Saildocs, LuckGrib, and Ocens. We come back to sources after looking at specific regions covered and associated notes. There is, for example, a 5-km set of CONUS data available, which is not in the Table as I cannot see a role for it. Most GRIB downloaders let us select the resolution we want, which is effectively downsampling the higher-res data.

The frequent updates of the CONUS data may not affect your specific data set. The full national set is updated every 30 min, but any one local area might not have updated in that time period, so the area you care about could be new only every hour or so, or maybe longer. The regional sets are in principle different, as they are controlled by their individual weather offices. 

Also the frequent updates seems to periodically affect the time steps where it is changing from say 3h steps to 6h steps. You may have a 4h step at the transition, or a 2h step. It will be up to the display or routing software to note this and handle it appropriately.  I have also seen individual steps missing at times. In short, if something does not look right, try downloading it again, or wait 30 min and do so. Ultimately the datasets are controlled by the local weather offices and perhaps even by the specific forecaster in charge at the time. The process of how the dataset is constructed is explained in this AMS article from 2002 that introduced the program... noting it was the first of its kind in the world.

When we have a choice of CONUS or oceanic, then the CONUS would be the obvious choice as it is four times higher resolution and is updated far more often. If your source does not identify the NDFD sector, then chances are it is oceanic. Saildocs and Ocens, for example, so far has only the oceanic sector. I anticipate that this data will become more readily available to mariners in the not-too-distant future, but there are a few hurdles to cross, as noted later.

In practice, the precise locations of the boundaries could be crucial. The oceanic region, for example, just makes it out to Bermuda as shown in Figure 2. 

Figure 2. Showing the corner of the oceanic NDFD near Bermuda

On the route Marion to Bermuda, for example, you could use the superior CONUS for most of the route, then switch to oceanic, as shown in Figure 3. 

Figure 3. Boundary between CONUS and oceanic coverage on a Bermuda route

The boundaries of the HI regional data and the HI oceanic data are the same (Figure 4), but the specs are quite different as shown in Table 1.  As of this writing, the regional HI data has to be extracted manually, as explained below... and again, we have to test this to see if that extra work is worth it.

Figure 4. HI Regional and oceanic coverage. In the oceanic data set the resolution is 5.4 nmi; in the regional data set resolution is 1.35 nmi.

The CONUS coverage is rather awkward in the Pacific Northwest as shown in Figure 5. The data end right at the Canadian border, in the middle of Strait of Juan de Fuca. So alternatives such as NAM or HRRR could be more convenient.

Figure 5. CONUS limits in the Strait of Juan de Fuca and San Juan Islands. The oceanic database has the same boundary in this region. Note that the HRRR data (updated hourly) does cover the full Strait and San Juan Islands, and is likely as good, but it goes out only 18h.

To help plan the potential use of the NDFD, we have made gpx files that mark the boundaries of the regions. These can be loaded into any navigation program to outline where the data are available, then just shown as needed. Download from this link:

 NDFD_gpx_boundaries.zip.

These gpx files are shown loaded below loaded into OpenCPN, followed by a related import to Expedition. OpenCPN will not yet display the regional or CONUS data (grb2 format), but it will show the oceanic sector provided by Saildocs, who have converted the grb2 to grb1.

Figure 6. NDFD boundaries imported into OpenCPN as a set of gpx files. Red is CONUS; blue oceanic.

Figure 7. Detail of Figure 6 at the approach to Strait of Juan de Fuca. Red is CONUS.

On US Inland waters, the CONUS and oceanic limits are the same, but in the ocean they differ in this region. 

Below shows these boundaries imported to Expedition as race notes. Download the link below (the borders made into Race Notes format), and save to c:\Program Data\Expedition\RaceNotes. Then in the Sail Panel, Race Notes section, Save all of your existing notes, clear the notes, then choose Open. Once imported, use Edit Notes to maybe change color of CONUS to highlight difference from oceanic; on the Oceanic note, change to rhumb lines rather than great circles; then lock them all, and save the set. Download from this link:

NDFD_borders.dat

Figure 8. NDFD regions displayed as race notes in Expedition. CONUS data are created in a Lambert projection (rather than mercator), which accounts for the shape shown. There are data on land for the CONUS sector—much better and more detailed than our local TV weather!—but there are no land data in the oceanic sector for Mexico, South America or Africa. These borders follow the coast lines, though not shown that way above.

Below shows a comparison of the CONUS and oceanic data for San Francisco Bay area.

Figure 9. Top, CONUS (2.5 km updated every 30 min). Bottom, Oceanic (10 km updated every 6 hr). Note there is no interpolation in either display; one wind arrow per grid point. 

Even though the winds in the early hours of a forecast might be very similar with just higher res in the CONUS, at later hours, the CONUS could be far superior if the forecast changes. The oceanic dataset cannot acknowledge such a change for 6h.  This can make a big difference with frontal passage anywhere, and could be really crucial in the Atlantic when tropical storms move north along the Gulf Stream. Recall that the National Hurricane Center (NHC) are contributors to the NDFD, so I would hope that their datasets include all information from the latest Advisories. (This will be a good research project the next time we have a TC moving north in the Atlantic. If there are any volunteers, let us know and we can explain how to set this up for the comparison.)

In principle, the h0 "forecast" of either dataset should be effectively a surface analysis at the time of issue, incorporating latest ship and other reports. This too should be easy to test by comparing the CONUS and Oceanic at synoptic times with the graphic maps of the surface analyses. Likewise, at these times, we can also compare the NDFD h48 and h96 forecasts with the graphic forecasts from the Ocean Prediction Center (OPC).

Sadly, there are no pressure forecasts in the NDFD. I am still trying to learn why. It would seem that if they can forecast anything, they could forecast pressure. I think the folks in charge do not appreciate how valuable pressure is to mariners. It is our best way to evaluate and monitor a forecast and indeed the actual weather. In contrast to wind speed and direction, it is easy to measure very accurate pressure.  When I learn more on why it is not included, I will edit this paragraph. Pressure forecasts have been a crucial part of maritime weather forecasting since the very concept of forecasting was invented by Robert FitzRoy, some 150 years ago.

The Nitty Gritty

First a minor detail. All the NDFD original data describe the wind provided as at the "surface," just as the WW3 model wind forecasts are presented. In contrast, all other models and txt forecasts give winds at 10m, and then we use various algorithms to convert that wind speed to other elevations in the optimum routing process.

Well... that is a bit misleading, and it is fair to assume that all reported winds are normalized to 10m.  Why the GRIB parameters for these use "surface" rather than "10 meter" is not clear, but here is the statement on WW3. The wind in the NDFD is also not on the surface, despite that statement in the grib files themselves. NDFD winds are indeed at 10m (see NDFD element definitions), as we might guess they would be.  This is just another example of how the government supports navigation schools.

NOAA provides the NDFD data in grib 2 format (see sector specs). There are not yet many grib viewers or navigation programs, however, that accept wind data in grb2 format.  Expedition is one popular navigation program that does support grb2, which means it will load and display the native CONUS data, but I do not know of others so far that will do so. If others are known, please post a comment with the info.

The Mac app LuckGrib has played a key role in the introduction of this data to mariners worldwide. Not only does this program display all the NDFD data, it is a super convenient source for the data files themselves, with a Grib File Explorer function that tells all about each file it shows.  If you use Expedition (PC only) for racing navigation and have access to a Mac, you can download the files with LuckGrib, then export them to use in Expedition. The process of getting the native NDFD files on your own with a PC, however, is easily done, as explained below.

The key datafile you can get this way (LuckGrib to Expedition, or download and process on your own) is the CONUS sector.  This data comes in a lambert projection, and Expedition can read and display that. The native oceanic sector and hi-res regional sectors are presented in a mercator projection, and the latest version Expedition will import these as well for routing or comparison with model output. 

There are other options for the oceanic sector.  Saildocs and Ocens have each independently converted the mercator oceanic data to a grb1 format, which will then read in just about any navigation program—OpenCPN is one popular free one. There is some loss in precision during the conversion process, so when possible the original grb2 mercator format would be preferred, but the existing conversions (from saildocs and Ocens) do allow mariners with just about any navigation program or grib viewer to see and use the oceanic dataset. Both sources refer to the product as simply "NDFD," but this is the NDFD oceanic sector data converted to grb1. Saildocs have rightfully changed the wind reference to 10m in their conversion.

We are currently doing a study of NDFD CONUS vs oceanic, NAM, HRRR, and Predict wind. The forecasts are then compared to buoy data at several locations near San Francisco. I will follow up with that when done. 

Downloading the data

For those who might want to experiment with the native NDFD sector files from NOAA, but do not have access to LuckGrib (which hands them to you with a button click) then here is an outline of the process. You can do this on a PC... or maybe other device then bluetooth to your PC (not sure of this yet.)

First note that in principle there is an easy solution for getting any Lat-Lon rectangle of the CONUS data at this link:

custom dgrib requests.

But for now it is not clear what they intend to give back with this, because I have not yet learned how to use this.  It could be a good solution. There could be an issue of http vs https, i.e., something was moved?

Failing that, one solution I have found so far takes just a few steps (short description, followed by a video illustration). A key factor here is to note that the grb2 files from NOAA do not have the .grb2 extension. They are in a .bin format, but what I have just realized is that the two programs I know of that can read these files (Expedition and LuckGrib), will import and read the .bin format directly.  (In an earlier version of this article, there was an extra step converting .bin to .grb2 using wgrib2.exe, but that is not necessary).

Next, the NDFD native data present wind speed and wind direction in two different files. Expedition, however, can read two files at once, so we do not even have to combine these externally.  We just get the two files and drag them onto the Expedition screen and they load and show proper wind vectors.

The links on this spreadsheet from NOAA list the data available. Every element of every sector is there.  Or go directly with this link once you understand the file names.

Index to NDFD products.

Once you see the directory structure, you might find easier ways to get what you want. The full CONUS sector is divided into sub-sectors defined here.  For example these two links get the latest hi-res CONUS wind direction, wind speed, and wave heights  for the SE US sub-sector, days 1 to 3:

  ds.wdir.bin              ds.wspd.bin          ds.waveh.bin        

Check Table 1 above for the forecast times and update schedule.  Remember the files always have the same name when downloaded, so we have to keep track of what we work with.  These are files you can practice with, as illustrated in the videos below. In short, download a set and load all three into Expedition Grib folder, or just drag them onto the open Expedition chart window.

Note: For some reason, not yet understood, when you download the CONUS data by regional sector, as in the above example, we get only half the promised resolution... ie we get 2.75 nmi per grid point, whereas if we download the same region from the full CONUS file, we get the promised 1.3 nmi. Still much to be learned about this dataset.

[Note added 5/12. I originally had as an example here the HI regional data, but that sector is not yet consistent in either time steps or in spacial coverage.  So we have to wait on that one.  I can only note that this observation is not consistent with published goals of the NDFD.  In the NDFD Operations Concept article we find this statement of one of its goals: to provide information that is "up-to-date, available on demand, and provided with spatial and temporal consistency to help users make more informed decisions."]

If the "custom degrib" link given above worked as we might hope, we could ask for specific Lat-Lon region within the CONUS, but in lieu of that we have to ask for a full file. The SE CONUS sector (covers both coasts of FL) is a total of 900 kb for wind speed and direction.

There is a stand alone executable of dgrib, which can be used to download the files, which will keep them sorted timewise. But I do not yet know how to specify Lat-Lon window with 
that.

I am very excited about the high value of this NDFD data for maritime use underway, but have to admit to learning more about it every day.  I will update or correct this article as needed. We have always stressed how important it is to use the forecasts of human intelligence to test the unvetted model forecasts, and this dataset offers an ideal way to do so, and in principle it should serve as the best data for optimum routing... in regions where the data are consistent in format.

Finally, to keep things really in perspective, once we get this NDFD data sorted out for marine use underway, we can look forward to supplementing it with the new National Blend of Models (NBM)! This is actual model output, so it is more uniform and consistent than the NDFD. It is also grb2, which is the WMO standard. The NBM will likely be the data source of choice for mariners in the near future. Its oceanic domain covers most of the Atlantic and Pacific (ie much larger than NDFD oceanic) and it includes some 60+ ensemble runs so it can offer probabilistic winds and MSLP (NDFD has no pressures.) We are at NBM v3.0 at the moment (NBM-v3.0 data at tgftp); we expect v3.1 in August, 2018, which is when the MSLP will be added.

_____

Note on getting archived NDFD data.