Wednesday, January 20, 2021

Introduction to GRIB files, Part 1: Numerical weather models and computed parameters

Beside model forecasts in GRIB format, there are several traditional formats of marine weather forecasts available to us for analysis and route planning. These were available before model forecasts became popular for general use, and they remain the primary resources we must rely upon. 


Traditional Weather Products

(1) Voice synopsis and forecasts over VHF radio for coastal and inland sailing, and over HF radio for ocean and offshore sailing. Zones covered, times, and frequencies are in Modern Marine Weather.

(2) Text version of the identical voice reports. We can get the text forecasts from NAVTEX radio receivers, radiofax receivers, or by email request to NWS or various 3rd party services.

(3) Graphic weather maps of the latest analysis at each of the synoptic times of 00z, 06z, 12,z 18z (z is an abbreviation for zulu, which indicates UTC), along with graphic forecasts for 24h, 48h, 72h, and 96h. We obtain these via radiofax, internet download, or by email request to NWS or various 3rd party services.

(4) We also have satellite cloud images that we can obtain from the same sources: radiofax, internet download, or by email request to NWS or various 3rd party services.

The above products are created by meteorologists at the NWS and OPC, and each is typically signed with the name of the author. To create these forecasts, the meteorologists use multiple resources including live or archived observations from many sources, at many altitudes, along with their own historical knowledge and records of how the surface weather behaved in various circumstances in the past. 

Numerical Weather Prediction

The human knowledge and intelligence that creates those weather products is also guided by numerical weather models of the entire atmosphere, which are effectively artificial intelligence. These are vast computer programs running on giant computers (some of the largest in the world) that can create a snapshot of any aspect of weather, at any altitude, at any location on earth, and they can make forecasts of things to come, as well.

The human meteorologist of yore might note from their knowledge of an area, that the wind speed at Point A will be 12 times the pressure difference in millibars between Point B and Point C... whenever the pressure at B was higher than C; but when C is higher, the factor is 10, and the wind blows the other way. Then they might know that if the cloud cover over the past 12 hours was more than 80%, then the wind at A would be 15% less, and so on. In short, they had a prescription for making the forecast of one parameter based on what they know about other parameters.

Weather models (numerical weather prediction) take such prescriptions to a whole new level; they are essentially elaborate mathematical equations that predict any weather parameter such as wind, rain, pressure, fog and hundreds more.  The wind speed at point A is then a complex function of just about everything! Not just the horizontal pressure gradient on the surface, but also the vertical gradient, and the humidity of the air, and certainly on the shape, texture, and temperature of nearby land, and more. And this equation has to encompass the whole global atmosphere, because what happens at Point A an hour or a day from now can be strongly dependent of the structure of the atmosphere hundreds of miles away and several miles straight up.

The various global weather models around the world are running night and day solving these equations based on the physics of fluids (air), along with various (physical and mathematical) approximations required to solve such complex interactions.

Every 6 hours, at the synoptic times, the models are fed thousands of inputs of global observations from buoys, lighthouses, airports, ship reports, land stations, balloon measurements, aircraft measurements, along with a vast array of data from satellite measurements, such as ASCAT ocean surface winds. These data seed the models, but the goal is not to make the forecasts match any specific data set, but rather to create a more dependable forecast overall.  

This process, called data assimilation, is just as complex as the models themselves, in that the data must undergo extensive consistency checks before it is allowed in, and it must be entered in ways that do not disrupt the main computations, which are so interdependent. Refer to Modern Marine Weather for more details.

Once synoptic-time observations are assimilated, it takes 3 or 4 hours for the global analysis map (valid at that synoptic time) to emerge. This then becomes the basis for the set of forecasts at later times. This  first file in the string of forecasts is called h0, which is effectively the surface analysis at that synoptic time. The 48-hr forecast would be h48, and so on.  Different global models have different forecast intervals and durations, but the h0 forecast will always correspond to the most recent synoptic time. 

The valid time of h0 is often referred to as the "run time" of the model, but this is a bit misleading. The model is running continuously,  and it will not be ready to account for the latest synoptic observations for several hours after that synoptic time. 

As soon as a forecast from any of the global models around the world is completed it is uploaded to a private international network that all national weather services around the world have access to. Thus each nation's meteorologists have access not to just their own nation's models, but to all of them. The meteorologists can then compare all these model forecasts with their own experience and local data sources to form their own best analysis and forecast for the areas they are responsible for.

When the OPC produces a graphic ocean surface analysis map or one of their forecast maps, they are using their human intelligence to evaluate the various solutions from artificial intelligence (the models), along with their own experience, and their own interpretation of the observations that went into the model solutions. The models do not try to fit specific observations in the data assimilation process, but meteorologists making their own map of what they think is right can use the data as they see fit. Many at the OPC know specific ship captain's reputations over years, and can believe their reports, especially if they get supporting evidence from the ASCAT wind data. I have seen quite a few cases where it appears that the OPC have taken a model forecast of an isobar and pulled it over to align with a ship report. 

The ongoing revolutions in practical marine weather over the past 10 years or so are (1) these numerical models are getting better and better every year, and (2) for quite a few years now, mariners can also directly access the raw model forecasts—they are not the private domain of the professionals any more. Individual mariners can now access the model forecasts in the form of digital weather maps that are distributed in a GRIB (gridded binary) vector format that can be displayed in navigation programs much as we display the ENC vector echarts. There are also numerous stand alone programs called GRIB viewers that do not show navigation charts, but do show the GRIB weather maps from the model outputs.


Figure 1. OPC graphic map on the left; corresponding GFS grib file on the right, viewed in LuckGrib

The ready availability of model forecasts has pros and cons. The cons are simply that when we use the model forecasts direct from the computers, we are looking at unvetted data, and it is known for certain that the models have weaknesses, and in some crucial cases they are wrong. The mainstay US global model called Global Forecast System (GFS), which we cover below, usually underestimates the wind speed in tropical storms, often seriously. It is also not dependable near coastlines. When we use GFS forecasts alone, without the filter of professional vetting, we run the risk of being led astray. Also the model output does not show such features as frontal lines, which the OPC fills in manually based in part on cloud images and ship reports.

The pros, on the other hand, are many, and certainly in the long run outweigh the cons, especially when we are aware of the limits of model forecasts and account for them. We cover these in Modern Marine Weather and we have procedures on how to evaluate a model forecast before we depend on it.


Values of GRIB Formatted Forecasts

Once we take those precautions into account, the pros of model forecasts include:

• Ubiquitous, easy access. Essentially every navigation program has the ability to download and display weather data from one or several models. Because of this, mariners who might not have taken the trouble to access proper ocean and coastal weather in the past, now have this data at their fingertips.

• Digital format. Just roll the cursor over a wind arrow on the model forecast to get digital wind speed, direction, pressure, rain, and so on.

• Versatile, informative displays of weather parameters. With digital data we can employ such features as meteogram plots of say wind speed or direction vs. time at a specific point;  we can color code the values of any parameters for a quick overview of the distribution; or we can display custom defined contours outlining ranges of values; and so on.

[meteogram and color code]

• Interpolated forecasts at any time. Standard NWS/OPC products are valid at the fixed synoptic times every 6 hours, with forecasts only at specific times, such as 12z 48 hr from an earlier 12z surface analysis. GRIB formatted model forecast maps, on the other hand, can be interpolated for any time at all. We can show how a map evolves in one hour steps, or we can ask to see the map at any specific time within its forecast range. With OPC maps we are interpolating between 6-hour intervals, but the models actually compute forecasts for much smaller intervals, down to one hour steps for many, and at least every 3 hours for most, and indeed some have sub-hourly forecast intervals.

• Consistent region coverage for analyses and forecasts. The OPC forecast maps vary in the Lat-Lon region covered depending on the forecast time. Surface analysis covers one region, the 24h forecast a different one, and the 48h and longer use still a different one.  When we digitally request a model forecast in GRIB format, we specify the region we want, and all forecasts are for the same region. The h0 or first map of any sequence will always correspond to the most recent synoptic time and thus represent a surface analysis.

• More parameters available. With standard OPC maps we get forecasts  for wind, pressure, sea state, and in the tropics we can also get tropical storm tracks. With model forecasts we can get very many  more parameters. The GFS model, for example, has over 300 parameters. Beside rain, cloud cover, we can also get simulated weather radar forecasts that are useful for squall forecasting, plus air temp and relative humidity, and many more. We can also request data for the boundary layer conditions at 850 mb. With OPC maps we get surface weather and winds at 500-mb level, although weather at other levels is available in aviation products.

• More than just model output. ASCAT satellite near-live ocean wind measurements are available in GRIB format. At present Expedition,  LuckGrib, and Panoply are the only programs I know that can show the ASCAT GRIB data. 

• Waves and currents. The standard GRIB viewers we use for weather data can now also be used for ocean model products such as detailed wave data from the WW3 model and ocean currents and SST from the RTOFS model.

• Longer term forecasts. OPC forecasts go out 96 hour (4 days) created once a day. This limit is largely do to the fact that this is about the reasonable limit to dependable forecasting for ocean weather. At modern standards, 3 days (72 hr) is typically very good forecasting, but at and beyond 96 hr becomes increasingly questionable. But that does not mean that the longer term forecasts are certain to be wrong, and not useful. As noted below, we can use these as a guide to what happens, and we do have ways now to assign some level of dependability to them using computed standard deviations (from the ensemble models) and specific probabilities (from the National Blend of Models). 

• Optimum weather routing. Another crucial value of the digital GRIB forecasts is we need data in this format in order to carry out computations of optimum routes for sailing vessels. This is a key component of the future of weather routing and the model data for wind, currents, and waves are the basis for it. The longer the forecast the better for this purpose, but we need procedures that emphasize the first few days of routing, with ways to update the route daily if not more often.


Grib2 Weather Parameters

The full list of Grib2 parameters and categories contains way more than we use at sea, but you may find some of interest for land based interests.  The primary ones we care about for marine weather are:

Atmosphere (GFS and others)

Pressure reduced to sea level (PRMSL)

Wind at 10m (as E-W and N-S components, or as scalar speed  WIND)

Wind gust at 10m

Rain, accumulated or rate (APCP, PRATE)

Cloud cover, usually total in % (TCDC), but low, medium, and high are available.

Temperature at 2m (TMP)

Relative humidity at 2m (RH %)

Dew point at 2m (DPT)

Convection, several parameters used by some (CAPE, CIN, LI, others) See Table 3.4-3 in 

Modern Marine Weather 

Simulated radar, composite radar reflectivity (REFC), good squall indicator. See 

Modern Marine Weather 

Visibility (VIZ)

500 mb, height of the surface and wind speed

850 mb, height of the surface and wind speed

Ensemble forecasts (GEFS and others)

Wind and pressure from individual components

Mean values of all components

Control value used 

Standard deviations for wind and pressure

 National Blend of Models

Probabilistic winds (10%, 25%, 50%, 75%, 90%) Oceanic domain

Standard deviations for wind and pressure, and other parameters. CONUS domain

Currents (RTOFS, OSCAR)

Current speed and direction

Sea surface temperature (SST)

Sea State (WW3 and others) Note these terms; they are sometimes mixed up online.

Significant Wave Height of wind weaves (WVHGT)

Direction of Wind Waves (WVDIR)  

Mean Period of Wind Waves (WVPER) 

Significant Wave Height of Swell Waves (SWELL)

Direction of Swell Waves (SWDIR) 

Mean Period of Swell Waves (SWPER)  

Significant Wave Height of the Combined Seas (HTSGW)  

Primary Wave Mean Period (PERPW)

Primary Wave Direction (DIRPW)

Wind Speed and Direction (offered in WW3, but best to get this from GFS. There is no special value of this wind data.)

 

Categories of Digital Forecasts in GRIB Format

Numerical weather prediction (NWP) models use essentially the same parameters defined by the grib2 standard, but they compute them using quite different physical and mathematical approximations. It is fair to ask if we really benefit from having so many different models, but they each argue they are unique in one way or the other. We consider here just an example or two from the most basic categories.

Global Models
One distinction or category is global model vs regional model. Global means just that, it covers the whole earth. The premier US version is the Global Forecasting System (GFS), which in its latest version could well be the best available worldwide in the most circumstances. Other global models include the UK Met version (  ), EU version (ECMWF), Canadian (  ), and a few others.  The US Navy also runs a global model (NOGAPS), independent of the NOAA products. Properties of the GFS are listed below.

Regional Models
Regional models start with a global model base and then fine tune it with higher resolution, smaller  forecast steps, and more frequent update times, using more thorough accounting for the effects of shape, texture, and temperature of adjacent terrain. They generally cover much smaller regions, and do not extend very far out in time. Notable US regional models include North American Mesoscale (NAM) and the High Resolution Rapid Refresh (HRRR). Properties of both are listed below.

The weather research and forecasting (WRF) is a regional model run by multiple universities and institutes around the world. They tend to focus on their own local regions, but often are indeed the best forecasts for their own regions.

It is fundamentally important that we choose the right model for the task at hand. We cannot rely on global models like the GFS to give us dependable forecasts for inland waters. We must use a regional model for that.

Ensemble Models
Another distinction between model forecasts is are they deterministic or probabilistic. Deterministic models compute (determine) specific parameter values, at a specific points, at specific times. The GFS and regional models mentioned are all deterministic.  In contrast to that are ensemble models that run the same computation many times with slight variations in each run and then average the results. The spread in the results (standard deviations) is a measure of the probability of the answer being right.

Each run varies some element of the computation that could impact the results. This usually involves varying some of the input data (which has a statistical uncertainty)  and varying some physical or mathematical approximations in the solutions, which work better in some cases than in others. The US ensemble model is called GEFS. This model has several outputs we can view:

Control run. This is the base model with no perturbations. Usually same as a GFS run

Mean run. This is the woking output being the average of all the individual runs. The mean run will also have a parameter showing the standard deviation (SD) of wind and one for pressure.  Generally we see the SD being low for a few days and then grow larger reflecting an increasing uncertainty in the results. In some cases, we see SD good in one part of a forecast region, but poor in another, which is our alert to be careful in those regions.

Member runs. These are the 20 or so individual runs, each with slightly different initial conditions. Depending on our GRIB viewer, we can look at these individually or all at the same time. When doing optimum routing computations, these member runs show the extremes of what the best route might be.

Model Blends
The National Blend of Models (NBM) is a combination of many models, including some international products. It is available in a near global version as well as a CONUS version and several regional products (AK, HI, etc).

NBM CONUS includes standard deviations, which  makes it one of the best for regional models where there is coverage. This combined with the HRRR model are the workhorses for inland and near coastal weather.

NBM Oceanic is a near global product that includes probabilistic forecasts. This is likely the best long term model (>4 days). For shorter periods, the GFS should be consulted, but still keeping in mind the probabilistic winds of this forecast.

National Digital Forecast Database (NDFD)
The NDFD is a unique GRIB formatted datasource and potentially the most important way to access US forecasts for local or ocean waters, but it is not yet fully implemented. It is not a model and not a blend of models, but rather it is the digitized versions of the human intelligence discussed above that makes the US forecasts, which are of course aided by all the model outputs. Since this dataset includes input from the local forecast offices there is sometimes a coordination conflict that impacts the GRIB formatting. Also, although the OPC does create their forecasts in a digital format, they do not offer this data to the public so far. See this note for further information on the NDFD.


Basic Properties of Selected Models

From our textbook Modern Marine Weather


Sources of Model Data in GRIB Format

We generally have easier ways to access GRIB files discussed below, but the primary source for GRIBBed forecasts is the NOMADS (NOAA Operational Model Archive and Distribution System) site. The third party sources we use go here for the data.

If we do have to come to this primary source because our favorite provider does not have what we want, then note there are three sources: grib Filter, https, and gds. The "gds" is  for unix based custom software that most mariners will not have access to. The "https" does link to GRIB files but they might not have a .grb or .grb2 extension. They will still load in some GRIB viewers but these are huge files (hundreds of MB) that cover the whole earth including all parameters and for one time only. In short, not a practical source for most mariners. 

The source marked "grib filter" lets users choose the lat-lon box and the parameters desired, but each file is one time only. There is also a learning curve to applying the filters. We have examples online, but this is still not a convenient source.

What we want ideally is a nav program or GRIB viewer where we can with a few button clicks specify exactly what we want, and then the program goes and fetches this for us and displays the results. Most nav programs include some level of this grib file request. State of the art nav programs like Expedition have the most options;  dedicated GRIB viewers such as LuckGrib for Mac and iOS have even larger sources of data, but these viewers do not show nautical charts and are intended for weather analysis with primary chart navigation being done in a different app.

A compromise solution is a program like the free OpenCPN chart navigation program (Mac and PC), which can display GRIB files, but you must obtain them from other sources. A common external source is Saildocs, who provide basic model data by email request including global and some regional models. See Modern Marine Weather for details. OpenCPN has an automated way to create a Saildocs request.

In our weather course we recommend starting with XyGrib. This is a combined GRIB viewer and data source that is free, with both a PC and Mac version. The folks that produce and support XyGrib also provide WRF model data for several locations in Europe as well as other global data.

The free program for Mac and PC called qtVlm is also an excellent resource. It incorporates accessing GRIB files from both XyGrib and from Saildocs. It is a full navigation program as well as GRIB viewer.


Here is a video that reviews the topics of this note.


[coming soon]


Below is a link to a video showing the use of XyGrib (as an example viewer) for getting started on the use of GRIB files, along with a discussion of the forecasts and how to use them.


[coming soon]





Saturday, January 16, 2021

Best App for Weather Forecasts Anywhere in the US

Usually at Starpath we concentrate on marine weather, being the weather on ocean, coastal, and inland waters, but this is a note about weather on land, anywhere in the US. 

The National Weather Service (NWS) is the primary source, and there are indeed free mobile apps to access NWS products. But there are ways for us to get more specific and more timely information than available in standard NWS products, and some of this information we cannot get from the NWS, although the NWS themselves use this same data when making their forecasts. 

I refer to the numerical forecasts of the National Blend of Models program, and especially their continental US version, called NBM CONUS. This model gives  a forecast every 1.35 nmi (2.5 km) across the US and large parts of Canada and Mexico. 


Figure 1. Coverage of the NBM CONUS forecasts.

The forecasts are for every hour, out to 18 hr, then every three hours for 2 days. It extends farther out in time, but that is not pertinent, because the main value of these forecasts are their immediacy. The model forecasts are updated every hour, so the primary value here is for shorter term forecasts of say a day or so, then we just update this same forecast as often as we like.

The reason I title this "best app" and not just "best model" is the app I refer to (LuckGrib for iOS) is the only app I know that  can access and display what I believe is the best data set for land based weather.  It is also well suited to be set up for your area of interest, after which you can just turn it on and refresh for latest forecast, with all parameters laid out in a nice meteogram format.


Figure 2. Two iPhone views of LuckGrib, showing NBM 18-hr forecast for central Rhode Island. Left is showing wind only for the time at the blue line. Other parameters can be selected with the second icon down on the top left. Numerical values shown are for the location of the cross hair. Right is a plot of wind, rain, and temperature over the time period of the forecast.  This type of plot is called a "meteogram." Turn on the meteogram by clicking the wind data in the map view. There is a thin blue line around it meaning it is a link.

The meteogram display is a very convenient way to look at the forecast. This can be for just 18 hours, or ask for 3 or more days. The screen can also be captured in your phone or tablet to refer back to once you learn what really occurred with the wind, rain, temperature, and so on.

The LuckGrib app (LG) includes many sources of data, meaning many different numerical models. It is my belief that this source (NBM CONUS) is the best for inland weather, but I will leave that argument for another article; everyone can establish their own conclusions with experience. Screen capture the forecast then compare with what subsequently happens. 

Besides the fact that the NBM encompasses all data and numerical models that are available (it is a blend, not a single model), and that it is updated every hour,  a key attraction is it also presents the statistical dependability of the forecasts in the form of a standard deviation (SD) parameter.  This was not shown in Figure 2 (to simplify that first look at a meteogram), but we can select it in the list of parameters when setting up the forecast. For daily use of this app and forecast it would seem logical to include this important information.

Figure 3. Landscape view of the temperature forecast (solid line) along with the SD (dashed line)  at two different times.

From this we can say that the forecasted temperature at 2 hr into the forecast is 37º ± 1.1º and at 17 hr into the forecast the temp is 48º ± 1.7º.  In other words, we can think of the SD as an error bar or uncertainty in that simple manner, but the information in the SD is actually much more specific. The diagram below shows how the SD is used to characterize a statistical distribution of results.

Figure 4. Standard deviation illustrated. From our textbook Modern Marine Weather.

In the case of model forecasts, we are not dealing with "measurements" as referred to in the caption above, but rather different computations of the forecast. The NBM includes many models, each with its own forecast, along with several groups of forecasts called ensembles. A wind, rain, or temperature forecast at any specific time and place will be the average of many, maybe a hundred, different forecasts. What they report as the forecast is actually the mean value of all the forecasts in the blend. 

The other values of the forecasts will typically be spread about this mean value in a normal distribution that looks like one of those in Figure 3. When the values from all the different models are all near the mean value, then the SD will be small, and that will be a good forecast, but when the the variation of the results is spread out,  meaning different models came up with different values, then the SD will be large, and the forecast will be more uncertain. 

We see in Figure 3 that the temperature forecast was very good, with a small SD, even in those cases where it was not as good as earlier. This is typical of normal atmospheric conditions. Air temperature can often be well forecasted, within a degree or two.

Wind, on the other hand, is much more difficult to forecast in many areas around the country and its coastlines. The wind forecast of Figure 2 was updated about 5 hour later to get the data below.


Figure 5.  NBM wind forecast also showing the standard deviation (SD).

The peak wind remains at about the same time, but is now with this fresher forecast is slightly higher now at 15 kts compared to 14 before. The direction has not changed.

What we see new in this display (which we could also have seen in the earlier one had we turned it on) is the uncertainty in the forecast.  The wind at the time of the forecast (forecast hour 0H) and again about a day out is very good at about ± 1.5 kts or so, but at the peak the wind is more uncertain at ± 2.5 kts.  

This is actually a very good and dependable forecast. A standard deviation of the wind forecast of 1 to 2 kts is essentially as good as it gets.  In other words, the state of the art in meteorology cannot forecast  wind speed any better than that. Also we cannot hope for a direction much better than ± 15 to 20º. The cases we want to look out for are winds forecasted to be 15 kts ± 12 kts! 

But you will not know that without something like the NBM. We might see many different model forecasts tell us the wind is expected to be 15 kts, and the NWS report on the radio might also be 15 kts, but when you look at the SD from the NBM you see this is ±12 kts. In other words, they really do not know the wind forecast at all.  This is why the NBM is so valuable. This also brings to mind what we stress often in our weather course: there will always be a forecast... and unless you look in the right place, they will not be marked good or bad.

To follow up a bit on the standard deviation concept, we can look at more specific implications with another graphic from our textbook.


Figure 6.
 Statistics  based on the standard deviation. 

With a forecast of  15 ± 2, we know that 68% of all the computations put the wind within that range, and that there is a 16% chance that the wind is higher than 17 or lower than 13. Often for land-based considerations, it is more valuable to know that there is only a 2% chance that the wind would be stronger than 19 kts (2 SD).

That was a quick overview of the data for the most basic land forecast. Needless to say, this app will do very much more, including, for example, displaying forecasted HRRR model's simulated weather radar display, which is a good way to see how rain is likely to move across your region.  We can add something on that later.

LuckGrib is in the iOS App Store. There is a 14 day free trial to see it if meets your needs. After that the cost is $25. (At the end of the 14 days it just stops working and you can delete or buy as you choose, or do nothing.  There is no automatic charge or pending charge at the end of the trial period.)

Here is a short video demo of the steps discussed above.



There will also be a follow up video to show how this works on subsequent days with just 3 clicks to get a new updated meteogram, and how to set up multiple locations for weather watching.