Monday, May 3, 2021

ASCAT News — The Good and the Bad.

Scatterometer wind measurements are the truth meter for weather work at sea. There are several sources, but the ASCAT data from EUMETSAT are the gold standard. We can see this data graphically at the STAR website—found easily with a Google search of "ASCAT." We have extensive coverage of this data and how to use it in Modern Marine Weather (MMWX), but the book links are out of date (being updated here!)—in part because there are now three satellites with more data to be presented, Metop A, B, and C.

So the first news is we have the updated links below, there are new data, and it is even better than before. The main bad news is the GRIB format of this data, which offered a super convenient and precise way to test model forecasts is at present no longer available from the Ocean Prediction Center (OPC). It is still listed at the OPC website, which might imply the loss is not permanent. The files look normal and are indeed updated as indicated, but they do not contain any data!

Granted, this wonderful source of digital data could only be conveniently viewed in two popular commercial programs, namely Expedition and LuckGrib—not to mention the free app Panoply, since that one takes extra work. Worse than that, even amongst the many users of these two leading programs, there were not many navigators actually taking advantage of this most important data, which might account for the lingering loss... ie there are not many people complaining. 

(For now, there is no GRIB source for ASCAT, but for those who want it enough to spend some extra work you can download the data in netcdf and then view it in Panoply, but this is not a practical, navigator's solution.)

Part of the problem in ASCAT popularity is the data are not plug and play like a GFS forecast is, for example. Once we did get the ASCAT grib, life was good, it showed the actual winds on the ocean, not a forecast, but real data as if we are looking at anemometers on a field buoys located every 25 km across the ocean. The problem is the data over some specific point, say 300 nmi in radius, might only be updated 2 or 3 times a day, and those times take some effort to predict. Refer to MMWX for these details.

The good news is, and the motivation of this article is, we have a new, free way to get georeferenced ASCAT data into a navigation program when underway using the program qtVlm. We already had one way described in MMWX, namely look up the file name for the data you want from the STAR site or from the table in MMWX, and then request that image from Saildocs. Once that is in hand, and you check the valid time of the satellite pass (MMWX), you can either just look at it to see the winds and compare to the forecasts, or in Expedition, OpenCPN, or qtVlm  you could manually georeference it and then overlay the model forecast at the corresponding tine right on top of it. In these programs you can store the georeference coordinates and then just periodically update to see if there is new data. The image size you are requesting is 20 to 60m kb. A sample is below.


ASCAT-A descending over Bermuda at 12:33z on May 2, 2021. It is May 2, and not May 3, because this image is from the 22-hr dataset that was most recently updated at 02:07 May 2. Had the small purple time at the bottom been between 00:00z and 02:07z then this would be May 2.


These files are always 10º Lat x 15º Lon so they are easy to georeference, plus the grid lines are clearly in view.

Below are samples georeferenced in Expedition and qtVlm with the GFS wind forecast at the corresponding time overlaid on it.


And then zoomed in below, we see the GFS is pretty good since it agrees with the scatterometer wind measurements. This is Metop A, descending. 


The same thing viewed in qtVlm looks like this:





The best comparison of overlaid data like this depends on our choices of transparency and wind barb  colors. This has not been optimized in either of these two programs.

The main new good news from qtVlm is their latest version allows us to enter a live internet link for the image, as opposed to a link to a static file on our hard drive, so that the data are updated automatically when we reload that image. The trick is to create an auto-updating KML file for image and then use that as the target file. This then not only updates automatically, it also brings with it all of the georeferencing information. With your computer linked to your satellite phone, this update should take place just as you do to obtain the latest model forecast.

We have videos online on how to do this.  We make the needed files in Google Earth, which is then another way to look at live, auto updating ASCAT images. The new step is just adding this functionality to a navigation program independent of Google Earth.

We made such KML files  some years ago for the Bermuda Race,  Transpac/Pacific Cup, and for Sydney-Hobart. Our next step will be to update the links and make a convenient place for mariners to download them.  Now that we have lost the GRIB format, these methods become of interest again.

Here is the generic image link using the Bermuda file as the base. Then the colored letters have to be changed for other locations, passes, and satellites.

https://manati.star.nesdis.noaa.gov/ascat_images/cur_25km_META/zooms/WMBds86.png

Change A to B or C for the 3 ASCAT satellites, and change the blue file name to match the location you care about, and then change the d (descending) to a for the ascending pass. 

Thus for each location you care about, there will be 6 files that might be of interest, keeping in mind that A and B are fairly close in time so the earth has not rotated much between them. In rare cases, you might need B having seen A, but not often.  

You can get these files (once you know the name of the ones you want) from Saildocs. For example, send an otherwise blank email to query@saildocs with this as the message:

send https://manati.star.nesdis.noaa.gov/ascat_images/cur_25km_META/zooms/WMBds86.png
send https://manati.star.nesdis.noaa.gov/ascat_images/cur_25km_META/zooms/WMBas86.png

This should get you those two files by return email. Sample file names below.


Click the images above for a better view.

Below are a couple How to videos.



How to create KML files of ASCAT wind data.



Viewing ASCAT wind data in qtVlm

Wednesday, March 17, 2021

Timing and Time Structure in Model Forecast Gribs

Analysis and forecast timing is crucial for good weather work. This is not just a challenge with model forecasts in grib format (the subject at hand), we also face this when using regular graphic weather maps and text forecasts. We cover this in Modern Marine Weather, under the heading "weather map sequencing" near Table 7.2-1. 

The timing involved for the GFS and other global models is very roughly outlined below. (Regional and rapid refresh models have a totally different timing pattern.)

(1) At precisely the synoptic times (HH = 00z, 06z, 12z, and 18z), observers and instruments around the world collect observations and report them via a global network to all the labs around the world running numerical weather prediction models. (A time label "z" means zulu time, which is a nickname for UTC, which was once called GMT.)

(2) The models collect the data and analyze it for quality and then assimilate it into their model programs and start a new global computation. We are now at HH + 2 hr or so

(3) The model completes the global surface analysis and passes it on to other labs. This is now roughly HH + 2.5 hr. Now, or a bit earlier, the model starts computing the various forecasts based on this analysis.

(4) The OPC receives this latest model analysis, which they combine with their own analysis and create the official OPC surface analysis graphic maps and text reports. This is now roughly HH +3.5 hr

(5) The model completes its full extent of forecasts and posts these online for third party access. This is now about HH + 5 hr 10 min. 

(6) Third party providers (Saildocs, Expedition, LuckGrib, XyGrib, etc) complete their download of the analysis and forecasts to their own servers and process it as needed so they can distribute this to their users. This takes us to about HH + 5 to 6 hr depending on several factors, discussed below.

Data are available to us following steps (4) and (6), with actual times depending on the format, product, and source we use, and we have to, of course, keep in mind both local times and dates, as well as the UTC times and dates of the products.  

In the following we first look at the time structure of what we get, and then address the question of when we can actually get it.


What Forecasts Are Available?

When we ask for a set of model forecasts, it is generally assumed that you want the most recent data. Some data sources let us choose between forecasts based on the latest, or forecasts based on a specific synoptic time over the past 24 hr, which may not be the latest. This latter option would only be used in special circumstances, such as comparing with past ASCAT wind data, or for filling in something we missed in the past 24 hr. Those offering this past data generally only go back 24 hr. (If we want even older grib data we have to look into archived or reanalyzed data, which is a topic of its own.)

When we ask for model forecasts such as GFS, we specify the Lat-Lon region, the extent of the forecasts, i.e., out 1 day, 2 day... on up to 16 day for GFS, and we specify the forecast interval, every hour, 3 hr, 6 hr, 12 hr etc. If we are asking for data from Saildocs by email, we must know ahead of time the extent and interval sizes that are available. When using data sources from within a grib viewer, they usually have the intervals and extents listed and we choose from what is available.

What is often not so clear, even when using a grib viewer with selection options is the available intervals typically change with the extent of the forecast, and the options we have depend on the source we are using. If we use the GFS model as an example,  NOAA creates these forecasts every hr for 120 hr (5 days), and then every 3 hr out to 16 days (384 hr), but outside of direct request from NOAA, there is no third-party source for data  over this full frequency range. Each third party provider makes a choice of what they believe best meets their user's needs. Samples are shown in Figure 1.

Figure 1. Sample availability of GFS forecasts.

LuckGrib and Expedition (via its direct NOAA link) are the only sources outside of NOAA direct that offer the 1hr data. XyGrib and those who use it only go out 10 days, but they offer 3 hr steps over that full range. Ten days is certainly more than adequate, except maybe for an initial long look ahead for planning. Generally GFS forecasts are good out 4 days, but then fall off. For longer term forecasts, the Oceanic National Blend of Models is likely best.

When planning a departure time, data every 12 hr could be adequate; when doing an optimum routing, you might want data as frequently as available within your airtime budget—but there are other factors to keep in perspective when making that choice.

The question of when we can actually download or request these various forecasts is another topic, covered below.


Time Format in Grib Forecasts

When we ask our grib viewer or other source for the latest forecast extending out, say, 48 hr with forecasts, say, every 6 hr, for wind, pressure, and rain, we can expect that each of the forecasts we get back will include the same parameters, except for the rain. A generic "rain" request is likely to be given in terms of accumulation, and there will not be any rain data in the first forecast, as none has accumulated at that point. (GRIB is a WMO weather based computer format, and rain is a deceptively complex parameter in that system. See notes in Modern Marine Weather.)

This 48 hr forecast in 6 hr steps will show up as one file containing 9 weather maps. The first is a GFS surface analysis, valid at the synoptic time of the model run, followed (in this example) by 8 GFS forecasts, one for every 6 hr past the synoptic time of the run, out to 48 hr. If the synoptic time of the run was 12z on Mar 13th, the last of the maps would be for 12z on Mar 15th.

These files might be identified in your grib viewer or nav program two ways. All programs will list them by their valid times, such as 18z Mar 14 or 00z Mar 15, but you may also see them identified by the extent of the forecast. In that presentation, the surface analysis is called h0, the 6-hr forecast is called h6, the 12-hr forecast h12, and so on. 

Figure 2. Time structure in a set of grib model forecasts.

In this example, based on a 12z run on Mar 13th, the 18z forecast on Mar 14 would be h30; 00z Mar 15 would be h36. In practice we really need to know both identities, when it is valid and how old is the forecast. Some programs (i.e., Expedition) use h6, h12, etc; others (i.e., LuckGrib) use 6h, 12h, etc for this notation.

Each grib or nav app will have its own unique way to step through these forecasts. There will be a way to go from one forecast to the next, or go to any specific specific forecast. Also, since these are digital data, the programs can interpolate the times. You may only have forecasts for 12z and 18z but you can ask for an interpolated map at say 14z, or even 1437z, which you might want, for example, to compare with the wind measurements from an ASCAT satellite that went by at that time. Or you could want to compare with data in your logbook, or data from a buoy given at some specific time. The programs do not specifically warn you that these intermediate times are interpolated, but they are, just as the wind data itself is interpolated at points between the actual grid points for the forecast resolution.


When Can We Get the Latest Model Forecasts?

This timing is important for a couple reasons. First we want to be sure we are comparing the right forecast with the right observations, and second, we do not want to download a new file and spend the satphone airtime just to get back the same forecasts we got a couple hours ago. Also when racing, you might want to make a decision as soon as possible, which means getting the newest forecast as soon as possible.

The fundamental fact we must live with is, the earliest possible weather analysis we can get is going to be some hours old, so to compare our observations with that analysis means we have to rely on our logbook records that go back to that time... or use saved tables or histograms of our instrument readings. The analysis will always apply to a synoptic time, so it is fundamental to record, one way or another, all weather data at each of the synoptic times. Wind speed, wind direction, and the barometer (and the trends of each) are the main data, but estimated sea state, measured ocean current, state of the rain, and sea surface temp are other parameters that can matter, as well as cloud cover. Air temp might be useful in the midlatitudes to investigate frontal passage.

To compare what we are seeing on the water at the moment, we have to look at an interpolated forecast that applies to the present time, as noted above.

If we assume we are not relying on HF rfax weather map transmissions, which are only available at specific times of day (a huge drawback!), then we are considering now the earliest times we can request the data (with the understanding that we could get it any time after that).

When OPC issues their analysis (Step 4 above), we know that they have the model data in hand and have had time to put all the data together. We can learn exact values of how long it takes them to issue the latest graphic analysis maps from the annotated rfax schedule at OPC, a sample of which is below.

Figure 3. Sample of an rfax schedule showing broadcast (left) and available times (right).

Part 2 (West North Atlantic along the US coast) was broadcast at 2138z on rfax (left-most time), but it was actually available to download direct from NWS via FTPmail or Saildocs at 2112z (right-most time) a bit earlier. Thus this graphic map analysis is 2112 - 1800 = 3 hr 12 min delayed, leading to the typical 3.5 hr delay for these graphic products—it will take about 15 min to request it and get it back if all goes well.

These 18z analysis and forecasts from the GFS model in grib format, however, will not be available on the boat or at home for another couple hours. These 18z GFS forecasts from LuckGrib were available at 2320z and available from XyGrib a bit earlier at 2317z. These were about 5h and 20m delayed, which is typical for global forecasts. You can see the most recent delay times for any model via LuckGrib at luckgrib.com/status. XyGrib delay times for the models they provide are available from a menu link within the program. LuckGrib will warn you if you attempt to download data that are not yet updated, providing the second request is identical to the first one.

A working estimate for data delays of OPC graphic analysis is about 3 to 3.5 hr after the synoptic time, and we get the first grib versions of global models such as GFS in about 5 to 6 hr after the synoptic times.

There are, however, a couple nuances to this latter delay that you might run into if you dive further into the process. If we look into the actual computation times of the various models, which we can see at this NCEP link, we note two things. First the GFS analysis computations are typically done at about HH+3h 22m, but the full range of forecasts out 16 days (384 hr) finishes at about HH + 5h 10m. The  intermediate forecasts are completed at intermediate times.

Thus our grib providers could actually provide the GFS analysis (h0) at about HH + 3.5 hr, but the full range of associated forecast would not be ready for another 3 hr or so. If you ask for a 6-day forecast at, say, HH+4 you would get the latest analysis and maybe a couple of the latest forecasts, but the later forecasts you got would be from the previous model run from HH-6, because they had not been completed by HH+4. 

To avoid that potential confusion LuckGrib, and likely other data providers, do not take the data as it completes, but wait the full 5h 10m to start their downloads and processing. This way users get all the latest forecasts referenced to the same model run at the cost of not getting any at the very earliest time possible. In the case of LuckGrib, the global model delays are about 5 to 6 hr.

Another way to learn the actual times the files are available is to look directly at the NOMADS link where they come from. The times listed are the times these files were posted. These selected forecasts, for example, were based on the 12z synoptic run (.t12z) with 0.25º resolution (0p25):

gfs.t12z.pgrb2.0p25.f000                  16-Mar-2021 15:27  289M  
gfs.t12z.pgrb2.0p25.f120                  16-Mar-2021 16:03  331M  
gfs.t12z.pgrb2.0p25.f240                  16-Mar-2021 16:33  330M  
gfs.t12z.pgrb2.0p25.f384                  16-Mar-2021 17:09  325M  

The 12z analysis (f000) was posted at 1527z; the 5-day (f120) was posted at 1603z; the 10-day (f240) was posted at 1633z, and the longest forecast at 16 days (f384) was done and posted at 1709z. This represents the average of 5h 10m for the full set. We anticipate a new GFS v16 in the near future, which might expand this delay another 10 minutes or so.

Grib Data Distribution is Not Guaranteed

NOAA uses two broad categories to describe the development and distribution of data: Operational and Experimental. The model forecasts we are using are all Operational; indeed a primary source of the data is the NOMADS site, which stands for NOAA Operational Model Archive and Distribution System. So the data are certainly operational, but the distribution (dissemination) of the data is still not guaranteed, as we read in this note at the NOMADS site. We see similar disclaimers at almost all NWS sites. The only "official NWS dissemination systems" are NOAA Weather Radio and data obtained directly from the local forecast offices. In short, the main data we care about is not guaranteed to be available to us, and that does not even account for the third-party processing that we also rely on. 

This means we have to be aware that the model forecasts in grib format and other grib products from  around the world may in rare cases not be complete or may be missing. Speaking with experts who know the details, I learned that NOMADS has been remarkably dependable over the years. A recent, rare snag in the system was addressed promptly and users were kept appraised of the situation at all times.

To me, a way more disconcerting example is the recent, unannounced loss of the ASCAT data in grib format. This has still not been announced, nor a remedy proposed.

Another fringe example is the grib version of the NDFD, which are the official NWS forecasts converted to digital format. These latest data include inputs from numerous local offices and as a result the format and timing in some user selected regions (that might span two offices) are occasionally inconsistent. This is not a global product, so we will cover it when we discuss regional grib files.

The main message is, although the data always look very official and consistent, we might periodically see glitches in the products, such as missing parameters, or fewer forecasts than anticipated. This is all based on computer technology, which is very good, but none is 100% dependable... if we want 100% we should go to NOAA Weather Radio, but we better use a radio that is 100% guaranteed to work right.

Friday, February 26, 2021

Say Goodbye to the First Paper Chart

Two years ago (Nov 14, 2019) we were told that NOAA was planning to do away with all paper charts and all electronic charts based on them (RNC), to be completed by end of 2024—46 months from now.  This was the famous Sunsetting article, complete, indeed, with a picture of the sunset. All charts after that date will be ENC or custom paper charts based on the ENC format.

Today NOAA announced the first printed chart that has been converted to ENC only, Lake Tahoe, chart 18665. From now on, this is ENC only—or design a paper chart of all, or any part of it in ENC format, and print it in color or have it printed for you. This is the new NCC (NOAA Custom Chart) program, which you can experiment with right now for any NOAA chart to see how it works. Here is the online NOAA NCC tool for this, along with a video on the process. I hope to supplement what they have with some instructions of our own in the  near future.



Goodbye Chart 18665.

I believe the NCC printed products will eventually be even better than the present paper charts, but this will take a while. Mariners have to get used to using ENC, which our book on Electronic Chart Navigation will help, and also the NCC tool will improve. Right now the main and maybe only drawback to ENC is their coverage of land. We don't sail there, but we do in fact look at it, and rely on it for navigation. Some nations do a better job on ENC land coverage than the US does, but overall the US has exceptionally good ENC.



A detailed reference on the use of ENC

But this is all digital business these days, GIS (geographic information systems) in particular, and NOAA has access to every possible set of GIS data for land information. Every building, every road, every elevation contour, and eventually this info will be incorporated into the new NCC.

So as of today we have fair warning. The camel's nose is under the tent. It is time to start thinking about ENC... with that in mind, we have a new online course we are about to announce on Electronic Chart Navigation. Maybe about a month out.


Friday, February 19, 2021

GRIB School

This document is an index to the several articles and videos Starpath has available on the subject of GRIB files and their use in marine weather analysis and routing.  You can skim through this to go directly to individual topics. Practice exercises are given at the end.


What is a GRIB File?

Use of numerical weather model forecasts by individual mariners is an ubiquitous component of modern marine weather. Using almost any navigation program these days connected to a wireless source on land or at sea, you can press a button to generate the latest wind, waves, and currents forecast across the chart with forecasts extending out a week or more.

The forecasts come to us in a digitized format called GRIB, standing for gridded binary. It is a vector product, meaning it is all numbers and symbols, but when rendered in an appropriate software program ("GRIB viewer app") it can appear as a graphic map of the isobars, wind vectors, rain distribution, and other parameters, laid out on a Lat-Lon grid. 

The GRIB standard was developed by the World Meteorological Organization (WMO) specifically as a way to transfer weather data in an efficient, standardized manner amongst meteorologists and mariners. The grid is a Lat-Lon lattice of points, with digital values (binary data) presented only at those specific points. The distance between points on the grid is the resolution of the file. This can vary from as large as 1º (60 nmi)  intervals to as small as 1.3 km (0.7 nmi), which is about the finest step size available outside of the laboratory.

A GRIB weather map at 0.5º resolution over a 10º x 10º area including surface wind and pressure every 6 hr for 3 days (a tremendous amount of information) will be about 30 kb in size. File size increases very roughly proportional to the number of parameters, but increases as the square of the resolution and area covered.

Several apps let the viewer request the latest grib forecast from a specific model from within the app, and these generally estimate the file size as you define what you wish to request.

Terminology: We often hear or say something like, "Have you downloaded the Grib?," or "What does the Grib tell us," and so on. Usually this is not ambiguous, but we should keep in mind that "Grib" is not a thing; it is a format.  It is like saying "What does the PDF tell us? The term Grib alone does not tell us at all what we are looking at. This could be a wind forecast from the GFS model or a wave forecast from the WW3 model. Use of the word Grib in such discussions requires clear understanding of the context. Likewise, the reference to "a" Grib file, generally refers a single file, but one that includes multiple forecasts over several hours or several days.

The following links contain information related to use of GRIB files


Background on GRIB files: Numerical weather models and computed parameters


Includes topics below plus a comparison with other forms of marine weather data. 

Traditional Weather Products

Numerical Weather Prediction

Values of GRIB Formatted Forecasts

Grib2 Weather Parameters

Categories of Digital Forecasts in GRIB Format

Basic Properties of Selected Models

Sources of Model Data in GRIB Format


Timing and Time Structure in Model Forecast Gribs

Details of this basic aspect of working with gribs

What Forecasts are Available?

Time Format in Grib Forecasts 

When Can We Get the Latest Forecasts?

Grib Data Distribution is Not Guaranteed 

 

Applications of Weather Data in GRIB Format 

Each topic has a short discussion along with video illustrations.

1. Global model weather forecasts for ocean navigation

2. Global ocean model forecasts of wind waves and swells

3. Ocean model for currents and water temperature

4. Regional model forecasts for inland and coastal sailing

5. Overlay model forecast winds on weather map and cloud photo images

6. Probabilistic forecasts from ensemble forecasts and model blends

7. View sea ice coverage from RTOFS in LuckGrib

8. Compute optimal sailing route 

9. View ASCAT scatterometer near-live satellite wind measurements

 

Introduction to using GRIB files with XyGrib

A few basic tips with a video example.


Introduction to using GRIB files with qtVlm

We have a couple videos on this, but the article is not done yet.


Optimum Weather Routing with qtVlm

A short discussion with video example


Optimum Weather Routing with OpenCPN

A short discussion with video example



Playlists of Related Videos on Use of GRIB Data


   OpenCPN

   qtVlm

   XyGrib

   Expedition

   LuckGrib  (Identifying frontal systems in GRIB files)

   Panoply


Homework (with answers)

Practice with Grib Files 01

Practice with Grib Files 02 

Practice with Grib Files 03

Please post comments or questions below on the exercises or other content of this topic.






    










Tuesday, February 16, 2021

Optimum Weather Routing with qtVlm

This note on qtVlm routing is part of our sequence of articles on the Applications of Weather Data in Grib Format. We have similar demos of routing with other applications.

qtVlm is a free, donation-supported navigation and weather software program for both Mac and PC computers, as well as mobile devices.  It is an internationally popular product with versions in multiple languages. It is the leading program worldwide for virtual tracking and taking part in ocean races presented online.  It is a versatile navigation program as well as weather resource. See also notes in the Starpath Glossary.

We cover charting aspects elsewhere; we have a video on using grib files in qtVlm; here we just look at the app for routing computations, which is of course tied to its display of grib formatted numerical weather and ocean predictions.  The program includes extensive functionality, which means there are numerous nested menus, some of which are interrelated. In short, there is a learning curve, as is with any such versatile program. So we start by just focusing on what we need to create a basic route, and then we will come back to the numerous ways it offers to fine tune and optimize the results further.

We will list the steps here and then add a video demo of the process.

Step 1. When you download and install the app, be sure to load the "maps," which in this case will be the high-res base maps. 

Step 2. The default display shows a daylight terminator which is very handy when underway or when planning a voyage, but for training and practice if you find it distracting it can be toggled off at menu View/Show-Hide/Night Zones.  

Step 3. Load a Grib forecast (review earlier notes on gribs) that will cover the range of the route you want to compute, and long enough to let the vessel finish with its known polar data.  In this example, we want to do summertime ocean routing using archived wind data from July, 201o that we have stored on the computer. Thus the steps are: menu Grib/Grib Slot 1/open and navigate to the file to load it into slot 1.

Doing these historic runs, you will get a warning that the grib data is old, and consequently it will not show on the screen. Click the clock icon in the middle of the menu bar and set the grib time to match the first forecast loaded. Using live data this will not likely arise, as the newest live data is some 4 or 5 hr old at best.

The second from the right magnifier icon (with a square inside) will center the view on the active grib data.

If you do not see the grib data, but you do see elevation and rivers on the land, then you have an overlay turned on that could be hiding the wind data. Click the menu chart icon "O" (online charts)  to toggle this on and off.

Step 4. Load the polar diagram you wish to use. It can accept files in the .pol format or the .csv format, with the separators being semicolons. See related polar format discussion.  To load the polar use menu Boat/Boat Settings.  Note we are not using menu Boat/Polars. That will be used to study the polar once it is loaded.  Once in Boat/Boat Settings, add a name and or model of your boat then open the Polars tab below it.  Navigate to your polar and import it. We can leave the other settings in default choices.

Step 5. Check the polar. Menu Boat/Polars/Wind polar analysis. Check all TWS (true wind speeds) to see if it looks as expected. Then go back to just one wind, and notice that you can click on the curve to read the data. We are not getting into this now, but if you want to change anything in the polar you can do it in Menu Boat/Polars/Wind polar editor. 

Step 6. Set start and end points. At the desired start point, right click within the grib area and make a mark. Give it a name and click OK. We can leave all defaults as is.  Do the same with the destination point.

Step 7.  Right click anywhere on the chart and chose Create a routing.

Give the route a name 
 
Unclick routing from boat
 
Confirm that start and finish are the points you want. The heading says "Finish and start points," but you want to have start first (top) and finish second (below it).

Turn on Keep Starting Date and type in the initial time of the grib forecast installed (default is month, day, year). Later we can use other start times. Select the month, type it in and just keep typing. 

Rest leave default and click OK, and you should see the isochrones being computed from the start point headed to the finish point. Convert to Route at the bottom should be set.  (If anything is not right, then when done just say OK, then Cancel, and from menu Routings select delete routing, and do it again.)

Step 8. Optimize the routing. When done, it presents the ETA, duration, and computation time. Click OK. Then we get the opportunity to convert this routing to a route.  The former is the result of an optimization using isochrones, the latter is a sequence of waypoints (POI). Unless we feel we need to study the solution in more depth, then the logical next step is create the route. 

We have the choice of Optimum or Maximum conversion. The process removes excess isochrone solution points along the route, for example, if three points are on a straight line, then you can remove the middle one. Optimum does that level of simplification. Maximum does more by readdressing the route decision between each POI left in the Optimum pass. Leaving the proposed setting of Maximum is best, but it might take a few more passes up and down the route to converge on the best.  Click OK and watch it optimize.

Once this is done, maybe with the offer to do more, then the routing will be moved from the routings list (menu Routings) to the routes list (menu Routes) and we can then look at details.

Step 9. In menu Routes/Edit Route  select your route. Logbook shows the conditions at each POI along the route. Histogram is an interesting way to look a plot of various parameters along the route. Statistics summarizes a few parameters of the whole route.   To export the route as GPX file, use menu Routes/Export route.

Below is a video sample of an optimum route computation.


Routing example with no special conditions [17m:33s].


qtVlm has many options and restrictions that can be placed on the computation. Like most other apps, you can define boundaries to block the route from certain areas or passes, but unlike other apps (except Expedition), qtVlm can route around a course of marks, or through specific gateways, which adds a layer of versatility to the solution.  This is accomplished by optimizing along a pathway.  In qtVlm, a pathway is a series of waypoints, what might be called a route in other programs. Routings, routes, and pathways have distinctions in qtVlm. 


References:  

Users Manual: http://download.meltemus.com/qtvlm/qtVlm_documentation_en.pdf

Forum details: https://wiki.v-l-m.org/index.php/QtVlm_Virtual_Race_Mode

Many videos in YouTube in French.


Monday, February 15, 2021

Optimum Weather Routing with OpenCPN

This note on OpenCPN routing is part of our sequence of articles on the Applications of Weather Data in Grib Format. We have similar demos of routing with other applications.

Optimum vessel routing across forecasted wind, seas, and currents patterns based on the known vessel performance in various conditions is the primary goal of modern marine weather technology. We can do this manually with graphic weather maps we obtain at sea by radiofax or email, and we can do it semi-manually using numerical weather model forecasts we receive in grib format by satcom or HF-radio. 

We can also now do this fully automatically using dedicated software that perform the optimum routing computations for us. In principle, if all input is correct, this would yield the best result and indeed be the optimum route. Experienced mariners know, however, that the wind forecasts are rarely right in all details, and that special conditions such as relative angles of wind and swells can notably affect the performance of the vessel relative to what is expected in its polar diagram of vessel speeds for various wind speeds and angles. Correcting for waves is even more problematic, and ocean currents remain elusive, even with the best ocean models to guide us. In short, the process is not going to be plug and play. Success will still depend on individual's knowledge of weather and their ability to evaluate not just the quality of the forecast, but also the quality of the computed solution.

All optimum routing apps require several inputs to be set correctly or the computation will not work. The various apps differ in how easy it is to insure this is done right and in the detail of the error messages given to help. In any event, for every app that does these computations, it is helpful to have a cheat sheet to follow until the process is mastered in your favorite app.

Furthermore, beyond the basic requirements to compute the route, there is a large layer of adjustments and scaling factors available to optimize the route. Although the apps have some overlap in the most basic corrections available, they do differ notably in the full range of such corrections.

But before we tackle these in future articles, we look at the basics of running a route computation in several apps, as in this example with OpenCPN. We also  have a couple videos showing the process, but these assume we have the basic details in place beforehand. Here is a summary of the basic steps followed by a video illustration.

OpenCPN Weather Routing Plugin
Mac and PC versions seem to behave the same.

(1) Polar diagram files can be any of three file types (.csv, .pol, or .txt), but they must all be in the same basic structure, shown below. (Note: sometimes Windows hides file extensions. If so, open File Explorer and under "View" turn on file extensions.)

Above is the CSV (comma separated values) format. You can use semicolons as well as commas for this. These would be in neat columns when viewed in a spreadsheet, which is one way to create the file, although the WeatherRouting plugin has a very convenient creator and editor for polar files—we do not need the separate plugin called Polar_pi.

If you replace the commas with tab you get the POL format. Example below.


Replace the comma in the CSV with a space and you get the TXT format.


These three are all really the same. True Wind Angles down the first column; True Wind Speed headers along the first row; then subsequent rows are the vessel speeds at the indicated TWA and TWS. The first element of row 1 is sometimes just "twa,"  but this does not matter. Sometimes the polar includes an inserted first line being the name of the sailboat model.

(2) Polars must be stored in the right place.

Mac:  HD\Users\username\Library\Preferences\opencpn\plugins\weather_routing\polars

PC:    C:\ProgramData\opencpn\plugins\weather_routing\polars

(3) Save only one polar per boat for now.  Later you can mix polars for various conditions (high wind, low wind, upwind, downwind, big seas, crossing seas).

(4) Be sure the polar covers all wind speeds along the route. If wind picks up to 25 and highest polar wind is 20, it might not run. Right click any place along the general route area on the grib file and choose "Weather Table" to get a digital meteogram of wind vs time at that location.

(5) Be sure the starting and destination points are within the wind field. Setting up all digitally within the plugin, these points are not shown on the chart as you set them, so typos may not be apparent. Recall S Lat and W Lon are negative values. See notes below on better ways to set up the route.

(6) Be sure the forecast duration is long enough to get to destination with the given polar. Else it might just fail and not say why. Try a quick test with shorter route. There are ways to fake this with extending last winds or using climatic winds, but you then get fake results.

If you get the message "polar failed" and it does not compute, then the cause is most likely in the list above.


Set up procedure
after weather route plugin is installed and enabled.

Step 1.  Drop a mark at the desired start point, then right click to properties, and give it a name; likewise with the end point.

Step 2. Open the WeatherRouting plugin, and with this open, right click the start point (be sure it highlights) and choose Weather Route Position. This will add it to the positions side of routing window. (You can do this at any point on the chart, it does not have to be a mark, but in that case you will not get a graphic confirmation of the location.)  Do the same thing with the end point, to get it into the positions list.

Step 3.  Check the positions list to make sure they are both there. If needed, edit the configuration window and the positions window from the main menu bar controls. Click anywhere in the routing window to show the menu. Note: you might have to try both right and left clicks to get an item highlighted.

Step 4.  Load a wind grib file that covers both start and end points, and be sure it extends far enough in time to get the boat to the end with the polar(s) chosen. You can also load waves and currents, as covered later. 

Step 5.  Set the forecast to the first of the sequence for now. This can be changed later.

Step 6. Viewing the routing panel (Positions and Configurations), use the position dropdown to load your start and end points. These must be in the positions panel to show up in the dropdown. 

Step 7. In the Start panel click Grib Time, and check is consistent with grib forecast showing.

Step 8.  In the Boat panel click Edit. You will initially see samples provided with the program. Do a select all and remove them, then press Add, and navigate to your polar file, which must be stored in the proper place explained below—it should default to that folder. Then "Save as Boat" and give it a name. You might also open the polar to confirm the maximum wind speed included, because the computation cannot proceed through higher winds.

Step 9.  Check in the Data Source Panel that we are using Grib data. Close the polar window and check the path in the Boat Panel to be sure you are using the one you saved. If not, click the three dot menu beside it to assign the right boat. Then close the configuration window. We can leave the rest of the settings on default to get started.

Step 10.  Leave Last Valid unchecked and Climatology disabled.  We will treat inadequate forecast duration another way.  Close config window.  Return to the main Weather Routing window, select the route we are working on, say two Hail Marys, and press Compute.

________

Once a route is computed, the parameters showing in the configuration window are chosen in the menu View/Settings. You must click the weather routing window to show the menu.

The menu View/Plot offers an insightful way to look over the route, both as computed (Current Route) or as you look over alternatives (Cursor Route).

Completed weather routes are not automatically saved when closing the program. The Export button at the bottom of the config window copies the route to the Routes Manager list. It arrives there as a track, but can be converted to a route. Either can be exported as a GPX file.

Here is a video illustration of the creation of an optimum route:


___________

Side Note: If you make a route on the chart, say C to D, then right click the route and choose Weather Analysis, it will both load these two points into the positions panel, and it will create what looks like a normal configuration in the configurations panel, but when you open it, you will see the start and end are filled in but grayed out. 

When you then compute that one, it will look at the course C to D, then look at the wind speed and direction at the start time from the grib, figure the true wind angle from that, then look into the polar to find the expected boat speed, and use that as the average speed for the whole course and give you that speed and the time it takes at that speed. That is not an optimum route; no isochrones were computed.  It is just a rough estimate of the time assuming the wind does not change. This result is roughly what qtVlm calls the VBVMG (very best VMG) route, but it uses only one step to get there.

To go from that set up to a real optimum route computation, just delete that configuration, make a new one, add C and D to it, and run as above. Then it will crank out the isochrones.

§







Wednesday, February 10, 2021

Applications of Weather Data in GRIB Format

We have several articles and videos on getting started with GRIB files cited at the end of this note. Here we take a look at various applications of this powerful technology. It would be too much to cover in any useful manner in one place, so we present this index to specific topics with linked video demos. We initially focus on a couple free viewers that we use in our online course, and then at the end show how these analyses can be expanded upon with several outstanding commercial products.

1. Global model weather forecasts for ocean navigation

This was the original application of the data, which was spearheaded by Saildocs using Airmail for HF radio communications and the PC program ViewFax for viewing the grib data. ViewFax remains a versatile way to view and obtain the files, along with other marine weather data.

We started our introduction to grib files using the grib viewer app XyGrib, so we will show samples from it whenever possible. We also look at GRIB files in the nav app OpenCPN that we use in both the coastal nav and marine weather courses. Our goal is then to also include qtVlm which is a powerful program that does both navigation, grib viewing, and grib file sourcing. All three apps have both a PC and a Mac version. We have an introductory article on XyGrib that includes a short video on loading a global model

The sample below looks at more details of using a global forecast of an ocean route.


We set up a global model display to look like the OPC maps. By comparing surface and 500 mb maps we get an insight into the dependability of a forecast [13m:32s].

The same files we download from XyGrib or by email request from Saildocs, can also be shown in  OpenCPN—recall that OpenCPN has a way to view the files, but does not directly download the files, although it does assist with preparing an email request for some of the saildocs files. See models available at Saildocs. For the most part, XyGrib offers a more informative way to display most GRIB files than does OpenCPN, but OpenCPN has a super convenient way to load georeferenced weather map images, upon which we can overlay the GRIB model forecasts for improved insight into both the model forecast and the graphic map image from a weather agency such as OPC. This process is illustrated in Section 5 below. 

There are  some 14 global weather models internationally, about half of which offer free access to their data in grib format. A couple of the popular ones are:

GFS (US)

NAVGEM (US Navy, see also)

ARPEGE (France)

ICON (Germany)

GDPS (Canada)

For historical analysis we can use re-analyzed model forecasts. These global data are available every 6 hr from ECMWF and GFS. When you use this data starting at say18z, July 10, 1990 for 10 days, you can get one map every 6 hr, but they are not forecasts, but the equivalent of surface analyses valid at those synoptic times. It is an excellent way to study past races or voyages.

Here is another example of viewing grib files in a free app with both Mac and PC versions called qtVlm.


Loading and viewing grib files in qtVlm [11m:35s].


2. Global ocean model forecasts of wind waves and swells

We can access the US WW3 model in almost any navigation  and grib viewer apps. The model has an extensive set of parameters (see definitions in the Background article.)  The German weather service, Deutsche Wetterdienst (DWD),  also have a high-quality ocean program with a global wave model (GWAM), European wave model (EWAM) and a higher resolution coastal wave model (CWAM).  XyGrib will download WW3, GWAN, and EWAM. Only WW3 is available from Saildocs. (All are available from Expedition or LuckGrib.)

Below we compare two global wave models in XyGrib. Recall the grib2 wave parameters listed below, which shows that both models use the same parameters, but they have different abbreviations for them.


Sample page from Panoply showing content of two grib files from XyGrib

Here I have loaded both files into Panoply, which not only displays the data graphically but also shows exactly what is included. We have videos on the use of Panoply.


A video comparison of WW3 and GWAN in XyGrib with notes on wave display [12m:13s].


3. Ocean model for currents and water temperature

The most popular ocean model for global currents is the US RTOFS model. It is available from Saildocs.  Other global ocean current models include the US Navy models HYCOM and NCOM, the latter of which is a regional model, generally offering the best current data when available. 

XyGrib v1.2.6 does not offer ocean current data download, but it does have a currents parameter to be shown if the data are obtained elsewhere. To get RTOFS grib file, send this in the body of a mail to query@saildocs.com

send RTOFS:42.1N,24.2N,34.3W,3.4W|0.08,0.08|0,6..144|CUR,WTMP

Change the Lat Lon box to what you want. This is for every 6h out to h144 (6 days). It provides current direction and speed as well as sea surface temperature (called WTM). The 0.08º (4.8 nmi, 8.9 km) is the resolution. 

The model is run once a day, completing at about 1700Z. Each run starts with a 24 hr hindcast and produces ocean surface forecasts starting at 12z the previous day for every 3h to h48, then every 6h till h48, then every 12h to h192 (8 days).


A video display of RTOFS data  in XyGrib [7m:43s].


4. Regional model forecasts for inland and coastal sailing

The main data sources for US regional models are HRRR, and the CONUS or Regional versions of NBM, NAM, and NDFD. In other parts of the world, the best regional data will typically be a WRF model run by a local university, institute, or commercial company. Samples can be seen at OpenSkiron and within XyGrib for Europe, and at Expedition Marine for other parts of the world, especially near AU and NZ.

HRRR is the only hi-res US dataset  that is readily available to free GRIB viewers, because we can get it from Saildocs. You might note that NAM and NDFD are also available from Saildocs, but you will discover that the NAM they offer is only 12-km resolution (we need the 3 km version) and the NDFD they offer is only the oceanic version (10 km). XyGrib also has only 12 km data for their version of NAM CONUS.  See our note on NDFD: Oceanic, CONUS, and Regional.

The HRRR model (High Resolution Rapid Refresh) is a 3-km model, updated every hour. The model is updated every hour with forecasts that go out 18h.  The runs that occur at the synoptic times are special in that they then go out 48h, with hourly forecasts to 18h, then 3h forecasts to 48h. We can get these extended HRRR forecasts from saildocs.  (The 2.5-km NAM data goes out for 2.5 days, but we need special sources such as LuckGrib to get these data.) 

For now, we can practice with regional forecasting using HRRR data in US waters (inland and coastal) or any of several hi-res models available from within XyGrib for European inland and coastal waters.  For HRRR data we can use this request to query@saildocs.com

send HRRR:47.9N,48.5N,123.4W,122.3W|0.03,0.03|0,1..18|WIND,PRMSL,REFC

It is important to use the decimals on the position, as these can be very large files at 3-km resolution. Even this area maybe more than we want.  Modify the Lat Lon coordinates for your location. For minimum files omit the PRMSL and REFC


A video demo of displaying HRRR data from Saildocs in the XyGrib viewer [9m:45s].


5. Overlay model forecast winds on weather map and cloud photo images

A major contribution that OpenCPN makes to weather work at sea is its ability to show selected model wind forecasts in grib format overlaid onto official, up-to-date weather maps made by the OPC or their counterparts around the world (UK Met, DWD, BOM, and others). Once the grib file is in hand, this overlay can be accomplished  in a few seconds.  It is a powerful way to evaluate the model forecasts. The procedure is implemented with the weatherfax plugin, which can be downloaded from within the program. We have several videos on the use of this tool and a note on how to add your own files to the list. Here is another example.

Overlay of GFS winds and pressure on an OPC map and on a GOES West satellite image [7m:40s].

The program qtVlm also has a sophisticated means of importing and displaying georeferenced weather maps and other related images. They are not shipped with the program, but they are very easy to import and save so that each run brings the latest maps. Here is a video of the process using qtVlm. Expedition also has a similar functionality.




Loading and viewing weather map images in qtVlm [11m:48s].

6. Probabilistic forecasts from ensemble forecasts and model blends

We get now to the modern aspect of using grib format of model forecasts, namely the use of standard deviations in the forecasts and probabilistic wind forecasts. In the past we would say "There will always be a forecast, and they are not marked good or bad," but that is no longer true—providing we have access to all the newest data.

Ensemble models, such as the US GEFS and the Canadian GEPS, compute 20 or so solutions (see Background notes) at each synoptic time and then provide us with an average of those solutions, along with the standard deviation of the set, which is in practice an uncertainty level. A wind forecast of 12 kts ± 2 kts is obviously much better forecast than 12 kts ± 10 kts.

The National Blend of Models (NBM CONUS) also includes similar standard deviation parameters. On the other hand, the NBM Oceanic domain includes actual wind probabilities deduced from the very many sources they take into account. 

This dataset includes these probabilistic winds: P25 is the wind at the high end of the lowest 25% of all wind solutions at that point; P75 is wind at the low end of the highest 25% of all wind solutions; and the mean wind called P50 is the average of all solutions between P25 and P75. The P50 wind represents the middle 50% of all wind solutions for that point. Likewise there is a P10 and P90. A P90 wind speed of 20 kts means that 10% of all solutions at that point were higher than 20 kts, and so on. Or you could say, there is only a 10% chance that the wind will be greater than 20 kts at that point and time. This is precisely the type of forecasting we need when we have crucial routing decisions to make.  See Evaluating a Weather Forecast.

These data are presented in standard grib2 format, but they are special parameters and not many apps have adapted to them yet. To my knowledge the only readily available source and display of this full range of data is the app called LuckGrib which is app for Mac computers and for iOS tablets and phones. Each app is a one time fee of $25 in their respective app stores; there is no charge for data downloads. The data can be obtained directly from NOAA and the viewed in a free program like Panoply, but this is very time consuming process.  LuckGrib has a very convenient interface and certainly state of the art file handling and graphic display. Below is a brief demo of these probabilistic forecasting features.  I am sure that more apps will adapt to these parameters once the utility of the data becomes better understood and tested. Several apps show the ensemble runs but not yet the standard deviations.


A quick look at GFS compared to GEFS standard deviations and NBM Oceanic probabilistic wind forecasts [12m:38s].


7. View sea ice coverage from RTOFS in LuckGrib

If you plan to travel to high latitudes, RTOFS as of this year also provides sea ice coverage. We can obtain and view this new ice data in LuckGrib—Saildocs has RTOFS currents but not yet the ice data. See the notes above on RTOFS model runs.

Looking at several northern waterways for sea ice coverage with LuckGrib [3m:44s].


8. Compute optimal sailing route

We come now to what is in one sense the ultimate goal of working with model forecasts in GRIB format, namely using that data along with the known performance of our boat in various wind conditions (polar diagrams) to compute the mathematically optimum route across the forecasted wind field, taking into account forecasted ocean currents and waves as they might apply. There is a section of Modern Marine Weather devoted to this topic that highlights the precautions we must keep in mind when relying on such results. It is crucial to go over those points before using such data, computed yourself or provided by third party services. Recall the joke of being told when asked for directions that "you cannot get there from here." For a sailor who does this routing incorrectly from the beginning (often traced to inadequate polar data), this might no longer be a joke!

There are several apps that provide optimum routing. The historic leader in this field has been the PC navigation program Expedition, which not only provides the GRIB files and does the routing computations, but has state of the art procedures for collecting and analyzing performance data so the best polar diagrams can be made. It also has many other custom features, which makes the investment worth it for the majority of racing sailors, worldwide. Cruising or day sailors may not need the full power of that app. There are several books on the use of Expedition



A short overview of optimum routing using Expedition [18m: 23m].

Another sophisticated routing solution is the add-on routing option to LuckGrib for Mac or iOS devices. That link also includes a detailed discussion of the routing process. The key to any good routing solution is the ability it offers to study the proposed route to learn why it was presented as the "optimum" and now sensitive that solution is to other factors. Luckgrib has several unique approaches to this function.


A short overview of optimum routing using LuckGrib [12m: 34m].


All good routing apps include numerous filters to be applied, such as avoid winds greater than a specified speed, or waves bigger than a specified height, motor if the wind speed is below something, or sailing speed below something, or scale the polar boat speeds up or down by a factor, or scale the model winds by some factor, or scale the effect of current by some factor, add so many seconds for every tack or jibe, don't sail closer upwind than some value or downwind by some value, and so on. There are many, depending on the app.  

We do not get into any of these details here, but leave those to our textbook and online courses, and to the several other books and online seminars on the topic. The job at hand is just to show a few solutions to see what they look like, and to note a couple options for practicing with this on your own. 

Also to consider is the Time Zero app, which is another popular commercial app for Windows. It  evolved from the collaboration of Nobeltec and the MaxSea app, which was one of the pioneers in optimum weather routing.  

As for open source or free products,  both OpenCPN and qtVlm have both Mac and PC routing functions. Below is an article and video demo on the OpenCPN version. 


A video demo of OpenCPN routing procedure [22m:55s]. Details are presented in a separate article.

Optimum routing is a key part of qtVlm, which is a popular app for following along and taking part in ocean races online. It is a free program for both Mac and PC, with grib file sourcing as well as routing. Even though this is a free program, its routing function is among the most sophisticated of the offerings. 



A video demo of qtVlm routing procedure [17m:33s]. Details are presented in a separate article.


Optimum sailboat routing in local coastal or inland waters is the exciting frontier of this technology. This must be done with the regional models, with special adjustments of the routing app routines. We will add notes on this process shortly.


9. View ASCAT scatterometer near-live satellite wind measurements

This, it turns out, is not a new technology. The Ocens company in Seattle had a grant to distribute the QuikSCAT data in GRIB format during the last few years of that program, which ended in 2009. The European program ASCAT has replaced that now, and these new data are available in graphic format and in GRIB format.  To my knowledge the only two packaged apps that can both download and display this data are Expedition and LuckGrib. Without these apps, you can download the files yourself and view them in Panoply. We have a video on how to do that. The latest data files are not large as they only include the latest passes. ASCAT wind is the truth meter for any forecast.

We do not have a video demo of this powerful tool because at the moment (early Feb, 2021) the data are not available. There is some technical snag going on. This is the first time we have seen this in years. We should note, however, that the GRIBed ASCAT data, since its inception nearly 15 years ago, has always been labeled "experimental," which distinguishes it from "operational."  The latter are intended to be guaranteed; the former are specifically not.  I will update this when it gets fixed.

We have ways to get this data onto your boat at sea by email request, but before rejuvenating that technology we will wait and hope that the NWS gets this sorted out. 

______________________

That concludes this brief overview of GRIB file applications. Please stand by the announcement of a new short online course we will offer on this topic with hands on practical details. 

Related topics:  The series of articles we have on related topics are in our blog index for Jan and Feb of 2021.