Friday, June 14, 2019

How to Obtain Custom GRIB Files

We usually try to get all of the GRIB files we need from within the navigation program or GRIB viewer we are using.  This solves a key issue of being sure that the ones they let us request will indeed be viewable in their viewer.  All such sources of GRIB data offer a filter for Lat-Lon region and desired parameters, as well as resolution, and time intervals and duration (ie every 3h out to h72).

Some navigation programs, such as Expedition, have many of the top model products coded into their system so there is less reason for getting external data, others have limited products, and it is not uncommon that popular charting software might only offer GFS parameters internally, maybe with the addition of RTOFS for currents and WW3 for waves.

The popular OpenCPN can create email requests to Saildocs, but the packaged interface options do not include many important models Saildocs offers. OpenCPN can view some grib2 files, but not all. Some of the new high-res data come in a Lambert projection which takes new programming.  There are also new parameters to be included. The forthcoming version of OpenCPN (newer than present 5.0.0+9065270) will include simulated radar (composite reflectivity, REFC), which is available in several WRF models, as wells as the NBM, and now globally in the new GFS version that came into effect on June 12, 2019.

REFC is not yet available as part of a GFS request to Saildocs, but we have other ways to get the data underway from Saildocs, as shown below.  I understand that REFC will be added as a request parameter in XyGrib in July or August. XyGrib will show the files if you get them elsewhere, i.e., as outlined below.  It is already available in the European WRF models from https://openskiron.org/en/.  We can also get this parameter from several models using LuckGrib, and then export the file to be loaded into a navigation program. LuckGrib has the high-res regional versions of REFC as well as the global.

Sample of REFC over SW Pacific viewed in LuckGrib

The notes below go beyond the new REFC parameter and show how to request GRIB files in general directly from their primary source, NCEP.  The link we start from is:


The procedure is outlined below, followed by a video link demonstrating the process. 

Step 1. The main link takes you to a list of many models, with various ways to access the data.  We want to use here the "grib filter." The header name is also a link to the process we are outlining here. The other data sources are huge files of global coverage and hundreds of parameters.

Step 2. Choose a model.  For now we will use GFS 0.25 degree.  This is now the new GFS, and it will as of yesterday include the RECF.  Click "grib filter" for that one.

Step 3. This takes us to an index of data files, and we note (at least for now) that there is a format change yesterday when the new GFS went online.  We have to choose one newer than 20190613.  This is yyyymmdd. We choose 20190614.

Step 4. The next page shows the synoptic time computations: 00, 06, 12, 18. Normally we would want the latest model run. Let's say the latest is 12z. That will be the top one on the list.  Some hours  after 18z, the 18z file will appear at the top of the list.  (More on time delays and latency later on.)

Step 5. Now we are on the request page, where we select, valid forecast time, parameters, levels, and Lat-Lon region. At every synoptic time, there will be 364 forecasts made. Let's assume we want every 6h starting at the 12z run for the next 24h.  Thus we want 12zf000, 12zf006, 12zf0012, 12zf0018, and 12zf0024. This is 5 files and we have to ask for these individually.



Parameters

Let's say we want wind, pressure, and simulated radar. We could get others, as discussed below.

For the wind we ask for UGRD, VGRD, which are the E-W and N-S components of the wind; the grib viewer will make these into a wind vector.

For the Simulated radar we ask for REFC

For the pressure we could ask for PRMSL, which is the common one used, or we could ask for MSLET, which is actually a more precise pressure parameter, but some grib viewers might not be set up to recognize this.  We do not want PRES, which is a parameter used for weather on land.

Other useful parameters are discussed below.



Levels

For wind we need "10 m above the ground."  For PRMSL we need "mean sea level."  For REFC we need plain "entire atmosphere" and not "entire atmosphere (considered as a single layer)."



Step 6. After selecting a file, parameters, and levels—you will see online that they present this in a different order—put a check in the box "Make a sub region" and then enter the bounds. Lon West is negative (-);  Lat South is (-). We will get Cape Verde region for a small sample, -30 to -14, and 20 to 10.

Step 7. Put a check in the Box "Show URL only for web programming," then press enter and you will get this link:

URL=
https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t12z.pgrb2.0p25.f000&lev_10_m_above_ground=on&lev_entire_atmosphere=on&lev_mean_sea_level=on&var_PRMSL=on&var_REFC=on&var_UGRD=on&var_VGRD=on&subregion=&leftlon=-41&rightlon=-16&toplat=40&bottomlat=26&dir=%2Fgfs.20190614%2F12

If you are on land with an internet connection, you can then take away the "Show URL..." check mark, then when you press Download you will get the file called: 

gfs.t12z.pgrb2.0p25.f000

At this stage you might want to add a .grb to the file to get  gfs.t12z.pgrb2.0p25.f000.grb, then you know what you have, but this is not necessary. We have shown that both OpenCPN and XyGrib will open the files as downloaded without the .grb added, or with it.

This file will then load into OpenCPN, but you won't see the REFC until the next version of OpenCPN. The latest version at the moment  (5.0.0+9065270) does not have the REFC implemented in the GRIB plugin, but the developers have it working in the next version, as shown in the video.

To get these files from Saildocs when underway you would send a mail to query@saildocs.com with the message being

send https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t12z.pgrb2.0p25.f000&lev_10_m_above_ground=on&lev_entire_atmosphere=on&lev_mean_sea_level=on&var_PRMSL=on&var_REFC=on&var_UGRD=on&var_VGRD=on&subregion=&leftlon=-41&rightlon=-16&toplat=40&bottomlat=26&dir=%2Fgfs.20190614%2F12

send https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t12z.pgrb2.0p25.f006&lev_10_m_above_ground=on&lev_entire_atmosphere=on&lev_mean_sea_level=on&var_PRMSL=on&var_REFC=on&var_UGRD=on&var_VGRD=on&subregion=&leftlon=-41&rightlon=-16&toplat=40&bottomlat=26&dir=%2Fgfs.20190614%2F12

send https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t12z.pgrb2.0p25.f012&lev_10_m_above_ground=on&lev_entire_atmosphere=on&lev_mean_sea_level=on&var_PRMSL=on&var_REFC=on&var_UGRD=on&var_VGRD=on&subregion=&leftlon=-41&rightlon=-16&toplat=40&bottomlat=26&dir=%2Fgfs.20190614%2F12

You could request as many as you like. These are about 56 kB each as returned by regular email (~twice actual size).  Using special satcom email they could get reduced quite a bit. They will come back as individual emails with the files attached.

The video below shows how to get these by email, and then view them in XyGrib and in a forthcoming edition of OpenCPN, along with some discussion of this new REFC parameter.  

With this method of requesting specific parameters and levels, navigators may also find it useful to look into the 850 mb level for both wind and relative humidity (RH). This is the boundary layer level at about 1000 m, where the winds become less, if at all, affected by surface friction. About the altitude of low clouds. These winds are a good indication of the direction of squall motion (compare with REFC forecasts), and in the tropics, the streamline view at this elevation can give indication of wind flow on the surface that does not show up in the actual wind data. We watched a prominent example of this off the coast of Queensland, AU earlier this month. The RH at this level can also show a good outline frontal systems, which are not always easy to discern from wind and pressure.  





Video demo of getting the files.




Video demo of viewing REFC in XyGrib.




Video demo of viewing REFC in OpenCPN.


On the list is to add a video devoted to the practical use of REFC and data we have to confirm it... which is easy to test because we have archived real weather radar online.... and  you can go to squally waters in real radar view, forecast some pattern and then see if it pans out.

We also want to add a short article and video note on use of the 850 mb data, as outlined above, which you can obtain with the methods outlined here.


Saturday, June 8, 2019

Best International Chart Deal Ever!

Well... that's a bold statement; let's clarify.  First, I am speaking of charts that are equivalent to official charts, as opposed to 3rd party charts, such as  Navionics or Garmin Blue charts, among others. Both of these can be good in some areas; both can be bad in some areas.  Sometimes these limitations are not just the chart, but the chart viewer. In most cases, these chart companies control the viewers, but some platforms are more versatile than others. Once these proprietary charts are then sold or licensed to other companies, more issues can arise, as they might, for example, license only some scale levels and not others. There have been notorious accidents traced at least in part to that issue.  But none of that matters here.

We can use paper charts or electronic charts (echarts), but for a "deal," we would have to mean echarts, because all paper charts are expensive, and rarely on special. Also of course, by "international" I mean waters other than the US waters, because all US echarts are free—the obviously best deal. A few other nations also offer free echarts for some of their waters (see charts section at OpenCPN).


Echarts come in two formats: raster navigational charts (RNC), which are graphic images of the paper charts, and vector charts, which are all digital data; the chart display is created 
from numerical parameters by the nav app as we view it. Vector charts can in principle include much more information than possible with raster charts, such as Light List and Coast Pilot data, as well as more descriptive attributes of the various charted objects. The bargain here is with the vector charts, because they are smaller files than RNC, and they are notably easier to maintain, manage and distribute, not to mention having the potential to include more information.

All maritime nations make a set of vector charts in the official IHO standard called S-57. These are called electronic navigational charts (ENC). The third party charts mentioned above are all "vector charts," but they are not ENC, and therefore not official charts.


To understand this "best deal ever" we need to look a bit more into how ENC are displayed in navigation apps, which is outlined below, but I should mention that we have a book on the design and use of ENC charts that includes all the practical detail needed to take advantage of the powerful ENC resource, as well as explaining relative roles of RNC and ENC. It also includes tips on use of echarts in general.






"Great book! I wish I had had this resource many years ago when just starting to craft OpenCPN. Would have saved me lots of effort pulling together the sometimes baffling set of standards and connected parts that is the current ENC landscape. Especially, having ENC Chart 1 available in clear color printed form is a huge benefit. And the annotations are a bonus. Well done! I will heartily recommend this volume to OpenCPN users as the opportunity presents."
—Dave Register, originator and lead developer of OpenCPN.


ENC products must include all the information specified in the IHO S-57 standard, but that standard does not tell us how the data should be displayed to the navigator. The display of the data is specified in IHO S-52 standard, and any navigation app or system that hopes to be certified "Type Approved" must meet the S-52 display standard—and indeed several other factors having to do with interface, chart updating, and back-up.


In passing, the ENC display in OpenCPN is as good as any we have seen outside of type approved apps, and indeed some of the choices made in OpenCPN that are not strictly standard could be considered an improvement by many navigators. The developers at OpenCPN have been very good about making improvements in the display whenever needs are detected.


The common factor among all nav apps capable of showing ENC, whether or not they are type-approved, is the way they process the chart files. Upon initial install, they load the standard ENC file issued by the producing Hydrographic Office, and the first thing they do is convert the ENC files to a format they can read and display efficiently in their own system, and then they store this new file on the hard drive along with the original ENC. The new file is called the System ENC, or SENC file. A SENC file is some 2.5 to 5 times larger than its base ENC, depending on the nav app and individual charts.


Thus every navigation app running official ENCs will have two copies of the chart installed. The government ENC and their own unique SENC file. The ENC files of a specific chart are the same loaded into any nav app, but the SENC files created in that app are unique to that specific nav app. 


Other than from the US and a few other nations offering free ENC in S-57 format, most maritime nations package their ENC into an encrypted format called IHO S-63. This is basically a protective envelope they wrap around the ENC, whose content is still determined by S-57.  OpenCPN has a plugin to allow for the display of S-63 (ENC) charts, but these charts are not the subject at hand now. The S-63 official charts are all fairly expensive.

Since international ENCs are encrypted (for copy protection and to guarantee accurate data), you cannot even create a SENC file without having needed credentials on your computer, which typically include some form of a fingerprint of your computer system along with a chart serial number that is proof of purchase of the chart. Therefore it is no surprise that you cannot just copy the SENC file of a chart and send it to a friend using the same navigation app.


The IHO has, however, approved the distribution of SENC charts that have been created by licensed providers, but these charts may not be referred to as "official charts," and indeed as such would not meet carriage requirements.  They do this in part to protect their official chart sales, but also because they did not make the actual charts themselves and therefore cannot guarantee that the charts are indeed identical to the official ENC, nor can they guarantee that the update system works as intended.


The special deal on international charts referred to here are SENC charts produced specifically for OpenCPN by o-charts.org, a company of sailors in Barcelona.  Since every nation has their own regulations on the use of their SENC charts, the License agreement from o-charts lists them all.

Not all nations offer this SENC option, but many do, and this does indeed look like the future of electronic charting. It has the advantage of being a machine language file that runs fast, and takes up half the space of a native ENC installation—and as we see here, companies have the option of providing these at very economical costs.

I must also add, another thing that contributes to them being the "best deal" in charts is that they are made for OpenCPN, which is a topnotch nav app. There are several very good commercial nav apps, but there are others that are not very good. Had one of the poor products offered the SENC charts it would not be such a good deal. As noted below, there are indeed international charts that cost less.

The limits on the SENC charts are they are unique to the navigation app they are made for, and second, they are not "official charts." The providers (o-charts) can and do state that the charts are identical to the latest official versions, and they include free updates for some period of time, typically one year. The SENC charts we have tested have indeed been identical to official ENCs, and the update process works well.

The primary reference for ENC and SENC related matters is the excellent IHO Pub. S-66 Facts About Electronic Charts and Carriage Requirements  (SENC is on pages 13 and 31-34).


So with that long introduction, the deal we are discussing here are SENC charts made for OpenCPN, available from o-charts.org.  This company is licensed by the nations that offer a SENC option. They produce the SENC chart files for OpenCPN (called 0eSENC, OpenCPN encrypted SENC) and then they sell and distribute these oeSENC directly to us without any interaction with the primary ENCs. One year of free updates is included. Nations update on different schedules.


We have tested this system, and it works well. Easy and efficient. And indeed a deal! You can, for example, buy all of the charts of Australia (889 charts!) for 35 euros (~$40). The ENC cost is about $580. The international selection of oeSENC charts available is given here.


For comparison shopping, o-charts offer all of West Coast Canada (BC) for 20 euros. The ENCs from Canada are some $1,200!; the best previous bargain I knew of was Rose Point price of $99, but they can only be used on their nav app Coastal Explorer (these are S-63 not SENC), which is an excellent navigation program but is costs $365. Rose Point does have the BC raster charts for $99 as well, again restricted to Coastal Explorer. O-charts says they are working on raster versions as well, but that is a big project... recall the notes above on why vector charts can be a bargain. 


In further contrast, I must note for completeness that the least expensive of all electronic chart  options are the Navionics apps for mobile devices, and in this case the one called "Boating US & Canada." That is just $22 for nav app and charts for all US and Canadian  waters, but the charts and app both have notable limitations. It would be stretching the point to call its use navigation... it is, as they describe it, "boating." To be fair, however, very many prudent navigators have in fact used the Navionics charts in one form or the other around the world, for short and long passages. But they have to be on top of things at all times, to deal with anomalies that might show up.




I am using OpenCPN 5.0 on a Mac, but users should double check that any crucial plugins they want to use are available for ver 5.0 on the Mac. At this moment, the Mac plugins are in process. All the standard plugins work for the PC ver 5.0. As it might be called for, there are multiple ways to run the PC version of OpenCPN 5.0 on a Mac, which I will review in another article. Several are free; others have just a modest cost.


The license you purchase from o-charts.org allows for two "installations," where each installation identifies the computer and its operating system. If you update your OS it will look like a new computer and you will have lost one install. Thus we have to think carefully on these matters. 


An alternative approach available only to PCs at the moment is to purchase a dongle from o-charts for 19 euros, and then one of the "installs" can be assigned to the dongle. This does not put the charts on the dongle, just your registration credentials for specific charts. Then you can plug the dongle into any PC or OS to make that system a legal viewer, but now it can be moved from one system to another. The system would of course have to be running a copy of OpenCPN, 4.5 or newer.


To use these SENC charts from o-charts you will need the free oeSENC plugin for OpenCPN. It is a simple install and the process of buying the charts and then creating the fingerprint is straight forward. 


I am going to order a dongle (total of 23 euros ~$26) to ship to the US. Then I will purchase the Canadian BC charts and make a video of the full process to add here.


In the meantime, there is an excellent video on the step-by-step process of obtaining and installing these charts made by o-charts.




There is no audio with the video, but that is not needed. It shows clearly every step.  Keep in mind the OpenCPN is an international product, with more language versions than any other navigation app, so no one language meets all needs in the demos.

...and somewhere down the line we do have to make our promised video series on the use of ENC, because ENC are not plug and play. They include much more info than raster or paper charts, but it takes a new approach to  chart reading to take advantage of them, which our book above provides.


Thus it is important that you know the type of chart you are getting and how to use them. You can download and work with the US versions (see starpath.com/getcharts) to see how they function, but our text on echarts (above) would help a lot with this (there are Kindle and Apple Books as well as paper). If you do not have that book, this reference from Claris on ENC objects and attributes will be helpful. This Russian version has a different format for the same data (
www.s-57.com).





Wednesday, June 5, 2019

Another Interactive Test of the Oceanic National Blend of Models

We recently completed a comparison of several models looking for the best extended forecast. The apparent winner in that case was the NBM. There are arguments on why this might in fact be expected to be best for extended forecasts; we will come back to that.

For now we have another, equally crucial test at hand. And in this case it looks like an ideal way to make this test, because the NBM is notably different at 78 hr out than the "competitors," namely GFS, FV3-GFS, and GEFS, and to the best we can estimate it, the ECMWF.

The vessel we are following is moored off of Flynn Reef in the Great Barrier Reef, 30 nmi east of Cairns in Queensland.  He needs winds solidly less than 20 kts to make progress on the last leg of his 9,000 mile row (actual miles rowed, not great circle) from Neah Bay, WA to Cairns AU.

We will use Sat Jun 8, 12z as our test point, which is 78h from the most recent model runs at 06z Wed, Jun 5.  Below are the meteograms for this period at that location that we obtain from LuckGrib.

Note: you can click one of these to zoom it, then subsequent clicks takes you through the set. 









The above is our guess of the ECMWF from windy.com. The 3 AM shown is AU local time; add 10h to 12z to get local time = 22 LT.

We will add more forecasts, every 12 hr or so. to see how things evolve.

For the moment, we would not have confidence to recommend chances of wind well below 20 on Fri / Sat, but that might change.


========== The next day:  Thursday Jun 6 ==========

Jacob needs winds well below 20 (~ 15 or so) from the SE, or he can also make progress to the west with winds of 20-25 if the direction is more easterly, that is ideally due east (090) or as far south at 110 to 120 at the lighter end of the speeds.

So far all forecasts are 20-25 at SE or even more southerly, which is no go.


 Below is the NBM meteogram for Jacobs present position


The test case we are following here is on Sat, but that wind, even if it pans out, is not a go situation.  On the other hand, we see some indication of winds going more easterly on Friday (tomorrow), still strong however.

The Cairns forecast for now also has indication of going a bit easterly on Friday... whereas all other forecasts now for before or after friday is the ole 20-25 going to 30 at SE.


Friday 7 June

Strong Wind Warning for Friday for Cairns Coast

Winds
East to southeasterly 20 to 25 knots, reaching up to 30 knots offshore at times.





We also get this indication of some more easterly from the FV3



A narrow window tomorrow.  This moment is just before daylight on Friday, so we will pass on this info to see what it looks like on the water.

----- added 04z Jun 9-------


In retrospect he did take off Friday, and indeed did get to Cairns, but it was a long hard row. We have not heard from him since arriving, but we do know the winds in the region from Cairns observations shown below.


The NBM from 84 hr out called for 21 from the SE, the actual winds were 23 or so. This is about as close as we could want. In this case the FV3 and the GFS were both over 25, also from SE, so this is not such a conclusive distinction between NBM and FV3 GFS.

On the other hand, a more subtle and encouraging verification is the wind did indeed back around to the east on Friday and called for in NBM and FV3, and that was what triggered Jacob to leave.  The wind did not at all go lighter and we are not sure of the direction at the boat, but it was a long tough row to get to cairns.  




Tuesday, June 4, 2019

FV3-GFS v. GFS—A comparison of real forecasts

On June 12, the GFS will be updated to the new FV3-GFS, which promises to be a big improvement in global forecasting.  We have been following the winds off the coast of Queensland, Australia in a real world example of needing the best possible long-range forecast. See Tools for Crucial Weather Routing: With an ongoing comparison of GFS and FV3-GFS. This note here is a summary of those results, which we can do now that we know what the winds turned out to be.

We started with a 90h forecast, using Friday May 31 at 12z as our test case.  

The boat was travelling at about 1.5 kts at the time, so the location did not effectively change relative to ocean wind patterns. But we see that at that location, the wind did indeed change a lot with time, so we stick with our 12z reference. Below are the winds for that day and location. 

Observed wind from Emerson on Fri May 31 located 154 nmi from Cairns, in direction 123T.
   
    00z to 08z  12 @ 070
   
    08z to 14z 20 @ 130-140
   
    14z  to 20z 15 @ 130-140
   
    20z  to 00z 20 @ 130-140

This location is 10h ahead of UTC, so the local times are 10h later than those shown.

Here is a summary of the results, keeping in mind the right answer is SE 20.



It is difficult to see any systematic behavior amongst this data—the meteograms in the original post might help with that.  It was a difficult situation to forecast, because the isobars off the NE coast of AU (Queensland) were affected by a huge ridge about a big High in the Bight at the southside of AU. The isobars moved around as this big high moved, and they were also influenced by fronts to the east of AU.

Except for a poor start on the speed at 90h, the FV3 did overall better than the GFS it will soon replace. That is good news.

We would have to call the Oceanic National Blend of Models as the winner of best extended forecasting. It includes the ECMWF, and after June 12 will likely include the FV3 GFS. 

The meteograms below show how the GFS and FV3 GFS evolved.



After the 90h miss, the FV3 homed in on the speed and held it, despite forecasted oscillations in wind speed, but it had trouble with direction. SE and E are not the same!


The GFS alone did not handle this situation very well.  Missing it at 90h and just getting worse.

Bear in mind that the GFS will be replaced permanently with the FV3 GFS on June 12, about a week from now... so we are in our last chances to make these comparisons.

Each of the individual meteograms is in the original study.




Tuesday, May 28, 2019

Tools for Crucial Weather Routing: With an ongoing comparison of GFS and FV3-GFS.

We will add new data at the end here until we figure out which model wins out!

It can happen that crucial decisions depend on details of a forecast. Usually this will be for a longer term forecast, since one or two days out are generally pretty good.  We have a specific case at hand now, which is a classic, real-world example.  The vessel has two optional routes, on each a crucial turn must be  made roughly 96 hr from now, meaning we are in interested in what happens on Friday, May 31.

The two main primary sources we have are the GFS and the new FV3 GFS, which will replace the former on June 12.  We see below that these two do not agree at all for Friday.




Again, Friday shown here is 4 days out from these  Monday 18z forecasts.  The displays are the meteograms from LuckGrib.  The standard GFS we have used for years shows roughly 20 kts from the east, or just above east.  The in-principle-better model FV3 GFS shows 30 kts from the SE.  For the vessel at hand here, and almost all vessels for that matter, this is a huge difference.

These LuckGrib meteograms are plotted with the vessel following a specific course at a specific speed. Thus the output applies to the time and location of the vessel.  Below we see these two data images with the vessel in the location shown by the blue line in each.



For a specific comparison at 12z on Friday, the GFS is 18.3 kts at 077 whereas the FV3 GFS is 27.5 kts at 139.

So what can we do faced with this? GFS itself offers no probabilities or other qualifications.  We have in our textbook Modern Marine Weather several ways to evaluate a forecast using the 500-mb maps and other means, but this situation is rather different from the examples in the book as this is for the Southern Hemisphere, off the coast of Queensland.

We got the above data from the powerful LuckGrib app, which offers many model options and very neat presentation of the data. Two models offered in LuckGrib that help us look at probabilities are the standard deviation outputs from the Global Ensemble Forecast System (GEFS) model and the probabilistic wind forecasts from the Oceanic National Blend of Models (NBM).

Below are the same meteograms (route and dates) for these two models, followed as above with the actual plot of the wind showing values at 12z on Friday.



And now the specific values at the vessel at 12z Friday.




These data require a bit of explanation.  The GEFS ensemble data comes from starting a computation with a base model forecast, called the Control run, which is similar but not identical to the base GFS forecast. Then they repeat this forecast 20 times, each time making some likely variation, such as changing the input data or trying different physics solutions. The idea is that the spread in these results is a measure of the uncertainty in the forecast. We could actually look at each of the 20 runs to see how widely they vary, or as LuckGrib has decided we look at the mean values of all of the runs along with the standard deviation of this set of runs.

The standard deviation (SD) is a measure of the spread in uncertainty assuming the different runs introduce more or less random differences. In the most basic sense, SD is defined in statistical terms with the curve below, from our book Mariners Pressure Atlas, in which we use these ideas for storm forecasting in the tropics based on pressure values.



The basic idea is the larger the SD, the broader the distribution of winds, with 68% of the observed winds expected to be within 1 SD of the mean, as shown below.

In the presentation of this GEFS data, we might expect the forecasted spread of wind speeds to be the mean wind ± 1 SD, but this is not what we see. Referring to the actual output of the GEFS (above) we see


which is not the same as 20.5 ± 6.2, which equals 14.3 to 26.7 kts, instead we see 16.2 to 25.5 kts.

This is a subtlety that might not matter too much in many practical situations, but since this output itself is a new concept, it might help to know the source of the apparent discrepancy. The issue is the GEFS model presents the SD as a vector quantity, meaning they do not give it for the wind speed alone, but rather they give separate values for the E-W component and the N-S component of the wind vector. Thus we have a situation like the one shown below.


In this sketch, the SD is called "sigma," shown as a dashed line.   u is the E-W SD and v is the N-S SD. The quoted value of SD in LuckGrib is sqrt (sima-u^2 + sigma-v^2).  The red vectors mark the spread of the wind speeds, and the blue lines mark the spread of wind angles reported.

Here is the latest data from Tuesday morning:




First we see that the GFS and FV3 GFS are now closer with the former at 14.3 @ 105 and the latter still higher at 22.1 @ 123.  The directions are closer, but speeds are still notably different at 3.5 days out.

 

The GEFS give us a spread of 15.0 to 23.5 kts, with directions varying from E to SE. So according to the ensemble analysis, this forecast is still up in the air.


The NBM calls for a mean wind of 21.1 kts from the SE, with 25% chance of being > 23.1 kts and 25% chance of being <17.1 kts. Again, no reassurance from this at all.

That is where we stand. No confidence for crucial decisions on or near Friday.

I will contact our meteorologist friends in Australia to see if they have any insights on what is taking place and what they think might happen.  This could all be tied to pressure systems moving across the continent of Australia which they will likely understand.

In the meantime the vessel (Jacob Adoram's Emerson) is going as slow as possible waiting for some guidance in the forecasts.

With the GEFS data we were only looking at the standard deviations, but this model also computes probabilistic winds, but the public presentation does not reach down to Australia.

Following on to see how these forecasts pan out...

======= New data from Wednesday morning =======

Update times from 06z to 18z, as explained below.



These two models are definitely closer now. We will also see  below that the ECMWF is right between these two forecasts.  We still have the FV3 calling for much stronger wind on Sat than GFS.

We can compare these to the ECMWF, which can be read from windy.com.


This is the summary:

GFS 13.6 @ 105
FV3 17.0 @ 120
EC    16 @ 105
and
GEFS 18.2 @115  with range  15.3 to 22.2 @ E to SE
NBM 21.6 @ 132  with a 25% chance of < 17.5

So both the probabilistic models (see below) cover the predictions of the 3 direct models.  The NBM looks to be the most off, but we have to keep this in perspective. The NBM for US waters updates every hour, but this oceanic version updates on a strange schedule. There is a new run at 00, 05, 07, 12, 17, and 19z. It then has a short latency and is available in about 1 hr after each run.  The GFS, for comparison, is run every 6 hr at the synoptic times, but it has a total latency of about 5 hr. GEFS has a latency of about 6 hr and FV3 is about 7 hr. So in principle, the Oceanic NBM could be the newest data... but this remains unclear till we learn more. It is blending models that are much older, so the full implications are not clear yet.

Now that we have this data so conveniently from LuckGrib, the next step will be to learn more about how to use it, which in a sense we are struggling with at the moment.





We will continue to add  data when we get it. In the meantime, we look at the overall weather patterns that are leading to this in another post.

====== Thursday morning ==========



For our 12z Friday test case and adding a Sun 12z new test case, we have:

FV3 19.9 @ 093       Sun 12z = 20 @ SE
GFS 16.1 @ 107       Sun 12z = 22 @ SE

EC1 17.5 @ 090       Sun 12z = 14.5 @ SE
EC2 23.5 @ 132       Sun 12z = 19.7 @ 107

We do not have direct access to ECMWF data as this is licensed data (ie the UK sells it), but we have two ways to access it, at least in principle.  We have an account with PredictWind, which offers a model called PWE that is based on the ECMWF data and we have the windy.com output, which they claim is the ECMWF. These are shown below.


Again, this is AU time, so 12z on Fri is at 22z, which is 17.5 @ E, with a GFS report of 15.5 @ 110, which is close to what we read directly for the GFS. The EC forecasts 20+ winds all the way through next Tuesday.


This PredictWind version of EC that we access via the nav program Expedition. This gives 23.5 @ 132 for Friday.  These two presumed sources of EC wind are so different we have to conclude we do not know what the official EC forecast is. Below are the actual numbers for the PredictWind version:

30-May-19 0300,,107,7.2
30-May-19 0600,,120,10.0
30-May-19 0900,,118,13.2
30-May-19 1200,,115,15.9
30-May-19 1500,,121,16.8
30-May-19 1800,,138,18.4
30-May-19 2100,,136,20.7
31-May-19 0000,,129,19.7
31-May-19 0300,,122,19.2
31-May-19 0600,,132,21.4
31-May-19 0900,,131,23.8
31-May-19 1200,,132,23.5  (windy.com value 17 E-SE)
31-May-19 1500,,130,23.2
31-May-19 1800,,127,22.9
31-May-19 2100,,121,23.6
01-Jun-19 0000,,111,23.3
01-Jun-19 0300,,107,22.3
01-Jun-19 0600,,110,22.4
01-Jun-19 0900,,118,25.3
01-Jun-19 1200,,121,27.1
01-Jun-19 1500,,121,26.0
01-Jun-19 1800,,123,25.6
01-Jun-19 2100,,118,26.6
02-Jun-19 0000,,111,25.1
02-Jun-19 0300,,111,21.5
02-Jun-19 0600,,112,21.9
02-Jun-19 0900,,111,21.5
02-Jun-19 1200,,108,20.0  (windy.com value = 15 SE)
02-Jun-19 1500,,105,18.8
02-Jun-19 1800,,109,19.0
02-Jun-19 2100,,114,18.6
03-Jun-19 0000,,127,18.2
03-Jun-19 0300,,134,19.3
03-Jun-19 0600,,136,20.1
03-Jun-19 0900,,138,22.4
03-Jun-19 1200,,142,25.0
03-Jun-19 1500,,153,26.7
03-Jun-19 1800,,156,26.8
03-Jun-19 2100,,162,28.0
04-Jun-19 0000,,160,27.8

With these model forecasts at hand (GFS, FV3, and some sort of EC) we can look at the probabilistic winds from this morning:

We see here a still surprising result, namely for tomorrow, less than 24h away, the uncertainty is still large. We have mean forecast of 17.7 @ 111, but with a spread of 15.4 to 20.4 over E to SE.  This range is significant to almost all sailboats, 15 kts vs 20 kts.


We get a bit more specification from NBM. A median wind of 19.3 from the SE, but only 10% chance of less than 14.7; 25% chance less than 16.2. So low wind side is unlikely, whereas we still have a high wind side.

At this point, I do not see any justification for gambling on the lighter side of the forecast, and indeed we should likely count on more rather than less.

With all this said, and still more to say, we can pause an look at how these models have done so far in predicting the winds that Jacob is actually reporting and we can measure with ASCAT.

This article will soon be a live link:  Compare ASCAT, Observations, and Model Forecasts.




Monday, May 27, 2019

Compare ASCAT and WindSAT Scatterometer Wind Data

We have two sources of scatterometer data these days, the 3 Metop satellites with ASCAT data and the US Navy Coriolis satellite with WindSAT data. The latter does not get much mention in official NWS forecast discussions, whereas ASCAT is referred to frequently. The reason for this is not clear, so in  a first step toward trying to understand this, we will compare the data whenever we can spot passes at about the same time, over the same region.  It could be we can calculate that conjunction of passes, which if so we will add to our satellite prediction time article, which is underway.

The OSWT site presents the ASCAT data in a 10º x 15º Lat-Lon grid and the WindSAT in a 20 x 30 grid. They both have in principle 25-km resolution, so the reason for this is not completely clear. It could be simply that the single WindSAT data swath is about twice as wide as one of the halves of the two ASCAT swaths, so it would likely take a different file layout to account for it.

If the WindSAT data are consistent with ASCAT, then this would be to many mariners a "new" and useful tool, because it has a wider swath of data without a nadir gap like ASCAT has. In that sense, the WindSAT scatterometer is more like the old QuikSCAT that provided so much valuable data before it failed—long after its design lifetime.

Below are several examples, the newest will be at the end where we are adding them. There is there a May 31 example that shows a larger difference than earlier ones... and we know what the answer is!

Example 1. Time difference 3h 05m


Figure 1. ASCAT valid 1153z on May 27.


Figure 2. WindSAT valid 0848z on May 27, about 3 hr earlier than the ASCAT pass


The data are not exactly at the same time, which has to be considered, but the reported winds are generally similar, though not at all identical. Yellow means above 20; green means below 20. Blue means below 15.  

So from this comparison we can say, if we assume the ocean had not changed, the wind data agree to within about ± 2.5 kts, which is all it takes to change the colors of the barbs. There is a yellow patch in WindSAT that is green seen in ASCAT, but the the green is a dark green, which is closer to yellow.  

In the next two examples, we put position markers at the colorbar wind-speed boundaries on one image and then read the wind at the corresponding positions on the other image.

Example 2. Time difference 3h 04m




Example 3. Time difference 3h 26m




Several Examples to be inserted here.


Example x. Time difference 2h 58m
This is one of particular interest, because we have been tracking Jacob Adorams's  ocean rowing vessel Emerson in this region at this time... and we know from him what the actual winds were at the time.  It is also the case where we see the largest difference so far between ASCAT and WindSAT.

Here is where Emerson was located (yellow stared marked) at the time of these passes:




And here is what the actual winds were on this day:

At the time of both satellite passes, the wind was 20 kts from the SE.

We are looking specifically at the region in the bottom of the outlined section, where ASCAT reports 20-22 kts from the S-SE (152T), whereas WindSAT shows rain contamination over this region but wind vectors from many directions. The ASCAT is consistent with observations well within the uncertainty of the observations.

At first glance this seems a big difference, but the WindSAT is more sensitive to rain than ASCAT and there are indeed many rain squalls in this region at this time.





To give WindSAT the benefit of the doubt, at first we thought maybe WindSAT was trying to show the unusual observation we got from the boat of 10 kt winds from 070 for Friday morning and much of Thursday.  But on the 30th, we could spot a local High in the region that would have accounted for this. It showed up nicely the BOM "gradient wind" streamline maps—we will be discussing these in another blog post.

With that in mind, we looked to the surface analysis maps shown below; one before these two satellite passes (18z), and one afterward (00z, the next day).


The magenta region shows the overlap in scatterometer data. We see the gradient actually weakening in the southern part where the winds differ most during this period, whereas the later ASCAT shows higher winds. Keep in mind this is SH, winds going counterclockwise around the High and out of the High into the Low, meaning this wind direction is SE, tending toward S-SE.  

In short, the maps do not support the WindSAT observations.  At this point we need to ask the experts what they can tell us about this. 

We will report back here when we learn more... and we will add several other examples above this one. They are done, just not posted yet.