FV3-GFS v. GFS—A comparison of real forecasts

On June 12, the GFS will be updated to the new FV3-GFS, which promises to be a big improvement in global forecasting.  We have been following the winds off the coast of Queensland, Australia in a real world example of needing the best possible long-range forecast. See Tools for Crucial Weather Routing: With an ongoing comparison of GFS and FV3-GFS. This note here is a summary of those results, which we can do now that we know what the winds turned out to be.

We started with a 90h forecast, using Friday May 31 at 12z as our test case.  

The boat was travelling at about 1.5 kts at the time, so the location did not effectively change relative to ocean wind patterns. But we see that at that location, the wind did indeed change a lot with time, so we stick with our 12z reference. Below are the winds for that day and location. 

Observed wind from Emerson on Fri May 31 located 154 nmi from Cairns, in direction 123T.
   
    00z to 08z  12 @ 070
   
    08z to 14z 20 @ 130-140
   
    14z  to 20z 15 @ 130-140
   
    20z  to 00z 20 @ 130-140

This location is 10h ahead of UTC, so the local times are 10h later than those shown.

Here is a summary of the results, keeping in mind the right answer is SE 20.

It is difficult to see any systematic behavior amongst this data—the meteograms in the original post might help with that.  It was a difficult situation to forecast, because the isobars off the NE coast of AU (Queensland) were affected by a huge ridge about a big High in the Bight at the southside of AU. The isobars moved around as this big high moved, and they were also influenced by fronts to the east of AU.

Except for a poor start on the speed at 90h, the FV3 did overall better than the GFS it will soon replace. That is good news.

We would have to call the Oceanic National Blend of Models as the winner of best extended forecasting. It includes the ECMWF, and after June 12 will likely include the FV3 GFS. 

The meteograms below show how the GFS and FV3 GFS evolved.

After the 90h miss, the FV3 homed in on the speed and held it, despite forecasted oscillations in wind speed, but it had trouble with direction. SE and E are not the same!

The GFS alone did not handle this situation very well.  Missing it at 90h and just getting worse.

Bear in mind that the GFS will be replaced permanently with the FV3 GFS on June 12, about a week from now... so we are in our last chances to make these comparisons.

Each of the individual meteograms is in the original study.