Friday, August 4, 2017

How to Combine Grib Files

It is not too often we have to do this. Some grib viewers and nav programs that show grib files will either let you select more than one at a time to show (ie OpenCPN and Expedition allows for this), or they might offer custom packages of say wind and current.

In Expedition, for example, you just load grib files from any source into their grib folder and put checkmarks next to the ones you want to combine.  In OpenCPN use the file open option then select multiple files with the cmd or ctrl key, and that will combine them as it loads them.... in most cases; there are exceptions.

The free navigation and routing program qtVlm (Mac or PC) has a neat grib combiner utility that combines grib files with option to rename and store the combination where you choose. It is a versatile program in its own right, but this utility within it lets you create files for use in any program. 

Nobeltec and Maxsea TimeZero apps provide custom packages of grib files (wind, current, seas) intended for optimum routing, but if you want to optimize the data using other than their choice of data we have to combine the ones we want on our own, as outlined below. For typical routing calculations we need wind, current, and wave data that come from different sources.

Zygrib (and XyGrib) allows for simultaneous download of ww3 wave data along with GFS wind data.  For inland and coastal waters, however, GFS wind data will rarely be the best choice. Zygrib does not allow for multiple file selection in their load from file option, so this is one that needs external combination as well. Likewise to include current with any file requires manual addition.

Saildocs offers a compromise or sorts, by adding a "waves" parameter to their GFS downloads. But waves are not part of the GFS model; they have taken one component of the ww3 model and just added it onto the GFS grib files themselves, so users can get both at once and not have to combine them manually. Again, though, this might not be the waves and wind combination we want. Furthermore, the "waves" addition alone does not bring wave direction, just height.  You can however append to a saildocs request the specific ww3 parameters you want it to add. 

One of our favorite grib viewers, LuckGrib, does not allow for opening multiple files, so to see combined data in that program, we have to add them externally.

(Beyond these more common calls for the need to externally combine grib files, we come to the high resolution NDFD wind data.  At least for the time being, I am only aware of two ways to access this data.  Either use LuckGrib (on a Mac or iPad) to download the wind and then export that (single) grib file to use in other programs, or get them directly from NOAA as ds.wdir.bin (wind direction) and ds.wspd.bin (wind speed) and then combine them as shown below—the .bin extension is a valid grb2 file for those nav programs that read grb2 data.  On the other hand, and the reason for the ( ) on this note is, this is a bit academic, in that, to my knowledge, Expedition and LuckGrib are the only two programs that can display these state of the art data files. And in Expedition, you can indeed just load the speed and direction files individually into their grib folder and it will combine them and show the data.)

And there are other subtitles we face when striving for the best data for specific locations. Meteo France, for example, offers a high quality waves and wind file (similar to the US WW3), but the wind part is coded like the NDFD wind data, meaning it is expressed as two values, wind speed and wind direction. OpenCPN and several other grib viewers will not read that type of wind data. They expect the more standard format of the two components being the E-W wind speed and the N-S wind speed and then they combine these for the vector winds they display. If you load one of these French .grib2 wind and waves files into OpenCPN you see only the waves.

In short, for optimum work we will likely have occasion to combine grib files manually, but with that said, it turns out to be an easy process, in either a Mac or a PC. The process is done in a command line statement, which may not be a familiar operation to many, so it is described here as if it had never been done before.

We use these files currents.grb, waves.grb, wind_pressure.grb from Norway Met Inst as an example. They have been renamed here. These files also illustrate other subtleties of grib file display, once we wander away from our comfortable US sources. We put these in the folder /Downloads/Norway.

Mac process
(1) in spotlight search, type Terminal, then click open the terminal app. It will look something like below, showing your computer name and home directory.

Then type: cd Downloads/Norway, and enter.... to get to the place where the files are located.

Then type: cat wind_pressure.grb waves.grb currents.grb >skaggerak080417.grb.  We make up the final file name ourselves. This one has the place and date. Can't put the time on this one, as these files do not cover the same time period, which is an issue we will have to deal with.

It won't say anything unless an error was made, but this new file should show up in Downloads\Norway and it should be about the size of the other three combined.

Now in principle we are ready to test our new combined file, which now includes wind, pressure, current, and waves. Current data typically has set, drift, and sea water temp; waves can have numerous parameters, which is more a topic for another post.

Above is combined file in OpenCPN. It will take some trials to figure best display choices with multiple parameters. This one is not optimized. For routing it does not matter what the picture looks like. Note that pressure does not show up, and this can be understood. I will cover that in the video or different article. Likewise, a video is easier to illustrate the timing issues, as these forecasts are not all for consistent times. A routing program, however, can interpolate these between forecasts as needed.

Below is the same file viewed in LuckGrib, using only arrows for display. Luckgrib has many display options for every parameter—best of any program—so we can do much better as needed.

Below we shut off all parameters but current for a better picture of what is taking place in this bay.

Note that as powerful as LuckGrib is, it will not display more than one grib file at a time, so this process is needed when we want to see, say, wind and current overlaid. OpenCPN, will indeed let us combine the three files we just did manually, so in this example our special process was not needed. We would get the same display by just selecting multiple files at the Open File dialog. 

Now we proceed to the process of combining grib files in a PC.

PC process

This time we put the Norway folder on the PC desktop. It does not matter where these are, but my PC is a virtual machine in a Mac, and I share the Mac Downloads folder which confuses this example, so we use this new location. In general, it is best to put the folder you are working on close to the root, so it is easier to get into it.

(1) In the main windows search box, bottom left, type CMD, enter.  This will find an app called Command Prompt.  Click that to open. It will look like below.

This is the PC counterpart to the Mac Terminal, and we start in the user's home directory.

(2) Type CD Desktop\Norway and enter. (Note the slashes in Mac and PC are opposite.)

(3) Then type: type currents.grb wind_pressure.grb waves.grb > skaggerat080417.grb and press enter, and you will get this, which means it worked.

Then if you check the Norway folder you will see your new file is there.

For both the PC and the Mac there are various short cuts and other commands that help with this if you care to learn more about this.  For example, in Mac or PC instead of listing the files to add, you can name them in the base folder with some initial ID, ie add z1 in front of every file name. Then the command cat z1* >z1.grb  or type z1* >z1.grb will do the job. Thus you can organize the files ahead of time with chosen prefixes.

Alternatively, you can download qtVLM and use it for a grib combiner!  Read about a popular use of this program.

The mentioned issue with the pressure from these Norwegian data is the reference level. Most grib files meant for maritime application use pressure at MSL (pressure@0-MSL or PRMSL@0-MSL), which is what we see on weather maps. The Norwegian files use a pressure above ground level (Press@0-HTGW). LuckGrib and Expedition take this in stride and show it, though we must be careful to remember that the isobars on land are relative the ground, not MSL. On the water it does not really matter, but you will see kinks in these as the isobars climb up the hills on land and not agree with the weather maps on land.

OpenCPN just ignores these pressure data and opens the files normally otherwise. Just with no pressure to see. The same with qtVlm. Zygrib will not open the files at all if they cannot identify all the the parameters, so you cannot use the program with that type of data.  When studying the effects of terrain on wind, the ground referenced isobars can be interesting to look at. The UW WRF model uses this, so we see the actual station pressure a person living on a hill would see.  Still a bit confusing for maritime work with a weather map in hand.

Here is a short video on how to adjust LuckGrib to create a draw style of this new pressure relative to ground level, which is not what we normally want in marine applications.

When this won't work

There are some limitations to the process. The normal goal is to add parameters from different models to a given region and time frame. If we wanted to combine several adjacent regions covering the time same time and same parameters we would find that this will not work in most viewers. Most viewer software would be conflicted on various stages of the display if it had, for example, wind speed at the same time in two different locations in the same data file. 

The only exception I know of is Expedition, that will allow for opening multiple adjacent regions covering the same time and parameters. Also it does not need external combining, it will do this itself.

In contrast, the program qtVlm, which does offer a way to combine GRIB files, will not allow for same parameters and time over adjacent regions.

No comments: