Getting Ready With The Data

* Preparing the Data

OBJECTIVE: The objective of this work flow is to take archive data provided by Environmental Canada for the Strathmore radar and create MDV formated data files that are viewable by the LROSE Rview application.

DATA SET: The summer 2017 Strathomore Radar data set was obtained from Environmental Canada and is currently available via ftp.

Create the projDir.strathmore directory:

mkdir ~/projDir.strathmore/

Create subdirectory

cd ~/projDir.strathmore/
mkdir CVS data display ingest params scripts system   

Create data subdirectories.

cd data
mkdir ascii images mdv NetCdf raw spdb verify 

Create data/raw subdirectories.

mkdir ~/projDir.strathmore/data/raw/radar
mkdir ~/projDir.strathmore/data/raw/radar/strathmore
mkdir ~/projDir.strathmore/data/raw/radar/strathmore/20170601/

NOTE: The “20170601” corresponds to the directory of particular day, in this case 1st June 2017, which uses the yyyymmdd format.

Create DOPVOL subdirectory.

mkdir ~/projDir.strathmore/data/raw/radar/stathmore/DOPVOL

Create additional data subdirectories.

mkdir ~/projDir.strathmore/data/NetCdf/radar
mkdir ~/projDir.strathmore/data/NetCdf/radar/strathmore/
mkdir ~/projDir.strathmore/data/mdv/radar
mkdir ~/projDir.strathmore/data/mdv/radar/cart
mkdir ~/projDir.strathmore/data/mdv/radar/polar/
mkdir ~/projDir.strathmore/data/mdv/vil/

Uncompress raw IRIS formatted data files.

unzip compressed_data.zip -d ~/projDir.strathmore/data/raw/radar/strathmore

NOTE: The directory strathmore should be populated with the raw data files.

Moving Raw Data files into specific directories.

mv 20000501?*CONVOL* ~/projDir.strathmore/data/raw/radar/strathmore/20000501  
mv 20000502?*CONVOL* ~/projDir.strathmore/data/raw/radar/strathmore/20000502

..and so on !!

mv 200005?*DOPVOL* ~/projDir.strathmore/data/raw/radar/strathmore/DOPVOL

Download the RadxConvert params file from here.

wget --trust-server-names http://wiki.atmos.und.edu/lib/exe/fetch.php?media=atmos:software:lrose:radxconvert_strathmore_20200318.params -O radxconvert_strathmore_20200318.params

NOTE: To generate a default params file use the “RadxConvert -print_params > ~/projDir.strathmore/display/params/RadxConvert.params” command can be used.

Convert a day's worth of data.

RadxConvert -f ~/projDir.strathmore/data/raw/radar/strathmore/20000501/*CONVOL* -params ~/projDir.strathmore/display/params/RadxConvert.params

Run similar command for all other days of data.

NOTE: The NetCdf/radar/strathmore directory should now have all the yyyymmdd directories containing the data in the NetCdf format.

Download the Strathmore params files from here.

NOTE: An example params file for Radx2Grid can be generated using “Radx2Grid -print_params > ~/projDir.strathmore/ingest/params/Radx2Grid.params”

Change your directory to:

~/projDir.strathmore/ingest/params

Use vi Radx2Grid.params to get inside Radx2Grid.params and make the following changes:

input_dir = "~/projDir.strathmore/data/NetCdf/radar/strathmore";
mode = ARCHIVE;
start_time = "2000 05 01 00 00 00";
end_time = "2000 07 31 23 59 59";
output_dir = "~/projDir.strathmore/data/mdv/radar/cart";

Go to command line and run the following:

Radx2Grid -params ~/projDir.strathmore/ingest/params/Radx2Grid.params

This should successfully convert the data from NetCdf to mdv format. The data/mdv/radar/cart directory should have the yyyymmdd directories with the data in mdv format now !!

This step is necessary only if your research involves working with vil.

The program Mdv2Vil is used for this conversion.

Make sure you are at the following location:

~/projDir.strathmore/ingest/params

Generate the params file for Mdv2Vil using the following command from command line:

 Mdv2Vil -print_params > ~/projDir.strathmore/ingest/params/Mdv2Vil.params
 

Run vi Mdv2Vil.params to get inside Mdv2Vil.params and make the following changes:

 input_dir = "~/projDir.strathmore/data/mdv/radar/cart";
 mode = ARCHIVE;
 output_dir = "~/projDir.strathmore/data/mdv/vil";
 

Run the following from the command line:

 Mdv2Vil -start "2000 05 01 00 00 00" -end "2000 07 31 23 59 59" -params ~/projDir.strathmore/ingest/params/Mdv2Vil.params

The data/mdv/vil directory should now have the yyyymmdd directories containing the vil of the data in mdv format.

VIL is a quantity computed throughout the vertical extent of a storm - right from its base all the way up to its top. In order to compute VIL only above a certain height, make the following change in the Mdv2Vil params file:

Under the min_height option in the COMPUTING VIL section in the params file,

min_height = 4;

As a result of the above step, Vil would be calculated from a height of 4 km all the way up to the storm top. If min_height = 0, Vil would be calculated from a height of 0 km (basically ground level) to the storm top.

For this project, only Vil above the melting layer is used. The average melting layer height of all the storms studied is a little under 4km. So, the value 4 is chosen in this example.

Now that the data is set up, it would have to be displayed using LROSE. For that a symbolic link should be created.

The computer might store data from a number of different radars. LROSE does not know which radar data it should display. Setting a symbolic link is a way of telling LROSE which radar data it should display. If the symbolic link points to the Strathmore data, LROSE would display the Strathmore data. If the symbolic link is made to point to the data from Hogwarts, LROSE would display the Hogwarts data.

To create the symbolic link, make sure you are in the yourfirstname.lastname directory.

Then run the following command from terminal:
ln -s projDir.strathmore projDir  

where, projDir.strathmore is the directory being pointed at and projDir is the name of the symbolic link that has been created.

To check if the symbolic link is created, run the following command from terminal:

ls -l projDir

If the link gets created successfully, the output should be:

projDir -> projDir.strathmore

NOTE: The above line may or may not be the only line of output. Other links and directories for which the user has read/write permission would also get displayed.

Now that the link is created, it should be displayed on the Rview window of LROSE by running the following command from terminal:

Rview -params ~/projDir/display/params/Rview.archive

or include start data/time.

Rview -params ~/projDir/display/params/Rview.archive -time "2017 07 08 12 09 56"