Note: Throughout this document, the characters “cluster>” indicate a line that will be executed from the command line. The “—“ indicate sections that are part of text files that need to be added or edited.

Document created by G. Mullendore for ATSC 405 (2009)

A. Setting up WRF/WPS

The first time you use WRF, you'll need to copy and untar the model and visualization software.

WRF: Weather Research & Forecasting Model

WPS: WRF Preprocessing System

RIP: “Read/Interpolate/Plot” Visualization Software

1. Copy the files to your home directory and untar them. The programs are located in

/home/data/wrf311_cluster.tar.gz (both WRF and WPS)

/home/data/RIP4.tar.gz

Note: If you ran WRF & RIP for Mesoscale class, you still need to copy the wrf311 file as it’s a newer version and includes the preprocessing system. You do not need to copy RIP.

2. Update your .bashrc

—

# User specific aliases and functions

#aliases

alias rm='rm -i'

alias cp='cp -i'

alias mv='mv -i'

#PORTLAND

export PGI=/opt/pgi

PATH=$PGI/linux86/7.2/bin:$PATH

PATH=$PGI/linux86/7.2/mpi2/mpich/bin:$PATH

MANPATH=$MANPATH:$PGI/linux86/7.2/mpi2/mpich/man

export PATH MANPATH

export LM_LICENSE_FILE=$PGI/license.dat

#NETCDF

export NETCDF=/usr/local/netcdf

export PATH=$NETCDF/bin:$PATH

#NCAR Graphics

export NCARG_ROOT=/usr/local

export PATH=$NCARG_ROOT/bin:$PATH

# for WPS

export JASPERLIB='/usr/local/lib'

export JASPERINC='/usr/local/include'

#RIP- Replace “full_path” with the path to your RIP4 directory

export RIP_ROOT=/full_path/RIP4

export PATH=$RIP_ROOT:$PATH

—

cluster>source .bashrc


B. Setting up your account to work with multiple processors- DO ONCE FOR EACH FORECAST CITY

If using multiple processors all the time, you would not need to reset your daemon for each city. However, because we are sharing the cluster with other users, your own set of node numbers may change for each extra credit group.

1. Start the mpd, or message passing daemon. This daemon is a program that controls the way the processors communicate when working together on the same problem.

1) Shut down any message passing daemons currently running

cluster> mpdallexit

SUBSECTION: ONLY IF FIRST FORECAST

2) Create configuration file.

cluster>gvim ~/.mpd.conf

—

MPD_SECRETWORD=abra2-du10o

—

Note: Make up a new secret word! Anything is fine for the secret word. If you have already run models, you will already have this file in your directory.

3) Change the permissions of the configuration file.

cluster>chmod 600 ~/.mpd.conf

END SUBSECTION: ONLY IF FIRST FORECAST

4) Start daemon.

cluster>mpdboot -n # -f ~/machines.boot

Replace # with the number of nodes in your machines.boot file plus one.

5) Test daemon.

cluster>mpdtrace

You should see a list of your subset of nodes (should be same names as in machines.execute, but may be in different order and will include the head node, called “cluster”). If you do not see a list, start at number 1 above and double-check all your steps. If still not producing the right list, email Gretchen the output you are getting from “mpdtrace”.

C. Initialize forecast

SUBSECTION: ONLY IF NEW LOCATION

1) Set domain

Open the cluster> wrf311/WPS directory.

namelist.wps contains a list of settings for the forecast run.

a) Always a good idea to keep a copy of the original namelist file around before editing a new one:

cluster>cp namelist.wps namelist.wps.orig

b) Edit namelist.wps

To set the domain for the region we want to simulate, edit the “&geogrid” section of the file. Notice that there are two columns. The first column is for the main simulations, additional columns are for “nested” grids. We will not be using nested grids for these forecasts.

—

geog_data_path = this is the location of the geographic data

set as '/home/data/WPS_GEOG'

dx, dy = this is the grid spacing in x (e-w) and y (n-s) in meters

the input data is 12km, so I recommend choosing 6 or 4 km resolution

geog_data_res = resolution of geographical data

set as '2m'

ref_lat, ref_lon = the center latitude and longitude locations for the forecast

e_we = ending grid point in the e-w direction

for example, it you want to run a forecast for a 100 km region, and your dx is set to 4000, this would be 25

e_sn = ending grid point int eh n-s direction

In &share section:

max_dom = 1,

—

Note on domain size and time step:

-I tried some small domain runs (25×25 grid, 4km resolution), and the near-boundary pressure gradients were very noisy. It's normal to get some noise near model boundaries- one should never use output from right near the boundary- a 25×25 grid is so small that noise may exist in the area of interest. There is still noise in the 50×50 grid (6km resolution, 24s time step) but the domain is larger, giving more confidence in the central area of the domain.

c) Check domain

./util/plotgrids.exe

idt gmeta

-scroll right arrow to see plot

-plotgrids.exe only shows state outlines, so for a small domain you may not see much. You can increase your domain size to see if you're centered in the right spot, and then decrease the domain size again to region you want. Note: plotgrids.exe runs on what is in “namelist.wps”, so you do not need to run geogrid.exe first.

d) After checking that the domain is setup correctly, run the domain setup:

mpiexec -l -machinefile ~/machines.execute -n # /full_path/WPS/geogrid.exe > geogrid.log 2>&1 < /dev/null &

-replace full_path by the path to your WPS directory and # by the number of nodes you want to run on (can't be higher than the number of nodes listed in ~/machines.execute! Also, # should be the number of processors you want to use, since cluster only have one processor on one node, it doesn't matter in this case. In other cases, e.g. hpc has 8 processors on 1 node, so you have to put the actual number of processors behind '-n'.)

-The above command runs on the processor list set in your own machines.execute file.

ssh to a node to check your job.

-When the job has completed successfully, you'll get a message saying so at the end of geogrid.log.

SUBSECTION: ONLY IF NEW TIME

3) Link 12 km NAM files

Note: We will always be running a forecast from 00Z (19 CDT). Because of the lag due to the NAM having to complete first, and then the time it takes “cluster” to download the NAM files, initialization files will not be available before midnight CDT.

NAM files are downloaded into /home/data/nam/YYYYMMDD, one for each day (the 00Z forecasts).

a) Edit namelist.wps

—

start_date = 'YYYY-MM-DD_00:00:00'

end_date = 'YYYY-MM-DD_00:00:00'

-set to 2 days after start_date

interval_seconds = 10800

-this is the time between NAM input files = 3 hours

—

b) link Vtable

ln -s ungrib/Variable_Tables/Vtable.NAM Vtable

c) link to NAM files

./link_grib.csh /home/data/nam/YYYYMMDD/nam.*tm00.grib2

-use the date the forecast starts from

d) pull needed variables from NAM data

mpiexec -l -machinefile ~/machines.execute -n # /full_path/WPS/ungrib.exe > ungrib.log 2>&1 < /dev/null &

-replace “full_path” with the path to your WPS directory

-replace # with the number of processors you want to run on (maximum is number in your machines.execute file)

-this step takes 4 hours on 5 processors- this long time is due to the high resolution of the NAM input files

SUBSECTION: IF NEW LOCATION OR NEW TIME

4) Create WRF init files

mpiexec -l -machinefile ~/machines.execute -n # /full_path/WPS/metgrid.exe > metgrid.log 2>&1 < /dev/null &

-for a 50×50 grid, this step about 30 minutes on 5 processors

5) Move init files to WRF run directory

mv met_em.d01.*.nc ../WRFV3/run

D. Run forecast

1) Edit namelist.input

cd ../WRFV3/run

namelist.input:

basic explanations of variables:

http://www.mmm.ucar.edu/wrf/OnLineTutorial/Basics/WRF/namelist.input.htm

more details on physics options (&physics):

http://www.mmm.ucar.edu/wrf/users/docs/user_guide/users_guide_chap5.html#Phys

At minimum, edit the following lines in namelist.input:

—

run_hours = 48,

start_year… end_hour: change to match the time period you are simulating

interval seconds: 10800

time_step: change to be 6*dx, but also good to pick a time_step that evenly divides into the output time step (history_interval). For example, if using dx=4000, 4*dx (in km) = 24. If history_interval = 180 (minutes), then 24s is good, because it divides into 10800s evenly.

e_we, e_sn, dx, dy = change to match setup values

num_metgrid_levels = 40

cu_physics = 0,

-turns off cumulus parameterization. However, 4 or 6km is at the edge of resolving convection, so in some cases you may get a better answer if you leave this on.

pd_moist = .false.,

pd_scalar = .false.,

—

2) Run WRF init

mpiexec -l -machinefile ~/machines.execute -n # /full_path/WRFV3/run/real.exe > real.log 2>&1 < /dev/null &

NOTE: if they exist, delete rsl files first: \rm rsl.*

3) Run forecast

mpiexec -l -machinefile ~/machines.execute -n # /full_path/WRFV3/run/wrf.exe > wrf.log 2>&1 < /dev/null &

NOTE: if they exist, delete rsl files first: \rm rsl.*

-test run of size 50×50 took 80 minutes on 5 nodes

E. Plot/analyze forecast.

1) Quick check

ncdump -h wrfout_d01_YYYY-MM-DD_00:00:00 | less

-header of netCDF file

2) Create new directory for RIP files

RIP will create a bunch of files, so create a new directory and then change to that directory to run RIP. For example, I created a directory /home/gretchen/wrf311/MMDD, substituting in the month and day of the forecast

3) Convert the model output data file(s) to RIP-format data files

ripdp_wrfarw NAME basic FULL_PATH/wrfout_d01_YYYY-MM-DD_00:00:00

-replace NAME with a nickname for this run, FULL_PATH with the path to your run directory, and YYYY-MM-DD with your initialization time. For example, “ripdp_wrfarw 1029 basic /home/gretchen/wrf/WRFV3/run/wrfout_d01_2008-10-29_00:00:00”

4) Create plots

a) temperature

cp /home/data/rip_temp.in .
rip NAME rip_temp.in

-creates multi-panel plot of temperature/SLP/winds → rip_temp.ps

-you can copy rip_temp.in to your own directory to change the plot parameters

-see the Appendices at http://www.mmm.ucar.edu/wrf/users/docs/ripug.htm for more help with RIP

b) precipitation

cp /home/data/rip_pcp.in .

rip NAME rip_pcp.in

-plots 24-hr accumulated pcp from 24hrs to 48hrs, every 6hr → rip_pcp.ps

-the contours are fixed to go from .25 mm (0.01 in) to 2.5 mm (0.1 in)

-if you need to plot higher rain rates, you will need to change the cbeg (beginning contour), cint (contour interval), and cend (ending contour) values in the file