WRF4G Project: Adaptation of WRF Model To Distributed Computing Infrastructures

http://www.meteo.unican.
es
A multidisciplinary approach to weather & climate
Santander Meteorology Group

A multidisciplinary approach for weather & climate
WRF4G project: Adaptation of WRF Model to

Distributed Computing Infrastructures
Antonio. S. Cofiño, V. Fernández-Quiruelas, M. García-Díez,
C. Blanco and J. Fernández

Department of Applied Mathematics and Computer Sciences
Universidad de Cantabria, Santander, Spain
Aknowledgments: This work is funded by the

Spanish PLAN NACIONAL de I+D+i 2008-2011
(Ref.# CGL2011-28864) and ERDF)
European Geosciences Union General Assembly 2013 (7 - 12 April), Vienna, Austria

http://www.meteo.unican.es

A multidisciplinary approach for weather & climate Outline
● Motivation
● WRF4G
● Accessing distributed resources
● Workflow
● Experiment types
● Projects
● Conclusions

A multidisciplinary approach for weather & climate WRF experimental setup scen.
Reanalysis/Reforecasts/Hindcast
●
High number (~104) of independent simulations
● High volume of output-data (>TB)
● Requires scalability
Regional climate simulation
● Long, continuous simulations; weeks of walltime
● High volume of output data (>TB)
● Recovering system for simulation restart
Weather Forecasting
● QoS and optimal resources: deadline for delivery
Sensitivity/ensemble studies
● Physical schemes, initial conditions and
boundary conditions: uncertainty sampling
● Resource demanding experiments composed
of many independent simulations

A multidisciplinary approach for weather & climate Examples: Santander MetGroup
Reanalysis/Reforecasts/Hindcast
● SEAWIND project
● 21 years of daily reforecasts (36h each)
● 7,665 independent simulations
Regional climate simulation
● ESCENA project
● 50 years (continuous run, 28-day restarts)
● 650 dependent simulations
Sensitivity/ensemble studies
● CORWES project
● Physics sensitivity study for CORDEX-Africa
● 8-member ensemble of 5-year continuous simulations
● 8 independent groups of 65 dependent simulations

A multidisciplinary approach for weather & climate Computer resource scenarios
Desktop/Laptop (UI)
● Low computational power and storage
● User interface to other computer resources
Workstation
● Multi-core, shared memory, moderate storage
● ssh access
Local group/institutional cluster
● Multi-node, distributed memory, large storage
● ssh access, batch system (PBS, SGE, ...) to submit jobs
Mainframe/HPC site
● Different architectures and memory arrangements
● ssh or kerberos and token based security
Grid infrastructure
● “Cluster of clusters”, geographycally distributed
● Huge amount of computational power and storage
(not trivial to take advantage of it for meteo/climate apps)
● PKI security

A multidisciplinary approach for weather & climate WRF4G
WRF4G, developed by the Santander Meteorology Group, provides:

● The ability to run these experiments on different computing
resources concurrently in a transparent way and ...
● … ease to design, execute and monitor WRF experiments


WRF4G provides a set of command line tools. An on-going work is been made
to create a Scientific Gateway: WRF4SG (see oral presentation on Tuesday at ESSI2.8
Session Room R14, 15:30).
WRF and third-party tools are deployed dynamically in the computing

resources by the framework. The local deployments on HPC can be used
The input/output and log files are centralized as a single repository
Broken experiments (due to a temporal disruptions) are automatically
handled by the framework: only the unfinished simulations will be re-submitted from
their last restart point


WRF4G software features:
● Components: WRFV3, MySQL, GridWay, CDO and NCO
● Compiled binaries for x86_64 Linux, are provided (OpenMPI). You only
need to download and run: tested on Ubuntu 10.04, 11.04 and 12.04; CentOS 5 and 6;
Debian 6.0
● Third-party dependencies like the middleware for accessing Grid

resources:
● gLite (old)
● UMD1/UMD2 (new)
● Globus toolkit (GT2, GT4 and GT5)



Access to
A multidisciplinary approach for weather & climate heterogeneous resources
DRM4G (Distributed Resource Manager) allows the user to
merge different computing resources at hand in a transparent way:
Local resources (UI)
● Directly in a shell session

● Interacting with LRMS
● PBS
● SGE
● SLURM
● ...

Access to

Remote resources (via ssh)

● PBS
● SGE
● SLURM
● ...

Access to

Remote resources (via ssh)
Grid infrastructures (via Globus or gLite)

● PBS
● SGE
● SLURM
● ...

A multidisciplinary approach for weather & climate Configuration files
wrf4gframework.conf
[ComputingResources]
mycomputer = local://localhost?
LRMS_TYPE=fork;
NODECOUNT=1;
myworkstation = ssh://user@workstation.unican.es?
LRMS_TYPE=fork;
NODECOUNT=16;
PBS_cluster = ssh://user@pbs.cluster.edu?
LRMS_TYPE=pbs;
QUEUE_NAME=long;
NODECOUNT=256;
ESR_EGI = ssh://user@ui.grid.es?
LRMS_TYPE=gLite;
VO=esr;
BDII=topbdii.grid.fr;









wrf4gframework.conf
resources.wrf4g
# WRF4G version to use (packed scripts must be in $WRF4G_APPS)

WRF4G_VERSION="1.0"
# Name of the packed WRF binaries (the file must be in $WRF4G_APPS)
WRF_VERSION="3.3.1_r832INTEL_OMPI"
# Common path to save all output and log files

WRF4G_BASEPATH="/oceano/gmeteo/WORK/ASNA/WRF/experiments"
# Path to the preconfigured WRF domains
WRF4G_DOMAINPATH="/oceano/gmeteo/WORK/ASNA/WRF/domains"
# Path to the global data for the boundary and initial conditions
WRF4G_INPUT="/oceano/gmeteo/DATA"
# Path to the packed binaries (WRF4G script, WRF, cdo (preprocessor), ..)
WRF4G_APPS="/oceano/gmeteo/WORK/wrf4g/repository/apps"
# Number of parallel processors (cores) per simulation

NP=8
# Computer resources to use
REQUIREMENTS = 'HOSTNAME = "myworkstation"'
# Fine tuning
ENVIRONMENT='WALLTIME = 10:00:00, MEMORY = 4000'

wrf4gframework.conf
resources.wrf4g

WRF4G_VERSION="1.0"

WRF4G_BASEPATH="gsiftp://my.storage.edu/path/to/WRF/experiments"
WRF4G_DOMAINPATH="gsiftp://other.computer.edu/path/to/WRF/domains"
WRF4G_INPUT="gsiftp://other2.computer.edu/path/to/input/DATA"
WRF4G_APPS="gsiftp://other3.computer.edu/path/to/apps"

NP=8
REQUIREMENTS = 'HOSTNAME = "ce01.macc.unican.es"'
# Fine tuning

wrf4gframework.conf
resources.wrf4g

WRF4G_VERSION="1.0"

WRF4G_BASEPATH="rsync://user@my.storage.edu/path/to/WRF/experiments"
WRF4G_DOMAINPATH="rsync://user@other.computer.edu/path/to/WRF/domains"
WRF4G_INPUT="rsync://user@other2.computer.edu/path/to/input/DATA"
WRF4G_APPS="rsync://user@other3.computer.edu/path/to/apps"

NP=8
REQUIREMENTS = 'HOSTNAME = "pbs.macc.unican.es"'
# Fine tuning

wrf4gframework.conf
resources.wrf4g
experiment.wrf4g
experiment_name = "MyExperiment"
domain_name = "Europe15km"
max_dom = 2
extdata_vtable = "ECMWF" # Vtables must exist as Vtable.[input_extdata]
extdata_path = "${WRF4G_INPUT}/ECMWF/INTERIM"
extdata_interval = 21600 # Seconds between global analysis input times Boundary data
extdata_preprocessor = "ECMWF"
postprocessor = "SEAWIND2"
start_date = "1989-01-01_06:00:00"
end_date = "2001-01-02_00:00:00"
chunk_size_h = 36
multiple_dates = 1 Experiment dates
simulation_interval_h = 24
simulation_length_h = 1*chunk_size_h
multiple_parameters = 0
multiparams_variables = "mp_physics,cu_physics,e_vert"
multiparams_nitems = "${max_dom},${max_dom},${maxdom}" Multi-parameter
multiparams_combinations = "3,1,28 / 3,3,28 / 4,1,36 / 3,1,36"
multiparams_labels = "WSM3_KF_L28/WSM3_GD_L28/WSM5_KF_L36/WSM3_KF_L36"
experiment
### Override namelist.input variables here ###
# Single valued:
NI_restart_interval = 2880 # minutes
NI_spec_bdy_width = 10
NI_spec_zone = 1
NI_relax_zone = 9
NI_sst_update__physics = 1
# One value per domain: namelist.input
NIM_history_interval = 180,60 # minutes
NIM_frames_per_outfile = 4,12 modifications
# One value per domain; but all equal (provide a single value here):
NIN_e_vert = 42

A multidisciplinary approach for weather & climate WRF4G framework
● WRF4G splits a regular WRF simulation experiment into:
● realizations
– A realization is any independent WRF simulation, each
simulation is one of the multiple_dates and multiple_parameters
● chunks
– For convenience, a WRF realization are split-ed into chunks.
By definition, a chunk is a dependent partial simulation and
requires the previous chunk to start.
– Chunks depend on computing resource limitations:
WALLTIME, RESOURCE DISK QUOTA, ...
– Chunks allow fine tuning of the size of the input files
(boundary and initial conditions).

A multidisciplinary approach for weather & climate Climate simulation (continuous)
multiple_dates = 0

A multidisciplinary approach for weather & climate Multi-parameter
multiple_dates = 0

A multidisciplinary approach for weather & climate Multi-Parameter & Multi-Date
multiple_dates = 1

A multidisciplinary approach for weather & climate Monitoring
Shell$ wrf4g_status -e example
Realization GW Stat Chunks Comp.Res WN Run.Sta ext %
example__ph1 - P 0/4 - - Prepared - 0.00
The experiment “example” is compose of five independent

realizations and all of them are in “Prepared” status. These
realization are divided into 4 chunks. They are going to be
submitted.

example__ph1 32 R 1/4 mycomputer legolas metgrid - 0.00
example__ph2 36 R 1/4 mycomputer legolas Down. Bound. - 0.00
example__ph3 40 W 1/4 - - Submitted - 0.00
The realization “example__ph1” is running metgrid binary (first

chunk) and the realization “example__ph2” is downloading the
boundary conditions (first chunk). The rest of realizations are
waiting for free computing resources.

example__ph1 32 R 1/4 mycomputer legolas real - 0.00
example__ph2 36 R 1/4 mycomputer legolas metgrid - 0.00
The realization “example__ph1” is running real binary (first chunk)

and the realization “example__ph2” is running “metgrid” binary
component (first chunk).

example__ph1 32 R 1/4 mycomputer legolas WRF - 0.00
Realizations “example__ph1” and “example__ph2” are running

WRF binary component (first chunk).

The first chunks of the realization “example__ph1” and

“example__ph2” have finished. Now, the first chunks of realization
“example__ph3” and “example__ph4” are running WRF binary
component.

A multidisciplinary approach for weather & climate Outline
● Motivation
● WRF4G
● Accessing distributed resources
● Workflow
● Experiment types
● Projects
● Conclusions

A multidisciplinary approach for weather & climate Projects
Community :
EELA2: E-science grid facility for Europe and Latin America

Partners: 52 institutions in Latin America and Europe
GISELA Project (Grid Initiatives for e-Science virtual communities

in Europe and Latin America)
Partners: 19 partners in Latin America and Europe
Infrastructure :
EGI: European Grid Infrastructure (ES-NGI)
WRF4G : http://appdb.egi.eu/store/software/wrf
RES: Spanish Supercomputing Network

Altamira and the Barcelona Supercomputing Center. Using
binaries compiled for these infrastructures

A multidisciplinary approach for weather & climate Projects
Scientific :
WRF model port to Grid infrastructures and proof-of-concept for a

high-resolution wind hindcast over Europe
Universidad de Cantabria
Coordinated regional climate downscaling experiment using WRF:

a contribution to the CORDEX initiative by the Spanish WRF community
Partners: 3 Spanish universities and a supercomputing center
CORDEX - Coordinated Regional Climate Downscaling Experiment :

a WCRP-sponsored program to produce regional climate change
scenarios globally

A multidisciplinary approach for weather & climate Conclusions
● The WRF user community usually:

● designs experiments where several (many or
huge ?) simulations are required
● has several distributed computer resources
available for running simulations
● WRF4G simplifies the design, execution and
monitoring of WRF on several computer
resources as an homogenous meta-computer.
● WRF4G is freely available for use. By, share-
alike and non-commercial license scheme
www.meteo.unican.es/software/wrf4g

Thank you!
Contact: antonio.cofino@unican.es
More info: www.meteo.unican.es/software/wrf4g

(“wrf4g” → Google's I'm Feeling Lucky button)
See oral presentation on Tuesday at ESSI2.8 Session Room R14, 15:30
Aknowledgments: This work is funded by the

Spanish PLAN NACIONAL de I+D+i 2008-2011
(Ref.# CGL2011-28864) and ERDF)

A multidisciplinary approach for weather & climate Useful tools developed
● vcp
● Virtual copy
● Provides transparent copy between any of:
– local file – ssh host (via rsync)
– gridftp URL – local link (as destination)
● E.g:
vcp /local/file /other/local/path/
vcp /local/file rsync://remote.comp.edu/remote/path/
vcp rsync://remote.comp.edu/remote/path ln:/local/path # (copies)
vcp /local/file ln:/other/local/path # (links)
vcp -r /local/dir gridftp://server:port/remote/path
vcp gridftp://srv1:port/rmt/file gridftp://srv2:port/other/file

A multidisciplinary approach for weather & climate Useful tools developed
● fortnml
● Fortran namelist
● Provides Fortran namelist manipulation from the
command line along with some WRF namelist
checks.
● E.g:
fortnml -f namelist.input
fortnml --wrf -f namelist.input
fortnml -f namelist.input -s variable value
fortnml -f namelist.input -s variable value1 value2 value3 ...
fortnml -f namelist.input -s variable@record value

A multidisciplinary approach for weather & climate Multi-parameter & multi-date

A multidisciplinary approach for weather & climate Multi-parameter & multi-date

A multidisciplinary approach for weather & climate Climate simulation (split)

A multidisciplinary approach for weather & climate Experiment definition







Three realizations split into two chunks each:
exp__rea1
exp__rea2
exp__rea3
time
exp__rea1
exp__rea2
exp__rea3
time
wrfbdy_d0x
wrfinput_d0x wrfrst_d0x
WPS+real wrflowinp_d0x
WRF Model time step wrfout_d0x
(no output)
exp__rea1
exp__rea2
exp__rea3
0 1 2 3 4 5 6
time
Storage Element - ${WRF4G_BASEPATH}/experiments/exp

exp__rea1 exp__rea2 exp__rea3
output restart wpsout output restart wpsout output restart wpsout
exp__rea1
exp__rea2
exp__rea3
0 1 2 3 4 5 6
time
Storage Element - METEO4G/WRF/experiments/exp

wrfinput_t0
wrfbdy_t0
exp__rea1
exp__rea2
exp__rea3
0 1 2 3 4 5 6
time

wrfout_t0 wrfinput_t0
wrfout_t1 wrfbdy_t0
exp__rea1
exp__rea2
exp__rea3
0 1 2 3 4 5 6
time

wrfout_t1 wrfbdy_t0
wrfout_t2 wrfrst_t2
exp__rea1
exp__rea2
exp__rea3
0 1 2 3 4 5 6
time

wrfout_t1 wrfbdy_t0
wrfout_t2 wrfrst_t2
wrfout_t3
wrfout_t4 wrfrst_t4
exp__rea1
exp__rea2
exp__rea3
0 1 2 3 4 5 6
time

wrfout_t1 wrfbdy_t0
wrfout_t2 wrfrst_t2
wrfout_t3
wrfout_t4 wrfrst_t4 wrfinput_t4
wrfbdy_t4
exp__rea1
exp__rea2
exp__rea3
0 1 2 3 4 5 6
time

wrfout_t1 wrfbdy_t0
wrfout_t2 wrfrst_t2
wrfout_t3
wrfbdy_t4
exp__rea1
exp__rea2
exp__rea3
0 1 2 3 4 5 6
time

wrfout_t1 wrfbdy_t0
wrfout_t2 wrfrst_t2
wrfout_t3
wrfout_t5 wrfbdy_t4
exp__rea1
exp__rea2
exp__rea3
0 1 2 3 4 5 6
time

wrfout_t1 wrfbdy_t0
wrfout_t2 wrfrst_t2
wrfout_t3
wrfout_t5 wrfbdy_t4
wrfout_t6 wrfrst_t6
wrfout_t7
wrfout_t8 wrfrst_t8
exp__rea1
exp__rea2
exp__rea3
0 1 2 3 4 5 6
time

wrfout_t0 wrfinput_t0 wrfout_t0 wrfinput_t0 wrfout_t0 wrfinput_t0

wrfout_t1 wrfbdy_t0 wrfout_t1 wrfbdy_t0 wrfout_t1 wrfbdy_t0
wrfout_t2 wrfrst_t2 wrfout_t2 wrfrst_t2 wrfout_t2 wrfrst_t2
wrfout_t3 wrfout_t3 wrfout_t3
wrfout_t4 wrfrst_t4 wrfinput_t4 wrfout_t4 wrfrst_t4 wrfinput_t4 wrfout_t4 wrfrst_t4 wrfinput_t4
wrfout_t5 wrfbdy_t4 wrfout_t5 wrfbdy_t4 wrfout_t5 wrfbdy_t4
wrfout_t7 wrfout_t7 wrfout_t7

WRF4G Project: Adaptation of WRF Model To Distributed Computing Infrastructures

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

WRF4G Project: Adaptation of WRF Model To Distributed Computing Infrastructures

Încărcat de

Drepturi de autor:

Formate disponibile

http://www.meteo.unican.

Santander Meteorology Group