wrf model: software architecture and code structure john michalakes, ncar ncar: w. skamarock, j....

WRF Model: Software Architecture and Code Structure

John Michalakes, NCAR

NCAR: W. Skamarock, J. Dudhia, D. Gill, A. Bourgeois, W. Wang, C. Deluca, R. LoftNOAA/NCEP: Tom Black, Jim Purser, S. GopalNOAA/FSL: T. Henderson, J. Middlecoff, L.Hart

U. Oklahoma: M. XueAFWA: J. Wegiel, D. McCormick

C. Coats (MCNC), J. Schmidt (NRL), V. Balaji (GFDL) S. Chen (UC Davis), J. Edwards (IBM)

Acknowledgement:Significant funding for WRF software development from DoD HPCMO CHSSI program

(CWO6)

Outline• Software architecture and implementation

– Goals and design aspects– Parallelism– Code Structure and Model Layer Interface– Registry– I/O

• Extending WRF: Example

WRF SoftwareGoals

• Community Model

• Good performance

• Portable across a range of architectures

• Flexible, maintainable, understandable

• Facilitate code reuse

• Multiple dynamics/ physics options

• Run-time configurable

• Nested

Aspects of Design• Single-source code

• Fortran90 modules, dynamic memory, structures, recursion

• Hierarchical software architecture

• Multi-level parallelism

• CASE: Registry

• Package-neutral APIs– I/O, data formats

– Communication

• IKJ Storage Order

OMP

Software Architecture

ConfigInquiry

I/O API

ConfigModule

WRF Tile-callableSubroutines

SolveMediation Layer

Model Layer

Driver Layer

DM comm

Thr

eads

External Packages

PackageIndependent

PackageDependent

Data formats,Parallel I/O

Mes

sage

Pas

sing

Driver

Registry

Model domains are decomposed for parallelism on two-levels

Patch: section of model domain allocated to a distributed memory nodeTile: section of a patch allocated to a shared-memory processor within a node; this is also the scope of a model layer subroutine.Distributed memory parallelism is over patches; shared memory parallelism is over tiles within patches

• Single version of code for efficient execution on:

– Distributed-memory– Shared-memory – Clusters of SMPs– Vector and microprocessors

WRF Multi-Layer Domain Decomposition

Logical domain

1 Patch, divided into multiple tiles

Inter-processor communication

Parallelism and you

• Some code you add to WRF won’t entail any extra effort on your part other than conforming to the WRF model layer interface specification

• Exceptions– Horizontal interpolations involving neighboring cells

– Upper radiative boundary conditions

– Dynamics and diffusion

– Fully 3D packages (that is, not “column physics”)

– Tell tale sign: if you have i+integer or i+integer array indices in right-hand side expressions in your code

http://home.nycap.rr.com/content/us_bio.html

Example

(dyn_eh/module_diffusion.F )SUBROUTINE horizontal_diffusion_s (tendency, rr, var, . . .. . . DO j = jts,jte DO k = kts,ktf DO i = its,ite mrdx=msft(i,j)*rdx mrdy=msft(i,j)*rdy tendency(i,k,j)=tendency(i,k,j)- & (mrdx*0.5*((rr(i+1,k,j)+rr(i,k,j))*H1(i+1,k,j)- & (rr(i-1,k,j)+rr(i,k,j))*H1(i ,k,j))+ & mrdy*0.5*((rr(i,k,j+1)+rr(i,k,j))*H2(i,k,j+1)- & (rr(i,k,j-1)+rr(i,k,j))*H2(i,k,j ))- & msft(i,j)*(H1avg(i,k+1,j)-H1avg(i,k,j)+ & H2avg(i,k+1,j)-H2avg(i,k,j) & )/dzetaw(k) & ) ENDDO ENDDO ENDDO . . .

!! 0 pt 4 pt 8 pt 24 pt!! * * * * *! * * * * * * * * *! + * + * * + * * * + * *! * * * * * * * * *! * * * * *!msft x!rr x!

• Halo updates

patch

ims imejms

jme

Distributed Memory Communications

patch

ims ime

memory on one processor memory on neighboring processor

• Halo updates


memory on one processor memory on neighboring processor

*+ **

* *

Halo update of second time level ofrr may be defined in Registry as:

halo HALO_DIFF dyn_em 4:rr_2

Halo update of second time level ofrr may be defined in Registry as:

halo HALO_DIFF dyn_em 4:rr_2

Halo update is invoked in the code (in the solve routine) as:

#include “HALO_DIFF.incl”

prior to the call to the diffusion routine

Halo update is invoked in the code (in the solve routine) as:

#include “HALO_DIFF.incl”

prior to the call to the diffusion routine

• Halo updates

• Periodic boundary updates


Period update of variable t may be defined in Registry as:

period PBDY dyn_em 3:t

Period update of variable t may be defined in Registry as:

period PBDY dyn_em 3:t

Period update is invoked in the code as:

#include “PBDY.incl”

Period update is invoked in the code as:

#include “PBDY.incl”

• Halo updates

• Periodic boundary updates

• Parallel transposes all z onpatch

all x onpatch

all y onpatch


Transpose of u defined in Registry as:

xpose TRANS_A dyn_em u,ux,uy

Note separate definitions of u, ux, and uy:

state real u ikj …state real ux ikjx …state real uy ikjy …

Transpose of u defined in Registry as:

xpose TRANS_A dyn_em u,ux,uy

Note separate definitions of u, ux, and uy:

state real u ikj …state real ux ikjx …state real uy ikjy …

Transpose from Z decomposition to X decomposition is invoked in the code as:

#include “TRANS_A_z2x.incl”

Transpose from Z decomposition to X decomposition is invoked in the code as:

#include “TRANS_A_z2x.incl”

OMP

Software Architecture

ConfigInquiry

I/O API

ConfigModule



Model Layer

Driver Layer

DM comm

Thr

eads

External Packages

PackageIndependent

PackageDependent


Mes

sage

Pas

sing

Driver

Registry

Code Structure

OMP

ConfigInquiry

I/O API

ConfigModule



Model Layer

Driver Layer

DM comm

Thr

eads

External Packages

PackageIndependent

PackageDependent


Mes

sage

Pas

sing

Driver

Registry

wrf (main/wrf.F)wrf (main/wrf.F)

integrate (frame/module_integrate.F)integrate (frame/module_integrate.F)

solve_interface (share/solve_interface.F)solve_interface (share/solve_interface.F)

solve_em (dyn_em/solve_em.F)solve_em (dyn_em/solve_em.F)

advance_uv (dyn_em/module_small_step_em.F)advance_uv (dyn_em/module_small_step_em.F)advance_uv (dyn_em/module_small_step_em.F)advance_uv (dyn_em/module_small_step_em.F)advance_uv (dyn_em/module_small_step_em.F)advance_uv (dyn_em/module_small_step_em.F)advance_uv (dyn_em/module_small_step_em.F)advance_uv (dyn_em/module_small_step_em.F)advance_uv (dyn_em/module_small_step_em.F)advance_uv (dyn_em/module_small_step_em.F)cumulus_driver (physics/cumulus_driver.F)cumulus_driver (physics/cumulus_driver.F)

WRF Model Layer Interface• Interface Mediation layer <> Model Layer

– All state arrays passed through argument list as simple (not derived) data types

– Domain, memory, and run dimensions passed unambiguously in three physical dimensions

– Restrictions on model layer subroutines• No I/O, communication, no stops or aborts (use

wrf_error_fatal in frame/module_wrf_error.F)• No common/module storage of decomposed data

(exception allowed for set-once/read-only tables)• Spatial scope of a Model Layer call is one “tile”• Temporal scope of a call is limited by coherency• Computation on halos is allowed and considered a

model-layer concern

• Benefits– Ensures that a sub-domain is tile-callable and

therefore portable to any decomposition supported by the driver

– Provides a common interface that• Allows use of other conforming driver layers (this

positions the code for new developments such as ESMF)

• WRF driver layer can (and is) being used to parallelize other models

OMP

ConfigInquiry

I/O API

ConfigModule


SolveDM comm

Thr

eads Data formats,

Parallel I/O

Mes

sage

Pas

sing

Driver

SUBROUTINE solve_xxx ( . . .!$OMP DO PARALLEL DO ij = 1, numtiles its = i_start(ij) ; ite = i_end(ij) jts = j_start(ij) ; jte = j_end(ij) CALL model_subroutine( arg1, arg2, . . . ids , ide , jds , jde , kds , kde , ims , ime , jms , jme , kms , kme , its , ite , jts , jte , kts , kte ) END DO . . .

END SUBROUTINE

SUBROUTINE solve_xxx ( . . .!$OMP DO PARALLEL DO ij = 1, numtiles its = i_start(ij) ; ite = i_end(ij) jts = j_start(ij) ; jte = j_end(ij) CALL model_subroutine( arg1, arg2, . . . ids , ide , jds , jde , kds , kde , ims , ime , jms , jme , kms , kme , its , ite , jts , jte , kts , kte ) END DO . . .

END SUBROUTINE

template for model layer subroutine

SUBROUTINE model_subroutine ( & arg1, arg2, arg3, … , argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims

IMPLICIT NONE

! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1, . . . REAL, DIMENSION (ims:ime,jms:jme) :: arg7, . . . . . . ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1, . . . . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO END DO END DO


SUBROUTINE model_subroutine ( & arg1, arg2, arg3, … , argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims

IMPLICIT NONE



SUBROUTINE model ( & arg1, arg2, arg3, … , argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims

IMPLICIT NONE




IMPLICIT NONE


• Domain dimensions• Size of logical domain• Used for bdy tests, etc.



IMPLICIT NONE




IMPLICIT NONE



• Memory dimensions• Used to dimension dummy

arguments• Do not use for local arrays



IMPLICIT NONE




IMPLICIT NONE



• Memory dimensions• Used to dimension dummy

arguments• Do not use for local arrays

• Tile dimensions• Local loop ranges• Local array dimensions

Registry

OMP

ConfigInquiry

I/O API

ConfigModule



Model Layer

Driver Layer

DM comm

Thr

eads

External Packages

PackageIndependent

PackageDependent


Mes

sage

Pas

sing

Driver

Registry

OMP

ConfigInquiry

I/O API

ConfigModule



Model Layer

Driver Layer

DM comm

Thr

eads

External Packages

PackageIndependent

PackageDependent


Mes

sage

Pas

sing

Driver

Registry

WR

F R

egistry• CASE-like “active data-dictionary” for managing WRF data structures

– Database describing attributes of model state, intermediate, and configuration data• Dimensionality, number of time levels, staggering

• Association with physics

• I/O classification (history, initial, restart, boundary)

• Communication points and patterns

• Configuration lists (e.g. namelists)

– Program for auto-generating sections of WRF from database: • 570 Registry entries 30-thousand lines of automatically generated WRF code

• Allocation statements for state data, I1 data

• Argument lists for driver layer/mediation layer interfaces

• Interprocessor communications: Halo and periodic boundary updates, transposes

• Code for defining and managing run-time configuration information

• Code for forcing, feedback and interpolation of nest data

• Automates time consuming, repetitive, error-prone programming

• Insulates programmers and code from package dependencies

• Allow rapid development

• Documents the data

# Registry file## table entries are of the form#<Table> <Type> <Sym> <Dims> <Use> <NumTLev> <Stagger> <IO>#state real ru ikj dyn-rk 2 X irhstate real u ikj dyn-rk 2 X irhstate real ru_m ikj dyn-rk 1 Xi1 real ru_tend ikj dyn-rk 1 X . . .## communicationshalo HALO_RK_A 24:u_2,v_2,w_2,t_2,tp_2,rw_2,rom_2;4:pp,piphalo HALO_RK_B 4:rtp_2halo HALO_RK_C 4:ru_2,rv_2,du,dv . . .

Registry Mechanics

%compile wrf

Registry/Registry registry program:tools/registry

inc/*.incl

WRF source*/*.F

Fortran90

wrf.exe

Registry Data Base

• Currently implemented as a text file: Registry/Registry

• Types of entry:– State – Describes state variables and arrays in the domain structure

– Dimspec – Describes dimensions that are used to define arrays in the model

– I1 – Describes local variables and arrays in solve

– Typedef – Describes derived types that are subtypes of the domain structure

– Rconfig – Describes a configuration (e.g. namelist) variable or array

– Package – Describes attributes of a package (e.g. physics)

– Halo – Describes halo update interprocessor communications

– Period – Describes communications for periodic boundary updates

– Xpose – Describes communications for parallel matrix transposes

State entry• Elements

– Entry: The keyword “state”

– Type: The type of the state variable or array (real, double, integer, logical, character, or derived)

– Sym: The symbolic name of the variable or array

– Dims: A string denoting the dimensionality of the array or a hyphen (-)

– Use: A string denoting association with a solver or 4D scalar array, or a hyphen

– NumTLev: An integer indicating the number of time levels (for arrays) or hypen (for variables)

– Stagger: String indicating staggered dimensions of variable (X, Y, Z, or hyphen)

– IO: String indicating whether and how the variable is subject to I/O

– DName: Metadata name for the variable

– Units: Metadata units of the variable

– Descrip: Metadata description of the variable

• Example# Type Sym Dims Use Tlev Stag IO Dname Descrip# definition of a 3D, two-time level, staggered state arraystate real ru ikj dyn_eh 2 X irh "RHO_U" "X WIND COMPONENT“

# definition of fields in 4D scalar array “moist”state real qv ikjft moist 2 - irh "QVAPOR" "Water vapor mix ratio“state real qc ikjft moist 2 - irh "QCLOUD" "Cloud water mixing ratio"

Dimspec entry• Elements

– Entry: The keyword “dimspec”

– DimName: The name of the dimension (single character)

– Order: The order of the dimension in the WRF framework (1, 2, 3, or ‘-‘)

– HowDefined: specification of how the range of the dimension is defined

– CoordAxis: which axis the dimension corresponds to, if any (X, Y, Z, or C)

– DatName: metadata name of dimension

• Example

#<Table> <Dim> <Order> <How defined> <Coord-axis> <DatName>dimspec i 1 standard_domain x west_eastdimspec j 3 standard_domain y south_northdimspec k 2 standard_domain z bottom_topdimspec l 2 namelist=num_soil_layers z soil_layers

Rconfig entry• Elements

– Entry: the keyword “rconfig”

– Type: the type of the namelist variable (integer, real, logical – no strings yet)

– Sym: the name of the namelist variable or array

– How set: indicates how the variable is set: e.g. namelist or derived, and if namelist, which block of the namelist it is set in

– Nentries: specifies the dimensionality of the namelist variable or array. If 1 (one) it is a variable and applies to all domains; otherwise specify max_domains (which is an integer parameter defined in module_driver_constants.F).

– Default: the default value of the variable to be used if none is specified in the namelist; hyphen (-) for no default

• Example

# Type Sym How set Nentries Defaultrconfig integer dyn_opt namelist,namelist_01 1 1

Package Entry• Elements

– Entry: the keyword “package”,

– Package name: the name of the package: e.g. “kesslerscheme”

– Associated rconfig choice: the name of a rconfig variable and the value of that variable that choses this package

– Package state vars: unused at present; specify hyphen (-)

– Associated 4D scalars: the names of 4D scalar arrays and the fields within those arrays this package uses

• Example

# specification of microphysics optionspackage passiveqv mp_physics==0 - moist:qvpackage kesslerscheme mp_physics==1 - moist:qv,qc,qrpackage linscheme mp_physics==2 - moist:qv,qc,qr,qi,qs,qgpackage ncepcloud3 mp_physics==3 - moist:qv,qc,qrpackage ncepcloud5 mp_physics==4 - moist:qv,qc,qr,qi,qs

# namelist entry that controls microphysics optionrconfig integer mp_physics namelist,namelist_04 max_domains 0

Comm entries: halo and period• Elements

– Entry: keywords “halo” or “period”

– Commname: name of comm operation

– Description: defines the halo or period operation• For halo: npts:f1,f2,...[;npts:f1,f2,...]*

• For period: width:f1,f2,...[;width:f1,f2,...]*

• Example

# first exchange in eh solverhalo HALO_EH_A dyn_em 24:u_2,v_2,ru_1,ru_2,rv_1,rv_2,w_2,t_2;4:pp,pip

# a periodic boundary updateperiod PERIOD_EH_A dyn_em 2:u_1,u_2,ru_1,ru_2,v_1,v_2,rv_1,rv_2,rw_1,rw_2

OMP

I/O Architecture

ConfigInquiry

I/O API

ConfigModule



Model Layer

Driver Layer

DM comm

Thr

eads

External Packages

PackageIndependent

PackageDependent


Mes

sage

Pas

sing

Driver

Registry

I/O Architecture

• Requirements of I/O Infrastructure– Efficiency: key concern for operations

– Flexibility: key concern in research

– Both types of user-institution already heavily invested in I/O infrastructure

• Operations: GRIB, BUFR

• Research: NetCDF, HDF

– “Portable I/O” – adaptable to range of uses, installations without affecting WRF and other programs that use the I/O infrastructure

I/O Architecture

• WRF I/O API– Package-independent interface to NetCDF, Fast-binary, HDF (planned)– Random access of fields by timestamp/name– Full transposition to arbitrary memory order– Built-in support for read/write of parallel file systems (planned)– Data-set centric, not file-centric (planned); Grid Computing

• Additional WRF model functionality– Collection/distribution of decomposed data to serial datasets– Fast, asynchronous, “quilt-server” I/O from NCEP Eta model

WRF I/O API• Key functions

– Initialization and shutdown– Opening and closing datasets– Reading and writing data– Reading and writing metadata

WR

F I

/O A

PI 7.4.1 Initialization and shutdown

SUBROUTINE ext_pkg_ioinit( Status ) INTEGER, INTENT(INOUT) :: Status Synopsis: Initialize the WRF I/O system. SUBROUTINE ext_pkg_ioexit( Status ) INTEGER, INTENT(INOUT) :: Status Synopsis: Shut down the WRF I/O system.

WR

F I

/O A

PI

7.4.2 Opening and closing datasets SUBROUTINE ext_pkg_open_for_read ( DatasetName , Comm , IOComm_io, & SysDepInfo , DataHandle , Status ) CHARACTER *(*), INTENT(IN) :: DatasetName INTEGER , INTENT(IN) :: Comm , IOComm CHARACTER *(*), INTENT(IN) :: SysDepInfo INTEGER , INTENT(OUT) :: DataHandle INTEGER , INTENT(OUT) :: Status Synopsis: Opens a WRF dataset for reading using the I/O package pkg. Arguments:

DatasetName: The name of the dataset to be opened.Comm: Not used.IOComm: Not used.SysDepInfo: A string-valued variable that provides a means to pass

additional control information to the I/O interface. Theformat of the string is a comma-separated list of variable=value pairs. Currently the only control variable now supported is DATASET, which may have the allowable values RESTART, HISTORY, BOUNDARY, or INITIAL (capitals required).For example, the WRF model will pass the string'DATASET=INITIAL' to open an initial dataset.

DataHandle: Returned to the calling program to serve as a handle to theOpen dataset for subsequent I/O API operations.

WR

F I

/O A

PI

SUBROUTINE ext_pkg_open_for_write_begin( DatasetName , Comm, IOComm, & SysDepInfo, DataHandle , Status ) CHARACTER *(*), INTENT(IN) :: DatasetName INTEGER , INTENT(IN) :: Comm, IOComm CHARACTER *(*), INTENT(IN) :: SysDepInfo INTEGER , INTENT(OUT) :: DataHandle INTEGER , INTENT(OUT) :: Status

SUBROUTINE ext_pkg_open_for_write_commit( DataHandle , Status ) INTEGER , INTENT(IN ) :: DatasetHandle INTEGER , INTENT(OUT) :: Status

Synopsis:

Begin data definition phase for the WRF dataset DatasetName using the I/O package Pkg.

Arguments:

DatasetName: The name of the dataset to be opened.Comm: Not used.IOComm: Not used.SysDepInfo: Same as open for read.DataHandle: integer handle.

Two-Stage Open For Write

There is significant performance advantage for netCDF and presumably other self-describing data format packages in going through a data-definition phase prior to

writing data through the package. Thus, the open-for-write is a two-stage operation. A WRF dataset is first opened for data-definition, at which point the model calls the

interface (predominantly the ext_pkg_write_field routine) with a complete serious of writes such as would occur during the output of an actual frame of data; however, no

data is actually written. This is strictly for the purpose of “training” the interface. Interface implementations that don’t need training can ignore this by simply opening the dataset on the open-begin call and setting an internal flag that disables writing in

the write_field routines. ext_pkg_open_for_write_begin must be paired with an ext_pkg_open_for_write_commit .

WR

F I

/O A

PI

SUBROUTINE ext_pkg_write_field ( DataHandle , DateStr , VarName , & Field , FieldType , Comm , IOComm & DomainDesc , MemoryOrder , Stagger , & DimNames , & DomainStart , DomainEnd , & MemoryStart , MemoryEnd , & PatchStart , PatchEnd , & Status ) INTEGER ,INTENT(IN) :: DataHandle CHARACTER*(*) ,INTENT(IN) :: DateStr CHARACTER*(*) ,INTENT(IN) :: VarName type ,INTENT(IN) :: Field(*) INTEGER ,INTENT(IN) :: FieldType INTEGER ,INTENT(INOUT) :: Comm INTEGER ,INTENT(INOUT) :: IOComm INTEGER ,INTENT(IN) :: DomainDesc CHARACTER*(*) ,INTENT(IN) :: MemoryOrder CHARACTER*(*) ,INTENT(IN) :: Stagger CHARACTER*(*) , DIMENSION (*) ,INTENT(IN) :: DimNames INTEGER ,DIMENSION(*) ,INTENT(IN) :: DomainStart, DomainEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: MemoryStart, MemoryEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: PatchStart, PatchEnd INTEGER ,INTENT(OUT) :: Status

WR

F I

/O A

PI


Data names and time stamps

The variable Name provides the name of the variable being read/written; the DateStr character string is the index into the series of values of the variable/array in the dataset. In WRF, this is a 23 character date string of the form “0000-01-01_00:00:00.0000” and is treated as a temporal index. An external implementation of the WRF I/O API is required to randomly access a dataset based on the

name and timestamp pair specified in the argument list.

From the point of view of a package implementing the I/O API, the timestamp string has no meaning or relationship with any other timestamp other than lexicographic: two strings are either the same or they are different; there is no meaning associated with the

timestamp string. The interface does not do time comparisons but it is aware of control breaks (change in the sequence of timestamps in the interface). The interface may also store a record of the sequence in which fields were written to a dataset and provide this

information in a way that allows sequential traversal through the dataset but datasets are fundamentally random access by variable name and timestamp string. The interface may assume and expect that all the fields in a frame of data have the same date string and

that on writing the calls to the interface to write them will have been called in consecutively.

WR

F I

/O A

PI


Full 2-D/3D Arrays

1-D (Column) Arrays

Non-decomposed

Boundary 3-D Boundary 2-D

‘XYZ’ ‘Z’ ‘C’ ‘XSZ’ (west) ‘XS’ (west)

‘XZY’ ‘XEZ’ (east) ‘XE’ (east)

‘ZXY’ 0-D (Scalars) ‘YSZ’ (south) ‘YS’ (south)

‘XY’ ‘0’ (zero) ‘YEZ’ (north) ‘YE’ (north)

‘YX’

WR

F I

/O A

PI


StaggeringStaggered means with respect to mass-point coordinates. For example, in WRF the

Eulerian height solver uses an Arakawa C-grid so that the U-velocity is staggered in X, the V-velelocity is staggered in Y, and the vertical velocity are staggered in Z. The stagger string is specified to the read_field and write_field using a string consisting of the characters X, Y, and/or Z that specifies which dimensions the field is staggered in

(or the empty string if the variable is not staggered in any dimension).

WR

F I

/O A

PI


Dimension Information

Each of the six arguments show are integer arrays whose length is the number of dimensions of the data item being written (specified

by the MemoryOrder string).

DomainStart and Domain are the starts and ends of the undecomposed array in each dimension.

MemoryStart and MemoryEnd are the starts and ends of each local array dimension as it is declared in the program.

PatchStart and PatchEnd are the starts and ends of the local subdomain on this process.

WR

F I

/O A

PI

call ext_ncd_ioinit(Status)

call ext_ncd_open_for_write_begin ( "WRF_DATA", 0, 0, "", dh1, Status )

start_memory(1) = ims ; end_memory(1) = ime start_memory(2) = kms ; end_memory(2) = kme start_memory(3) = jms ; end_memory(3) = jme

start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","T",t,WRF_REAL,0,0,0, & "XZY"," ", & dimnames , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !patch Status)

start_domain(1) = 1 ; end_domain(1) = ide start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","U",u,WRF_REAL,0,0,0, & "XZY","X", & dimnames_x , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !patch Status)

start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","V",v,WRF_REAL,0,0,0, & "XZY","Y", & dimnames_y , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !patch Status)

call ext_ncd_open_for_write_commit ( dh1, Status )! file now open for writing

WR

F I

/O A

PI

! file now open for writing

start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","T",t,WRF_REAL,0,0,0, & "XZY"," ", & dimnames , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !domain Status)

start_domain(1) = 1 ; end_domain(1) = ide start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","U",u,WRF_REAL,0,0,0, & "XZY","X", & dimnames_x , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !domain Status)

start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","V",v,WRF_REAL,0,0,0, & "XZY","Y", & dimnames_y , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !domain Status)

call ext_ncd_ioclose(dh1, Status )

call ext_ncd_ioexit( Status )

. . .

WR

F I

/O A

PI

netcdf WRF_DATA {dimensions:

Time = UNLIMITED ; // (1 currently)DateStrLen = 19 ;west_east = 199 ;south_north = 299 ;bottom_top = 34 ;west_east_stag = 200 ;south_north_stag = 300 ;

variables:char Times(Time, DateStrLen) ;float T(Time, bottom_top, south_north,

west_east) ;T:FieldType = 104 ;T:MemoryOrder = "XYZ" ;

float U(Time, bottom_top, south_north, west_east_stag) ;

U:FieldType = 104 ;U:MemoryOrder = "XYZ" ;

float V(Time, bottom_top, south_north_stag, west_east) ;

V:FieldType = 104 ;V:MemoryOrder = "XYZ" ;

data:

Times = "2000-01-24_12:00:00" ;

. . .

Output from ncdump

Additional Notes on WRF I/O• Calls for reading and writing time-dependent and

time-independent meta data associated with datasets and individual variables

• A sample reader program is distributed with WRF in external/io_netcdf/diffwrf.F90

• The WRF framework provides higher level I/O routines (see Sec. 7.2):

• Higher level calls to read/write fields from an entire domain• Separate Initial, History, Restart and Boundary I/O streams; 4

auxiliary Initial and History streams• Separately run-time assignable I/O formats to I/O streams• Parallel/Quilted I/O• Integrated and controllable through the Registry; little to

actually code

Example: Adding a new core• Conceptual

– WRF framework can slot in new dynamics as run-time selectable option

• Changes to:– Mediation layer, model layer

– Registry

• Reuse:– Top-level driver layer

– I/O infrastructure

– Parallel infrastructure

OMP

ConfigInquiry

I/O API

ConfigModule


SolveDM comm

Thr

eads Data formats,

Parallel I/O

Mes

sage

Pas

sing

Driver

OMP

ConfigInquiry

I/O API

ConfigModule


SolveDM comm

Thr

eads Data formats,

Parallel I/O

Mes

sage

Pas

sing

Driver

Registry

Example: Adding a new core• Steps

– Develop new or convert existing code:

• Mediation layer routine: solve• Model layer subroutines called

by solver

– Add to WRF • Add code to source tree• Incorporate into build

mechanism• Registry entries: data, solver

options, comms• Some additional splicing

– Single processor testing– Analyze data-dependencies,

define and implement communication for parallelism

– Multi-processor testing

OMP

ConfigInquiry

I/O API

ConfigModule


SolveDM comm

Thr

eads Data formats,

Parallel I/O

Mes

sage

Pas

sing

Driver

OMP

ConfigInquiry

I/O API

ConfigModule


SolveDM comm

Thr

eads Data formats,

Parallel I/O

Mes

sage

Pas

sing

Driver

Registry

PSEUDO CODE FOR NEW WRF SOLVER:

Time loop outside of solver (part of WRF driver: integrate)DO 1 <- number of iterations

subroutine solve_exp ( x_1 , x_2 )

for each i,j X1,i,j+1

X2,ij <= X1,i-1,j X1,i+1,j

X1,i,j-1

for each i,j

X1,ij <= X2,ij

end subroutine solve_exp

End time loop

SOLVE_EXP

!WRF:MEDIATION_LAYER:SOLVER

SUBROUTINE solve_exp ( grid , &!#include "exp_dummy_args.inc"! )

CALL set_tiles ( grid , . . . )

#ifdef DM_PARALLEL# include "HALO_EXP_A.inc"#endif

!$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL comp_1_into_2 ( x_1, x_2, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO

!$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL copy_2_into_1 ( x_2, x_1, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO

SOLVE_EXP




by solver







OMP

ConfigInquiry

I/O API

ConfigModule


SolveDM comm

Thr

eads Data formats,

Parallel I/O

Mes

sage

Pas

sing

Driver

OMP

ConfigInquiry

I/O API

ConfigModule


SolveDM comm

Thr

eads Data formats,

Parallel I/O

Mes

sage

Pas

sing

Driver

Registry

!WRF:MODEL_LAYER:DYNAMICS!

MODULE module_exp

USE module_state_description

CONTAINS

!-------------------------------------------------------------------

SUBROUTINE comp_1_into_2 ( x1, x2, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & its, ite, jts, jte, kts, kte )

DO j = jts, jte IF ( j > jds .AND. j < jde-1 ) THEN DO k = kts, kte DO i = its, ite IF ( i > ids .AND. i < ide-1 ) THEN x2(i,k,j) = 0.25*(x1(i+1,k,j)+x1(i-1,k,j)+ & x1(i,k,j+1)+x1(i,k,j-1)) ENDIF ENDDO ENDDO ENDIF ENDDO

END SUBROUTINE comp_1_into_2

SUBROUTINE copy_2_into_1 ( x2, x1, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & its, ite, jts, jte, kts, kte ). . .

module_exp




by solver







WRFV1/

dyn_em/

share/physics/

dyn_eh/

Registry Makefile clean

. . .

WRFV1/

dyn_em/

share/physics/

dyn_eh/


. . .dyn_em/

share/physics/

dyn_eh/


. . .

Makefile

solve_exp.F

module_exp.F

Makefile

solve_exp.F

module_exp.F

dyn_exp/




by solver







• Create dyn_exp/Makefile• Edit top-level WRFV1/Makefile• Additions to clean script




by solver







Additions to Registry/Registry file:

# define the state variable for new core

state real x ikj dyn_exp 2 - ih "TOYVAR"

# value of namelist variable dyn_opt for this core

package dyn_exp dyn_opt==4

# four-point halo-exchange on first time level of x

halo HALO_EXP_A dyn_exp 4:x_1




by solver










by solver







four processor run




by solver







!WRF:MEDIATION_LAYER:SOLVER

SUBROUTINE solve_exp ( grid , &!#include "exp_dummy_args.inc"! )

CALL set_tiles ( grid , . . . )

#ifdef DM_PARALLEL# include "HALO_EXP_A.inc"#endif

!$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL comp_1_into_2 ( x_1, x_2, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO

!$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL copy_2_into_1 ( x_2, x_1, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO




by solver







Additional Information• [email protected]

• www.wrf-model.org

• WRF Design and Implementation Document

mailto:[email protected]

mailto:[email protected]

http://www.wrf-model.org/

wrf model: software architecture and code structure john michalakes, ncar ncar: w. skamarock, j....

Documents

j rri

j msfti

j mrdy

j mrdx

j h2avgi

code slide

j dzetawk enddo

distributed memory parallelism