modeling and computational issues for air/water quality problems g. giunta, r. montella, a. murli,...

Modeling and computational issues for air/water quality

problems

G. Giunta, R. Montella,A. Murli, A. Riccio

Dept. of Applied Sciences , University of Naples “Parthenope”

Dept. of Mathematics, University of Naples “Federico II”

ICAR-CNR

Past experiencesPast experiences• Coupling of a meteorological model (MM5v3), an Coupling of a meteorological model (MM5v3), an

emission model (STdEM) and an air quality model emission model (STdEM) and an air quality model (PNAM or CAMx) to simulate the transport and (PNAM or CAMx) to simulate the transport and chemical reactivity of atmospheric pollutants in urban chemical reactivity of atmospheric pollutants in urban areas from anthropogenic and natural sourcesareas from anthropogenic and natural sources

• Use of an oceanographic circulation model (POM) to Use of an oceanographic circulation model (POM) to study the circulation patterns and biogeochemical study the circulation patterns and biogeochemical processes in the Gulf of Naples, and mixed layer processes in the Gulf of Naples, and mixed layer temperature in the Mediterranean area.temperature in the Mediterranean area.

• Grid flow/Grid aware computational experiences in Grid flow/Grid aware computational experiences in water/air quality simulationswater/air quality simulations

– Simulation and forecasting of:Simulation and forecasting of:

• Meteorological circulation (MM5-MPP)Meteorological circulation (MM5-MPP)

• Dispersion and reactivity of atmospheric Dispersion and reactivity of atmospheric pollutants (PNAM-CAMx)pollutants (PNAM-CAMx)

• Oceanographic circulation near coastal areas Oceanographic circulation near coastal areas (POM-pPOM)(POM-pPOM)

• Wave propagation (WW3)Wave propagation (WW3)

Primary goalsPrimary goals

1.1. Forecasting and MonitoringForecasting and Monitoring

2.2. Scenario simulationScenario simulation (“What if …?” (“What if …?” analysis) analysis)

3.3. Coupling with GIS systemsCoupling with GIS systems

Expected usesExpected uses

The meteorological model (MM5v3)

• Limited-area, non hydrostatic, terrain following sigma-coordinate Limited-area, non hydrostatic, terrain following sigma-coordinate modelmodel

• Based on the integration of continuity, energy and momentum Based on the integration of continuity, energy and momentum balance equationsbalance equations

• Includes nesting capabilities, FDDA, and many kind of physical Includes nesting capabilities, FDDA, and many kind of physical parameterization (more boundary layer, radiative and atmosphere-parameterization (more boundary layer, radiative and atmosphere-surface exchange submodels)surface exchange submodels)

• Multitasking capability on shared- and distributed-memory machines Multitasking capability on shared- and distributed-memory machines

• Initially developed in the late 1970's at the PENN State University, Initially developed in the late 1970's at the PENN State University, but continuously updated at the NCAR Microscale and Mesoscale but continuously updated at the NCAR Microscale and Mesoscale DivisionDivision

• One of the most used meteorological model for real-time forecastingOne of the most used meteorological model for real-time forecastingTo know morehttp://www.mmm.ucar.edu/mm5/mm5-home.html

G. Grell, J. Dudhia and D. R. Stauffer, A Description of the Fifth-Generation Penn-State/NCAR Mesoscale Model (MM5), NCAR Technical Note, NCAR/TN-398+STR

Weather forecasting on the Mediterranean region using the MM5

http://imfa.uniparthenope.it

Weather forecasting on the Mediterranean region using the MM5

http://imfa.uniparthenope.it

z

wc

y

vc

x

uc

t

c iiii

Tendency= Advection

Turbulent diffusion+ Ri + Si + Li

Chemistry EmissionsDry/wet deposition

z

cK

zy

cK

yx

cK

xi

Vi

Hi

H

The air quality model (CAMx)

Example of ozone dynamics (CAMx)

Example of ozone dynamics (CAMx)

FuelConsumption

CORINAIR METHODOLOGY

COPERT III

Number ofVehicles

ClimateData

(temperature)

Urban emission

sNOx

CO

CO2

VOC

SO2

Rural emission

sNOx

CO

CO2

VOC

SO2

Highwaysemission NOx

CO

CO2

VOC

SO2

(age, speed, engine technology)

(mileage, mean speed)

Emission time trends

Spatial disaggregation

•Population density•Kind of land use

•Population density•Kind of land use•Main road net

• Highways net

Time disaggregation

• NOx and VOC speciation• Regridding

Urban emission

sNOx

CO

CO2

VOC

SO2

Rural emission

sNOx

CO

CO2

VOC

SO2

Highwaysemission NOx

CO

CO2

VOC

SO2

Emission factors from urban road traffic

Emission factors from rural road traffic

Spatial disaggregation

Product yieldEmissionfactors

Ei,j / year = (Ai / year) * Fei,j

Simple Methodology

Industrialemissions NOx

CO

CO2

VOC

SO2

Number of employers

NOx emissions from industrial processes

VOC emissions from industrial processes

SOx emissions from industrial processes

CO emissions from industrial processes

CO2 emissions from industrial processes

z

wc

y

vc

x

uc

t

c iiii

Tendency= Advection

Turbulent diffusion

+ Ri + Si + LiChemistry Emissions Dry deposition

z

cK

zi

V

The air quality model (PNAM)The air quality model (PNAM)

Implementation tools Implementation tools

• Fortran 90 (dynamic memory allocation,

modules, pointers, recursion,…)

• Run-time System Library (RSL), an MPI-

based parallel software package for finite

difference regular-grid problems

http://www-unix.mcs.anl.gov/~michalak/rsl/

• domain decomposition and refinement

• local address space computation

• local/global index translation

• specialized communication for updating ghost regions

• inter- and intra-domain communication for nesting

• dynamic remapping for load balancing

RSL provides facilities for RSL provides facilities for

in each domain combine simple tasks in a coarse task subdomains not necessarily of the same

shape subdomains not necessarily of regular

shape

map coarse tasks to processors

Domain decomposition

one-way nesting

mother domainforcing

nested domains

forcing = provide Dirichlet boundary conditions on the child domain

recursive descent

parallelism at domain level, iteration over domains

recursive descent

parallelism at domain level, iteration over domains

recursive descent

parallelism at domain level, iteration over domains

recursive descent

parallelism at domain level, iteration over domains

Workload per task: CPU time each vertical grid

column

the stiffness of the diffusion + chemical kinetics ODE systems

varies in time and space

variable time step feature of the ODE solver causes different workloads in

vertical columns

load imbalance = inefficiencyG. Barone, P. D'Ambra, D. di Serafino, G. Giunta, A. Riccio, A. Murli, “PNAM: Parallel software for air

quality simulations in Naples area”, J. Environ. Health and Manag., 1999, 10, 209-215

G. Barone, P. D'Ambra, D. di Serafino, G. Giunta, A. Riccio, A. Murli, “Application of a parallel Air Quality model to the Campania region”, Environ. Modelling and Software”, 2000, 15, 503-511

dynamic load balancing strategy

run-time agglomeration and remapping of simple tasks

For each domain

do assess processors’ work loads

determine the average workload

greedy algorithm for local work flow scheduling

done

Dynamic load balancing: remapping of tasks over processors (4x4 processors mesh,24 h

simulation)

cpu time (vertical diffusion + chemistry) per processor

16 processors24 h simulation

24 h simulationone nest

efficiency

speedup

beowulf 16 procs

IBM SP 16 procs

24 h simulationone nest

efficiency

speedup

The Princeton Ocean Model (POM)

• Sigma coordinate, free surface, primitive equation ocean model, which includes a turbulence sub-model (Mellor-Yamada level 2.5). 

• Developed in the late 1970's by Blumberg and Mellor, with subsequent contributions from other people. (Blumberger and Mellor 1987)

• Used for modeling of estuaries, coastal regions and open oceans

• The most used ocean model in the coastal applications.

To know morehttp://www.aos.princeton.edu/WWWPUBLIC/htdocs.pom/

Blumberg, A. F. and G. L. Mellor, A description of a three-dimensional coastal ocean circulation model. Three-Dimensional Coastal ocean Models, edited by N. Heaps, 208 pp., American Geophysical Union., 1987

POM’s applications• Used in many hindcast/nowcast/forecast systems projects……

http://splash.princeton.edu/WWWPUBLIC/PROFS/ http://chartmaker.ncd.noaa.gov/csdl/op/

http://www.jamstec.go.jp/frsgc/jcope/

http://www.mar.dfo-mpo.gc.ca/

http://superior.eng.ohio-state.edu

• … and in many process studies.

Example of applications• Gulf of Naples climatology

• A marine ecological study for the Campania Region

References

Mariani P., Esposito S., Ribera M., Interaction between seasonal cycle and terrestrial runoff in determining the time course of plankton biomass in a coastal embayment: a numerical study. Chemistry and Ecology. Submitted

Example of applications

• The study of the mixed layer depth in the Mediterranean

Example of applications

POMpn validation

SEAMOUNT test

east west z = 4500 m h = 450 m 21 σ vertical levels stretched horiz. grid ρ (z) V0 = 0.2 m/s

Simulation timeSimulation time

8,42 h

4,36 h

2 domains vs 1 domain

1,14 h

POMpn performance

POMpn performance

EfficiencyEfficiency E(p) =T(1)

T(p)

p

Efficiency loss < 10%

Speed-upSpeed-up

0

1

2

3

4

5

6

7

8

9

0 2 4 6 8 10

numero di processori

spee

d-u

p

Speed-up misurato Speed-up ideale

S(p) =T(1)

T(p)

POMpn performance

Thinking in… grid

Computational resource sharing;

Distributed storage;

Security infrastructure;

Complex problem solving environment.

Ubiquitous (high performance) computing

The Application Workflow

Offline model coupling

Weather driven simulations

Operational/On Demand modes

User Request

Initial and boundary conditiondata file collecting

MM5 weather simulation

Other Environmental Model

Visualization Tools

NOAA

ECMWF

POM ocean circulation model PNAM air quality model

ScheduledRequest

Cache trigger

Full modular framework

High performance computing tools

Some trivial parallel process

Our grid approachComputational unit decomposition:

MM5 Modules POM Pre & Post Processing STdEM / PNAM coupling

Parallel process takes care of:

Nested domain pre-processing Loosely coupled models

Optimize inter-process communications:

Limiting time and data dependences

Optimize data exchanging

Only node/node data transfers No buffer node usage

MM5 module decomposition

Grid middleware wraps around each model and model component

both master managed job factory service (mmjfs)

and ad hoc grid service components

Grid parallelism achieved in nested domains preprocessing

The MM5 job consumes other jobs produced data

TERRAIN

REGRIDdomain 1

INTERPFdomain 1

Mm5

INTERPFdomain 2

INTERPFdomain 3

INTERPFdomain 4

REGRIDdomain 2

REGRIDdomain 3

REGRIDdomain 4

The Application Job flow (1/2)

The MM5 job produces data for dependent jobs

No coupling between MM5 coupled models

Grid parallelism achieved by weather driven model concurrent runs

Asynchronous results publishing

Synchronized metadata update to ensure stored data coherence

Mm5

INTERPPOM

PUBLPOM

PUBLPNAM

PUBLMm5

UPDATEdb-metadata

POM

STdEM

PNAM

The Application Job flow (2/2)

Grid EnablingEnvironmental Models

• Environmental models as computational resources

• Developing a wrap around the resource using a high level programming language;

• Developing a grid wrap using a middleware grid toolkit as Globus GT3;

• Exposing common interface through the job submitting system or an ad hoc developed grid service.

HardwareOperating System

ResourcesJVMGlobus ToolkitGrid Services

interfaces

Sta

ndar

d

Cus

tom

grid

wra

p

Developing the Grid Application

Remote resource virtualization with grid service consumer;

The grid application as glue between remote services and the local machine;

Nothing than application setup could be executed on the user interface (thin client possibility).

HardwareOperating System

Java Virtual MachineGlobus Toolkit

Grid Application

Grid ServiceConsumer

interfaces

HardwareOperating System

Java Virtual MachineGlobus Toolkit

Grid Application

Grid ServiceConsumer

interfaces

the Internet

grid middleware

HardwareOperating System

ResourcesJVMGlobus ToolkitGrid Services

interfaces

Sta

nda

rd

Cu

sto

m

grid

wra

p

HardwareOperating System

ResourcesJVMGlobus ToolkitGrid Services

interfaces

Sta

nda

rd

Cus

tom

grid

wra

p

HardwareOperating System

ResourcesJVMGlobus ToolkitGrid Services

interfaces

Sta

nda

rd

Cus

tom

grid

wra

p

HardwareOperating System

ResourcesJVMGlobus ToolkitGrid Services

interfaces

Sta

ndar

d

Cus

tom

grid

wra

p

MesoscaleModel 5v3

PrincetonOcean Model

Parallel NaplesAirquality Model

S-T DistributionEmission Model

MM 5 POM PNAM STdEM

Grid Application Building Blocks

Up and running

http://imfa.uniparthenope.it

Operational Web Portal 1/3

Weather Forecast

Surface sea current forecast

Operational Web Portal 2/3

Interactive time series graph

Operational Web Portal 3/3