advanced uncertainty evaluation of climate models by monte ......marko laine [email protected]...

Advanced uncertainty evaluation of climatemodels by Monte Carlo methods

Marko [email protected]

Pirkka Ollinaho, Janne Hakkarainen, Johanna Tamminen, HeikkiJärvinen (FMI)

Antti Solonen, Heikki Haario (LUT)Alexander Ilin, Erkki Oja (Aalto)

FMI – Finnish Meteorological InstituteLUT – Lappeenranta University of Technology

SIAM UQ 2012

Contents

Adaptive Markov chain Monte Carlo methodEfficient MCMC for short chainsParallel chainsEarly rejection

MCMC for ECHAM5 climate modelFormulating the cost functionStochastic Lorenz 95 test case

EPPES – ensemble prediction and parameter estimationLorenz 95 experiment with EPPESECHAM5 experiment with EPPES

M.Laine: Advanced uncertainty evaluation of climate models by Monte Carlo methods 2/27

Adaptive MCMC

The current work is based on our success on applying anddeveloping adaptive Markov chain Monte Carlo (MCMC) forchemical kinetics, ecological models, and satellite retrieval. In2010 we started a project on applying MCMC for climate modelclosure parameter uncertainty evaluation.

• Haario H., Saksman E., Tamminen J.: An adaptive Metropolis algorithm.Bernoulli 7(2), pp. 223–242, 2001.

• Haario H., M. Laine, M. Lehtinen, E. Saksman, J. Tamminen: Markovchain Monte Carlo methods for high dimensional inversion in remote sensing,with discussion, J.R. Statist. Soc. B, 66, part 3 pp. 591–607, 2004.

• Haario H., M. Laine, A. Mira, E. Saksman: DRAM: Efficient adaptiveMCMC. Stat. Comput. 16, pp. 339–354, ISSN 0960-3174, 2006.

• Laine, M., J. Tamminen: Aerosol model selection and uncertainty modellingby adaptive MCMC technique. Atmos. Chem. Phys., 8, 2008.


Terminology for modeling with MCMC methodsThe observation model in general form is

y = f (x|θ) + ε,observations = model + error.

Likelihood function for Gaussian errors corresponds to aquadratic cost function, with

p(y|θ) ∝ exp

{−1

2∑n

i (yi − f (xi|θ))2

σ2

}

= exp{−1

2SS(θ)

σ2

},

where SS(θ) = −2 log(p(y|θ)), the log-likelihood in"sum-of-squares" cost function format. For calculating theposterior, we also need to account SSpri(θ) = −2 log(p(θ)), theprior "sum-of-squares".


Metropolis-Hastings algorithm

Random walk Metropolis-Hastings algorithm with Gaussianproposal distribution (and Gaussian likelihood).

• Propose new parameter value θprop = θcurr + ξ, whereξ ∼ N(0, Σprop) is drawn from the proposal distribution.

• Accept θprop with probability α,

α(θcurr, θprop) = 1∧ exp{− 1

2

(SS(θprop)− SS(θcurr)

σ2

)

− 12

(SSpri(θprop)− SSpri(θcurr)

)}

• Efficient proposal distribution⇒ adaptive tuning of Σprop,AM, DRAM algorithms.


Short chains and adaptation• It is important to make short chains as efficient as possible.• Efficient: produce estimates with small Monte Carlo error.

0 1000 2000 3000 4000 50000

0.1

0.2

0.3

0.4

0.5

simulation index

mean of the 1. parameter

mhamdramram

0 1000 2000 3000 4000 50000.94

0.95

0.96

0.97

0.98

0.99

195% quantile

simulation index

mhamdramram

Short MCMC chain repeated 1000 times with differentalgorithms.Gaussian 10 dimensional target, too large initial covariance.


Short chains and adaptation

• But, adaptation might slow the convergence.

0 1000 2000 3000 4000 50000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

simulation index

mean of the 1. parameter

mhamdramram

0 1000 2000 3000 4000 50000.94

0.95

0.96

0.97

0.98

0.99

195% quantile

simulation index

mhamdramram

Same as in the previous slide, but now with more optimalinitial proposal.Gaussian 10 dimensional target, near optimal initial covariance.


Faster MCMC: parallel chains

• Random walk MCMC isby nature sequential, andit is generally moreefficient to run one longchain than many shortindependent chains.

• In parallel adaptiveMCMC, the adaptation isdone over the points in allchains and they share onecommon adapted proposalcovariance.

• Communication betweenthe chains can beasynchronous.

θi+1 ∼ N(θi, Σi)

α = min(1, π(θi+1)/π(θi))

θi+1 = θi +1

i + 1(θi − θi)

Σi+1 =i− 1

iΣi +

1i(θi − θi)(θi − θi)T





π(θi+1) ∝ p(θi+1)L(y|θi+1) π(θi+1) ∝ p(θi+1)L(y|θi+1) π(θi+1) ∝ p(θi+1)L(y|θi+1)


Faster MCMC: early rejectionIdea: evaluate the likelihood in parts and check after each partif the proposed parameter value can be rejected.

Cumulative cost function evaluated after each monthduring one year climate model simulation

0

10

20

30

40

50

60

2 4 6 8 10 12

COST FUNCTION VALUE

MONTH1 2 3 4 5 6 7 8 9 10 11 12

0

200

400

600

800

1000

1200

Early rejection month

time to stop the simulation proportion of stopped runs by month

This simple trick saved 10%–80% of CPU time in different testcases.


Early rejection

• In many cases SS(θ) is a monotonically increasing functionwrt. adding new observations or simulating the modelfurther in time.

• With the acceptance probability α(θcurr, θprop) defined inprevious slides, we draw u ∼ U(0, 1), and accept if

−2 log(u) <SS(θprop)− SS(θcurr)

σ2 + SSpri(θprop)−SSpri(θcurr).

• If we write this as

SScrit = −2 log(u) + SS(θcurr)/σ2 + SSpri(θcurr)

< SS(θprop)/σ2 + SSpri(θprop),

then we can stop evaluating the model whenSS(θprop) ≥ (SScrit − SSpri(θprop))σ2.


MCMC for ECHAM5 climate model

Studies on feasibility of MCMC for large scale climate models,on the formulation of the likelihood and on the problem ofchaotic behaviour of the models.

• H. Järvinen, P. Räisänen, M. Laine, J. Tamminen, A. Ilin, E. Oja, A.Solonen, H. Haario: Estimation of ECHAM5 climate model closureparameters with adaptive MCMC, Atmospheric Chemistry and Physics,10(1), pages 9993–10002, 2010. doi:10.5194/acp-10-9993-2010

• J. Hakkarainen, A. Ilin, A. Solonen, M. Laine, H. Haario, J. Tamminen,E. Oja, H. Järvinen: On closure parameter estimation in chaotic systems,Nonlinear Processes in Geophysics, 19(1), pages 127–143, 2012.doi:10.5194/npg-19-127-2012

• A. Solonen, P. Ollinaho, M. Laine, H. Haario, J. Tamminen, H. Järvinen:Efficient MCMC for climate model parameter estimation: paralleladaptive chains and early rejection, Bayesian Analysis, 7(2), 1–22 2012.


Climate model parametrization• In climate models, the atmosphere is divided into cells.• The scale of many important processes is smaller than a

single cell (e.g. clouds and rain).• These processes are parametrized: for example clouds and

rain are calculated based on the knowledge of thetemperature and humidity in the cell.2.3. Performance Metrics for Climate Models Chapter 2. Modeling Climate

Figure 2.2: The grid points that ECHAM5 uses in horizontal resolution T21.

Roeckner et al. (2006) have tested the sensitivity of the model as to the choice ofspatial resolution and concluded that the convergence of the model to a realisticclimate depends both on the choice of the horizontal and the vertical resolution.Intuitively, the denser the grid, the better the model performs in producing a realisticclimate. However, according to the sensitivity analysis, this is not entirely true. Forinstance, when vertical resolution is L19, it does not matter if the horizontal resolutionis increased above T42. On the other hand, with the vertical resolution of L31, themodel performance becomes monotonically better when the horizontal resolution isincreased.

Running ECHAM5 is very demanding for today’s computers. For instance, Max-Planck Institute for Meteorology lists on their website computer resources needed tointegrate ECHAM5 at specified resolutions1. In their experiment, they have used thesupercomputer NEC SX-6 with four CPUs for the resolution T21L19, with which ittook about 16 minutes to simulate the climate for one year. When the resolution wasincreased to T31L19, the time needed was doubled.

2.3 Performance Metrics for Climate Models

2.3.1 Key Concepts and Notation

This section introduces some basic methodology that can be employed to assess thesuccess of climate simulations. Determining the accuracy of short-term weather fore-casts or long-term climate projections is broadly referred to as forecast verification.2

1http://www.mpimet.mpg.de/en/wissenschaft/modelle/echam/echam5/runcontrol/echam5-computer-resources.html

2This thesis makes distinction only between short-term weather forecasts and long-term climateprojections. Here the distinction is that short-term weather forecasts are comparable to observations

10


Stochastic Lorenz 95 test case

We have 40 slow state variables -•- and 320 fast state variables-•-, whose effect is parametrized in the forecast model.

Good test case to study:• Estimation

methodologies.• Different

parameterizationsg(xk, θ).

• Modeling error.• Filtering and

ensemble methods.

3 2 1 40 39

dxk

dt= −xk−1 (xk−2 − xk+1)− xk + F−hc

b

Jk�

j=J(k−1)+1

yj

dyj

dt= −cbyj+1 (yj+2 − yj−1)− cyj +

c

bFy +

hc

bx1+� j−1

J �

NATURE:

FORECAST MODEL: dxk

dt= −xk−1 (xk−2 − xk+1)− xk + F−g(xk, θ)

Wilks, D.: Effects of stochastic parametrizations in the Lorenz ’96 system, Quart. J. Roy. Meteor. Soc., 131(606),389–407, 2005. doi:10.1256/qj.04.03


Lorenz 95 – true forcing vs. parameterization

10 5 0 5 10 153

2

1

0

1

2

3

4

5

6

7

slow variables

forc

ing

due

to fa

st v

aria

bles

The true effect of the fast variables wrt. the values of the slowvariables in full model simulation. Red lines give estimatedoptimal linear parameterization and its uncertainty.


Climate model parametrization

• Currently, best expert knowledge is used to define theoptimal closure parameter values, based on observations,process studies, etc.

• Closure parameters act as ’tuning handles’ of thesimulated climate.

• Our goal is to come up with an objective, algorithmic wayto determine the closure parameters.

ECHAM5 parameters estimated by MCMC and ensemble methods:Parameter DescriptionCAULOC A parameter influencing the accretion of cloud droplets

by precipitation (rain formation in stratiform clouds).CMFCTOP Relative cloud mass flux at the level above non-buoyancy

(in cumulus mass flux scheme).CPRCON A coefficient for determining conversion from cloud water to rain

(in convective clouds).ENTRSCV Entrainment rate for shallow convection.


Cost function - likelihood

• For climate model parameter estimation, it is essential todefine a metric with which we can measure the ’goodness’of the parameters.

• In our MCMC experiments, we have specified differentcost functions based on the net radiation at the top of theatmosphere.

• Example: global F and zonal Ft,x modeled mean fluxes arecompared to the observed ones, Fo and Fo

t,x.

J(θ) =(F− Fo)2

σ2 +12

∑t=1

∑y

(Ft,x − Fot,x)

2

σ2t,x


ECHAM5 – MCMC results

chain histograms pairwise chains

0 5 10 15 20 25 30

CAULOC

0 0.05 0.1 0.15 0.2

CMFCTOP

0 0.005 0.01 0.015

CPRCON

0 1 2 3 4 5

x 10−3

ENTRSCV

4 4.5 5 5.5 6

sqrt(costf.)

0

0.05

0.1

CM

FC

TO

P

CAULOC

5

10

15x 10

−3

CP

RC

ON

CMFCTOP

2

4

x 10−3

EN

TR

SC

V

CPRCON

0 10 20 30

4.5

5

5.5

sqrt

(cos

tf.)

0 0.05 0.1 0 5 10 15

x 10−3

0 2 4

x 10−3

ENTRSCV

Conducting a MCMC experiment of about 3000 one yearclimate model runs (ECHAM5 in T21 resolution), required 3months worth of super computer time.


EPPES – ensemble prediction and parameterestimation system

• Estimating closure parameters using ensemble runs.• Initially aimed for estimating numerical weather

prediction (NWP) model closure parameters using existingoperational ensemble prediction (EPS) infra-structure.

• But usable for climate model uncertainty analysis, too.

• M. Laine, A. Solonen, H. Haario, H. Järvinen: Ensemble prediction andparameter estimation system: the method, Quarterly Journal of the RoyalMeteorological Society, 138(663), 2012. doi:10.1002/qj.922

• H. Järvinen, M. Laine, A. Solonen, H. Haario: Ensemble prediction andparameter estimation system: the concept, Quarterly Journal of the RoyalMeteorological Society, 138(663), 2012. doi:10.1002/qj.923

• P. Ollinaho, H. Järvinen, M. Laine, A. Solonen, H. Haario: NWP modelforecast skill optimization via closure parameter variations, QuarterlyJournal of the Royal Meteorological Society, in revision, 2012.


The EPPES concept for NWP

EPPES = EPS + parameter estimation.

• In addition to initial valueperturbations, modelparameters θ are sampledfrom a proposaldistribution.

• The parameters areweighted according to acost function that dependson forecast skill.

• The backgrounduncertainty in θ is modeledas Gaussian θ ∼ N(µ, Σ)with µ and Σ estimatedsequentially.

model space

* * *

parameter space

1.2 1.4 1.6 1.8 2 2.20.2

0.1

0

0.1

0.2

0.3

0.4

1

2


Cost function and likelihoodCost function for model F, observations yk, initial values xk,and parameter θk for forecast time window k:

(yk − F(xk; θk))′ Σ−1

obs (yk − F(xk; θk))

+ (θk − µ)′ Σ−1 (θk − µ)= Jobs + Jpri,

interpreted as a statistical inverse problem:

p(θk|yk) ∝ exp(−12

Jobs) exp(−12

Jpri)

∝ p(yk|θk)p(θk)= likelihood · prior.

In EPPES we are estimating local parameters θk by importancesampling for p(θk|yk), and also the optimal global hyperparameters µ and Σ that describe the variability in θ betweenthe time windows by sequential updates.


Random effect model

• The true value of theparameter θ istreated as random.

• The distributiongeneratingindividual θk foreach forecast timewindow is assumedGaussian and theparameters of this"meta" distributionare estimatedsequentially.

The model:

θk ∼ N(µk, Σk)µk ∼ N(µk−1, Wk−1)Σk ∼ iWish(Σk−1, nk−1).

with the following update formulas:

Wk =(

W−1k−1 + Σ−1

k−1

)−1

µk = Wk

(W−1

k−1µk−1 + Σ−1k−1θk

)

nk = nk−1 + 1Σk =

(nk−1Σk−1 + (θi − µk)(θk − µk)′

)/nk.


Stochastic Lorenz 95 test case• We parameterize the effect of the fast states -•- using linear

model with two closure parameters in g(xk, θ) of theforecast model.

• We run 50 member ensembles sequentially by perturbingboth the initial values and the closure parameters.

• We observe some ofthe slow variablesevery 2nd "day" ofthe truth run withadded observationerror.

• Cost function isconstructed using6 day forecast skill.

3 2 1 40 39

dxk

dt= −xk−1 (xk−2 − xk+1)− xk + F−hc

b

Jk�

j=J(k−1)+1

yj

dyj

dt= −cbyj+1 (yj+2 − yj−1)− cyj +

c

bFy +

hc

bx1+� j−1

J �

NATURE:

FORECAST MODEL: dxk

dt= −xk−1 (xk−2 − xk+1)− xk + F−g(xk, θ)


Stochastic Lorenz 95 experiment

Evolution of two parameters in the L95 experiment. On left,each column of points corresponds to proposed parametervalues in one time window of the sequential estimationprocedure. On right, forecast skill is calculated over a grid,with an ellipse and dot showing the final estimated Σ and µ.

parameter evolution "6 day" forecast skill

50 100 150 200 250

0.5

1

1.5

θ 1

50 100 150 200 250

−0.2

0

0.2

0.4

0.6

θ 2

Ensemble number


ECHAM5 climate model experiment

ECHAM5 atmospheric general circulation model was run inensemble prediction (EPS) mode to estimate of 4 closureparameters related to cloud formation.

• Control run and 50 perturbed members.• Initial states from ECMWF EPS system, with data from Jan

2011 to March 2011, 00 and 12UTC daily.• Model resolution: T42 truncation with 31 levels.• Total of 2× 90× 51 = 9180 parameter sample points.• Cost function: RMS of 500 hPa geopotential height 10 days

forecast error.


ECHAM5 experiment

First iteration vs. last iteration as 2 dimensional plots showingthe proposed parameter values, the weights they havereceived, and the matrices Σ and W of the EPPES algorithm.

0

0.2

0.4

0.6

0.8

11: 2011−01−01 00:00:00

CM

FC

TO

P

0

0.005

0.01

0.015

CP

RC

ON

0 20 400

1

2

3

4x 10

−3

EN

TR

SC

V

CAULOC0 0.5 1

CMFCTOP0 0.01 0.02

CPRCON

0

0.2

0.4

0.6

0.8

11: 2011−03−31 12:00:00

CM

FC

TO

P

0

0.005

0.01

0.015

CP

RC

ON

0 20 400

1

2

3

4x 10

−3E

NT

RS

CV

CAULOC0 0.5 1

CMFCTOP0 0.01 0.02

CPRCON


ECHAM5 validationRMSE variability using default parameters in blue and usingthe EPPES optimized parameter values in red. Independentdata but with the same cost function as in the optimization.

ECHAM5 validation

RMS variability using default parameters in blue and using theEPPES optimised parameter values in red. Independent data butwith the same cost function as in optimization.

0.2

0.3

0.4

0.5

0.6

0.7

0.8

5 6 7 8 9 10 05060708090

100110120

5 6 7 8 9 10 0

2010

jfm

Forecast length (days)

0.20.30.40.50.60.70.8

5 6 7 8 9 10 05060708090

100110120

5 6 7 8 9 10 0

2011

a0.20.30.40.50.60.70.8

5 6 7 8 9 10 0

500hPa geopotential height ACC

5060708090

100110120

5 6 7 8 9 10 0

2011

jfm

Expreriment: l2l3-def 500hPa geopotential height RMSE

M.Laine: Numerical weather prediction model tuning via ensemble prediction system 10/14

500hPa geopotential height RMSE

forecast length (days)

2011, April

optimizedparameters

defaultparameters

about 6 hour increase in the forecast skill


Conclusions and ongoing research• MCMC can be applied to climate models with the help of

adaptation and other speed up tricks, but the problem liesin the formulation of the cost function.

• Existing ensemble run infrastructures can be used to inferabout model parameters in NWP and this methodologycan be used with climate models, also.

• MCMC runs, even short ones, are useful to pinpointproblems in parametrizations, formulation of the costfunction, etc.

• Which climate model fields to include in the cost functionand how to scale and weight multi-criteria cost functionterms?

• Can filtering method be used in defining climate modelcost function?

• How well do locally tuned parameter work in longersimulations?


advanced uncertainty evaluation of climate models by monte ......marko laine [email protected]...

Documents