eumetcal nwp-course 2007: the concept of ensemble forecasting renate hagedorn european centre for...

EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting

Renate Hagedorn European Centre for Medium-Range Weather Forecasts

The General Concept of

Ensemble Forecasting in Theory and Practice


Goals

• Students to learn:

Why & how are probabilistic forecasts produced & used?

• Teacher to learn:

What are your greatest needs & expectations from an EPS?

• Achieve together:

What is the best way forward to integrate uncertainty information

as an integral component into public weather forecasts?


Outline

• Why do we need Ensemble Prediction Systems?

Chaos theory and its consequences for weather prediction

• How are probabilistic forecasts made in practice?

How do we represent uncertainties?

From ensemble members to PDF’s and CDF’s

• Good ensembles – bad ensembles?

How to verify probabilistic forecasts


The Philosophical Point of View…

To know what you know, and to know what you do not know,that is real knowledge

ConfuciusThe Analects


The Practical Point of View…

orPredicting predictability is as important as predicting rainfall

No forecast is complete without a forecast of forecast skill


The Practical Point of View…

orPredicting predictability is as important as predicting rainfall

• Weather Forecasts have errors (are uncertain)

• Ultimate goal of weather forecasting is to improve user decisions, i.e. decision-making based on forecast information should be superior to decision-making without forecast information

• Decision-making can be improved when uncertainty information is available

WHY?

No forecast is complete without a forecast of forecast skill


Weather Forecasting: How does it work?

Numerical modelto describe the processes

in the earth system



Observationsto start the forecast



Computer

Observations

Model


Deterministic Forecasting

Forecast time

Tem

pera

ture

Initial condition Forecast

Is this forecast “correct”?

Initial Uncertainty

Model Error


The Lorenz Attractor

“… one flap of a sea-gull’s wing

may forever change the

future course of the weather”

(Lorenz, 1963)


The Lorenz Attractor…

bZXYZ

YrXXZY

YXX

...is the visualization of the

time-evolution of a three-

dimensional non-linear

dynamical system described

by the ‘Lorenz-63‘ equations


Scientific basis for Ensemble Predictions

In a non-linear dynamical system, the growth of uncertainties in initial conditions is flow dependant

bZXYZ

YrXXZY

YXX

IC IC IC

cold warmcold warm cold warm


Ensemble Forecasting

Forecast time

Tem

pera

ture

Complete description of weather prediction in terms of aProbability Density Function (PDF)

Initial condition Forecast


Flow dependence of forecast errors

If the forecasts are coherent (small spread) the atmosphere is in a more

predictable state than if the forecasts diverge (large spread)

aa

34

30

28

26

24

22

20

18

16

14

12

10

0 1 2 3 4 5 6 7 8 9 10Forecast day

UK

Control Analysis Ensemble

ECMWF ensemble forecast - Air temperatureDate: 26/06/1994 London Lat: 51.5 Long: 0

30

28

26

24

22

20

18

16

14

12

10

8

0 1 2 3 4 5 6 7 8 9 10Forecast day

UK

Control Analysis Ensemble

ECMWF ensemble forecast - Air temperatureDate: 26/06/1995 London Lat: 51.5 Long: 0

26th June 1995 26th June 1994


Goal of Ensemble Prediction

• Represent/predict uncertainty of prediction

• Move from deterministic to probabilistic forecast

• Ensemble Spread should

capture “truth” (spread ~ RMS error)

indicate range of uncertainty


Outline









Sources of uncertainties

• Initial conditions: limited accuracy of observations and data assimilation

Run ensemble of forecasts from slightly different conditions. Initial perturbations generated via singular vector technique, breeding vectors, ETKF, etc…


Initial Perturbations

• Problem: How to construct “good” initial perturbations, given that only a

number of limited integrations can be carried out?

• Solution: Find initial perturbations with maximum amplification rate

• Singular vector approach: Perturbations with the fastest growth over a finite time intervall

(SV) can be identified in solving an eigenvalue problem of the product of the tangent forward and adjoint model propagator

22/101

**2/10 PMEEPME


Example of initial perturbations

150OW 100 OW 50OW 0O 50OE 100OE 150OE

50.0N

1000800600500400300

200

Cross section of temp 20060321 00 step 0 Expver 0001

60°S60°S

30°S30°S

0°0°

30°N30°N

60°N60°N

120°W

120°W

60°W

60°W

0°

0°

60°E

60°E

120°E

120°E21/03/2006 00UTC, Temperature (every 0.2K) @~700hPa

@ 50°N





• Model error parameterisations: how to represent unresolved processes

o stochastic physics approach physical parameter values: inaccurate knowledge of parameter space

o perturbed parameter approach model structure: how to represent physical processes in models

o multi-model approach


Stochastic Physics

• Background:

Assume that parameterization is tuned to give correct ensemble mean Account for statistical fluctuations using random numbers

• ECMWF implementation: assign random numbers [0.5,1.5] to 10º lat/lon boxes multiply model parameterization tendencies by these random numbers assign new random numbers every 6 hours

Skill measure: area under ROC curve Event: precipitation > 40 mm/day

Top curves: winter performance

Bottom curves: summer performance

Buizza et al, 1999

SP: yes

SP: no


RMS error

Spread

No SPBS

SPBS

New Stochastic Physics (SPBS)

Courtesy: Judith Berner

under-dispersion reduced for all forecast ranges

RMS error reduced

u-component 850hPa, Tropics





• Model error parameterisations: how to represent unresolved processes

o stochastic physics approach physical parameter values: inaccurate knowledge of parameter space

o perturbed parameter approach model structure: how to represent physical processes in models

o multi-model approach

• Boundary conditions: SST, soil moisture, sea ice, etc.

Unknown changes in boundary conditions are source of uncertainty, however, known (or well modelled) external forcing can be source of predictability for extended range forecasts


Outline









Ensemble Prediction System

• 1 control run + 50 perturbed runs (TL399 L62)

added dimension of ensemble members

f(x,y,z,t,e)

• How do we deal with added dimension when

interpreting, verifying and using EPS output?


From ensembles to PDF’s

6 8 10 12 14Temperature [Degree Celsius]

0.0

0.5

1.0

1.5

2.0


Count members per bin


0

5

10

15

20

25

num

ber

of m

embe

rs in

1 d

eg b

in


Discrete probability distribution


0.0

0.1

0.2

0.3

0.4

0.5

prob

abili

ty o

f T

emp

in 1

deg

bin


Continuous probability density function


0.0

0.1

0.2

0.3

0.4

0.5

prob

abili

ty d

ensi

ty f

unct

ion

(PD

F)

f (x)1

2exp

x 2 2

2


Continuous PDF


0.0

0.1

0.2

0.3

0.4

0.5

prob

abili

ty d

ensi

ty f

unct

ion

(PD

F)

f (x)1

2exp

x 2 2

2

f (x) is the “probability density” function,

or PDF

is the “mean”

is the “standard deviation”


-5 0 5 10 15 20 25Temperature [Degree Celsius]

0.0

0.2

0.4

0.6

0.8

1.0

prob

abili

ty d

ensi

ty f

unct

ion

(PD

F)

Continuous PDF’s

f (x)1

2exp

x 2 2

2

Which forecast has a mean of 10 degree celsius?



0.0

0.2

0.4

0.6

0.8

1.0

prob

abili

ty d

ensi

ty f

unct

ion

(PD

F)

Continuous PDF’s

f (x)1

2exp

x 2 2

2

Which forecast has the highest σ (standard deviation)?


Continuous PDF’s


0.0

0.2

0.4

0.6

0.8

1.0

prob

abili

ty d

ensi

ty f

unct

ion

(PD

F)

f (x)1

2exp

x 2 2

2

= 0.0, = 0.5

= 10.0, = 1.0

= 15.0, = 2.0


What information can we get from PDF’s?

• PDF’s describe the probability density of a continuous spectrum of possible outcomes

• Probability density describes relative likelihood to be near a particular value

• We have to distinguish between: Continuous events: unlimited number of possible outcomes

(temperature, windspeed, …) Discrete events: limited number of possible outcomes

(rain/no-rain, temperature below/above freezing,…)

• Probabilities are only meaningful for discrete events P(9≤T≤11) can be determined from PDF (or CDF) P(T=10°C) = 0


Probabilities related to event


0.0

0.1

0.2

0.3

0.4

0.5

prob

abili

ty o

f T

emp

in 1

deg

bin


Probabilities related to event


0.0

0.1

0.2

0.3

0.4

0.5

prob

abili

ty o

f T

emp

in 0

.2 d

eg b

in



0.0

0.1

0.2

0.3

0.4

0.5

prob

abili

ty d

ensi

ty f

unct

ion

(PD

F)

Probability density function

f (x)1

2exp

x 2 2

2



0.0

0.1

0.2

0.3

0.4

0.5

prob

abili

ty d

ensi

ty f

unct

ion

(PD

F)

From PDF’s to probabilities

a

f(x)dxa)P(X

f (x)1

2exp

x 2 2

2


From PDF’s to CDF’s


0.0

0.2

0.4

0.6

0.8

1.0

PD

F &

CD

F

X

f(x)dxF(X)

f (x)1

2exp

x 2 2

2

a

f(x)dxa)P(X


Probabilities from a CDF


0.0

0.2

0.4

0.6

0.8

1.0

CD

F

What is the probability that the temperature will be 10°C?




0.0

0.2

0.4

0.6

0.8

1.0

CD

F

What is the probability that the temperature will be 10°C?

P(X=10) = 0




0.0

0.2

0.4

0.6

0.8

1.0

CD

F

What is the probability that the temperature will be ≤10°C?




0.0

0.2

0.4

0.6

0.8

1.0

CD

F

What is the probability that the temperature will be ≤10°C?

P(X≤10) = 0.5




0.0

0.2

0.4

0.6

0.8

1.0

CD

F

What is the probability that the temperature will be >11°C?




0.0

0.2

0.4

0.6

0.8

1.0

CD

F

What is the probability that the temperature will be >11°C?

P(X>11) = P(X≤∞) – P(X≤11) = 1. - 0.85 = 0.15




0.0

0.2

0.4

0.6

0.8

1.0

CD

F

What is the probability that the temperature will be between 9-11°C?




0.0

0.2

0.4

0.6

0.8

1.0

CD

F

What is the probability that the temperature will be between 9-11°C?

P(9≤X≤11) = P(X≤11) – P(X<9) = 0.85 – 0.15 = 0.70


From PDF’s to CDF’s


0.0

0.2

0.4

0.6

0.8

1.0P

DF


0.0

0.2

0.4

0.6

0.8

1.0

CD

F


Ensemble Prediction System

• 1 control run + 50 perturbed runs (TL399 L62)

added dimension of ensemble members

f(x,y,z,t,e)

• How do we deal with added dimension when

interpreting, verifying and using EPS output?

• Transition from forecasting local events (22°C) to categorical events (>20°)

deterministic (yes/no) to probabilistic (x%)


Outline









Objective of diagnostic/verification tools

Assessing the goodness of a forecast system involvesdetermining skill and value of forecasts




A forecast has skill if it predicts the observed conditions well according to some objective or subjective criteria.

A forecast has value if it helps the user to make better decisions than without knowledge of the forecast.




A forecast has skill if it predicts the observed conditions well according to some objective or subjective criteria.

A forecast has value if it helps the user to make better decisions than without knowledge of the forecast.

• Forecasts with poor skill can be valuable (e.g. location mismatch)

• Forecasts with high skill can be of little value (e.g. blue sky desert)


Assessing the quality of a forecast

• The forecast indicated 10% probability for rain

• It did rain on the day

• Was it a good forecast?

□ Yes

□ No

□ I don’t know

• Single probabilistic forecasts are never completely wrong or right (unless they give 0% or 100% probabilities)

• To evaluate a forecast system we need to look at a (large) number of forecast–observation pairs


Assessing the quality of a forecast system

• Characteristics of a forecast system:

Consistency*: Do the observations statistically belong to the distributions of the forecast ensembles? (consistent degree of ensemble dispersion)

Reliability: Can I trust the probabilities to mean what they say?

Sharpness: How much do the forecasts differ from the climatological mean probabilities of the event?

Resolution: How much do the forecasts differ from the climatological mean probabilities of the event, and the systems gets it right?

Skill: Are the forecasts better than my reference system (chance, climatology, persistence,…)?

* Note that terms like consistency, reliability etc. are not always well defined in verification theory and can be used with different meanings in other contexts


Rank Histogram

• Rank Histograms asses whether the ensemble spread is consistent with the assumption that the observations are statistically just another member of the forecast distribution

Check whether observations are equally distributed amongst predicted ensemble

Sort ensemble members in increasing order and determine where the observation lies with respect to the ensemble members


Rank Histogram




Temperature ->

Rank 1 case


Rank Histogram




Temperature ->

Rank 1 case Rank 4 case

Temperature ->


Rank Histograms

A uniform rank histogram is a necessary but not sufficient criterion for determining that the ensemble is reliable (see also: T. Hamill, 2001, MWR)

OBS is indistinguishable from any other ensemble member

OBS is too often below the ensemble members (biased forecast)

OBS is too often outside the ensemble spread


Reliability

• A forecast system is reliable if: statistically the predicted probabilities agree with the observed

frequencies, i.e. taking all cases in which the event is predicted to occur with a

probability of x%, that event should occur exactly in x% of these cases; not more and not less.


Reliability




• A reliability diagram displays whether a forecast system is reliable (unbiased) or produces over-confident / under-confident probability forecasts


Reliability




• A reliability diagram displays whether a forecast system is reliable (unbiased) or produces over-confident / under-confident probability forecasts

• A reliability diagram also gives information on the resolution (and sharpness) of a forecast system

Forecast PDFClimatological PDF


Reliability Diagram

Take a sample of probabilistic forecasts: e.g. 30 days x 2200 GP = 66000 forecasts

How often was event (T > 25) forecasted with X probability?

FC Prob. # FC “perfect FC”OBS-Freq.

“real” OBS-Freq.

100% 8000 8000 (100%) 7200 (90%)

90% 5000 4500 ( 90%) 4000 (80%)

80% 4500 3600 ( 80%) 3000 (66%)

…. …. …. ….

…. …. …. ….

…. …. …. ….

10% 5500 550 ( 10%) 800 (15%)

0% 7000 0 ( 0%) 700 (10%)

25

25

25


Reliability Diagram

Take a sample of probabilistic forecasts: e.g. 30 days x 2200 GP = 66000 forecasts

How often was event (T > 25) forecasted with X probability?

FC Prob. # FC “perfect FC”OBS-Freq.

“real” OBS-Freq.

100% 8000 8000 (100%) 7200 (90%)

90% 5000 4500 ( 90%) 4000 (80%)

80% 4500 3600 ( 80%) 3000 (66%)

…. …. …. ….

…. …. …. ….

…. …. …. ….

10% 5500 550 ( 10%) 800 (15%)

0% 7000 0 ( 0%) 700 (10%)

OB

S-F

req

uency

0 100

100

••

•

••FC-Probability

0


Reliability Diagram

over-confident model perfect model


Reliability Diagram

under-confident model perfect model


Reliability diagram

Reliability score (the smaller, the better)

imperfect model perfect model


Components of the Brier Score

2

1

)(1

ii

I

ii ofn

NREL

N = total number of casesI = number of probability binsni = number of cases in probability bin i

fi = forecast probability in probability bin I

oi = frequency of event being observed when forecasted with fi

Reliability: forecast probability vs. observed relative frequencies


Sharpness

Reliability Diagram

0.0 0.2 0.4 0.6 0.8 1.0Forecast Probability

0.0

0.2

0.4

0.6

0.8

1.0

Obs

erve

d F

requ

ency

Event: 500 hPa Geopotential anom. > 0.00 sigma

Area: West Africa (land only)

Model: DEMETER I

Start dates: Feb / 1980-2001

Avg. over FC period: 2-4 months (MAM)

Brier (Skill) Score: 0.203 ( 0.189)

B(S)S_Reliability: 0.026 ( 0.897)

B(S)S_Resolution: 0.073 ( 0.292)

Uncertainty: 0.250

0.0 0.2 0.4 0.6 0.8 1.00.0rel FC distribution

0.000.02

0.040.060.080.10

0.12

Reliability Diagram


0.0

0.2

0.4

0.6

0.8

1.0

Obs

erve

d F

requ

ency


Area: Northern Extratropics (land+sea)

Model: DEMETER I






Uncertainty: 0.250


0.00

0.01

0.02

0.03

0.04

Diagrams show the distribution of issued forecast probabilities

FC Probability FC Probability

Rel.

Fre

qu

en

cy

Rel.

Fre

qu

en

cy

Sample A Sample B


Sharpness

Reliability Diagram


0.0

0.2

0.4

0.6

0.8

1.0

Obs

erve

d F

requ

ency


Area: West Africa (land only)

Model: DEMETER I






Uncertainty: 0.250


0.000.02

0.040.060.080.10

0.12

Reliability Diagram


0.0

0.2

0.4

0.6

0.8

1.0

Obs

erve

d F

requ

ency


Area: Northern Extratropics (land+sea)

Model: DEMETER I






Uncertainty: 0.250


0.00

0.01

0.02

0.03

0.04

Diagrams show the distribution of issued forecast probabilities

FC Probability FC Probability

Rel.

Fre

qu

en

cy

Rel.

Fre

qu

en

cy

Sample A Sample B

Which sample contains the sharper probability forecasts?


Reliability diagram

Poor resolution Good resolution

Reliability score (the smaller, the better)

Resolution score (the bigger, the better)

c c

Size of red bullets represents number of forecasts in probability category (sharpness)


Components of the Brier Score

2

1

)(1

ii

I

ii ofn

NREL

N = total number of casesI = number of probability binsni = number of cases in probability bin i

fi = forecast probability in probability bin I

oi = frequency of event being observed when forecasted with fi

c = frequency of event being observed in whole sample

Reliability: forecast probability vs. observed relative frequencies

Resolution: ability to issue reliable forecasts close to 0% or 100%

2

1

)(1

conN

RES i

I

ii

Uncertainty: variance of observations frequency in sample

)1( ccUNC

Brier Score = Reliability – Resolution + Uncertainty


Brier Score

• The Brier score is a measure of the accuracy of probability forecasts

N

nnnN

BS op1

2

)(1

with p: forecast probability (fraction of members predicting event) o: observed outcome (1 if event occurs; 0 if event does not occur)

• BS varies from 0 (perfect deterministic forecasts) to 1 (perfectly wrong!)

• Considering N forecast – observation pairs the BS is defined as:

• BS corresponds to RMS error for deterministic forecasts


Brier Skill Score

• Skill scores are used to compare the performance of forecasts with that

of a reference forecast such as climatology or persistence

cBS

BSBSS 1

• positive (negative) BSS better (worse) than reference

• Constructed so that perfect FC takes value 1 and reference FC = 0

Skill score = score of current FC – score for ref FC

score for perfect FC – score for ref FC


Assessing the quality of a forecast system

• Characteristics of a forecast system:

Consistency: Do the observations statistically belong to the distributions of the forecast ensembles? (consistent degree of ensemble dispersion)

Reliability: Can I trust the probabilities to mean what they say?

Sharpness: How much do the forecasts differ from the climatological mean probabilities of the event?

Resolution: How much do the forecasts differ from the climatological mean probabilities of the even, and the systems gets it right?

Skill: Are the forecasts better than my reference system (chance, climatology, persistence,…)?

Relia

bili

ty D

iag

ram

Rank Histogram

Brier Skill Score


Brier Score -> Ranked Probability Score

5 10 15 20 25

f(y)

• Brier Score used for two category (yes/no) situations (e.g. T > 15oC)

5 10 15 20 25

• RPS takes into account ordered nature of variable (“extreme errors”)

F(y)1


Ranked Probability Score

category

f(y)

category

F(y)1

PD

F

CD

F



K

kkOBSkFC CDFCDF

KRPS

1

2,, )(

1

1

category

f(y)

category

F(y)1

PD

F

CD

F



category

f(y)

PD

F

RPS=0.01sharp & accurate

category

f(y)

PD

F

RPS=0.15sharp, but biased

category

f(y)

PD

F

RPS=0.05not very sharp, slightly biased

category

f(y)

PD

F

RPS=0.08accurate, but not sharp

climatology



• Measures the quadratic distance between forecast and verification probabilities for several probability categories k

K

kkBS

KRPS

11

1• It is the average Brier score across the range of the variable

• Ranked Probability Skill Score (RPSS) is a measure for skill relative to a reference forecast

cRPS

RPSRPSS 1

• Emphasizes accuracy by penalizing large errors more than “near misses”• Rewards sharp forecast if it is accurate


Example of RPSS for ECMWF’s EPS


Goals

• Students to learn:

Why & how are probabilistic forecasts produced & used?

• Teacher to learn:

What are your greatest needs & expectations from an EPS?

• Achieve together:

What is the best way forward to integrate uncertainty information

as an integral component into public weather forecasts?


Assignment-I

• What are your greatest needs and/or expectations from the probabilistic products of an EPS?

Are there any areas in your day-to-day work which benefit from probabilistic forecasts (now or in the future)?

Which aspect of the output of an EPS is most valuable for you?

Can you think of any information valuable for you which is currently not available as EPS product?


Assignment-II

• How would you present a probabilistic weather forecast to the general public?

• Prepare one or more examples of a weather forecast containing probabilistic information for:

TV Radio Newspaper Internet Governmental agency (weather warning) Commercial company …

• Some more hints next week, but you might think already this week about your general concept


References and further reading

• Palmer, T. and R. Hagedorn (editors), 2006: Predictability of weather and climate. Cambridge University Press, pp.702

• Jolliffe, I.T. and D.B. Stephenson, 2003: Forecast Verification. A Practitioner’s Guide in Atmospheric Science. Wiley, pp. 240

• Wilks, D. S., 2006: Statistical methods in the atmospheric sciences. 2nd ed. Academic Press, pp.627

• ECMWF newsletter for updates on EPS performance

• Hamill, T., 2001: Interpretation of Rank Histograms for Verifying Ensemble Forecasts. Monthly Weather Review, 129, 550-560

• Buizza, R., Bidlot, J.-R., Wedi, N., Fuentes, M., Hamrud, M., Holt, G., and Vitart, F., 2007: The new ECMWF VAREPS (Variable Resolution Ensemble Prediction System). Q. J. Roy. Meteorol. Soc., 133, 681-695

• Leutbecher, M. and T.N. Palmer, 2007: Ensemble forecasting. J. Comp. Phys., in press


Web links:

• ECMWF products and training:

http://www.ecmwf.int/products/forecasts/d/charts

http://www.ecmwf.int/newsevents/training/meteorological_presentations/MET_PR.html

• NCEP ensemble training:

http://www.emc.ncep.noaa.gov/gmb/ens/training.html

http://www.hpc.ncep.noaa.gov/ensembletraining/

• Interactive learning on probabilities:

http://www.shodor.org/interactivate/activities/

eumetcal nwp-course 2007: the concept of ensemble forecasting renate hagedorn european centre for...

Documents