1 wmo swfdp macau 9 april 2013 anders persson decision making process and blending ensemble and...

1 WMO SWFDP Macau 9 April

2013 Anders Persson

Decision making process and blending ensemble and deterministic forecasts


2013 Anders Persson19/04/23

1. What do good forecasters do?


2013 Anders Persson

Blending deterministic and probabilistic forecast information has been a challenge since Fitzroy started weather forecasting 150 years ago

An overcast evening outside London in January 1863:

Low clouds over snow covered ground with +2° C

The clouds will disperse and the temperature drop to -6° C

But will the clouds disperse?

Probably? (=60% chance?)

How will this affect the forecast?



The same situation in our days:

The NWP tells the clouds will clear and the temperature drop

+2º -6º

A classical, physical-meteorological, deterministic problemThe skilled weather forecaster is invited to “add value” to the NWP by modifying the -6° forecast

However, the real added value might be of some other kind. . . assume the clouds do not lift?

→



Assume the probability of clearing = 60% Three different forecasts might be provided

I. A compromise forecast -3º for verifications

II. A missed event is considered worse than a false alarm so -4º or -5º is forecast

III. Special customers are told that there is a slightly higher probability (60%) for the clouds to disperse with -6º, rather than not (40%) with +2º

All of these involve clever use of intuitive statistics



Asymmetric cost- or penalty functions

Error Error

“Pain”

Possible errors Possible errors

● ●

●

●

“Pain”Forecasts for unspecified

customer or for verification purposes

Specific customer

with sensitivity for missed cold

events

Forecast

too mild

Forecast too cold



Expected mean cost/d

£30

£20

£10

£0

Value of probabilities : The school book example

Loss=£100 and average probability of bad weather pclim=30%

£0 £30 £60 £90 protection cost

£30

£20

£10

0

gain

Never protect

Always protect

Deterministic forecast

Perfect forecasts

Useful

forecasts

Ob Fc R _

R 20 10 - 10 60


2013 Anders Persson

Ob Fc R _

R 20 10 - 10 60

ObProb R _

100 10 0 80 8 2 60 6 4 40 4 6 20 2 8 0 0 50

Ob Fc R -

R 10 0

?? 20 20

- 0 50

Categorical Non-categorical

The value of uncertain weather forecasts

Probabilistic




gain

Never protect

Always protect

Deterministic forecast

Probabilistic forecasts

Ob%

R _

100 10 0

80 8 2

60 6 4

40 4 6

20 2 8

0 0 50


£30

£20

£10

£0£0 £30 £60 £90 protection cost

£30

£20

£10

0


2013 Anders Persson

The intuitive-statistical nature of routine forecasting

The forecasters work in an environment with a flow of information from different sources that might be incorrect, contradictory and unrepresentative



2. The need of good “statistical intuition” has been the subject of learned books



Not only meteorologists are concerned with risks and uncertainties

http://www.amazon.co.uk/gp/reader/0140167269/ref=sib_dp_pt/279-6152049-7318633



Time constrains, limited and sometimes misleading information, stress and outside distraction

(Almost) unlimited time, a wide range of reliable information

and full concentration

Fast thinking: Meteorologists in the forecast office

Slow thinking: Meteorologists attending a seminar

The title of Kahneman´s book refers to



3. Five points where we humans have to improve on how to deal with uncertainties


2013 Anders Persson19/04/23 15

Common human intuitive weaknesses

1. Over-confidence

2. Underestimation of randomness

3. Problems estimating uncertainty

4. Communicating this uncertainty

5. Drawing the conclusions from uncertainty



3.1 Overconfidence


2013 Anders Persson19/04/2319/04/23 17

3.1 Overconfidence: Before 2000 Concord was regarded as the safest airplane

Concorde Other company

__0___ 100 000Flight hours

< __1_____ 1 000 000Flight hours

__1___ 100 001Flight hours

> __1_____ 1 000 000Flight hours

. . .after the 2000 crash the most unsafe

accidents

accidents


2013 Anders Persson

⎟ ⎟ ⎟Mon 00 Mon 12 Tue 00 Tue 12 Wed 00 Wed 12

Model 1

Model 2

Model 3

Model 1

Model 2

Model 3

⎟ ⎟ ⎟

Mon 00 Mon 12 Tue 00 Tue 12 Wed 00 Wed 12

Over confidenceThree forecasts from different NWP models valid at the same time

in 5% of the cases

Surely dry!

in 80% of the cases

Surely rain!



3.2.Underestimating randomness


2013 Anders Persson19/04/2319/04/23 20

Conditional sampling

Comments from ECMWF Member States:

1. You overforecast Portuguese cut-offs at D+5, only 50% verify

2. You overforecast >25 mm/day events at D+3, only 50% verify

3. You overforecast gales at D+4, only 50% verify

OBFC

OB OB

FCFC

Many hitsUnder forecasting

Few missesOver forecasting

Well tuned forecasts



3.3 Estimating uncertainty (probabilities)


2013 Anders Persson


Forecast from model A

Forecast from model B

Forecast from model C

The Halo Effect

“…the atmosphere is inherently unpredictable due to the chaotic nature of its motions (Ørgård, 1963)…”

“…the atmosphere is inherently unpredictable due to the chaotic nature of its motions (Lorenz, 1963)…”

In forecasting meteorology: to weight one’s favourite NWP too much


2013 Anders Persson19/04/23Barcelona 5 Sep 2012

Anders Persson

15 UTC chart 15 UTC forecast

03 UTC chart

(≡) =

=15 UTC forecast

☼Δ

Δ

Δ

Δ+24 h forecast ☼☼

The availability effect

( )( )

TS-risk 70%

TS-risk 30%

+12 h forecast


2013 Anders Persson


Model 1

Model 2

Model 3

Model 1

Model 2

Model 3

⎟ ⎟ ⎟


The primacy effectTo order of arrival of the NWP may also affect the assessment

“Yes, rain is possible”

“No, I do not believe in rain”


2013 Anders Persson

in 63% of the cases


Forecast 1

Forecast 2

Forecast 3

Forecast 1

Forecast 2

Forecast 3

⎟ ⎟ ⎟


Misleading consistencyThree consecutive NWP from the same model valid at the same time

in 58% of the cases

Very “jumpy”

Rather consistent


2013 Anders Persson

Consecutive forecasts tend to be correlated since the new observations do not change the “first guess” entirely

The two best on average but also the most correlated ones i.e. their

mutual agreement is less significant

The best and the worst on average but also the least correlated ones. Their mutual agreement becomes more significant

+24h

+36h

+48h

+24h

+36h

+48h



3.4 Communicating uncertainty



People react differently to a statement like:

“-There is a 30% risk of rain”

compared to

”- A 70% chance of dry weather”

This is the “Framing effect”

3.4 Example of communicating probabilities



An example of a meteorological framing effect:

The authorities react more appropriately to a probability forecast of 60% for a whole region (Midlands) than to 10-20% for an individual location (Birmingham)

20%60%15%

10%

Thunderstorm risk


2013 Anders Persson19/04/2319/04/23 30

The Base rate effect

50% probability means different things Base rate

1. Tossing a coin: 50-50? = I do not know 50%

2. Snowfall in Barcelona: 50% very high risk! 2%

3. <4/8 clouds in Barcelona: 50% is a low “risk”! 80%

It all depends on the “base rate”


2013 Anders Persson

The base rate in meteorology is the climatology

The ECMWF:s Extreme Forecast Index (EFI) relates the probabilities to the climatology

19/04/23 31

ECMWF’s new EFI chart 16 August 2012 12 UTC +60 to +84 h


2013 Anders Persson

Excessive rain risk

Excessive hot



D+8 forecast 7 December




The predicted arrivals of the 15-16 December storm(ECMWF and UKMO alike)



The cyclone has changed track several times - we have revised our

calculations

No blame on the computer for the “jumpiness”


2013 Anders Persson19/04/2319/04/23 35

The way the Met Office and BBC forecasters handled the weather situation was “very well received by senior managers in the BBC and the Met Office….and had been praised by the section of government which is responsible for the Met Office. “

No direct surveys of public opinion were made, “but informal feedback has been positive.”

I’ll come back into more detail what the BBC/Met Office did



3.5 Drawing conclusions from probabilities

-What do you prefer?

-An 80% chance of winning £1000 or

-Get £700 directly?



Summary of part 1-3:

• The forecasters will increasingly deal with forecast uncertainty and risk assessments, which will increase the public’s confidence and improve the weather forecasters´ reputation

• A five-point program is suggested on how to change the current deterministic culture:

• The greatest “threat” to the meteorological weather forecaster is not the computer but the growing number of non-meteorological weather forecasters with a modern outlook

a) Reduce forecast over-confidenceb) Understand the effects of randomnessc) To estimate forecast uncertainty d) To convey probabilistic informatione) To help the customers to make decisions



4. Updating deterministic and ensemble forecasts


2013 Anders Persson

Meteorologists have five sources of information:

1. Observations

2. Deterministic NWP

3. Statistical interpretation

4. Ensemble forecasts

5. Climatological information

Let’s start with the last one, point 5.

(systematic errors and “jumpiness”)

(irregular, varying quality and representative)

(outdated or unrepresentative)

(a lot of information with “probs”)

(“only” background information)



4.1 The problem seen from a typical PDF (probability density function) perspective – climate distribution


2013 Anders Persson

Ψ2% 14% 34% 34% 14% 2%

Mean value+1 SD +2 SD-1 SD-2 SD

Most likely

valuesHigher than normal

Much higher than normal

Lower than normal

Much lower than

normal

A typical climatological distribution(temperature)


2013 Anders Persson

very tricky for bi-modal distributions

More tricky for rain or wind

Mode (the most likely value) Median (divides the data into two equal halves)

Mean (the average)

Mode

MedianMean

Mode

Median

Mean

What is one single “representative” value?


2013 Anders Persson

Ψ

Climatological average

Probability

The analysisobservations

The forecast


2013 Anders Persson

Ψ

Larger area = more certain

Smaller area = less certain

If the forecast is wrong it is more likely to be wrong “to the left” (less anomalous) than “to the right” (even more anomalous)

The “Regression to the Mean

Effect”

Probability


2013 Anders Persson

ΨClimatological average

More probable

Less probable

Probability


2013 Anders Persson

Ψ

Probability

How certain is this NWP?

We do not know!

It could be very certain

. . . or very uncertain

NWP


2013 Anders Persson

Ψ

Probability

Investigations show that “jumpiness” correlates badly to the accuracy of the last forecast

NWP today

NWP yesterdayNWP the day

before yesterday

We might an opinion by looking at the last NWPs from the same model, so called “lagged” forecasts


2013 Anders Persson

Ψ

Probability

The “jumpiness” correlates fairly well to the accuracy of the weighted average of the three NWP

NWP today

NWP yesterdayNWP the day

before yesterday

This “poor man’s ensemble captures the essentials of the “rich man’s” ensemble

1. Ensemble mean2. Spread3. Rough “probs”

Weighted average of the three

NWP


2013 Anders Persson

Ψ

Probability

Consensus NWP

There are no exact rules on how to merge manual and NWP information

Subjectively weighted average of the

manual and NWP

ManualFinal?


2013 Anders Persson

Ψ

Probability

Then arrives the EPS, the

Ensemble Prediction System forecast

FinalAgain, there are no exact rules on how

to merge manual and

NWP forecasts


2013 Anders Persson

Ψ

Probability

Again, there are no exact rules on how to merge manual and EPS information

EPS

Manual+NWP

New


2013 Anders Persson

Ψ

Probability

The change in the most representative value, from the manual+NWP to the new forecast, is not much affected

Manual+NWP

EPSNew

Minor change of deterministic value


2013 Anders Persson

Ψ

Probability

The major change is in the spread of the forecasts, the (un)certainty

Major change of probabilistic values

Increased risk for extremes



4.2 The problem seen from an EPS meteogram perspective



Medium range forecasting…with deterministic and EPS information

Latest three NWP

Latest EPS

Most common case with good agreement between EPS spread

and NWP “jumpiness”

Most common case



Rather poor agreement between larger EPS

spread and small NWP “jumpiness”.

The analysis system has obviously managed to

avoid possible problems because the NWP is not

very “jumpy”

Should the forecasters be more certain than the EPS indicates?

Rather common case



Rather poor agreement between small EPS

spread and large NWP “jumpiness”.

The perturbations have not been quite

able to cover the analysis uncertainties

Should the forecasters be more uncertain than the EPS indicates?

Not uncommon case



Poor agreement between the main

directions of the EPS and the NWP

This puts the forecasters in a very difficult situation and there is not enough experience or investigations about this situation

Rare case

Best choice: create a “super ensemble”



4.3 The same seen from a PDF-perspective


2013 Anders Persson

Ψ

Probability

Again, there are no exact rules on how to merge manual and EPS information

EPS

Manual+NWP

New


2013 Anders Persson

Ψ

Probability

EPS

NWP

Final

Case 1: Lagged NWP agree with EPS and about the same spread


2013 Anders Persson

Ψ

Probability

EPS

NWP

Final

Case 2: Lagged NWP agree with EPS but has smaller spread


2013 Anders Persson

Ψ

Probability

EPS

NWP

Final

Case 3: Lagged NWP agree with EPS but has larger spread


2013 Anders Persson

Ψ

Probability

EPS

NWP

Final

Case 4: Lagged NWP agree with EPS and about the same spread but quite different means


2013 Anders Persson

Summary of part 4

1. The forecaster has an increasing role to play as an “intuitive statistician”

2. The EPS must be compared and blended with more than one NWP, preferably 3-4 NWP

3. In the blending the spread and probabilities will normally be affected more than the “representative” value.

Ensemble information tend to make us more uncertain – is that good?



5. The value of uncertainty “per se”


2013 Anders Persson

Ob Fc R _

R 20 10 - 10 60

Ob Fc R -

R 10 0

?? 20 20

- 0 50

Categorical Non-categorical

The value of uncertain weather forecasts

Ob Fc R _

R 30 20 - 0 50

Ob Fc R _

R 10 0 - 20 70

Low protection cost

High protection cost




gain

Ob Fc R _

R 20 10- 10 60

Low protection

costHigh protection cost

Ob Fc R -

R 20 10

?? 20 20

- 10 60


£30

£20

£10

£0£0 £30 £60 £90 protection cost

£30

£20

£10

0

This is not just playing with mathematics – this

was the approach actually used by the Met

Office and BBC in December 2011



“Some terrible weather will come on Thursday-Friday”

The BBC forecasters avoided going into

detail and did not show any isobar

maps



On Wednesday 14 Dec still large uncertainty about the storm track


2013 Anders Persson

. . . and then finally the day before



The Met Office repeated the approach 1 ½ month later



The Met Office and the BBC didn’t hide, but made use of the uncertainty


2013 Anders Persson

My conclusions from the Met Office and BBC experience

1. Uncertainties can be communicated without numbers

2. The meteorologist must appear to be in control

3. Tell the background to the uncertainty – tell a “story”

4. Do not hesitate to give advice such as “if I were you…”

5. Follow up the forecast – but do not take to much credit


2013 Anders Persson19/04/2319/04/23 75

End


2013 Anders Persson

Observed rain 9-12 July 2004

50 mm

Prognosis

8 July

30 mm30 mmPrognosisPrognosis

7 July7 July

40 mmPrognosis

5 July

30 mmPrognosis

6 July

Expected rain for 9 July 2004

Example from Sweden

The meteorologists’ forecasts


2013 Anders Persson

Observed rain 9-12 July 2004

Somewhere 50 mm

Somewhere 30 mmSomewhere 30 mmSomewhere 30 mm

Somewhere 30 mm

Expected rain for 9 July 2004

The hydrologists’ forecasts


2013 Anders Persson

1 wmo swfdp macau 9 april 2013 anders persson decision making process and blending ensemble and...

Documents

wmo swfdp macau

anders persson decision

anders persson ob fc

cold slide

unrepresentative slide

deterministic forecasts

mild forecast

compromise forecast