training course 2009 – nwp-pr: how to communicate uncertainties 1/33 how to communicate...
TRANSCRIPT
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 1/33
How to Communicate Uncertainties
Renate Hagedorn European Centre for Medium-Range Weather Forecasts
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 2/33
Motivation
• Main reasons for not using (probabilistic) predictions in decision-making processes include:
forecasts are not “accurate” enough
fluctuation of successive forecasts
competing or conflicting forecast information
history of previous forecasts not available
procedures for acquiring and integrating forecasts into decision-making processes have not been defined
external constraints forbid flexible response to forecast info
local information may be more important
value of forecast has not been demonstrated
All forecast system or impact system related impediments
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 3/33
Motivation
• Additionally, “non-rational” thinking or cognitive illusions affect the optimal use of (probabilistic) forecasts
Capability of human mind for solving complex problems is limited compared with the size of problems
Lack of objectively rational behaviour in real world
Use of simple “rules of thumb” to simplify decision making
Heuristics are often helpful, but can lead to biases, especially in uncertain situations where probabilities are encountered
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 4/33
Main messages
• “Nothing is certain”
• In many situations, decisions have to be based on probabilities
• Interpretation of probabilities is sometimes not straightforward
• Appropriate presentation can help to make the right decisions
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 5/33
The illusion of certainty…
…or how we construct a single certainty from uncertain cues
Do these two table surfaces have the same area and shape?
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 6/33
Understanding uncertainties in the real world
• Examples of well-known sources of cognitive bias
formulating the problem: - probabilities vs. frequencies - the framing effect - the anchoring effect
underweighting base rates
hindsight and confirmation bias
belief persistence: Primacy and inertia effect
group conformity and decision regret
• A practical test… (the “Monty Hall” Problem)
• Strategies to reduce impact of cognitive illusions
• Examples of communication/visualization of probabilities
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 7/33
Conditional probabilities
• Breast cancer screening: The facts: - Probability that a woman aged 40-50 has breast cancer = 0.8% - If a woman has breast cancer, probability of positive test = 90% - If a woman does not have breast cancer, prob. of positive test=7%
• Imagine a woman with a positive test. What is the probability, that she actually has breast cancer?
• Solution (with Bayes Theorem): - p(disease) = 0.008 - p(pos|disease) = 0.90 - p(pos| no disease) = 0.07 p(disease) * p(pos|disease) - p(disease|pos) = --------------------------------------------------- p(disease) * p(pos|disease) + p(no disease) * p(pos| no disease)
0.09
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 8/33
Frequency formulation
• Breast cancer screening: The facts: - Probability that a woman aged 40-50 has breast cancer = 0.8% - If a woman has breast cancer, probability of positive test = 90% - If a woman does not have breast cancer, prob. of positive test=7%
• Solution: 1000 women
8: disease 992: no disease
7: positive 1: negative 69: positive 923: negative
p(disease | pos) = 7 / (7+69) 0.09
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 9/33
Probabilities vs. frequencies
Estimated chances of breast cancer given a positive screening mammogram (from Gigerenzer, 2002)
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 10/33
The framing effect
• The way a problem (or forecast) is formulated can affect a decision
• Imagine that London faces an unusual disease that is expected to kill 600 people. Two alternative programs to combat disease:
- Program A: 200 people will be saved
- Program B: 1/3 probability 600 saved, 2/3 probability nobody saved
Tests indicate that 72% would select program A (risk-averse)
• Slightly changed wording:
- Program C: 400 people will die
- Program D: 1/3 prob. that nobody will die, 2/3 prob. that 600 will die
Tests indicate that 78% would select program D (risk-taking)
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 11/33
The framing effect in real life
• Professionals, experienced in decision-making, are still affected
• E.g., information for doctors:
- mortality rate of 7% within 5 years -> hesitant to recommend
- survival rate after 5 years of 93% -> more inclined to recommend
• For weather predictions this suggests different response to forecasts expressed as likelihood of drought or non-likelihood of wet conditions
• E.g., different response to: 30% chance of drought and 70% chance of normal or wet conditions
• Worded vs. numerical forecast: - 11% judge forecast “rain is likely” as poor if it did not rain - 37% judge forecast “70% chance of rain” as poor if it did not rain although they associate the word “likely” with probability of 70%
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 12/33
Test your knowledge of history
• What are the last three digits of your phone number?
Range of initial anchor Average estimate
400 – 599 629
600 – 799 680
800 – 999 789
1000 – 1199 885
1200 – 1399 988
• The correct answer is: A.D. 451
• In what year would you guess Attila the Hun was defeated?
• Do you think Attila the Hun was defeated in Europe before or after that year?
• Add 400 to this number
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 13/33
Underweighting base rates
• Imagine a climate model (with 90% accuracy) predicts drought
• Historically, there is 10% chance of drought
• What is the chance that drought will occur in next season?
• Solution: 100 seasons
10: drought 90: no drought
9: drought FC 1: no-drought FC 81: no-drought FC 9: drought FC
p(drought | drought FC) = 9 / (9+9) = 0.50
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 14/33
Underweighting base rates
Challenge to convince user that
Model was correct 90% of time
the probability of a drought next season was only 50%
Remember: only for equally likely events,
accuracy translates into probabilities
• Imagine a climate model (with 90% accuracy) predicts drought
• Historically, there is 10% chance of drought
• What is the chance that drought will occur in next season?
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 15/33
Underweighting base rates
• Imagine a climate model (with 90% accuracy) predicts warmer than normal conditions
• There is a 50% chance of above normal
• What is the chance that warmer than normal conditions will occur?
• Solution: 100 seasons
50: warmer 50: colder
45: warm FC 5: cold FC 45: cold FC 5: warm FC
p(warmer | warm FC) = 45 / (45+5) = 0.90
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 16/33
Hindsight and confirmation bias
Men mark where they hit, and not where they miss. (Jevons, 1958)
• After finding out whether or not an event occurred, individuals tend to overestimate the degree to which they would have predicted the correct outcome
• Reported outcomes seem seem less surprising in hindsight than in foresight
• Example: El Nino 1997 regarded as “stunning success”, although only one model was reported in the March 1997 NOAA Long-Lead Forecast Bulletin predicting more than slight warming. Some of the very poor forecasts simply ignored in hindsight.
• Considerable evidence that people tend to ignore (and not search for) disconfirming information of any hypothesis
• Introduce “double-blind test” for model assessment?
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 17/33
Belief persistence
• Primacy and inertia also tend to weight evidence inaccurately.
• People tend to weight more heavily evidence presented first, e.g. persons described as: - intelligent, industrious, impulsive, critical, stubborn, envious are more favourable perceived than persons described as - envious, stubborn, critical, impulsive, industrious, intelligent
• Inertia may lead people to ignore evidence that contradicts their prior belief (e.g. that a particular forecast system produces useful forecasts)
• Forecast producers may not recognise the disparity of model predictions, and instead rely too heavily on a forecast that supports their intuitive understanding of the current state of climate
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 18/33
Group conformity
• The “Asch” test: Is the test line equal to line A, B, or C?
Test Line A B C
individual test 1 person in
front: A
2 persons in
front: A
3 persons in
front: A
monetary
reward
error rate 1% 2% 13% 33% 47%
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 19/33
Probabilities in Gambling
• Monty Hall: Let’s Make a Deal - in one of the boxes is a bottle of wine - choose 1, 2, or 3
- after choosing, one of the empty boxes will be opened, so that only one empty and one full box are left - you can choose again (stay with first choice or switch) - what is the best strategy?
1 2 3
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 20/33
Probabilities in Gambling
• Monty Hall: Let’s Make a Deal - in one of the boxes is a bottle of wine - choose 1, 2, or 3 stay switch
1 2 3
1 2 3
1 2 3
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 21/33
Strategies to reduce CI influence
• Recognition that decision-making is inherently biased
• Understanding how written forecasts, and numerical probability forecasts are interpreted by potential users
• Try to reduce impact of cognitive illusions by
encouraging forecaster groups to de-bias forecasts by e.g. reducing overconfidence or hindsight bias
taking care that media reports and forecasts do not cause anchoring to extreme events (e.g. El Nino 82/83)
taking care in wording forecasts to avoid framing
avoid “intuitive” approach when combining forecasts, objective approaches exist and are more successful
ensuring that base-rates are not ignored
using additional visual aids to convey real levels of skill
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 22/33
Transformation of probabilities to words
Terminology Likelihood of theoccurrence
Virtually certain Greater than 99%Probability
Very likely Greater than 90%Probability
Likely Greater than 66%probability
About as likely as not 33% to 66% probability
Unlikely Less than 33% probability
Very unlikely Less than 10% probability
Exceptionallyunlikely
Less than 1% probability
Table 1: IPCC Likelihood Scale
Terminology Likelihood of theoccurrence
Extremely likely Greater than 99%Probability
Very likely 90%-99% probability
Likely 70%-89% probability
Probably – more likely than not
55%-69% probability
Equally likely as not 45%-54% probability
Possible – less likely than not
30%-44% probability
Unlikely 10%-29% probability
Very unlikely 1%-9% probability
Extremely unlikely Less than 1% probability
Table 2: Forecast Likelihood Scale
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 23/33
Use of colour
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 24/33
Visualization of Timeseries
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 25/33
Probability Maps (medium range)
30°N
40°N
50°N
60°N
60°W
60°W 40°W
40°W 20°W
20°W 0°
0° 20°E
20°E 40°E
40°E 60°E
60°E
5
5
5
5
5 5
5
5
5
5
55
5
5
35
35
35
35
35
35
35
35
35
35
35
35
35
35
35
65
65
65
65
65
65
65
65
65
65
65
65
65
65
65
95
95
95
95
95
95
95
95
95 95
95
100100
5
5
5
5
5 5
5
5
5
5
55
5
5
35
35
35
35
35
35
35
35
35
35
35
35
35
35
35
65
65
65
65
65
65
65
65
65
65
65
65
65
65
65
95
95
95
95
95
95
95
95
95 95
95
Surface: Total precipitation of at least 1 mmSunday 13 April 2008 00UTC ©ECMWF Forecast probability t+036-060 VT: Monday 14 April 2008 12UTC - Tuesday 15 April 2008 12UTC
30
40
50
60
60
60 40
40 20
20 0
0 20
20 40
40 60
60
5
35
65
95
100
30°N
40°N
50°N
60°N
60°W
60°W 40°W
40°W 20°W
20°W 0°
0° 20°E
20°E 40°E
40°E 60°E
60°E
5
5
5
5
5
5
5 5
5
5
5
5
5
5
5
5
5
5
35
35
35
35
35
35
35
35
35
3535
35
35
35
65
65
65
65
65
65
65
65
95
95
95
5
5
5
5
5
5
5 5
5
5
5
5
5
5
5
5
5
5
35
35
35
35
35
35
35
35
35
3535
35
35
35
65
65
65
65
65
65
65
65
95
95
95
Surface: Total precipitation of at least 5 mmSunday 13 April 2008 00UTC ©ECMWF Forecast probability t+036-060 VT: Monday 14 April 2008 12UTC - Tuesday 15 April 2008 12UTC
30
40
50
60
60
60 40
40 20
20 0
0 20
20 40
40 60
60
5
35
65
95
100
30°N
40°N
50°N
60°N
60°W
60°W 40°W
40°W 20°W
20°W 0°
0° 20°E
20°E 40°E
40°E 60°E
60°E
5
5
5
5
5
5
5
5
5
5
5
5
5
35
35
35
35
3535
35
35
65
65
5
5
5
5
5
5
5
5
5
5
5
5
5
35
35
35
35
3535
35
35
65
65
Surface: Total precipitation of at least 10 mmSunday 13 April 2008 00UTC ©ECMWF Forecast probability t+036-060 VT: Monday 14 April 2008 12UTC - Tuesday 15 April 2008 12UTC
30
40
50
60
60
60 40
40 20
20 0
0 20
20 40
40 60
60
5
35
65
95
100
30°N
40°N
50°N
60°N
60°W
60°W 40°W
40°W 20°W
20°W 0°
0° 20°E
20°E 40°E
40°E 60°E
60°E
5
5
5
5
5
35
5
5
5
5
5
35
Surface: Total precipitation of at least 20 mmSunday 13 April 2008 00UTC ©ECMWF Forecast probability t+036-060 VT: Monday 14 April 2008 12UTC - Tuesday 15 April 2008 12UTC
30
40
50
60
60
60 40
40 20
20 0
0 20
20 40
40 60
60
5
35
65
95
100
RR>1mm RR>5mm
RR>10mm RR>20mm
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 26/33
Summary of probability of 4 events
Courtesy:Gjermund Haugen, Magnus Ovhed, met.no
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 27/33
Ensemble size = 41, climate size = 275Forecast start reference is 01/03/09Prob(most likely category of 2m temperature)ECMWF Seasonal Forecast
No significance test appliedJJA 2009System 3
75°S 75°S
60°S60°S
45°S 45°S
30°S30°S
15°S 15°S
0°0°
15°N 15°N
30°N30°N
45°N 45°N
60°N60°N
75°N 75°N
150°W
150°W 120°W
120°W 90°W
90°W 60°W
60°W 30°W
30°W 0°
0° 30°E
30°E 60°E
60°E 90°E
90°E 120°E
120°E 150°E
150°E
Forecast issue date: 15/03/2009
<---- below lower tercile above upper tercile ---->70..100% 60..70% 50..60% 40..50% other 40..50% 50..60% 60..70% 70..100%
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 28/33
Ensemble size = 41, climate size = 275Forecast start reference is 01/03/08Prob(most likely category of 2m temperature)ECMWF Seasonal Forecast
No significance test appliedJJA 2008System 3
75°S 75°S
60°S60°S
45°S 45°S
30°S30°S
15°S 15°S
0°0°
15°N 15°N
30°N30°N
45°N 45°N
60°N60°N
75°N 75°N
150°W
150°W 120°W
120°W 90°W
90°W 60°W
60°W 30°W
30°W 0°
0° 30°E
30°E 60°E
60°E 90°E
90°E 120°E
120°E 150°E
150°E
Forecast issue date: 15/03/2008
<---- below lower tercile above upper tercile ---->70..100% 60..70% 50..60% 40..50% other 40..50% 50..60% 60..70% 70..100%
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 29/33
Ensemble size = 41, climate size = 275Forecast start reference is 01/03/09Prob(most likely category of precipitation)ECMWF Seasonal Forecast
No significance test appliedJJA 2009System 3
75°S 75°S
60°S60°S
45°S 45°S
30°S30°S
15°S 15°S
0°0°
15°N 15°N
30°N30°N
45°N 45°N
60°N60°N
75°N 75°N
150°W
150°W 120°W
120°W 90°W
90°W 60°W
60°W 30°W
30°W 0°
0° 30°E
30°E 60°E
60°E 90°E
90°E 120°E
120°E 150°E
150°E
Forecast issue date: 15/03/2009
<---- below lower tercile above upper tercile ---->70..100% 60..70% 50..60% 40..50% other 40..50% 50..60% 60..70% 70..100%
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 30/33
Unified Prediction System
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 31/33
EPS in the Media
German TV
Dutch TV
high
normal
low
Predictability
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 32/33
Summary
…in this world there is nothing certain but death and taxes.(Benjamin Franklin)
•“Nothing is certain”
…the theory of probabilities is at bottom only common sense reduced to calculus.
(Pierre-Simon, Marquis de Laplace)
• In many situations, decisions have to be based on probabilities
…math is hard, let’s go shopping.(Barbie)
• Interpretation of probabilities is sometimes not straightforward
…solving a problem simply means representing it so asto make the solution transparent.
(Herbert A. Simon)
• Appropriate presentation can help to make the right decisions
Training Course 2009 – NWP-PR: How to Communicate Uncertainties 33/33
Further Reading:
• Nicholls, Neville, 1999: Cognitive illusions, heuristics, and climate predictions. BAMS, 80, 1385 - 1397
• Gigerenzer, Gerd et al., 1989: The empire of chance: How probability changed science and everyday life. Cambridge University Press, pp. 340.
• Gigerenzer, Gerd, Peter M. Todd, and the ABC research group, 1999: Simple heuristics that make us smart. Oxford University Press, pp. 416
• Gigerenzer, Gerd, 2002: Reckoning with risk. The Penguin Press, pp. 310
• WMO, 2007: Guidelines on communicating forecast uncertainty. WMO/TD No.1422 (WMO website)
• http://www.cut-the-knot.org/probability.shtml