understanding and using uncertainty information in weather forecasting susan joslyn university of...
Post on 20-Dec-2015
216 views
TRANSCRIPT
Understanding and Using Uncertainty Information in Weather
Forecasting
Susan Joslyn
University of Washington
Acknowledgements
Earl Hunt
David Jones
Limor Nadav-Greenberg
John Pyles
Adrian Raftery
Karla Schweitzer
McLean Slaughter
Meng Taing
Jeff ThomassonThis research was supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745
Forecast Uncertainty
• Available for some time
• Rarely communicated in public forecasts
• Underused by weather forecasters
Forecast Uncertainty
• Difficult to understand- Forecasters claim
• People make mistakes when reasoning with probability
• Format: Frequency (1 time in 10) is better than
Probability (10% chance)
Forecast Uncertainty
• Useful for deterministic forecasts decision?
Theoretically
Practically useful?
• It doesn’t matter how good the information if people can’t or won’t make use of it.
Goals for Psychology Team
• Establish uncertainty information is useful Threshold forecast (forecasters & general public)- high wind advisory for boater safety
• What is best presentation format to enhanceUnderstanding?Decisions?
Three Major Lines of Inquiry
1. Does probability information improve threshold forecast? Study 1
2. Does display format (visualization) matter?
Study 2
3. Does the wording matter? Studies 3-4
(probability/ frequency)
Study1 Does Probability Information Improve
Threshold Forecast?
Participants: Advanced atmospheric science students
Task: • Forecast wind speed and direction
• Decide whether to issue high wind advisory
(winds > 20 knots)
Within Subject Design
Historical data Radar Imagery
Satellite Imagery
TAFs and current METARs
Model output
(AVN, MM5 & NGM)
Historical data Radar Imagery
Satellite Imagery
TAFs and current METARs
Model output
(AVN, MM5 & NGM)
+
Chart showing probability
of winds > 20 k
Condition 1 Condition 2
Within Subjects Design
Historical data Radar Imagery
Satellite Imagery
TAFs and current
METARsModel output (AVN, MM5 & NGM)
Historical data Radar Imagery
Satellite Imagery
TAFs and current
METARsModel output (AVN, MM5 & NGM)
+Chart showing probability
of winds > 20 k
Condition 1 Condition 2
• Same participants, same weather• Only difference is probability product
Results
Threshold Forecast: • People posted fewer wind advisories with
probability product.
• Similar ability to discriminate between high wind and low wind event (sensitivity).
Results: Percent Advisories
Y= % times
forecasters posted advisory
X= probability
of winds
> 20K
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0-10 10-30 30-50 50-70 70-90 90-100
Probability (winds > 20 k) range given by Model
Percent Advisories
With Probability ProductWithout Probability Product
Conclusion: Uncertainty Information IS Beneficial for Threshold
• Increased advisories when high winds were very likely
• Decreased advisories when high winds were unlikely-fewer false alarms
• Increase trust in warnings!
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0-10 10-30 30-50 50-70 70-90 90-100
Probability (winds > 20 k) range given by Model
Percent Advisories
With Probability ProductWithout Probability Product
Study 2 Does Display Format Matter?
• 3 different visualizations of 90% predictive interval
• Range of likely wind speeds
• All conditions included median wind speed chart
deterministic forecast
3 Visualizations: Between subjects
1. 90% Upper bound: warmer colors = higher wind speed
• “ observed wind speeds will be higher only 1 time in 10”• worse case scenario: highest likely winds
3 Visualizations
1. 90% Upper bound:• wind speeds will be higher only 1 time in 10warmer colors = higher wind speed
2. Margin of error:
• range of wind speeds between UB & median • display of uncertainty in the forecast
warmer colors = more uncertainty
3 Visualizations
1. 90% Upper bound:wind speeds will be higher only 1 time in 10warmer colors = higher wind speed
2. Margin of error:
range of wind speeds between upper bound and median warmer colors = more uncertainty
3. Box plot:
median 90% Upper bound90% lower boundWind Speed in knots
Wind speed in knots
Method
• Participants: Atmospheric Science students (replicated on NOAA Forecasters)
• Practice: Learned how to read charts
• Test: - Forecast wind speeds - Threshold: high wind advisory (winds >20 knots)
- Rate uncertainty in forecast
Results: Wind Speed Forecast
• UB forecast significantly higher wind speeds• Display provided a high anchor (Tversky & Kahneman, 1982)
0 0.5 1 1.5 2 2.5
Box Plot
Upper bound
Margin of Error
Knots above the Median
1.55
2.02
1.17
Results: High Wind Advisories
Likelihoodof high winds
Box PlotUpperBound
Margin of Error
HIGH Median > 20K 98.44% 94.45% 91.67%
MEDIUM Median 15-20K 32.40% 31.24% 27.95%
LOW Median <15 K
3.57% 3.97% 2.38%
People in the box plot condition:• posted significantly more advisories• most in high likelihood situations
Results: Uncertainty Rating
Box plot.81
Upper Bound .89
Margin of Error .97
• MoE best for detecting relative uncertainty• They learned: “The wider the range the greater the uncertainty”
• Ratings in the MoE significantly more highly correlated to range
correlation
Conclusion: Format Matters
Box Plot better threshold forecast wind speed: no bias (salient high and low anchors)
MoE detect relative uncertainty in forecast
Upper higher winds speeds: bias (anchor)
Bound no benefit to threshold forecast
Study 3 & 4 Does Wording Matter?
• Participants:
Psychology undergraduates
• Frequency is easier to understand than probability (Gigerenzer, 1995, 1999, 2000)
– Research on complex problems – Is that true of simple expressions of uncertainty?
Method
Procedure: Fill out questionnaire rating expressions of uncertaintyDecide whether or not to post a high wind advisory
Suppose that there is a 10% chance that the wind speeds will be greater than 20 knots.
“How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble)” Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No
Method
Procedure: Fill out questionnaire rating expressions of uncertaintyDecide weather to post a wind advisory
Suppose that there is a 10% chance that the wind speeds will be greater than 20 k.
“How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble)” Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No
Method
Procedure: Filled out questionnaire rating expressions of uncertaintyDecide weather to post a wind advisory
Suppose that 1 time in 10 the wind speeds will be greater than 20 knots.
“How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble)”Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No
Study 3
2 Variables: Wording & Likelihood
Probability Frequency
10% chance = 1 time in 10
90% chance = 9 times in 10
Study 3: Likelihood of High Wind Held Constant
1 time in 10 wind speeds = 9 times in 10 wind speeds
will be greater than 20 knots will be less than 20 knots
Results: Reversal Error
• Rate from wrong side of scale
Suppose that there is a 90% chance that the wind speeds will be less than 20 knots.
“How likely are the wind speeds to be less than 20 knots? (please fill in a bubble)”
O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O <---very unlikely very likely ------>
• They completely misunderstand the phrase
• Most in “90% (9 in 10) less than” wording
• Which is it? High likelihood? Less than?
Reversal error
Added 2 levels of likelihood
Less Greater10% chance less 10 % chance greater1 in 10 less 1 in 10 greater30% chance less 30% chance greater 3 in 10 less 3 in 10 greater 70% chance less 70% chance greater 7 in 10 less 7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater
Equivalent Expressions
Less Wording Greater Wording10% chance less 10 % chance greater1 in 10 less 1 in 10 greater30% chance less 30% chance greater 3 in 10 less 3 in 10 greater 70% chance less 70% chance greater 7 in 10 less 7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater
Equivalent Expressions
Less Wording Greater Wording10% chance less 10 % chance greater1 in 10 less 1 in 10 greater30% chance less 30% chance greater 3 in 10 less 3 in 10 greater 70% chance less 70% chance greater 7 in 10 less 7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater
Results: Reversal Error
More often in “less than” wording (4x as likely)
Mean reversal error
per person
Less than .41
Greater than .10
High vs. low likelihood does not matter
Frequency wording does not help
Results: Wind Advisories
Percent of Advisories Posted
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4
Likelihood of winds exceeding 20 k
Percent of Advisories Posted
freq greater
10% 30% 70% 90%
Percent of Advisories Posted
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4
Likelihood of winds exceeding 20 k
Percent of advisories.
freq greater
freq less
Results: Wind Advisories
10% 30% 70% 90%
Percent of Advisories Posted
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4
Likelihood of winds exceeding 20 k
Percent of advisories.
freq greater
freq less
prob greater
Results: Wind Advisories
10% 30% 70% 90%
Results: Probability “less” is worst
10% 30% 70% 90%
Reversal error subjects eliminated from this analysis
Percent of Advisories Posted
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4
Likelihood of winds exceeding 20 k
Percent of advisories.
freq greater
freq less
prob greater
prob less
10% 30% 70% 90%
Conclusion: Wording Matters
• “Less than” wording is difficult (reversal errors)
• Wind speed advisories in “probability less” - too many advisories in low ranges
- too few in high ranges
• Frequency protects against posting errors generated by “less than” wording
Conclusions
• Probability information improves threshold forecasts– Many end-user weather decisions are yes/no threshold decisions
• The right display format – Improves understanding
• MoE communicates relative uncertainty
– Improves weather decisions• Box Plot increases warnings in high likelihood• Box Plot unbiased wind speed forecast
• Wording matters– “Less than” is confusing– Frequency helps sometimes
• NOT in reversal errors• HELPS in posting advisories
Results: Percent Advisories
Y= % times
forecasters posted advisory
X= probability
of winds
> 20K
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0-10 10-30 30-50 50-70 70-90 90-100
Probability (winds > 20 k) range given by Model
Percent Advisories
With Probability ProductWithout Probability Product
Results: Percent Advisories
• Y= % times
forecasters posted advisory
• X= probability
of winds
> 20K
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0-10 10-30 30-50 50-70 70-90 90-100
Probability (winds > 20 k) range given by Model
Percent Advisories
With Probability ProductWithout Probability Product
Results: Percent Advisories
• Y= % times
forecasters posted advisory
• X= probability
of winds
> 20K
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0-10 10-30 30-50 50-70 70-90 90-100
Probability (winds > 20 k) range given by Model
Percent Advisories
With Probability ProductWithout Probability ProductExpected Response
Results: Percent Advisories
• Y= % times
forecasters posted advisory
• X= probability
of winds
> 20K
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0-10 10-30 30-50 50-70 70-90 90-100
Probability (winds > 20 k) range given by Model
Percent Advisories
% Observed Winds
> 20 k
With Probability ProductWithout Probability Product% times observed winds> 20ktsExpected Response
Study 1: Rating
• 10% was rated significantly higher
Probability condition:
10% chance (M=1.32) 90% chance (M=.99)
O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O
Frequency condition:
1 in ten (M=1.06) 9 out of 10 (M=.98)
O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O
Study 2: Rating
10 was rated higher--did not reach significance
10% (1 in 10) greater (M=1.25) 90% (9 in 10)less (M=.97)
O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O
10% (1 in 10) less (M=.98) 90% (9 in 10)greater (M=.88)
O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O
Study 1: Reversal Error
Mean reversal error per person
90% (9 times)
less than.83
10% (10 times) greater than
.33
User Needs & Understanding
• Naval Forecasters • Terminal Aerodrome Forecast (TAF) posted at regular intervals while fulfilling other duties
Numerical Model: MM5
Satellite
Synoptic Pattern Comparison
1. Compare position of low in the model & satellite
2. Assess differences in movement and position
3. Adjust forecast accordingly
Compare Predicted to Observed Values
1. Access NOGAPS predicted pressure for current time 29.69
2. Access current local pressure and 29.69
subtract from NOGAPS - 29.64
.05
3. Access NOGAPS predicted pressure for 29.59
forecast period and subtract error amount - .05
4. Forecast 29.54
Results
• Naval forecasters rely heavily on models (1/3-1/2 source statements referred to models)
• Statements implying understanding of model uncertainty
Model biases and strengths
Initialization of model run Strategies for determining uncertainty
Evaluation of degree of uncertaintyAdjusting model predictions
Conclusions
• Uncertainty?• Error in deterministic forecast?• Subsequent questionnaire study: confidence
is related to their assessment of model performance
Probability ProblemThe probability that a woman getting
a mammogram has breast cancer is 1%. If the woman has breast cancer the probability is 80% that she will have a positive mammogram.
If the woman does not have breast cancer the probability that she will still have a positive mammogram is 10%.
You have a patient that has a positive mammogram (no symptoms)--what is the probability she has breast cancer.
Frequency ProblemTen out of every 1,000 women
have breast cancerOf those 10 women with breast
cancer 8 will have a positive mammogram
Of the remaining 990 women without breast cancer, 95 will still have a positive monogram
You have a sample of women who have positive mammograms in your screening (no symptoms)
How many of these women will actually have breast cancer?