nws verification team meeting 05/05/08 the ensemble ... · 2 1. introduction to evs software •...
TRANSCRIPT
![Page 1: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/1.jpg)
1
James [email protected]
The Ensemble Verification System (EVS): an introduction
NWS Verification Team meeting 05/05/08
![Page 2: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/2.jpg)
2
1. Introduction to EVS software• Mechanics of EVS (structure, I/O etc.)• Brief lecture followed by demo.
2. Overview of metrics in EVS• Which metrics are available in EVS?• What can they tell us (focus on exercises)?
3. Brief introduction to exercises
Goals for today
![Page 3: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/3.jpg)
3
1a. Overview of EVS
![Page 4: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/4.jpg)
4
Diagnostic verification• Problem-focused: what/where errors & why?• Distinguished from real-time verification
Diagnostic questions include….• Are ensembles reliable?• Prob[flood]=0.9: does it occur 9/10 times?• Operational forc. vs. hindcasts (e.g. MODS)• What are the major sources of uncertainty?
Scope of EVS
![Page 5: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/5.jpg)
5
Verification of continuous time-series• Temperature, precipitation, streamflow etc.• > 1 forecast point, but not spatial products
Forecast products at different scales• Any lead time (e.g. 1 day – 2 years or longer)• Any forecast resolution (e.g. hourly, daily)• Temporal aggregation (e.g. hourly to daily)• Aggregation across forecast points
Design goals of EVS
![Page 6: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/6.jpg)
6
Flexibility to target data of interest• Two target variables: 1) forecast; 2) observed• Two conditions: 1) time; 2) variable value • e.g. observed winter flows > flood stage • e.g. ensemble mean temperature < freezing
Carefully selected metrics• From very detailed to highly summarized• Documented and explained
Design goals of EVS
![Page 7: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/7.jpg)
7
Example of workflow
How biased are my winter flows > flood
level at dam A?
![Page 8: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/8.jpg)
8
Data I/O and archiving
Files:• CS binary (flow forecast)• OHD Datacard (temp.
and precip. forecast)• Observed (Datacard)
File:• XML
File:• XML
Files• Graphical (jpeg/png)• Numerical (xml)
![Page 9: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/9.jpg)
9
1b. Demonstration of EVS
![Page 10: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/10.jpg)
10
2. Verification metrics
![Page 11: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/11.jpg)
11
Many ways to classify metrics 1. Tests for single-valued property (e.g. mean)2. Tests of broader forecast distribution• Both may involve reference forecasts (“skill”)
Caveats in testing probabilities• Observed probabilities require many events• Big assumption 1: we can ‘pool’ events• Big assumption 2: observations are ‘good’
Metrics for probabilities
![Page 12: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/12.jpg)
12
Discrete/categorical forecasts• Many metrics rely on discrete forecasts• e.g. will it rain? {yes/no} (rain > 0.01)• e.g. will it flood? {yes/no} (stage > flood level)
What about continuous forecasts?• An infinite number of events• Arbitrary event thresholds (i.e. ‘bins’)?• Typically, yes (and choice will affect results)
Continuous prob. forecasts
![Page 13: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/13.jpg)
13
Observation-centered metrics (discrim.)• “What do forecasts do when observed do X”? • i.e. “binning” in terms of observed• e.g. Relative Operating Characteristic
Forecast-centered metrics (reliability)• “What do observed do when forecasts do Y”? • i.e. “binning” in terms of forecasts• e.g. Reliability Diagram
Metrics vary by design
![Page 14: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/14.jpg)
14
Detail varies with verification question • e.g. inspection of ‘blown’ forecasts (detailed) • e.g. avg. reliability of flood forecast (< detail)• e.g. rapid screening of forecasts (<< detail)
Metrics vary in detail
![Page 15: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/15.jpg)
15
Greatest + ve
90 percent.80 percent.
50 percent.
20 percent.10 percent.
‘Errors’ for 1 forecast
Greatest - ve Observation
Ense
mbl
e fo
reca
st e
rror
s (le
ad h
our 6
)
Most detailed (box plot)
0 2 4 6 8 10 12 14 16 18 20
Time (days after first forecast)
![Page 16: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/16.jpg)
16
Greatest + ve
90 percent.80 percent.
50 percent.
20 percent.10 percent.
‘Errors’ for 1 forecast
Greatest - ve
Observation
Ense
mbl
e fo
reca
st e
rror
s (le
ad h
our 6
)
Observed value (increasing size)
Most detailed (box plot)
![Page 17: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/17.jpg)
17
Pro
babi
lity
that
X fa
lls in
win
dow
60% of time, observation should fall in window ±30%
“Underspread”
“Hit rate” = 90%GFS-EPP precipitation ensembles (w/o zero observed)
Cumulative Talagrand
Error window (percentile around median)
60%
![Page 18: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/18.jpg)
18
ROC at Flood Action StagePr
obab
ility
of D
etec
tion
[TP/
(TP+
FN)]
Probability of False Detection [FP/(FP+TN)]
0.00 1.0
1.0
Climatology
F TP FP
!F FN TN
O !O
Each point represents a prob.threshold at which forecast says event will occur
e.g. Prob(Y>AS) = 0.6
Perfect
![Page 19: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/19.jpg)
19
Least detailed (a score)
0 5 10 15 20 25 30
Riv
er s
tage
Time (days)
2.0
1.6
1.2
0.8
0.4
0.0
Flood stage Forecast Observation
Brier score = 1/5 x {(0.8-1.0)2 + (0.1-1.0)2 +(0.0-0.0)2 + (0.95-1.0)2 + (1.0-1.0)2}=0.8528
4
![Page 20: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/20.jpg)
20
Least detailed (a score)
0.0 0.1 0.2 0.3 0.4 0.5 0.6
Cum
ulat
ive
prob
abili
ty
Precipitation amount (inches)
1.0
0.8
0.6
0.4
0.2
0.0
Forecast (F)
Observed (O)
CRPS = (F-O)2
• Then average acrossmultiple forecasts
• Small scores = better
![Page 21: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/21.jpg)
21
3. Exercises
![Page 22: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/22.jpg)
22
See EVS User’s Manual (pp. 6-8)• Will run under any OS (tested for Lx/Win.)• Software provided in folder• Recommend JRE version 1.6.0 (1.5.0_12 min.)
Installation
Executable
![Page 23: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/23.jpg)
23
All data/instructions by COB 9th May• Word document containing exercises• Folder containing data for each exercise• Folder containing software
Data/instructions
![Page 24: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/24.jpg)
24
Three exercises (increasingly complex)• First two exercises deal with synthetic data…• ….linear regression model for temperature• Exercise 1: forecasts unbiased• Exercise 2: forecasts biased in mean/spread• Exercise 3: deals with real flow (MARFC)• ‘Real’ biases are less easy to detect!• Need to create plots and analyze them
Exercises
![Page 25: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/25.jpg)
25
Go through EVS results• What did you learn?• What did you find difficult?• What were the main problems with EVS?• What were the main conceptual problems?
Use list server for data/software issues!!• We will respond to technical/software issues• Conceptual issues addressed in next meeting
Next meeting (06/12)
![Page 26: NWS Verification Team meeting 05/05/08 The Ensemble ... · 2 1. Introduction to EVS software • Mechanics of EVS (structure, I/O etc.) • Brief lecture followed by demo. 2. Overview](https://reader034.vdocuments.us/reader034/viewer/2022042104/5e8203c514649257ed3493fe/html5/thumbnails/26.jpg)
26
Discuss the COMET training module• Available in early June• …..E-mail from Matt Kelsch• Feedback from the team• What aspects were easy/difficult?
Verif-hydro list server for questionsEmail: [email protected]: http://infolist.nws.noaa.gov/read/login
Next meeting (06/12)