verification methods for high resolution model forecasts
DESCRIPTION
Verification Methods for High Resolution Model Forecasts. Barbara Brown ([email protected]) NCAR, Boulder , Colorado Collaborators : Randy Bullock, John Halley Gotway, Chris Davis, David Ahijevych, Eric Gilleland, Beth Ebert Hurricane Diagnostics and Verification Workshop May 2009. Goals. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/1.jpg)
Verification Methods for High Resolution Model Forecasts
Barbara Brown ([email protected])NCAR, Boulder, Colorado
Collaborators: Randy Bullock, John Halley Gotway, Chris Davis, David Ahijevych, Eric Gilleland, Beth Ebert
Hurricane Diagnostics and Verification WorkshopMay 2009
![Page 2: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/2.jpg)
2
GoalsDescribe new approaches for evaluation
of high-resolution spatial (gridded) forecasts • Potentially useful for evaluating hurricane
rainfall (and wind) patterns
The aim of these methods is to • Provide more meaningful information about
performance than traditional approaches• Overcome some of the insensitivity of
traditional approaches
![Page 3: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/3.jpg)
3
Relevance for tropical cyclones Meaningful evaluation of precipitation evolution
for land-falling hurricanes• Short-term predictions (e.g., extreme events;
precipitation distribution; flooding)• Total storm precipitation predictions
Meaningful comparisons of high- and low-resolution models• Low resolution tends to “win” using traditional
measures even if pattern is not as good… Evaluation of oceanic precipitation using satellite Evaluation of wind pattern forecasts Other??
![Page 4: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/4.jpg)
4
Hurricane Rita precipitation
![Page 5: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/5.jpg)
5
Hurricane precipitation verificationSwath-based precip
verification approach (Marchok et al. 2007)
QPF pattern matching Mean, volume, and
distribution of rain values
Production of extreme amounts
![Page 6: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/6.jpg)
6
Hurricane Rita precipitation9/20 9/21
9/22
9/239/24
9/259/26
![Page 7: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/7.jpg)
7
Consider gridded forecasts and
observations of precipitation
Traditional approach
Which is better?
OBS1
2 3
45
![Page 8: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/8.jpg)
8
Traditional approach
OBS1
2 3
45
Scores for Examples 1-4:Correlation Coefficient = -0.02Probability of Detection = 0.00
False Alarm Ratio = 1.00Hanssen-Kuipers = -0.03
Gilbert Skill Score (ETS) = -0.01
Scores for Example 5:Correlation Coefficient = 0.2
Probability of Detection = 0.88False Alarm Ratio = 0.89Hanssen-Kuipers = 0.69
Gilbert Skill Score (ETS) = 0.08
Forecast 5 is “Best”
![Page 9: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/9.jpg)
9
Traditional approach
OBS1
2 3
45
Some problems with the traditional approach:(1) Non-diagnostic – doesn’t tell us what was wrong with the forecast – or what was right
(2) Utra-sensitive to small errors in simulation of localized phenomena
![Page 10: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/10.jpg)
10
Spatial verification techniques aim to:
Account for• Uncertainties in
timing and location• Spatial structure
Provide information that is
• Able to characterize error in physical terms
• Diagnostic• Meaningful to
forecast users
Weather variables (e.g., precipitation, wind fields)
defined over spatial domains have coherent structure and features
Spatial forecasts
![Page 11: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/11.jpg)
11
Spatial verification approachesFiltering1. Neighborhood2. Scale
separation
Displacement3. Feature-based4. Field
deformation
![Page 12: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/12.jpg)
12
Field deformation approaches• Measure distortion and
displacement (phase error) for whole field How should the forecast be adjusted to
make the best match with the observed field?
Scale decomposition methods• Measure scale-dependent error
Neighborhood verification methods• Give credit to "close" forecasts
Object- and feature-based methods• Evaluate attributes of
identifiable features
New spatial verification approaches
Keil and Craig, 2008
![Page 13: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/13.jpg)
13
Intensity-scale methodCasati et al. (2004)
Evaluate forecast skill as a function of the precipitation intensity and the spatial scale of the error
![Page 14: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/14.jpg)
14
Scale wavelet decomposition of binary error
Scale l=8 (640 km)
Scale l=1 (5 km)
mean (1280 km)
Scale l=6 (160 km)
Scale l=7 (320 km)
Scale l=5 (80 km) Scale l=4 (40 km)
Scale l=3 (20 km) Scale l=2 (10 km)
1
0
-1
L
lluu EE
1,
L
lluu MSEMSE
1,
From Ebert 2008
![Page 15: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/15.jpg)
15
MSE skill score LMSE
MSEMSEMSEMSE
SS lu
randomlubestlu
randomlululu /12
1 ,
,,,,
,,,,
Sample climatology(base rate)
From Ebert 2008
![Page 16: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/16.jpg)
16
Neighborhood verification Also called “fuzzy”
verification Upscaling
• Put observations and/or forecast on coarser grid
• Calculate traditional metrics
Provide information about scales where the forecasts have skill
![Page 17: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/17.jpg)
17
Neighborhood methods
From Mittermaier 2008
Fractional Skill Score (FSS)
![Page 18: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/18.jpg)
18
Feature-based verification Composite approach
(Nachamkin) Contiguous rain area
approach (CRA; Ebert and McBride, 2000; Gallus and others)
Error components• displacement• volume• pattern
![Page 19: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/19.jpg)
19
Spatial verification method: MODE MODE: Method for
Object-based Diagnostic Evaluation
Goals• Mimic how a human
would identify storms and evaluate forecasts
• Measure forecast “attributes” that are of interest to users
Steps• Identify objects• Measure attributes• Match forecast
attributes• Measure differences in
attributes
Example: Precipitation; 1 Jun 2005
Forecast Observed
User inputs: Object identification; attribute selection and weighting
![Page 20: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/20.jpg)
20
Object-based example: 1 June 2005 MODE quantitative
results indicate• Most forecast areas too
large• Forecast areas slightly
displaced• Median and extreme
intensities too large• BUT – overall – forecast
is pretty good
In contrast:• POD = 0.40• FAR = 0.56• CSI = 0.27
1
23
Forecast precipitation objects with Diagnosed objects overlaid
![Page 21: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/21.jpg)
21
Back to the original example… MODE “Interest”
measures overall ability of forecasts to match obs
Interest values provide more intuitive estimates of performance than the traditional measure (ETS)
But – even for spatial methods, Single measures don’t tell the whole story!
![Page 22: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/22.jpg)
22
Spatial Method Intercomparison Project (ICP)
Goal: Compare the various approaches using the same datasets (real, geometric, known errors)
Includes all of the methods described here; international participants
Collection of papers in preparation (Weather and Forecasting)
http://www.rap.ucar.edu/projects/icp/index.html
![Page 23: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/23.jpg)
23
What do the new methods measure?
Attribute Traditional Feature-based
Neighbor-hood Scale
Field Defor-mation
Perf at different scales
Indirectly Indirectly Yes Yes No
Location errors No Yes Indirectly Indirectly Yes
Intensity errors Yes Yes Yes Yes Yes
Structure errors No Yes No No Yes
Hits, etc. Yes Yes Yes Indirectly Yes
![Page 24: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/24.jpg)
24
Applicability to ensemble forecasts
From C. Davis
![Page 25: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/25.jpg)
25
Statistical inference
Confidence intervals are required to provide• Meaningful evaluations
of individual model performance
• Meaningful comparisons of model performance
Threshold
24-h QPF, 24-h lead
![Page 26: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/26.jpg)
26
Method availability Many methods available
as part of the Model Evaluation Tools (MET)• MODE• Neighborhood• Intensity-scale
MET is freely available• Strong user support
Software for some others is available on the intercomparison website or from original developers http://www.dtcenter.org/met/users/
![Page 27: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/27.jpg)
27
Conclusion New spatial methods provide great
opportunities for more meaningful evaluation of precipitation forecasts – and other forecast fields• Feed back into forecast development• Provide information to users
Each method is useful for particular types of situations and for answering particular types of questions
![Page 28: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/28.jpg)
28
Topics for discussion Consider how new methods may be
beneficial (and adaptable) for high-res NWP for hurricanes• Can these methods help?• How do they need to be
adapted/altered?• Would they be useful for other fields
(e.g., winds)?• Are other kinds of new methods
needed? Use of aircraft observations –
incomplete grids (reflectivity) Methods for evaluation of genesis? A
global problem… Need to consider false alarms as well as misses
Use of confidence intervals
From Bender et al. 2007
![Page 29: Verification Methods for High Resolution Model Forecasts](https://reader036.vdocuments.us/reader036/viewer/2022062410/568162af550346895dd33701/html5/thumbnails/29.jpg)
29