model evaluation tools (met) - dtcenter.org · ... (software engineer) z [lacey holland...

Copyright 2008, University Corporation for Atmospheric Research, all rights reserved

Model Evaluation Tools (MET)

MET Development TeamDevelopmental Testbed Center

16 July 2008

1


MET Development Team

Dave Ahijevych (scientist)Barbara Brown (scientist/statistician)Randy Bullock (software engineer) Eric Gilleland (scientist/statistician)John Halley Gotway (software engineer)[Lacey Holland (scientist)]

With thanks to the Air Force Weather Agency (AFWA) and NOAA for their support

2


Outline

MET Overview – BarbPoint-stat and grid-stat – Barb Confidence intervals – Eric MODE – Randy VSDB and MODE analysis tools – Dave Technical details – John Point- and grid-stat practical – John MODE Tool Practical – John Questions…

3


MET: A community tool

Goal: to provide a set of forecast verification tools that are

Openly available to the community“Created” by the community, through contributed methods and capabilities

Evaluation methodsGraphical methods

Community includes diverse usersWRF DevelopersDevelopment Testbed Center (DTC)University researchersOperational centers

4


MET status

MET implementation initiated in fall 2006Version 1.0 released in January 2008Version 1.1 released July 11, 2008

ASCII observationsNeighborhood methodsNew confidence interval estimates for non-Gaussian measures

Version 2.0 in early 2009

5

http://www.dtcenter.org/met/users/

MET isA modular set of verification tools that can be freely downloadedFully documentedSupported through an e-mail help address

6


Main MET components

Data reformatting modulesStatistics modulesAnalysis modules

7


8


Input formatsGRIB model output (WRF post); GRIB gridded analysesASCII or PrepBufr point observations

Re-formattingPCP combine: Adds or subtracts accumulated precipitation from several GRIB files into a single NetCDF fileASCII2NC: Reformats ASCII point obs into the intermediate NetCDF format used by Point-StatPB2NC: Reformats PrepBufr obs into NetCDF format used by Point-Stat

9


Stats toolsMODE: Method for Object-based Diagnostic Evaluation

Object-based spatial verification approach

Grid-Stat: Compares gridded forecasts and observations

Now includes “neighborhood methods”

Point-Stat: Compares gridded forecasts and point observations (e.g., rawinsonde output)

Includes several methods for matching forecasts to point obs

StatisticsMODE:

Comparisons of object attributesGeometric (size, location, orientation, etc.)Intensity

Grid-Stat and Point-Stat: Traditional stats (POD, FAR; RMSE, MAE, Bias, etc.)Confidence intervals

ParametricBootstrap

10


Analysis tools:Aggregate and stratify MET output files according to user-specified criteriaReport aggregated statistics and distributions of statisticsAll output in ASCII format

11


Main MET components

Data reformatting modulesMove data into the format(s) expected by MODE (ascii2nc; pb2nc)Combine precipitation values across time periods (e.g., 24-h totals) (pcp_combine)Subtract precipitation values to create values for finer sub-periods (pcp_combine)

Statistics modulesObject-based spatial verification method (MODE)Verification of grids (Grid-stat)Verification at points (Point-stat)

Analysis modulesAggregate results across cases; stratify results by categoriesVSDB analysis tool (for Grid-stat and Point-stat)MODE analysis tool

12


Grid-stat and Point-stat scienceStats tools

MODE: Method for Object-based Diagnostic EvaluationGrid-Stat: Compares gridded forecasts and observations

Includes “neighborhood methods”

Point-Stat: Compares gridded forecasts and point observations (e.g., rawinsonde output)

Includes several methods for matching forecasts to point obs

StatisticsGrid-Stat and

Point-Stat: Traditional statistics

Contingency table statistics (POD, FAR, etc.)Continuous statistics (RMSE, MAE, Bias, etc.)

Confidence intervals

ParametricBootstrap

13


Point Stat: Grid-to-Point Verification

Input Grib forecasts and NetCDFobservations from PB2NCSelect multiple…

Variables, levels, thresholds, masking regions, matching methods, confidence interval (CI) method and alpha values for CIs

Output VSDB and ASCIIContingency table counts and statistics with CIsContinuous statistics with CIsPartial sums

14


Point-stat: Matching approaches

User-specified region of gridpoints around the observation point:

1 - nearest point2 - 2x2 box around point3 – 3x3 box around pointEtc.

Several metrics can be used to create matched forecast value

Min, Max, Median, Unweighted mean, Distance-weighted mean, Least-squares fit

User-defined min number of valid data points

15


Grid Stat: Grid-to-Grid verification

Output VSDB and ASCIIContingency table counts and statistics with confidence intervalsContinuous statistics with CIsPartial sums (L1L2, etc.)Neighborhood methods

Output NetCDFMatched pairs and difference fields for each variable, level, masking region

Input Grib or NetCDF from PCP CombineSelect multiple…

Variables, levels, thresholds, masking regions, smoothing methods, confidence interval (CI) methods and alpha values for CIs

16


Statistics for discrete variables

Measures for 2x2 contingency tablesEx: based on applying a

threshold to a continuous variable

Number of observationsFHO statisticsContingency table countsContingency table proportionsAccuracyBias

Probability of Detection of Yes (PODy)Probability of Detection of No (PODn)False Alarm Ratio (FAR)Critical Success Index (CSI)Gilbert Skill Score (GSS = ETS)Hanssen and KuipersDiscriminant (H-K = TSS)Heidke Skill Score (HSS)Odds Ratio (OR)

17


Statistics for continuous variables

Forecast/observation meanForecast/observation standard deviation Correlation coefficients (Pearson, Spearman, Kendall’s tau)Mean error (ME)Mean Absolute Error (MAE)Mean Squared Error (MSE)Bias-corrected Mean Squared Error (BCMSE); also known as “standard deviation of the error” (ESTDV)

Root-Mean Squared Error (RMSE)Error percentiles (10th, 25th, 50th, 75th, 90th)

Partial sums (1st and 2nd

moments of the forecasts, observations, and errors)

ScalarAnomalyVectorVector anomaly

18


Neighborhood verification methods

Also called “fuzzy” verificationUpscaling

Put observations and/or forecast on coarser gridCalculate traditional metrics

Provide information about scales where the forecasts have skill

19





Provide information about scales where the forecasts have skill

20





Provide information about scales where the forecasts have skillOutput

Standard contingency table statistics for different scales and thresholdsFractions skill score

21



From Mittermaier 2008

observed forecast

Ebert (2008; Met Applications) describes the neighborhood methods in MET

Example: Fractions skill score (Roberts and Lean, MWR, 2008)

22

Motivation/Background Parametric Intervals Bootstrap Future

Confidence intervals and MET

E Gilleland Confidence Intervals 1 / 9

c©University Corporation for Atmospheric Research. All rights reserved.

23


Background: What is a confidence interval?

Level α hypothesis test vs. (1− α) · 100% confidence intervalInterpretation: if experiment is run 100 times (i.e., 100 CIsestimated), then the true parameter should fall within (1− α) · 100of those limits. For example, if α = 0.05, then we expect, onaverage, that the true parameter would fall inside 95 of theintervals.



24


Background: Types of confidence intervals used in MET

ParametricNormal approximation

θ ± zα/2 · se(θ)

t-distribution (mean and small samples)

µ± tα/2,n−1 · se(θ)

Nonparametric: IID BootstrapPercentileAdjusted percentile (BCa)



25


Parametric confidence intervals: Normal Approximation

Assumption is that the sample is independent and identicallydistributed (iid) following, at least approximately, a normaldistribution. For some verification statistics (θ), the normalapproximation can then be used (i.e., θ ± zα/2 · se(θ)).

NotationForecast values are x1, . . . , xn and are iid normal with variance σ2

x

Observations are y1, . . . , yn and are iid normal with variance σ2y

Errors (F −O) are z1 = x1 − y1, . . . , zn = xn − yn and are iidnormal with variance σ2

z

H = hit rate, F = false alarm rate, nH = hits and misses, andnF = false alarms and correct negatives.a, b, c, and d are usual contingency table counts.



26


Parametric confidence intervals: Normal Approximation in MET

Verification statistics using the normal approximation in MET.The first two also have the t-distribution for small samples.

θ se(θ)

Forecast/Observation Mean (θ = 1n

n∑i=1

xi = x) σx/√

n

Mean Error (θ = x− y = z) σz/√

n

Peirce Skill Score (PSS)√

H(1−H)nH

+ F (1−F )nF

(logarithm of) odds ratio (OR)√

1a + 1

b + 1c + 1

d



27


Other parametric CIs used in MET

All of these still have an assumption of approximate normality for theunderlying sample, but the intervals derived are not in the same formas the normal approximation intervals.

Forecast/Observation VarianceForecast/Observation standard deviation(Linear) CorrelationHit Rate, False Alarm Rate (Wilson’s interval for proportions)



28


Bootstrap Confidence Intervals

Assume the sample is iid and representative of the populationResample with replacement from the sample B timesEstimate the statistic(s) for each replicate sample to get a sampleof the statistic(s).Calculate confidence intervals from this sample of the statistic(s)

PercentileBCa (adjusted percentile)



29


Verification statistics CIs relying on the bootstrap

All statistics in MET can obtain CIs through bootstrap resampling,and at least for the mean, bootstrap CIs are more accurate than thenormal approximation.



30


Summary and Possible Future Additions to MET

Parametric intervals for: Mean, ME, Variance/std. dev.,Correlation, PSS, log OR, hit/false alarm rates.

Strong assumptions about the samples (iid normal)Fast and easy to computeCan be computed knowing only the single realization of the pointestimate and its standard error

Nonparametric bootstrap intervals for all verification statisticsNo particular population distribution is assumedSample must be iidPotentially more accurate intervals than parametricRequires the entire original sample to computeSlower than parametric intervals

Uncertainty obtained in MET is sampling uncertainty only



31

xxxxxx

2008 University Corporation for Atmospheric Research All Rights Reserved

32

xxxxxx


33

xxxxxx


34

xxxxxx


35

xxxxxx


36

xxxxxx


37

xxxxxx


38

xxxxxx


39

xxxxxx


40

xxxxxx


41

xxxxxx


42

xxxxxx


43

xxxxxx


44

xxxxxx


45


Point Stat

MET Analysis tools and examples

MODE

ASCII

Grid Stat

PSASCII

NetCDF

ASCIINetCDF

GET STATS

OUTPUT

VSDBAnalysis

MODEAnalysis

ASCII

ASCII

AGGREGATE & STRATIFY

GraphicsPackage

GraphicsPackage

User-defined display

46


MODE Analysis - usage

Usage:mode_analysis -lookin path -summary | -bycase

[-column name] [-dump_row filename] [-out filename] [-help] [MODE file list] [-config config_file] | [MODE LINE OPTIONS]

example:mode_analysis -lookin ./mode_output/wrf2caps/24km -summary \

-column area -column centroid_lat -column axis_ang \-dump_row rowdump.txt \-single -simple \-obs_thr \>=3.000 -area_min 1000

MODEAnalysis

ASCIIMODE ASCII

MODE ASCIIMODE ASCII

MODE ASCIIMODE ASCII

MODE ASCII

9 cases x 3 models x 7 thresholds x 10 conv_radii

47


MODE Analysis - outputoutput of previous command:

Total mode lines read = 9,352Total mode lines kept = 26

Field N Min Max Mean StdDev

P10 P25 P50 P75 P90 Sum------------

--

-------

-------

-------

------

-------

-------

-------

-------

-------

--------area

26 1098.00 7200.00 2191.77 1357.42 1175.50 1247.00

1720.50 2796.25 3421.00 56986.00centroid_lat

26 31.28 46.48 39.08 3.54 34.38 36.27

39.69 41.26 42.58 1015.95axis_ang

26 -89.83 89.63 26.33 58.82 -67.19 -26.91 58.10 68.69 76.39 684.64

with slightly different options: -summary -column angle_diff

-column interest

-pair -simple -obs_thr

\>3.000 \-interest_min

0.9


Field N Min Max Mean StdDev

P10 P25 P50 P75 P90 Sum----------

-

----

-----

-----

------

----

----

-----

-----

-----

------angle_diff

9 0.53 59.08 21.89 17.76 2.87 8.29 18.42 25.62 45.71 197.04interest

9 0.91 1.00 0.96 0.03 0.93 0.94 0.96 0.99 1.00 8.65

with -bycase

option:


Fcst

Valid Time Area Matched Area Unmatched # Fcst

Matched # Fcst

Unmatched # Obs

Matched # Obs

Unmatched----------------------

------------

--------------

--------------

----------------

-------------

---------------Apr 26, 2005 00:00:00 3576 1257 7 10 2 4May 13, 2005 00:00:00 12546 1592 3 5 2 5...

48


MODE Analysis – by radius,threshold

convolution radius

prec

ip th

resh

old

1-h precipitation forecast, 24-h lead time, valid 1 June 2005

49


MODE Analysis – quilt plotPercent observed objects matched - Jun 1, 2005

convolution radius [km]0 20 40 60 80 100 120

prec

ip th

resh

old

[0.0

1”]

50


MODE object displacement analysis

9 cases from 200548 km convolution radius, 3 mm rain threshold

displacement of composite forecast objects from matched observed objects [degrees]

wrf2caps model wrf4ncar model

mean displacement

51


VSDB Analysis

VSDBanalysis

ASCIIGrid Stat ASCII genericgraphicspackage

Gilbert skill score for 7 geometric test cases with 4 rain thresholds

Pearson correlation coefficient for 9 cases from 2005,3 different models with 95% bootstrap confidence intervals

Grid Stat ASCIIGrid Stat ASCII

Grid Stat ASCII

52


Technical InformationMET distributed as a tarball to be downloaded and compiled locally.

METv1.1 to be released in July, 2008.Updated User’s Guide and Online Tutorial.Register and download: www.dtcenter.org/met/users

Language:Written primarily in C++ with calls to a Fortran library

Supported Platforms and Compilers:Linux machines with GNU compilers

g++ and gfortran or g77Linux machines with Portland Group (PGI) compilers

pgCC and pgf77IBM machines with IBM compilers

xlC and xlf

53


Dependencies

Required to compile:GNU Make UtilityC++ and Fortran compilers (GNU, PGI, or IBM) NetCDF LibraryBUFRLIB LibraryGNU Scientific Library (GSL)F2C Library (f2c or g2c)

Recommended for use:WRF Post-ProcessorCOPYGB (included with WRF-Post) CWORDSH

54


Building METSteps for building MET:1. Build the required libraries with the same family

of compilers to be used with MET.2. Select the appropriate Makefile.

GNU, PGI, or IBM3. Configure the Makefile.

C++ and Fortran compilersPaths for NetCDF, BUFRLIB, GSL, and F2C libraries

4. Execute the GNU Make utility.5. Run the test script and check for runtime errors.

Runs each of the MET tools at least once.Uses sample data distributed with the tarball.

55


METv1.1 Flowchart

GriddedNetCDF

Point-Stat

GriddedGrib Input:

ObservationAnalyses

ModelForecasts

MODE

PrepBufrPoint Obs PB2NC

PCPCombine

VSDB

Grid-Stat

NetCDFPointObs

PSASCII

NetCDF

ASCIIVSDB

NetCDF

INPUT RFMT INTERMED STATS OUTPUT

= optional

VSDBAnalysis

MODEAnalysis

ASCII

ASCII

AGGREGATE

ASCIIPoint Obs ASCII2NC

56


Configuration FilesMET is a set of command line tools which are controlled using ASCII configuration files passed to the tools on the command line.Configuration files control things such as:

Fields/levels to be verified.Thresholds to be applied.Interpolation methods to be used.Verification methods to be applied.Regions over which to accumulate statistics.

Well commented and documented in User’s Guide.Easy to modify.Use the version of the configuration files distributed with the tarball.

57


Use of Configuration Files

GriddedNetCDF

Point-Stat

GriddedGrib Input:

ObservationAnalyses

ModelForecasts

MODE


PCPCombine

VSDB

Grid-Stat

NetCDFPointObs

PSASCII

NetCDF

ASCIIVSDB

NetCDF


= optional

VSDBAnalysis

MODEAnalysis

ASCII

ASCII

AGGREGATE


58


PB2NC (18)MODE (65)Grid-Stat (21)Point-Stat (20)VSDB-Analysis (23)MODE-Analysis (89)Only need to modify a few!

Sample File 59


Data Reformatting Tools

GriddedNetCDF

Point-Stat

GriddedGrib Input:

ObservationAnalyses

ModelForecasts

MODE


PCPCombine

VSDB

Grid-Stat

NetCDFPointObs

PSASCII

NetCDF

ASCIIVSDB

NetCDF


= optional

VSDBAnalysis

MODEAnalysis

ASCII

ASCII

AGGREGATE


60


PCP-Combine Tool

Functionality:Sum precipitation across multiple files.New for v1.1:

Add precipitation in 2 files (i.e. NMM output).Subtract precipitation in 2 files (i.e. ARW output).

Specify field name on the command line.No configuration file.

Data formats:Reads GRIB.Writes gridded NetCDF as input to stats tools.

61


PB2NC Tool

Functionality:Filter and reformat PREPBUFR point observations into intermediate NetCDF format.Configuration file specifies:

Observation types, variables, locations, elevations, quality marks, and times to retain or derive for use in Point-Stat.

Data formats:Reads PREPBUFR using NCEP’s BUFRLIB.Writes point NetCDF as input to Point-Stat.

CWORDSH utility for FORTRAN blocking

62


ASCII2NC Tool (new in v1.1)

Functionality:Reformat ASCII point observations into intermediate NetCDF format.For v1.1, one input ASCII format supported (10 columns):

Message_Type, Station_ID, Valid_TimeLat(Deg North), Lon(Deg East), Elevation(msl)Grib_Code, Level, Height(msl), Observation_Value

No configuration file.Data formats:

Reads ASCII.Writes point NetCDF as input to Point-Stat.Support additional ASCII formats based on user input.

63


Practical: Grid-Stat Tool

Functionality:Computes a variety of statistics for comparing a gridded forecast to a gridded observation.One time step at a time.Data must reside on a common grid.

Recommend COPYGB for regridding GRIB data.Configuration file specifies:

Fields/levels, thresholds, vx regions (grids or polylines)Smoothing options, neighborhood sizesVx methods, confidence interval options, output types

Data formats:Reads gridded GRIB and NetCDF output of PCP-Combine.Writes ASCII statistics and NetCDF matched pairs.

64


Practical: Grid-Stat Usage

Command Line:Required: fcst_file, obs_file, config_fileOptional: -outdir, -v

65


Practical: Grid-Stat Line Types

Statistics line types: 11 possibleCategorical - apply threshold

Contingency table counts (FHO, CTC, CTP, CFP, COP)Contingency table statistics (CTS)

Continuous - raw fieldsContinuous statistics (CNT)Partial Sums (SL1L2)

Neighborhood – choose size (new for v1.1)Neighborhood categorical (NBRCTC, NBRCTS)Neighborhood continuous (NBRCNT)

Ten header columns common to all line types.Data in remaining columns specific to each line type.

66


Practical: Grid-Stat Output

Output files:ASCII statistics file containing all line types (File ends with “.vsdb”).Optional ASCII files sorted by line type with a header row (File ends with “_TYPE.txt”).Optional NetCDF matched pairs and difference fields. (File ends with “_pairs.nc”).

Naming conventions:grid_stat_HHMMSSL_YYYYMMDD_HHMMSSV[.vsdb | _pairs.nc | _TYPE.txt]Ex: grid_stat_120000L_20050807_120000V.vsdb

67


Practical: Point-Stat Tool

Functionality:Computes a variety of statistics for comparing a gridded forecast to point observations.One time step at a time.Configuration file specifies:

Fields/levels, thresholds, vx regions (grids, polys, stations)Vx message types, interpolation methodsVx methods, confidence interval options, output types

Data formats:Reads gridded GRIB and NetCDF output of PCP-Combine.Reads point NetCDF output of ASCII2NC and PB2NC.Writes ASCII statistics.

68


Practical: Point-Stat Usage

Command Line:Required: fcst_file, obs_file, config_fileOptional: -climo, -ncfile, -outdir, -v

69


Practical: Point-Stat Line Types

Statistics line types: 12 possibleCategorical - apply threshold

Contingency table counts (FHO, CTC, CTP, CFP, COP)Contingency table statistics (CTS)

Continuous - raw fieldsContinuous statistics (CNT)Partial Sums (SL1L2, SAL1L2, VL1L2, VAL1L2)

Matched Pairs (new for v1.1)Raw matched pairs – a lot of data! (MPR)

Ten header columns common to all line types.Data in remaining columns specific to each line type.

70


Practical: Point-Stat Output

Output files:ASCII statistics file containing all line types (File ends with “.vsdb”).Optional ASCII files sorted by line type with a header row (File ends with “_TYPE.txt”).

Naming conventions:point_stat_HHMMSSL_YYYYMMDD_HHMMSSV[.vsdb | _TYPE.txt]Ex: point_stat_120000L_20050807_120000V.vsdb

71


Practical: MODE Tool

Functionality:Identifies objects in two fields, computes single object attributes and pair-wise differences, matches/merges objects, writes object attributes and differences.One time step at a time.Data must reside on a common grid.

Recommend COPYGB for regridding GRIB data.Configuration file specifies:

Forecast and observation specified separatelyField/level, raw filter value, mask region (grid, polyline)Object definition parameters (convolution radius and threshold)Object filtering parameters (area, intensity)

Flags for matching/merging logicFuzzy engine weights, interest functions, and confidence mapsTotal interest thresholdPostScript plotting options

Data formatsReads gridded GRIB and NetCDF output of PCP-Combine.Writes ASCII statistics, NetCDF object fields, PostScript summary plot.

72


Practical: MODE Usage

Command LineRequired: fcst_file, obs_file, config_fileOptional: -config_merge, -outdir, -vDisable output: -plot, -obj_plot, -obj_stat, ct_stat

73


Practical: MODE Output

Output files:ASCII object statistics file (File ends with “_obj.txt”).ASCII Contingency table statistics file (File ends with “_cts.txt”).NetCDF object file (File ends with “_obj.nc”).PostScript summary plot (File ends with “.ps”).

Naming conventions:mode_FFIELD_FLVL_vs_OFIELD_OLVL_HHMMSSL_YYYYMMDD_HHMMSSV_HHMMSSA [.obj.txt | _cts.txt | _obj.nc | .ps]Ex: mode_APCP_12_SFC_vs_APCP_12_SFC_120000L_20050807_120000V_120000A_obj.txt

74


Practical: MODE Object Stats

Four object statistics line types (contents of OBJECT_IDcolumn):

Simple forecast and observation objects (FNNN and ONNN)Pairs of simple objects (FNNN_ONNN)Composite forecast and observation objects (CFNNN and CONNN)Pairs of composite object (CFNNN_CONNN)

Same number of columns for each line type (50 in total):18 header columns.20 columns applicable to SINGLE simple and composite line types.12 columns applicable to PAIRS of simple or composite line types.Columns which do not apply to a given line type contain fill data (-9999)

May be disabled using the –obj_stat command line argument.

75


Practical: MODE CTStats

Contains traditional contingency table counts and corresponding statistics computed in three ways:

Scoring the RAW fields by applying the convolution thresholds.Scoring the FILTERed fields by first applying any raw filters and then applying the convolution thresholds.Scoring point-wise using the resolved OBJECT fields.

Meant simply as a point of reference for the MODE method.Differs from the VSDB CTS line type.

Does not include confidence intervals.May be disabled using the –ct_stat command line argument.

76


Practical: MODE NetCDF

NetCDF output file contains 4 fields:Indices for the simple forecast objects.Indices for the simple observation objects.Indices for the composite forecast objects.Indices for the composite observation objects.

May be disabled using the –obj_plotcommand line argument.

77


Practical: MODE PostScript

MET does not generally provide plotting tools.Exception for MODE to illustrate the method.Configuration file plotting options:

Specify colortables to be used for plotting raw and object fields.61 colortables provided in data/colortables.Specify how to rescale an existing colortable.Or explicitly define you own.

Option for how colorbar is plotted.Draw lines as great circle arcs or straight lines in the grid.Zoom plot up to only the valid region of data.

Number of pages of PostScript output based on configuration fileselections:

At least 4 depicting object definition and matching.Additional pages for:

Merging using the double-threshold technique.Merging using the fuzzy engine technique.

78


MODE PS Pages 1 and 2 79


MODE PS Pages 3 and 4 80

model evaluation tools (met) - dtcenter.org · ... (software engineer) z [lacey holland...

Documents