main plans (or projects)

Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007

COSMO strategy for Verification

Adriano RaspantiCOSMO WG5 Coordinator – “Verification and Case studies”

Head of Verification Section at Italian Met Service ([email protected])

with contributions by WG4 (Interpretation-PP) and WG5 people


MAIN PLANS (or projects)

• Advanced interpretation and verification of very

high resolution models (project by Pierre Eckert)

• Conditional Verification-VerSUS project

• COSI “The global Score” (COSMO Index)

COSMO strategy for Verification


Advanced interpretation and verification of very high resolution models

BackgroundBackground

The increase in resolution of the models will lead to a “proliferation” of grid points and also to an increase of noise in the forecasts.

The effects of the so-called “double penalty” also will increase for events not predicted exactly at the right place at the right time.

Ways to extract the most valuable information out of high density fields have to be found.

The connection with various fuzzy verification methods will be explored in this project.



MAIN Goal of the projectMAIN Goal of the project

Data with a very high spatial (and temporal) variability like precipitation have to be treated with special care in order to avoid the double penalty syndrome.

Following methods have been identified in a first stage: Fuzzy verification, Contiguous Rain Area (CRA), Neighborhood methods, Fraction skill score, Intensity scale technique and similar

When the aggregation region is small, the scores are usually poor, but with an increasing averaging area the scores become very good

The goal is to find the smallest area in which the benefit of running a very high resolution model is present. This will be called the reliable scalereliable scale.

Not only the verification will be carried out at this “optimal” scale, but the products for forecasters and customers should also be designed at this scale (or scales).



Other aspects of the projectOther aspects of the project

1. Application of “boosting” method for the detection of “special" weather parameters

• This method finds optimal choices for predictors which are proposed by the meteorologists. Good results with weather parameters not directly included in the model like fog or visibility are expected.

2. Use of very high resolution precipitation as input of the hydrological models. Studies and verification on the impact of this coupling


Which rain forecast would you rather use?

Mesoscale model (5 km) 21 Mar 2004

Sydney

Global model (100 km) 21 Mar 2004

Sydney

Observed 24h rain

RMS=13.0 RMS=4.6


Some early results

Picture From B. Ebert


A Fuzzy Verification Toolbox

Fuzzy method Decision model for useful forecast

Upscaling (Zepeda-Arce et al. 2000; Weygandt et

al. 2004)Resembles obs when averaged to coarser scales

Anywhere in window (Damrath 2004), 50%

coveragePredicts event over minimum fraction of region

Fuzzy logic (Damrath 2004), Joint probability

(Ebert 2002)More correct than incorrect

Multi-event contingency table (Atger 2001) Predicts at least one event close to observed event

Intensity-scale (Casati et al. 2004) Lower error than random arrangement of obs

Fractions skill score (Roberts and Lean 2005) Similar frequency of forecast and observed events

Practically perfect hindcast (Brooks et al. 1998)Resembles forecast based on perfect knowledge of

observations

Pragmatic (Theis et al. 2005) Can distinguish events and non-events

CSRR (Germann and Zawadzki 2004) High probability of matching observed value

Area-related RMSE (Rezacova et al. 2005) Similar intensity distribution as observed


Some early results


Perturbati

on

Type of forecast error Algorithm

PERFECTNo error – perfect

forecast!-

XSHIFT Horizontal translationHorizontal translation

(10 grid points)

BROWNIAN No small scale skill

Random exchange of

neighboring points

(Brownian motion)

LS_NOISEWrong large scale

forcing

Multiplication with a

disturbance factor

generated by large scale 2d

Gaussian kernels.

SMOOTHHigh horizontal diffusion

(or coarse scale model)

Moving window arithmetic

average

DRIZZLEOverestimation of low

intensity precipitation

Moving Window filter setting

each point below average

point to the mean value


Some early results


Effect of „Leaking“ Scores

observation forecast

Problem: Some methods assume no skill at scales below window size!

pobs=0.5 pforecast=0.5

Assuming random ordering within window

yes no

yes 0.25 0.25

no 0.25 0.25

An example: Joint probability method

ForecastO

BS Not perfect!


Some early results


Summary

-0,1

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0

0,1

0,2

Up-

scaling

Any-

where in

Window

50%

cover-

age

Fuzzy

Logic

Joint

Prob.

Multi

event

cont. tab.

Intensity

Scale

Fraction

Skill

Score

Prag-

matic

Appr.

Practic.

Perf.

Hindcast

CSSR

Area

related

RMSE

Leaking ScoresXSHIFT

BROWNIAN SMOOTH

LS_NOISE DRIZZLE„Sensitivity Score“

STD

good

good

• Leaking scores show an overall poor performance

• “Intensity scale” and “Practically Perfect Hindcast” perform in general well, but …

• Many score have problem to detect large scale noise (LS_NOISE); “Upscaling” and “50% coverage” are beneficial in this respect

• Leaking scores show an overall poor performance

• “Intensity scale” and “Practically Perfect Hindcast” perform in general well, but …

• Many score have problem to detect large scale noise (LS_NOISE); “Upscaling” and “50% coverage” are beneficial in this respect


Some early results


CV Project - VerSUS - Verification System Unified Survey

MAIN Goal of the Versus projectMAIN Goal of the Versus project

Development of a common and unified verification “package” including a Conditional Verification tool.

METHODMETHOD

The typical approach to CV could consist of the selection of one or several forecast products and one or several mask variables or conditions, which would be used to define thresholds for the product verification (e.g. verification of T2M only for grid points with zero cloud cover in model and observations). After the selection of the desired conditions, a classical verification tool for statistical indexes can be used.

The more flexible way to perform a selection of forecasts and observations is to use an “ad hoc database”, planned and designed for this purpose, where the mask or filter could be simply or complex SQL statements.



Main DB Modules

RDBMS features :• OBS e FCS data

• Data configuration to perform verification

• Verification results, Scorse and images

“daemon” process (Loader) to load data from different sources (e.g. MARS, districo DB, File system): BUFR format for obs and GRIB format for fcs

processes performing verifications through specific requests (Integration with “R” statistic package) and storing of resulting data

WEB GUI (server-client architecture)


OBS dataConfiguration data

for verification

Verification results(Scores and images)

Verification R

Web GUI

FCS dataLoader

MARS Districo DB

Usermanagement

Versus-DB

VerSUS - Architectural Design


StationForecast User/FE

Observation

Index



VERSUS DB has the following main areas:

• Users managing area

• Front-End area for Front-End setting up. Two main FE: the

loader FE for data ingestion, and scores FE for the execution

of verification indexes by means of “R” package library.

• Meteorological data area, for handling of observations

(surface and upper air) and forecasts data and their lookup

tables.

• Score criteria area that manages the definition of scores and

their applications.

• Output area that stores the scores and graphical output.



Main lookup tables:•Station: the list of punctual meteorological station that provides surface or upper air

observation data to VERSUS system. The attributes are name, nationality, latitude, longitude,

height, the WMO and/or ICAO code (if they exist) of the station. Moreover there is an unique

identifier of the station that VERSUS DB automatically assigns when a new station is defined

by means of Graphic User Interface (GUI)

•Obs_type: the list of observation types (templates) such as synop, temp, any other

observation data coded in BUFR format. That table is modified by means of a GUI

•Obs_parameter: the list of BUFR parameter codes, the meaning and input measurement.

This table is automatically updated whenever a new occurrence of BUFR parameter code

comes to the system.

•Model: the list of meteorological models verified VERSUS

•Grid: the list of grids that are defined in the section 3 of the grib.

•Fcs_parameter: data defined in the section 1 of the grib.

The lookup tables are managed by GUI or loader FE of the system, automatically, The lookup tables are managed by GUI or loader FE of the system, automatically,

whenever a new instance of them occurs.whenever a new instance of them occurs.



The selection criteria of the forecast and observation data is setting up

by

means of a GUI. The information that must be define are:

•Stratification (lat/lon, WMO name, morphological,….)

•The list of R-verification indexes to apply

•The observed parameter and its condition/filter, if any

•The forecast parameter (model, grid, parameter) and its condition/filter, if any

•The method of getting forecast data, such as nearest point, mean on a given

radius,…

•The start date and end date of the data or the frequency (monthly, weekly,

seasonal)

•Steps

•Pressure Levels (for upper air)


• Continuous parameters: Reduction of variance

RV = 1 – (RMSE prog / RMSE ref)2

where ref = persistence

• Categorical parameters: ETS

ETS = (R – „chance“) / (T –“chance“)R= number of obs events correctly forecastT = number of events which were either observed or forecasted

global score S like

COSMO-index COSI = S/S0 x100

Where S0 is the value of S the first year of computation

iii

ii

SSww

1S

COSI “The global Score” (COSMO Index)


Parameters

• total cloud amount [threshold: 0-2, 3-6, 7-8

• temperature [t2m, later: tmin, tmax]

• 10m- windvector

• precipitation [thresholds: 0.2, 2, 10 mm/6h]



Verification frequency

• All 3h− T2m, 10m-wind and cloudiness:

• @ 00, 03,…, 18, 21 UTC later on: tmin & tmx over 12h

• 6h-sums: precipitation



Which models ? Aggregation ?

• Start with COSMO-7

• But programming also for COSMO-2

• Temperature and windspeed: 1 gridpoint

• Precipitation: mean in a radius of 15km

• Cloudiness: mean in a radius of 30 km




List of stations:

• starting point: EWGLAM station list for verification

• selection based on availability of cloudiness each 3h per day

• plus „some more“ representative stations for COSMO-countries

THE_Score will be computed for each COSMO-country and different regions (W/N/E/S-Europe, Alps, smallest common region of all COSMO-xx, …)


COSMO strategy for VerificationConclusions

Advanced interpretation and verification of very high resolution modelsAdvanced interpretation and verification of very high resolution models

• Search for the “optimal scale” for verification and for representation of precipitation fields

• Fuzzy Verification score are a promising framework for verification of high resolution precipitation forecasts.• • Not all scores indicate a perfect forecast by perfect scores (Leaking scores).

• Choice of the scores: Upscaling, Intensity scale, Fraction skill score (?)

• End of the project expected for 2008



VerSUS projectVerSUS project

• One tool for Verification and Conditional Verification

• DB powerful

• No “ad hoc” application to create verifications: only simple selections

• R-Integration (to add statistical Indexes only the “Verification Package” can be updated) – Community Knowledge

• User configurable using the GUI (Graphical User Interface)

• GUI WEB-based

• End of the project expected for 2008 (delivery of the package)



COSI “The global Score” (COSMO Index)COSI “The global Score” (COSMO Index)

• Next future implementation

• Included in Common Verification Suite package (common fortran package for standard verifications, delivered in 2006 for COSMO community)

• Will be included in VERSUS package

• First results hopefully for COSMO GM of 2008

main plans (or projects)

Documents