verification highligths by wg5

VERIFICATIONHighligths

by WG5

2

Outlook

• Some focus on Temperature with common plots and Conditional Verification

• Some Fuzzy verification

• Long trends

SON 2009

DJF 2009-2010

5 Verification results at MeteoSwiss in 2010

COSMO GM / WG5 Parallel Session, 06.09.2010

T2m: mean diurnal cycle (first 24h forecasts)domain Switzerland (hourly SYNOP‘s)

Autumn 2009

Winter 2009/2010

Spring 2010

P. Kaufmann, V. Stauch

OBS

COSMO-7

COSMO-2

Summer 2010

T2m COSMO-I7 00UTC: LAST YEAR

WAM

WG5 COSMO General Meeting, Moscow 2010

Conditional VerificationExtracting information for relevant

performance of weather parameters

The input from modelers and forecasters is necessary for identifying and testing

hypotheses.

F. Gofa - HNMS

Temp in overcast conditionsTemp in overcast conditions

Fall Winter

Spring Summer

F. Gofa - HNMS

COSMO General Meeting – Moscow 06-10 Sept 2010

Conditional Verification Temp – TCC obs >=75%

SON

MAM

DJF

Better behaviour for all the seasonsCompare to no condition model

Temp in clear sky conditions

Fall Winter

Spring Summer

F. Gofa - HNMS

Conditional Verification Temp – TCC obs <=35%

Worse behaviour for all the seasonsCompare to no condition model

SON

MAM

DJF

Temp in ‘calm’ conditions (<2 m/s)

Fall Winter

Spring Summer

WG5 COSMO General Meeting, Moscow 2010F. Gofa - HNMS

WG5 COSMO General Meeting, Moscow 2010

Fall Winter

Spring Summer

Temp in ‘high wind’ conditions >10m/s

F. Gofa - HNMS

14

Some conclusion

• A problem with Temp is clear.

• RMSE between 2-3 °C it is not so small.

• Diurnal cycle too cold during the day and too warm during the night

• Clear different behaviour with conditions on TCC and with different wind conditions

15

Outlook

• Some focus on Temperature with common plots and Conditional Verification

• Some Fuzzy verification

• Long trends

16 Verification results at MeteoSwiss in 2010

COSMO GM / WG5 Parallel Session, 06.09.2010

results for 20093h accumulated precipitation sumsover the domain of the swiss radar composit

models: COSMO-2 and COSMO-7leadtimes 04 – 07h for all 8 daily forecast runs

obervation precipitation estimates of the swiss radar composit

in case of a missing value, the full date will not be evaluated(total of 28 days)

Neighborhood verification for precipitation

17 Verification results at MeteoSwiss in 2010

COSMO GM / WG5 Parallel Session, 06.09.2010

COSMO-2 COSMO-7 COSMO-2 - COSMO-7

- =

- =

goodbadCOSMO-7 better COSMO-2 better

Fractions Skill Score

Upscaling

Neighborhood (fuzzy) verification 2009, FSS and UP T. Weusthoff

18 Verification results at MeteoSwiss in 2010

COSMO GM / WG5 Parallel Session, 06.09.2010

FractionsSkill Score

FSS

UpscalingETS

Upscalingfreq. bias

FBI

Neighborhood (fuzzy) verification: Spring 2010COSMO-2/COSMO-7: 3h acc, leadtime +4 to +6 for all models

COSMO-2 COSMO-7 IFS

T. Weusthoff

19

Outlook

• Some Common Plots (Task 6 Versus)

• Conditional Verification

• Some Fuzzy verification

• Long trends verification

Introduction of new Z0

Introduction of new Z0

Introduction of new Z0

Introduction of new Z0

Total cloud cover_____ Cloud cover above 2 Octa (Cl.1).......... Cloud cover above 6 Octa (Cl.2)

Valid time 00 UTC

Cloud cover of low clouds because incorporation of AWS

Stand Mai 2010

Time series of the COSI: State May 2010

Stand Mai 2010

Time series of the COSI: State May 2010

Time series of the COSI: Temperature day 1

Stand Mai 2010

Time series of the COSI: Temperature day 2

Stand Mai 2010

Time series of the COSI: Temperature day 3

Stand Mai 2010

Time series of the COSI: State May 2010 (STDV used for T2m instead of RMSE)

• Long period verification (seasonal trend) (from djf’04 to mam’10)

1. Some Statistical indices for low thres (0.2mm/24h)

2. Some Statistical indices for high thres (20mm/24h)

• Verification ovest last year (DJF 2009-MAM2010)

1. Driving model comparison:

ecmwf/Cosmo-I7/Cosmo-I2

2. Driving model comparison:

ecmwf/Cosmo-ME/Cosmo-IT

COSMO General Meeting – Moscow 06-10 Sept 2010

• All the versions present a seasonal cycle with an overestimation during summertime (except COSMO-7 and I2)• COSMO-7 and I2 underestimate• Overestimation error decreases in D+2 (spin-up effect vanished)

QPF verification of the 4 model versions at 7 km res. (COSMO-I7, COSMO-7, COSMO-EU, COSMO-ME) with the 2 model versions at 2.8 km res. (COSMO-I2, COSMO-IT)Dataset: high resolution network of rain gauges coming from COSMO dataset and Civil Protection Department 1300 stationsMethod: 24h/6h averaged cumulated precipitation value over 90 meteo-hydrological basins

Seasonal trend - low thresholds

COSMO General Meeting – Moscow 06-10 Sept 2010

• Very light improvement in trend

• Seasonal error cycle: lower ets during winter and summertime

• no significant differences between D+1 and D+2

• Last winter (very snowy particularly in Northern Italy): low ets value (D+1 and D+2) model error or lack of representativeness of the rain gauges over the plain during snowfall ?

Seasonal trend - low thresholds

COSMO General Meeting – Moscow 06-10 Sept 2010

Driving model comparison: ECMWF/COSMO-ME/COSMO-IT, low thresholds

• ECMWF tendency to forecast low rainfall amounts big overestimation, big false alarms, very low ets, quite good pod

• Better prediction for COSMO-models (no strong differences between ME and IT)

• Seasons DJF2009 – MAM2010

COSMO General Meeting – Moscow 06-10 Sept 2010

• ECMWF tendency to forecast low rainfall amounts big overestimation, big false alarms, very low ets, quite good pod

• Better prediction for COSMO-models BUT bad performance during summertime

•Seasons DJF2009 – MAM2010

Driving model comparison: ECMWF/COSMO-I7/COSMO-I2, low thresholds

37 Verification results at MeteoSwiss in 2010

COSMO GM / WG5 Parallel Session, 06.09.2010

Precipitation (12h-sums +12 to +24h):Spring 2010 over Switzerland (SYNOP‘s)COSMO-7 & COSMO-2

V. Stauch

38 Verification results at MeteoSwiss in 2010

COSMO GM / WG5 Parallel Session, 06.09.2010

Precipitation (12h-sums +12 to +24h):Spring 2010 over Switzerland (SYNOP‘s)COSMO-7 & IFS

V. Stauch

COSMO General Meeting – Moscow 06-10 Sept 2010

• Slight bias reduction during latest seasons

• Last winter: all the versions overestimate (probably due to lack of representativeness of the rain gauges over the plain during snowfall)

• Strong COSMO-7 underestimation BUT slight improvement during latest seasons

Seasonal trend - high thresholds

COSMO General Meeting – Moscow 06-10 Sept 2010

• Low values during summertime

• In general, quite stationary error since son2008 up to now

• All the versions present a jump around son2008: ets increases from 0.2-0.4 up to 0.3-0.5

• Skill decreases with forecast time

Seasonal trend - high thresholds

COSMO General Meeting – Moscow 06-10 Sept 2010

• ECMWF difficulty to forecast high rainfall amounts bias around 1 BUT big false alarms, very low ets and pod

• Better prediction for COSMO-models

• Seasons DJF2009 – MAM2010

Driving model comparison: ECMWF/COSMO-ME/COSMO-IT, high thresholds

COSMO General Meeting – Moscow 06-10 Sept 2010

• ECMWF difficulty to forecast high rainfall amounts bias around 1 BUT big false alarms, very low ets and pod

• Better prediction for COSMO-models

• Seasons DJF2009 – MAM2010

Driving model comparison: ECMWF/COSMO-I7/COSMO-I2, high thresholds

12h Precipitation – Sep2009-Aug2010

WG5 COSMO General Meeting, Moscow 2010

COSMOECMWF

Really strong overestimation of lower preci amounts up to 3mm and lower ETS scores for

ECMWF

F. Gofa - HNMS

44

Some conclusion

• Long term trends show a general (sometimes light) improvements for all the considered models

• Comparison between COSMO models and IFS shows a general clear better behaviour for COSMO implementations