cosmo general meeting 2011 wg5 parallel session 5 september 2011

Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss

Verification resultsat MeteoSwiss in the year 2011

Francis Schubiger, Vanessa Stauch,Tanja Weusthoff, Pirmin Kaufmann

MeteoSwiss

COSMO General Meeting 2011

WG5 Parallel Session 5 September 2011

2 Verification results at MeteoSwiss in 2011

COSMO GM / WG5 Parallel Session, 05.09.2011

Content

• results of the operational verification in 2011

• separate verification of all eight daily COSMO-2 forecasts

• neighborhood verification (also weather type dependant)

• COSMO-2 03 UTC forecasts up to +48h

• extension of Beth Ebert‘s package: -> „fuzzy in time“



• Operational (COSMO-2, COSMO-7, COSMO-LEPS [mean], IFS):• Surface (3h-steps SYNOP and 1h-steps SMN [Swiss Met Net])

• psred, ps, T2m, Td2m, 10m-wind, cloud cover, gusts,precipitation (1h, 12h), global radiation

• Upper-air (TEMP)• T, RH, wind,

• Quasi-operational (COSMO-2, COSMO-7, IFS):• precipitation with RADAR (neighborhood/fuzzy verification, Ebert’s

package) also weather-type dependant• Monitoring: SMN (swiss SYNOPs) , RADAR, windprofilers• Ongoing in experimental modus:

• windprofiler over Switzerland• fluxes at Payerne

• New / Next developments:• VAD (locations of swiss radars)

=> All these packages should be replaced by VERSUS 2

Verification of COSMO-2, COSMO-7, COSMO-LEPS and IFS



T2m: mean diurnal cycle (first 24h forecasts)domain Switzerland (hourly SYNOP‘s)

V. Stauch, P. Kaufmann

OBS

COSMO-7

COSMO-2

Autumn 2010

Winter 2010/2011

Spring 2011

Summer 2011



T2m: mean diurnal cycle Spring 2011COSMO-7 vs IFSSYNOP‘s over COSMO-7 domain

OBS

COSMO-7

IFS




T2m: mean diurnal cycle Spring 2011COSMO-7 vs COSMO-LEPSSYNOP‘s over COSMO-7 domain


OBS

COSMO-7

COSMO-LEPS



Global radiation (inclined): Spring 2011COSMO-7 mean diurnal cycleSYNOP‘s over Switzerland




Global radiation (horizontal): Spring 2011COSMO-7 mean diurnal cycleSYNOP‘s over Switzerland




Precipitation: Spring 2011geographical distribution

precipitation12h sums

frequency bias(1 mm/12h)

Spring 2011

COSMO-7+24 & +48h




Precipitation (12h-sums +12 to +24h):Spring 2011 over Switzerland (SYNOP‘s)frequency bias: COSMO-7 & COSMO-2

observed frequency

V. Stauch



Precipitation (12h-sums +36 to +48h):Spring 2011 over Switzerland (SYNOP‘s)frequency bias: COSMO-7 & IFS

observed frequency

V. Stauch



Precipitation (12h-sums +12 to +24h):Spring 2011 over Switzerland (SYNOP‘s)COSMO-7 & COSMO-2

V. Stauch

for both models mean over 9 gridpoints foreach station



Precipitation (12h-sums):Spring 2011 over Switzerland (SYNOP‘s)COSMO-7 & IFS

V. Stauch



Hourly precipitation: diurnal cycleSwiss SYNOPs (1h acc) CH-radarcomposit (3h acc)

July

201

0

03 09 15 21 00

fcst: +12..+24h fcst: +3..+6hRADAR

COSMO-7

COSMO-2

COSMO-2

COSMO-7

SMN

Au

gu

st 2

010

RADAR

COSMO-7

COSMO-2

COSMO-2

COSMO-7

SMN

03 09 15 21 00

fcst: +12..+24h fcst: +3..+6h

V. Stauch



• 2m-temperature:

• cold bias in winter and warm bias in summer• positive bias more pronounced during nighttime• COSMO-2 ~0.5 K warmer (higher values) than COSMO-7

• 2m-dewpoint:• negative bias (~1 K) in the summer and autumn period

• 10m-windspeed:• negative bias along the coast and on mountains, positive bias inland

• total cloudiness:• mean daily cycle not well represented (mainly overestimation during night

and underestimation during day)• precipitation:

• overestimation (except in Summer ~ no bias) most pronounced in Winter (over COSMO-2 domain: 30-35% ; over Swizerland ~45%)

• higher amounts in COSMO-2 / higher amounts over Alpine area• low amounts (0.1 mm/12h): overestimated in Winter (over the Alps all

seasons)• high amounts (10 mm/12h): underestimated in COSMO-7 (over full domain

by ~30% in summer, 20% in winter)

Verification of surface weather parameters



TEMPS verification: Windspeed +24hWinter 10/11

COSMO-7

COSMO-2

all TEMPsCOSMO-2domain



• temperature: cold bias (~ 0.5 K) in winter and warm bias (~ 0.5 K) in summer from ground up to 300 hPa

• windspeed: positive bias in PBL (up to +1 m/s in winter) and slight negative bias above 400 hPa

Verification with vertical profiles: main results



Separate verification of the eight daily COSMO-2 forecasts• questions of interest:

• which is the best forecast for the afternoon/evening convection ? – is it the latest forecast or perhaps a forecast with a longer „spin up“ time ?

• are there significant differences between the eight forecasts ?

• period:COSMO-2 forecasts from 1 May to 31 July 2011• Verification with hourly obs over Switzerland



COSMO-2: precipitation diurnal cycleforecasts of 00, 03, 06 and 09 UTC

Vanessa Stauch



COSMO-2: precipitation diurnal cycleforecasts of 12, 15, 18 and 21 UTC

Vanessa Stauch



COSMO-2: precipitation diurnal cycleall eight daily forecasts

Vanessa Stauch



COSMO-2: 2m-temperature diurnal cycleall eight daily forecasts

Vanessa Stauch



COSMO-2: 2m-temperature biasall eight daily forecasts

Vanessa Stauch



COSMO-2: 10m-wind speed diurnal cycleall eight daily forecasts

Vanessa Stauch



results for 20103h accumulated precipitation sumsover the domain of the swiss radar composit

models: COSMO-2 and COSMO-7for all 8 daily forecast runs, precipitation sums from +3 to +6h

observation precipitation estimates of the swiss radar composit

in case of a missing value, the full date will not be evaluated

Neighborhood verification for precipitation



COSMO-2 COSMO-7 COSMO-2 - COSMO-7

- =

- =

goodbadCOSMO-7 better COSMO-2 better

Neighbourhood Verification, January–December 2010 3h sums (+3 ..+6h) Fractions Skill Score (top) and Upscaling (bottom)

Fractions Skill Score

Upscaling

T. Weusthoff



FSS, COSMO-2 minus COSMO-7, 2010

Winter Spring

AutumnSummer

numbers = FSS-Score of COSMO-2

colours = differences COSMO-2 minus COSMO-7

COSMO-2 better

COSMO-7 better

• COSMO-2: similar skill in all seasons

• COSMO-2 better than COSMO-7 on almost all scales

• COSMO-2 better than COSMO-7 especially in Winter and Summer

Tanja Weusthoff



NE

(11x)

N

(18x)

NW

(38x)

SE

(4x)

S

(10x)

SW

(49x)

E

(4x)

W

(56x)

F

(78x)

H

(73x)

L

(25x)

COSMO-7 better COSMO-2 better

differences in Fractions Skill Score for weather-type dependant verif

COSMO-2 minus COSMO-7

YEAR 2010



Summary neighbourhood Verifikation precipitation in 2010

• COSMO-2 better than COSMO-7 on all scales,differences become less with increasing leadtime

• good forecast of the spatial structure on higher scales

• The skill of the models varies for different weather types and the differences between COSMO-2 and COSMO-7 varies also:- best skill: Autumn and Spring, south to northwest weather types- greatest difference COSMO-2 minus COSMO-7: Summer and Winter, north- and east types, convective cases

Tanja Weusthoff



COSMO-2 03 UTC up to +48h:Comparison COSMO-2 to COSMO-7• period (full year): 1 June 2010 – 31 May 2011• Main interest:

• advance of COSMO-2 as compared to COSMO-7• question: “are the COSMO-2 forecasts at +48h

still better than those of COSMO-7” ?• Verification with:

• SYNOP over Switzerland (hourly):comparison +0 to +24h vs +24 to +48h

• Swiss RadarComposit: neighborhood Verifikation for four 12h-Summen (0..+12, +12..+24, +24..+36, +36..+48h)

• TEMPs for +21h and +45h



Verification SYNOP-CH: 2m-temperatureCOSMO-2 vs COSMO-7

COSMO-2COSMO-7

advance of COSMO-2 also in the second day

BIAS

MAE

VAL

STDE

V. Stauch



Verification SYNOP-CH: precipitation COSMO-2 vs COSMO-7 [+0 to +24h]

Frequency bias (mm/h)

COSMO-2COSMO-7

0.1 mm/h

10 mm/h

1.0 mm/h

V. Stauch



COSMO-2COSMO-7

0.1 mm/h

10 mm/h

1.0 mm/h

V. Stauch

Verification SYNOP-CH: precipitation COSMO-2 vs COSMO-7 [+24 to +48h]

Frequency bias (mm/h)



Verification with swiss radarcomposit

second forecast day

first forecast day+0.. +12h

+24..+36h

+12..+24h

+36..+48h

daytime: 03-15 UTC nighttime: 15-03 UTC

T. Weusthoff

COSMO-2 COSMO-7 COSMO-2 COSMO-7

COSMO-2 COSMO-7 COSMO-2 COSMO-7



FRACTIONS SKILL SCORE: COSMO-2 – COSMO-7precipitation sums of 12h

+ 0 .. + 12h

+ 24 .. + 36h

+ 12 .. + 24h

+ 36 .. + 48h

T. Weusthoff

COSMO-2 better

COSMO-7 better



UPSCALING: COSMO-2 – COSMO-7precipitation sums of 12h; score: ETS (equitable threat score)

+ 0 .. + 12h

+ 24 .. + 36h

+ 12 .. + 24h

+ 36 .. + 48h

T. Weusthoff

COSMO-2 better

COSMO-7 better



3. „Fuzzy in Time“

• Extension of the spatial window with a window in time volume (dx * dy * dt)

• Evaluation of the forecasts in this volume• Time-window ntm = [1h,3h,5h,7h,9h]

dt

dx

dy

T. Weusthoff



dxdy

dx

dymodel

observation

e.g. FSS

fraction blue pxiels model = 12/75 (dt=3), 5/25 (dt=1)

fraction blue pxiels obs = 12/75 (dt=3), 3/25 (dt=1)

t0

t0-1

t0+1

t0

t0-1

t0+1

T. Weusthoff



00 01 02 03 04 05 06 07 08 09 10 11 12 13 1420 21 22 23

00-02

01-03

02-04

03-05

04-06

05-07

…

23-01

22-00

21-23

20-22

hourly accumulated precipitation

Evaluation of 3h accumulations, time window ntm = 5 (t0 +-2h)

00 UTC 03 UTC 06 UTC 08 UTC

22 UTC 02 UTC

01 UTC 05 UTC

3 hourly accumulated precipitation

Evaluation of 3h accumulations, time window ntm = 1

09 UTC00 UTC 03 UTC 06 UTC

23 UTC 01 UTC 02 UTC 04 UTC

T. Weusthoff



FSS for different time-windowsCOSMO-2, July 2010

T. Weusthoff


COSMO GM / WG5 Parallel Session, 05.09.2011 T. Weusthoff

FSS for different time-windowsCOSMO-7, July 2010



FSS for different time-windows COSMO-2 minus COSMO-7, July 2010

T. Weusthoff



Summary „fuzzy in time“

• FSS increases on all scales with increasing time-window• greatest effect for small spatical scales• lowest effect for high threshods

• Both models show a similar increase• difference COSMO-2 minus COSMO-7 stays equal, resp.

becomes littler for high time-windows

• For Upscaling the influence of a time tolerance is relatively low und restricted on low thresholds ( effect of the avergaing)

Application of time-windows on the gridsclae would make sense; simultaneous application with space tolerance brings no great change

T. Weusthoff

cosmo general meeting 2011 wg5 parallel session 5 september 2011

Documents