experiences concerning fuzzy-verification and pattern recognition methods ulrich damrath

Experiences concerning fuzzy-verification and pattern recognition methods

Ulrich Damrath

U. Damrath: Experiences concerning fuzzy-verification ... - COSMO GM, Offenbach 2009

2

Outlook

• Results on operational verifcation for winter and summer month

• An approach concerning significance test of „fuzzy“-verification results

• Estimation of consistency of forecasts using a pattern recognition method (CRA method by Beth Ebert)


3

Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time 06-18 hours

GME COSMO-EU

COSMO-DE


4

ETS upscaling for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time 06-18 hours

GME COSMO-EU

COSMO-DE


5

Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for August 2009, forecast time 06-18 hours

Global Europe

Germany


6

ETS upscaling for forecasts of GME, COSMO-EU and COSMO-DE for August 2009, forecast time 06-18 hours

Global Europe

Germany


7

Examination of statistical significance of „fuzzy“-verification results using bootstrapping

• Basic idea of bootstrapping:• Repeat a resampling all elements of a given in a sample of

forecasts and observations as often as necessary (N times) and calculate the relevant score(s)

• Calculate from N scores statistical properties of the sample such as mean value standard deviation, confidence intervals and quantiles

• Application to „fuzzy“-verification• Resampling is done using „blocks“.

• Blocks are defined as single days.

• Number of resampling cases: N=Days*100

• Calculation scores from N samples for NT thesholds and NW windows

• Calculation of quantiles for each window and threshold


8

Values and quantiles 0.1 and 0.9 for Upscaling ETS GME, period June - August 2009

Germany


9

Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-EU, period June - August 2009

Germany


10

Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-DE, period June -August 2009

Germany


11

Next step: Evaluation of significance

• First impression: Is the result of Model 1 better than the result of Model 2?

• Significance hypothesis checked using a Wilcoxon-test (IDL-code RS_TEST)


12

Differences between GME and COSMO-EU

ETS(COSMO-EU) - ETS(GME) Significance test

COSMO-EU better than GMECOSMO-EU worse than GME

Germany


13

Differences between GME and COSMO-DE

ETS(COSMO-DE) - ETS(GME) Significance test

COSMO-DE better than GMECOSMO-DE worse than GME

Germany


14

Differences between COSMO-DE and COSMO-EU

ETS(COSMO-DE) - ETS(COSMO-EU) Significance test

COSMO-DE better than COSMO-EUCOSMO-DE worse than COSMO-EU

Germany


15

ETS(COSMO-DE) - ETS(COSMO-EU)

COSMO-DE better than COSMO-EUCOSMO-DE worse than COSMO-EU

Significance test

Differences between COSMO-DE and COSMO-EU


16

Example of good precipitation forecast of COSMO-DE

Zeigten die numerischen Modelle und die statistischen Prognose----------------------------------------------------------------verfahren Signale für das Ereignis?-----------------------------------Die Numerik zeigte im Vorfeld vermehrt Signale für kräftige Konvektion. Während diese bei GME und COSMO-EU recht breit gestreut und pauschal auftraten, signalisierten mehrere COSMO-DE-Läufe eine linienartige Struktur mit unwetterartigen Zellen (auf Basis der 1- bzw. 3-stündigen RR-Prognosen) im Grenzbereich von Hessen zu NRW und Niedersachsen. Diese Linie trat dann in den Mittags- und frühen Nachmittagsstunden tatsächlich auf, wenn auch nicht 100%ig kongruent, aber doch in der Nähe, so dass in diesem Fall von einer guten Prognose gesprochen werden kann (mehr dazu siehe "Zentraler UW-Sofortbericht" der VBZ).


17

Example of good precipitation forecast of COSMO-DE

3h-precipitation forecast of COSMO-DEvalid 10.08.2009 12 UTC, left 03 UTC +9h, right 06 UTC +6h.

3h-precipitation observation10.08.2009 09-12 UTC


18

Example of good precipitation forecast of COSMO-DE compared to other models


19

Example of good precipitation forecast of COSMO-DE compared to other models


20

About consistency and inconsistency

• Forecasters are interested in consistent model forecasts.• But due to growing of errors during forecast time forecasts

consistency cannot be expected concerning all properties of the forecasted fields!

• Inconsistency: Differences between forecasts that are valid for the same time concerning different properties of the forecasted fields (properties of the pattern, values at special points of interest, extreme values, ...)

• Differences between the forecasted fields concerning• phase,• amplitude• and the remaining part

Entity-based QPF verification (rain “blobs”) by E. Ebert (BOM Melbourne)Verify the properties of the forecast rain system against the properties of

the observed rain system:

• location• rain area• rain intensity

(mean, maximum)

Observed Forecast

CRA error decompositionThe total mean squared error (MSE) can be written as:

MSEtotal = MSEdisplacement + MSEvolume + MSEpattern

Configuration for the current study:

- “Observations”: forecasts: 06-30 hours

- Forecasts : forecasts: 30-54 hours and

forecasts: 54-78 hours


22

Dark :forecasts 30-54 hLight:forecasts 54-78 h


23



24



25



26

Summary• Scores like Fractions skill score and ETS from upscaling show in general

advantages of COSMO models compared to GME.• This is true especially for summer months.

• For winter months all models have nearly the same quality for low precipitation amounts and large window sizes for averaging.

• Significance test lead to the results, that:• The advantages of COSMO models compared to GME are statistically significant for

most window sizes and precipitation amounts.

• The differences between COSMO-EU and COSMO-DE are not significant altough there are systematical differences for different precipitation amounts and window sizes.

• There are some cases with very useful precipitation forecasts of COSMO-DE compared to COSMO-EU from the view of forecasters.

• A study about the consistency of precipitation forecasts showed - it could be expected , but now it is proved - that:

• Forecasts of high precipitation amounts are less consistent than those for low precipitation amounts.

• Pattern errors contribute most to forecast errors.

• During winter months volume errors are higher than displacement errors.

• During summer months displacement errors are higher than volume errors.

experiences concerning fuzzy-verification and pattern recognition methods ulrich damrath

Documents

cosmo gm

cosmoeu slide

cosmoeu germany

etscosmoeu cosmo

cosmoeu etscosmoeu

gme cosmoeu worse

upscaling ets cosmoeu

hours gmecosmoeu cosmo