„detective“ miss marple...„detective“ miss marple in „murder at the gallop“ m2 slide 15...
TRANSCRIPT
-
13rd verification workshop, ECMWF 2007, Göber
„Detective“ Miss Marple in
„Murder at the Gallop“
m1
-
Slide 1
m1 Now, this is how we verify the forecasters at DWD: we watch them intensily and when they get it wrong, they'll get a bang over their head. OK, I'm slightly exaggerating and I will explain later what Detective Miss Marple has to do with warning verification.mgoeber, 03/01/2007
-
23rd verification workshop, ECMWF 2007, Göber
Comparing peaches and apples
On the accuracy of gust warnings issued by forecasters and the accuracy of the model
guidance
Martin Göber
Department Weather ForecastingDeutscher Wetterdienst
Acknowledgements: R. Kirchner, T. Kratzsch, D. Richardson, G. Schweigert, S. Tremmel
-
33rd verification workshop, ECMWF 2007, Göber
-
43rd verification workshop, ECMWF 2007, Göber
Heidke skill score
00,10,2
0,30,40,50,60,7
0,80,9
1
near gale gale storm violent storm hurrican force
p
forecaster Local model
-
53rd verification workshop, ECMWF 2007, Göber
hit rate
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
near gale gale storm violent storm hurrican force
p
forecaster Local model
-
63rd verification workshop, ECMWF 2007, Göber
false alarm ratio
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
near gale gale storm violent storm hurrican force
p
forecaster Local model
-
73rd verification workshop, ECMWF 2007, Göber
Frequency bias
0
1
2
3
4
5
6
7
8
9
10
near gale gale storm violent storm hurrican force
p
forecaster Local model
-
83rd verification workshop, ECMWF 2007, Göber
relative value for C/L=0,1
-0,4
-0,2
0
0,2
0,4
0,6
0,8
1
near gale gale storm violent storm hurrican
p
forecaster Local model
-
93rd verification workshop, ECMWF 2007, Göber
relative value for C/L=0,01
-0,4
-0,2
0
0,2
0,4
0,6
0,8
1
near gale gale storm violent storm hurrican
p
forecaster Local model
-
103rd verification workshop, ECMWF 2007, Göber
Signal Detection Theory
-
113rd verification workshop, ECMWF 2007, Göber
1 2 3 4 5 6 7 8 9 10all cases
0
5
10
15
20
25
30
35
frequ
ency
indicator
all
all cases
(=CAPE, wind speed, finger prints, ....)
-
123rd verification workshop, ECMWF 2007, Göber
0 1 2 3 4 5 6 7 8 9event
all cases0
5
10
15
20
25
30
35fre
quen
cy
indicator
eventno eventall cases
-
133rd verification workshop, ECMWF 2007, Göber
0 1 2 3 4 5 6 7 8 9event
all cases0
5
10
15
20
25
30
35
frequ
ency
indicator
eventno eventall cases
thresholdmisses
False alarms
POD=70%FAR=15%ETS=50%Bias=80%
-
143rd verification workshop, ECMWF 2007, Göber
0 1 2 3 4 5 6 7 8 9event
all cases0
5
10
15
20
25
30
35
frequ
ency
indicator
eventno eventall cases
thresholdmisses
False alarms
POD=90%FAR=40%ETS=42%
Bias=150%
-
153rd verification workshop, ECMWF 2007, Göber
„Detective“ Miss Marple in
„Murder at the Gallop“
m2
-
Slide 15
m2 Now, this is how we verify the forecasters at DWD: we watch them intensily and when they get it wrong, they'll get a bang over their head. OK, I'm slightly exaggerating and I will explain later what Detective Miss Marple has to do with warning verification.mgoeber, 03/01/2007
-
163rd verification workshop, ECMWF 2007, Göber
1
10
100
1000
10000
1000000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
m/s
frequ
ency
n(F|O=29)
Violent storm warning: gusts>= 29 m/s
-
173rd verification workshop, ECMWF 2007, Göber
1
10
100
1000
10000
1000000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
m/s
frequ
ency
correct NO missed false alarm hit
model at “face value”
Violent storm warning: gusts>= 29 m/s
-
183rd verification workshop, ECMWF 2007, Göber
1
10
100
1000
10000
1000000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
m/s
frequ
ency
correct NO missed false alarm hit
Violent storm warning: gusts>= 29 m/s
-
193rd verification workshop, ECMWF 2007, Göber
Bias freehit rate>90%
Violent storm warning: gusts>= 29 m/s
0
50
100
150
200
250
300
11 13 15 17 19 21 23 25 27 29 31 33 35 37
threshold in m/s
%
hit rate
false alarm ratio
Bias
Heidke skill score
-
203rd verification workshop, ECMWF 2007, Göber
Re-labeling„model bias = forecaster bias“
model in m/s ----> „model gust interpretationfor warnings “
13 ----> 14 (near gale)16 ----> 18 (gale)22 ----> 25 (storm)25 ----> 29 (violent storm)30 ----> 33 (hurricane force)
Verification of heavily biased model ? Quite similar to forecaster !
-
213rd verification workshop, ECMWF 2007, Göber
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
0,0 0,2 0,4 0,6 0,8 1,0
F
H
model: near gale (>14m/s)forecaster: near gale (>14m/s)no skill
Model at face value
smaller mode
l values=
overforecast
ingla
rger
mod
el v
alue
s=un
derf
orec
astin
g
forecaster
Relative Operating Characteristics (ROC)
-
223rd verification workshop, ECMWF 2007, Göber
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
0,0 0,2 0,4 0,6 0,8 1,0
F
H
model: violent stormforecaster: violent stormno skillmodel: near gale (>14m/s)forecaster: near gale
Face value
overforecasting
unde
rfor
ecas
ting
forecaster
-
233rd verification workshop, ECMWF 2007, Göber
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0.0 0.2 0.4 0.6 0.8 1.0
F
H
near galegalestormviolent stormhurrican forceforecaster: galeforecaster: stormforecaster: violent stormforecaster: hurrican forceno skill
-
243rd verification workshop, ECMWF 2007, Göber
My conclusions
End user forecast verification: face value (incl. space-time point)
Guidance verification: measure potential of the guidance usingFuzzy, Object, ........., Signal detection Theory (ROC)
Comparing peaches and apples�� On the accuracy of gust warnings issued by forecasters and the accuracy of the model guidance