U.S. DEPARTMENT OF COMMERCENATIONAL OCEANIC AND ATMOSPHERIC ADMINISTRATION

NATIONAL WEATHER SERVICE

OFFICE NOTE 210

A Modified Threat Score and a Measure of Placement Error

Frederick G. ShumanNational Meteorological Center

FEBRUARY 1980

This is an unreviewed manuscript, primarilyintended for informal exchange of informationamong NMC staff members.

4-~

-1 b , .- - 7 ....-...-...

r2

Introduction

It is well known and can easily be shown that, with present forecasttechniques and accuracy, higherthreat scores can be achieved by increasingthe bias above unity. Figure 1 indicates the effect of increased bias onthreat score. For instance, it indicates that a forecast with unit bias andwith threat score of 0.2, can be improved to a threat score of 0.27 byincreasing the bias to 2.6. Skill, therefore, cannot be judged by threatscores alone, but only in consideration with other statistics, especially bias.

Records of bias as well as threat score, prefigurance, and post agree-ment are kept at NMC but in comparing techniques and forecasters, if biasesdiffer substantially, relative technical skill is unclear. The same can besaid for determining progress in technical skill from a time series of suchstatistics.

It is important to distinguish between measures of technical skill andmeasures of the usefulness, or service value, of forecasts. Measures ofboth should be available. The need for our keeping watch on the utilityof our forecasts is self evident. As for our technical skill, it may be toolow in some forecast products to be useful for certain purposes. In suchcases, we want to know how we are progressing toward the goal of utility,especially in the present era of steady advances in the state of the art.The modified threat score and measure of placement error that I willdevelop here, are intended to measure technical skill, not necessarilyservice value.

*~ ~ I should also add that a verification statistic is only useful to theextent that it agrees with the judgement of skilled practicing meteorologists.Thus, verification statistics are not entirely objective, for they are designedand selected subjectively. Like any other statistic, the modified threatscore and measure of placement error must be judged useful in order tobe useful to us.

My aim in modifying the threat score is to remove the effect of biasin overforecasting, and likewise to exact a penalty for underforecasting.I will introduce the notion of complex numbers for threat scores, althoughin practice they will occur only in individual cases, or for very shortrecords, or for very large amounts of precipitation. I will suggest that,in the case of a complex number, its magnitude be taken as a negativethreat score.

Th ethreat score is

TS= H (1)F+Q.-Hwhere F and Q are, respectively, forecast and observed areas or numbersof events, and H is "hits," i.e., the area or number of events correctlyforecast.

One of my aims is to be able easily to compute the modified threatscore from the past record. The only parameters that have been measuredand for which records have been kept over the years at NMC are F, Q, and H.

II

VP

2

000* Zr-.

.4 000Woo

cc..

000

000.

0.-c

L .4 I I~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ..... F I $ I rz~~~~~~~~~~~~~~~~~~~;aS

71~~~~~~~~~~~ 3aI ~~~~~~~~~~~~~~LL

ix ,~~~4

1- lw.I-I II tI ~~~~~~~~~~~~~~~~~~~~~~~~~1

.~~~~~ '4 / ~~~~~~~~~~~~~. t.-I-,-'l,..-..~~~~~~~~~~~~~~~~~~~~~~~~~~~m ~

13 ~ ~ ~ ~ ~ / I t[4 ... 1 iiV ~~~~~~~~vt.2. Tr 2

l44

;Billy: -. -., WS: i ..! , -- . - .. . ..Ci_t ~~~~h&%__ I I h;'b 1ri S~~-4 r At [h -.B ; -cur~ .': r ....'.lle eff '..bia.. _'n . . . .... ' '- ..'.. .....;+: . -'r e " _ -,' :"Bs' w-;-,~: c : ~r ... ;Ti~n]Y~nsode. __ '.......Lc.urv-?],ab:...... .... ....

RSB fis thi~maxir~l l x hriatscl - -tia- can be6 acnieved b r increasing: .is .

.. ....... .. . .... ..

' _,-, -__ -- .: .o .' ''For, < .i 'ali ti e ----4'--

iure .:', T ie: effc t bio : . as on treatscoe :' : . hcuras ere l a-cale r:acur lat-dr ot ai-''b'v' t[4n{iy nio dol: [Figule, '2); ;:te: 'curve: ] abeledli T S :Shows: ithreat. score: w:9i.:1 nrntii: b~as i'-J{J

'1 S~'is -th hex~~r threat' saetd i eahee yicesn is ie~F-!-B:as .curize shown F' thisat:-scores die lowemr than i-:4: .V:ar biases above the .c -:?Btasl cu re:the thre tscore is +ess thanrS.::;-: '-: ; :... :

tt|*'t'1isX~l+I ''-' 'I 1- s .i .,. ........, +.....t ---;X=-I X .. ...r' ... . . . ... 1- ............."; :-:,1 ~--' 'L ,-~ .......', ' "i... ::,;¥L,.-"4 44i1 j½4

_

;

.

A

E :

W.X

.

.

I -I Ii

3

The modified threat score is designed to require only these threeparameters, and only these are needed for placement error for anindividual case. For a set of cases the number, N, of cases for whichbothF and Q are not zero in the set will also be needed to calculateplacement error, but N can easily be counted from available records.

The approach I take is to model F and Q as twoequal to F and Q, respectively; H is thus modeled aslap, as shown in Figure 2.

circles withthe area of

Figure 2. Model for calculating the modifiedthreat score and placement error. Forecast eventsF, and observed events, Q, are represented asareas of circles with radii a and b, respectively.The placement error, c, is the distance betweenthe centers of the circles. "Hits," i.e., eventscorrectly forecast is represented as H, theoverlapping area.

It is evident from Figure 2 that, given a and b, H will vary monoton-ically with c, so long as circles A and B intersect. Thus, givenH, F, and Q, it is possible to determine c.

To modify the threat score, the placement error, c, is firstdetermined. Then holding c constant, the larger of F(a) and Q(b)is reduced to equal the other, thus changing the bias to unity. Themodified threat score is then calculated from the new configuration.

K

areasover-

4

Determination of placement error

In Figure 2, H is the area, b2 a, of circular sector CAD; plus thearea, a2 3, of circular sector CBD; less the area of quadrilateral ACBD.Because of the symmetry of ACBD about line segment AB, its area is twicethe area, ½ ab sin y, of triangle ABC. But y = wr - (a + ~),thussin y = sin (a +f). Therefore

H = b2 a + a2g - ab sin (a+3) (2)

The cosine law gives

'cos a = c 2 - (a 2 - b 2) (3)2 bc

COS f = c 2 + (a 2 - b2 )2 ac

and therefore a and 6 are functions of a, b, and c. The parametersa and b are given by

F = .a 2

Q = 2-b2t b

H is thus a transcendental function of c, and given F, Q, and H,(2) can be solved for c. The method I use, based on Newton's iteration, is

V+- cV : '[b2V+ a2 H c- (c - = :L 51 ;;s 9 1 - cv 1 4)

where a o where -t ~ cos" l (c v) 2 _ (:a'2'_ b 2) v2 b c:V

6V - -c V. -= cos c- - b cos o-

a

and where v is the iteration count. As a flag to readers, the law of sineswas used in deriving (4):

a - b = csin a sin sin Y

As a first guess, I use

C = = a '2 b' 2 (cV = 1 7r) if F >Q

c = .b 2 a2 v=o = Tr) if F < Q

There are three cases for which c cannot be determined with my modelof circles, without further assumptions. They are when the circles do notintersect:

i

5

F

Case I. H=0. Case II. H=Q. Case III. H=F.

of these cases, I assume that the two circles are tangent to eachThus I assume the configuration most favorable to the forecast inand least favorable in the other two cases. The result is

For Case I.For Case II.For Case III.

H; = 0H=QH=F QH = F

c = a +bc=a-bc =b -a

For these cases, iteration (4) is not required to find c.

Determination of the modified threat score

If Fparameters.

> Q, then F' = Q and a' = b, where the primes indicate modifiedThen, from (1)

HITS' - H'2Q - H'

From (2),

H' = b 2 (2 a' - sin 2 ca')

Thus,

2a' - sin 2 ca'2 r - (2a'c - sin 2 a')

where from (3)

-12.l = cos 1 c2b

Q

'I

In eachother.Case I,

(5)

(6)

I'

MIP--

6

Likewise, if Q > F, then Q' = F and bT = a, and

2' - sin 2 '2 Tr - t2B' - sin 2 6')

where

W = cos- 1 C

2a (7)

Cases can and do occur in which the two circles in the modifiedconfiguration do not overlap nor are tangent. The figure below illustratesan example.

In such cases,

if F > Qif Q > F

c -> 2b and cos a' > 1c > 2a and cos ,' > 1

and a' =number.

B' is thereforeI note that

an imaginary number, and TS' is a complex

cos iz = cosh zsin iz = i sinh z

for any z. I let: 7 . : 0~~~~~~.

:a = . I= i z

where z is real. The parameter, z, is found, in the case of F > Q, bysubstituting iz for a' into (6),

Z = cosh'- 1 c2b

and in the case of Q > F, by substituting iz for 3' in (7),

z = cosh- 1 c2a

The positive root is always taken for z. Equation (5) then gives

TS' -iq27r + iq

*:de

i 0 : f 0 : :

7

where q = sinh 2z - 2z

Thus,o 2 + 2iiq _

TS' = - 4. +

The function, q, by the way, is always positive for positive z. Insteadof dealing with complex numbers, I take the magnitude of TS' as a negativemodified threat score:

TS" = -__:4r2 + q2

TS" is always negative, and it can easily be shown that DTS"/ 3cis always negative, so that a forecast is penalized in the modified threatscore by placement error, c. Also, DTS"/a a or MTS" / b is always positivefor F > Q or Q > F; respectively, so that the smaller the precipitationareas, the lower the algebraic threat score. In the limit when Q or F,but not both, are zero the modified threat score is TS" = - 1, the minimumalgebraic value for TS". When both Q and F are zero, such cases are simplynot counted. The modified threat score is thus nicely limited:

-1 <TS", TS' <+ 1

Figure 3 shows the variation of TS" with c/b (F > Q). TS" = -0.996for c/b = 12, and approaches -1 asymptotically as c/b aDpproaches infinityV

Some examples

To get a feeling for how the threat score is modified in practiceusing my model, I calculated the modified threat score from the annualrecords of QPF for the 1" area of accumulated precipitation in the first24 hr for the 10 years, 1970 to 1979. The records are in the form of annualsums of F, Q, and H. I had the Quantitative Precipitation Branch count thenumber of days, N, in each year for which both F and Q were not zero, andthen divided F, Q, and H by N before calculating. This was done so thatthe placement error, c, would be in the same unit of length that was usedin the daily measurements of the area, F, Q, and H. That unit of lengthis a latitude degree.

Figure 4 shows the result. Although the forecasts are substantiallypenalized in varying amounts in the modified threat score for biases greaterthan one, especially so for the years 1974-6, there remains a high correlationbetween the two threat scores. The table below shows the correlation co-efficient for that relationship, as well as for others.

.

,.

8

-� a -

a.,'

404

4.

-I .1- . . ..... .. . ....

J7 : p -r

I 1- IJa,'� a- .4I;i.uiL4LhTT:: .� . . � . �4

R 1.1-1~~ .

Tt. a-.. 44V aa,.L Va±. .... ila4a'L .* .... a

_�, I I , ,., ['- , 44 -:!-�T :_ " _' -,i- 7--- - - .' ['-' ttt "i r,-i-HE�:1-I- ,\-' 7 1 'I [_ �7L ft, L!=n I-11, .4,,. :j�, ,, 7' , 14:1 ; I H, I - I ... !� , - _,i. , I -al, - -!L. 1_4t'. 4,- ! � i�,

, .. . -, ;i '. _ala. � ... L � I .

£lTj. a-a-a .4 a� .�ljt� a- �

�iKL1J�ikfl<i L�..VaTA�i �a-�L 1.!�i -. 1

4.t.1.2 .4.. .L�a. . I �i•j Ifl±i ! � �..,aai .. ,...a..�... --- ba-I. -t�

4-Th � LI

I dI�j� LtK. K" a.,, -;z-�a-* A

& I I L4b

'I4±v�Lf .1 1 ... i i �1

��-' �I �

....4;:7:jjj:2I::.� a

.Ta�ur.e The. yariation .TS with � 3Ia-"j�� -aaaI.'. a-I'�a-

I � 4 Ia � .j -Ia � � L

14

a.La.I.a. .a.IL.2......aji �7,______ _______ �.a �.Ia....jiUj

'1

I A.I -"

*~'*~j, ~*a .,. 1 ~. .1 ~ ~ 'a if ~ .:~l~i~tET:I:.

LIII . .

.�

4,:.

C-

F.XaI

Ij�jiJ

II-1IAI

i

i

iI4

i

iJ

II

I

ii

IiIII

m

I

, ,---,-_ - - ' l

_:t. :_ " -t -___ -.11-t. - __ _:_ - 1___ , M - [__ El 1!

itt-i "P iTI ;�L;4 'FIAT �:,ji

II :

r-i-'_. -.;..'. . T. . .-:-:i:Tji.:,':w:-:_;,-T-~':_ i..q.: !';.:r'-_,::, ; : i!:,:-.'%~.,:;s !:..'':W!F . .:~L::,':-- -!:2," it- F 5! Th.: rL:T:[]:::.qr];,' - oT-*,. ,J-;!:":i : .!.1- ....- ; - -* ',., 1;:.[ ;;;:ti::l:t+ZI:;' *;:N:I:2 ;\' i::' :::: ;: : : ];:,:: ,- '.jt,,t~. -; I li.Jtit I~;}:;: ':FITH " 7,j :*' -,, !!- I .....;.r-..i1, '"i :t: t...,:i i ::.:.: t.;;^- :i ;;:. ' :: ' "/ t.:!':i-..1::;-, i j -! !:.,: F'-::t -.: .... :.. .....

-..... . . .............t ........ ...f......[-t. -* .. +..... ' 't--- ' c* .. _.._.... L.:....,;.' e_:. -_:_ .... r--_ 'I t.'-A .... =--' ......--- ' '---~ ... ...- ',----- '- '-'~_- ..,__i.'-.-:-'i -- -, j .''':: ... .!.:: -: :.'t': T --1 ' . . .;-:.I... . . ..... :.... i: *. . , i; .. t';' -.-..- i...-

1 7'! ' I I ... I 4~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ T1 :

:,it' :::: :,' -- ; .. : :::i:.: : i : ... -: j; ! -.,i----l-- ...--;[¢ i:' ' .: :::.i : ,.1.-::'-::'.:: :'::::m:i:::: ; :: : :::t:'.-:: 5~.-[:::,:!'::;'~ [::i::::;:::-]: :t::.::.:x...- :.:z~!t: : [:: ::: :.:.:

,' ......j ... + ......, .,.j..r..;'"..7, -- li'-"' 1 i -'RH i T -i ''i ,I '.' ' '. *\ ..... '; ....i'--''

4-i----i- . . :-. .;zitvt:U::-it.. T/. . '4 ... ., .+ ..:. .. . .... q ....st~~~~~~~ .... t,,~ .% - ' :<.i ' - :/--- -=', ; :/' r } - 1 , % . . . i - ' +, . ,- :- , * .... I= -" '~-i- -::--;'.- - .-:?=-S:-:znz"F- ....... ;i--' xt*i i; -i -,+ .t :----. ,.-: ;l.,i- ;t.:;i--:t:; _,- -';r;X; t= ,,, -4~~~~~~~~~~~ ....... :: " .... : E 4 , \.-.........' " '.... ,; ...

_z,,., r i . . ,, '-uz4xh~::'u:' -- '?' 'T-~-%.- %:-X; c xzx: FUiv "-~--'-'- Ft..4- i, .....

h1 4 ~~~~~~~~~-. .: <. ? ?i !j ;~ ]L: |. t ,r0:S~'

441; t;4t~~f zt J -ft'--- ~ r iE il~ii 4 ~ ----4 -;4-rH --.+ ____...4!=

irl ~~~~~~~~~

t~~~~~~~~~~~~~~~~~~

.%H-,.:- : --~-Z --. ::t_ ::. --. . ' %.'-~ --T- T.C. ::-z-7-'z-t..~ J:-l i! ,*--' T44' ~-- '' -'* -''' - --1-- ~-,~ * 4 :

<{{ ,~!] .. ............2 :I....____~~~~~~~~ % : A. :::T---'*%, :d~ '-f~

F[ ';'.]:-:~L tW i/L-": '!y:T4Tl; $7!;isL: FE-*--:i-'-~ '; {!H f- } :+ ''

j - r .j -.t L-I-i r a - U-I-, .4-:84 :'~ .... ~ -....:-.:1.-,I--=:':::- ~ 4......LA...i. ..... t-+-t..'"...

i~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.

4tttl.~.rI- I' '; -LLI2-4 [ ·L1--- - sM*-r4,-. ..... I......1 Z.~ ........ 4 _ TI....:...Z -r/Z . .. ~...l. ....... ,- ..

q

I.1iI

4I14.1

I

i

' ·'Z 47 r, ' h .: - ' ' ·' '' -. ·: ' .- · :..... .... .... ..-

p~ ~ ~ ~~~~~~~~~~::- .t.1,t-j--~-I;'.;'': '*....-..p. :i{::'-'*-':r *ri''-:'* t?:;-. ;It': z :; ±* igure. 4i -..erflicatior: statistics for :accumulation:-o-t?:-re ci tataron: {-[ ; -[':: -,: j.th:'e: fi[ st.24;hr2 'arDacr. than, one nch |- . . .

Li_2tiStXtUIH~· .. ........ 11 L':':....t-: .. .i,,'tLIl: J ....______ iitit+t---tt}=:-,,+ ::-'':::~ ::--; .m:' ___:__' hr" ......:..

.8.

7 2

000

.0.00

0 Ott

0000d

r..

:

.1

I

I

tiI

IIIII4

i4i

1

1

11I-1

I

-iI

1i

i

I

I

.... , I . . ............. -. - - - .- --. - - -- -_ _ . .

10

TST

Bias

C

c/b

TS

1

+0. 821

+0.222

- 0. 417

-0.799

Table of correlation coefficients

TS' Bias c/bC

1:.

-0. 374

-0. 688

-0. 9981

.1

+0.502

+0. 408

1.,+0.702 1

The low correlation, 0.222, between Bias and TS, and relatively high

correlations between c and TS and between c and Bias, support the notion

that the forecaster tends to include more misses than hits when he increases

the bias. Also, to the extent that this notion is correct, these correlationslend some credibility to the placement error as a valid concept. The remark-ably high correlation between c/b and TS' is explained by the fact that TS'

is a function of c/b only, if F >, Q, which was true for all years except1972, when F = 0. 988 Q. The departure of the correlation coefficient from

unity mostly reflects the departure of the function from linearity, whichis small because of the relatively small range of the numbers given (seethe curve, TS1 , Figure 1). Note the high negative correlation coefficientbetween c/b and TS. This points up the fact that there is far more to

be gained, in terms of threat score, by the forecaster reducing placement.error relative to the dimensions of Q, than by increasing the bias.

I also went to the daily NMC records for 24 hr accumulated precipitationamounts verifying at 1200 GMT for the first week of January 1979. The table

below shows a few of my more interesting calculations.

Sample of the daily record (1979)

AmountDate

Jan 1 Day 1

:(>) F. q

" 148.2 114.4 103" 3.3 1.5

H Bias TS TS' ,TS" c c/b

0.8 1.295 0.623 0.623 2.214 0.3670 2.200 0 -0.074 1.716 2.483

Jan 2 Day 1

Jan 3Jan 3Jan 3

Day 1Day 1Day 1

Jan 5 Day 1Jan 5 Day 2

Jan 6 Day 1

1" 59.53" 3.2

1 i22"11

311

1121 i2

III

61.58.4

0

51.7 39.;60 0

51.21.90.1

50.51.7

0

1.151 0.553 0.545 1.895X 0 -1 1.009

1.2014.421

0

0.8120.198

0

0.8410.110

-1

0.5481.0710.178

0.467co

0.1361. 3770. 998

10.5 2.6 0.2 4.038 0.016 -0.134 2.469 2.71417.1 2.6 0 6.577 0 -0.419 3.243 3.565

1.0 7.2 0.4 0.139 0.051 -0.153 1.569 1.036

TS

'4

r~

I., 0 - . -1 1 -.---- D .- 1- -._ A .1 W .... ,,. .~ I .1-1.1- . - ...__.

11

Note that TS' may be greater than TS, for example, in the case of theexcellent forecast on Jan 3 for Day 1, for the > ½" area. In that case,Figure 1 shows that the forecaster actually hurt his threat score, evenwith a bias as close to unity as 1.2. On the other hand, with a forecastnot quite as good, on Jan 1, Day 1, > ½1", TS' is the same as TS with theaccuracy shown, although the bias is higher, 1.3.

In conclusion, use of the modified threat score developed here appearsto be a good way of removing the effect of bias on threat score, and theplacement error appears to be related to skill in locating precipitation areas.

A--

i h:

i

U.S. DEPARTMEWBT OF COIiMMERCEFIational Oceanic and Atmospheric Administratiorn.NATIONAL WEATHER SERVICE

National 'Meteorological CenterWashington, D. C. 20233

March 5, 1980

TO:

FROM:

SUBJECT:

Distribution

OA/W3 - Frederick G. Shuman ~ I-

Correction to NMC Office Note 210

The iteration given on page 4 will not converge if the bias exactlyequals one (a = b). The problem is-the first guess given near thebottom of page 4. A better first guess is given on the correctedpage 4, attached.

Attachment

Distribution --

Mr. SaylorHarry BrownDr. PhillipsDave OlsonVern BohlBob HiranoDr. KleinDr. BonnerJerry LaRueDr. CressmanMr. BedientMr. Carlstead /Dr. WinstonDr. StackpoleDr. BrownDr. GlahnSSD, Western RegionDr. Gerrity

4

Determination of placement errorI

In Figure 2, His the area, b2a, of circular sector CAD; plus thearea, a2 6, of circular sector CBD; less the area of quadrilateral ACBD.Because of the symmetry of ACBD about line segment AB, its area is twicethe area, I ab sin y, of triangle ABC. But y = r - (a + ),thussin Y = sin (cc +6). Therefore

H = b 2ca + a2 6 - ab sin (cx+3) (2)

The cosine law gives

cos a = c2 - (a 2 -b 2)- (3)*2 b c

COS 6 - C2 + (a 2 - b2 )2acand therefore c and $ are functions of a, b, and c. The parametersa and b are given by

-n2

Q= rb 2

H is thus a transcendental function of c, and given F, Q, and H,(2) can be solved for c. The method I use, based on Newton's iteration, is

(;00 00 XHe 0:0j ; :- c -C= i [ bfcVair~ - xH l -0 cvj0 - 1 (4) 'b c+ 0 -1.

. -

:0 where a cos- (cv)2 _ (a 2 b2) :.... ~~~~~~~~~~~~~~~~~2 b c:¥

c -= C Fs cY b cosa a

and where v is the iteration count. As a flag to readers, the law of sineswas used in deriving (4):

a = b _ c_ . .0 t f 0 0 0 ; 000 tsin si n sin 'y

; As a first guess, I useG~~~~~~ (- :=o 'v)oVa-I i=:,_ + Sb-° = ) - A:[Corrected March 5, 1980]

There are three cases for which c cannot be determined with my modelof circles, without further assumptions. They are when the circles do notintersect: