a superior alternative to the modified heidke skill score for verification of categorical versions...

A Superior Alternative to the Modified Heidke Skill Score for Verification of Categorical Versions of CPC OutlooksBob LivezeyClimate Services Division/OCWWS/NWS

28th Climate Diagnostics and Prediction Workshop

Reno, October 20, 2003

OutlineIntroduction2. Contingency Tables & NotationCommon Scores & Score AttributesGandin & Murphy Equitable ScoresGerrity Scores Recommendations

Contingency Tables and Notationpij: Joint relative frequenciespi:: Observed relative frequenciesqi: Forecast relative frequenciespi*: Prescribed relative observed frequencies (climatology)Table 4.2. Contingency table giving pij in percent (total sample size n=788) for U.S. mean temperature forecasts for June through August 1983-90.

Seasonal Mean Temperature Observed

Forecast

Below Normal

Near Normal

Above Normal

Forecast Dist.

Below Normal

3

8

4

15

Near Normal

8

13

18

39

Above Normal

7

14

25

46

Observed Dist.

18

35

47

100

Some Simple Categorical Skill Scores: Heidke, CPC Heidke, and Pierce

Some Simple Categorical Skill Scores: Heidke, CPC Heidke, and PierceSkill scores for U.S. mean temperature forecasts in three categories for February through April and June through August 1983-90.

Desirable Attributes of ScoresEquitable;Equitable without dependence on the forecast distribution;Rewards for correct forecasts inversely proportional to their event frequencies;Penalties for incorrect forecasts directly proportional to their event frequencies;Penalties for incorrect ordinal forecasts with equal event frequencies proportional to degree of miss;Consistent with an underlying linear association and insensitive to type or number of categories used.

Note: 2 &3 imply that all information in the contingency table is taken into account.

Figure 4.1

Chart3

-0.98-0.04-0.32-0.04-1

-0.72-0.16-0.2-0.12-0.9

-0.57-0.23-0.16-0.12-0.8

-0.49-0.22-0.15-0.11-0.7

-0.42-0.2-0.13-0.1-0.6

-0.34-0.17-0.115-0.09-0.5

-0.27-0.14-0.1-0.08-0.4

-0.2-0.1-0.08-0.06-0.3

-0.14-0.08-0.06-0.04-0.2

-0.08-0.04-0.04-0.02-0.1

00000

0.060.040.030.020.1

0.120.080.060.040.2

0.190.130.090.080.3

0.260.180.120.120.4

0.330.240.170.160.5

0.40.30.220.20.6

0.490.380.30.260.7

0.590.480.380.340.8

0.730.620.540.50.9

0.980.960.960.961

2-class

3-class

4-class

5-class

One-to-one

CORRELATION SCORE

NWS SKILL SCORE

Figure1

abcd

CorrelationHeidkeHeidkeHeidkeHeidke

-1-0.98-0.32-0.04-0.04-1

-0.9-0.72-0.2-0.16-0.12-0.9

-0.8-0.57-0.16-0.23-0.12-0.8

-0.7-0.49-0.15-0.22-0.11-0.7

-0.6-0.42-0.13-0.2-0.1-0.6

-0.5-0.34-0.115-0.17-0.09-0.5

-0.4-0.27-0.1-0.14-0.08-0.4

-0.3-0.2-0.08-0.1-0.06-0.3

-0.2-0.14-0.06-0.08-0.04-0.2

-0.1-0.08-0.04-0.04-0.02-0.1

000000

0.10.060.030.040.020.1

0.20.120.060.080.040.2

0.30.190.090.130.080.3

0.40.260.120.180.120.4

0.50.330.170.240.160.5

0.60.40.220.30.20.6

0.70.490.30.380.260.7

0.80.590.380.480.340.8

0.90.730.540.620.50.9

10.980.960.960.961

Figure1

2-class

3-class

4-class

5-class

One-to-one

CORRELATION SCORE

NWS SKILL SCORE

Figure2

Leps score

Correlation3-class5-class

-1-0.7-0.64-1-0.06

-0.9-0.62-0.57-0.9-0.05

-0.8-0.54-0.52-0.8-0.02

-0.7-0.48-0.46-0.7-0.02

-0.6-0.42-0.4-0.6-0.02

-0.5-0.34-0.33-0.5-0.01

-0.4-0.28-0.27-0.4-0.01

-0.3-0.2-0.2-0.30

-0.2-0.14-0.14-0.20

-0.1-0.08-0.08-0.10

00000

0.10.060.060.10

0.20.120.120.20

0.30.20.20.30

0.40.280.280.40

0.50.350.350.50

0.60.440.440.60

0.70.520.520.70

0.80.620.620.80

0.90.740.740.90

10.980.9810

Figure2

3-class

5-class

One-to-one

CORRELATION SCORE

LEPS SCORE

Figure2 Jacky

5-class

Difference

One-to-one

LEPS 3-class skore

LEPS 5-class score

Difference (3class - 5class)

Figure 3

r2 classes3 classes4 classes5 classes

-1-1-0.6641666667-0.6666666667-0.601845

-0.9-0.718-0.5913666667-0.5633666667-0.5432

-0.8-0.599-0.5316166667-0.5034833333-0.48282

-0.7-0.5002-0.4650666667-0.4383333333-0.42591

-0.6-0.4186-0.3961166667-0.3779666667-0.366345

-0.5-0.339-0.3310166667-0.3148333333-0.306665

-0.4-0.271-0.2641666667-0.2554333333-0.24654

-0.3-0.2056-0.1970166667-0.1935-0.18683

-0.2-0.1356-0.1326166667-0.12725-0.123315

-0.1-0.0686-0.0642166667-0.0642333333-0.06417

0-0.00680.0005333333-0.00165-0.000825

0.10.06040.07148333330.06606666670.06227

0.20.12540.13803333330.13383333330.13278

0.30.1950.20223333330.206050.199705

0.40.26340.27388333330.27966666670.272235

0.50.32880.34698333330.349550.35403

0.60.40920.43248333330.43246666670.43455

0.70.48680.52073333330.52273333330.52369

0.80.58840.62193333330.627250.62699

0.90.72020.73318333330.74953333330.75496

110.998333333311.00378

Figure 3

2-class

3-class

4-class

5-class

One-to-one

CORRELATION SCORE

LEPS SKILL SCORE

-10.25

-0.5-0.5

00

1-1

-10.250.25-1-0.5-0.5

-0.5-0.500-0.50

1-1-1100

0-0.5

-0.5-0.5

Gandin and Murphy Equitable Scores

A scoring matrix

, is used to define a general form of a skill score using the contingency table:

Conditions for equitability and scale of score:

ADVANCE \u 18;

ADVANCE \u 16

Symmetry for S:

Correct forecast reward greater than or equal to incorrect forecast one:

;

n-class error penalty less than or equal to n+1-class error penalty:

;

For three by three tables this determines all but two sij , for symmetric categorizations all but one.

_974258201.unknown

_1076861518.unknown

_1076862210.unknown

_1076862274.unknown

_1076861928.unknown

_1076861209.unknown

_974258196.unknown

_974258200.unknown

_974258195.unknown

Gerrity Scores

The Gerrity scores are one formalization to rationally determine the ndetermined Gandin and Murphy reward/penalty coefficients:

The elements of S are then given by

Recall that all Gandin and Murphy scoring matrices (including these) are symmetrical. Note also that the summation entry for those cases above when the upper index is less than the lower is zero. Finally observe in that ADVANCE \d 6

ADVANCE \u 6 always.

_1076865978.unknown

_1076866555.unknown

_1076866666.unknown

_1076866195.unknown

_974258238.unknown

Event Probabilities (p1,p2,p3,)

(0.33,0.33,0.33)

(0.3,0.4,0.3)

ADVANCE \d 2

ADVANCE \u 2

ADVANCE \d 2

ADVANCE \u 2

Gandin and Murphy (1992) ADVANCE \d 28 ADVANCE \d 28

ADVANCE \u 28

ADVANCE \d 2

ADVANCE \u 2 ADVANCE \d 2

ADVANCE \u 2

Gerrity (1992) ADVANCE \d 27

ADVANCE \u 27 ADVANCE \d 28

ADVANCE \u 28

Potts et al. (1996)

Equitable scoring matrices for three-category forecasts with two different event probabilities.

EMBED Equation.COEE2 \* MERGEFORMAT \s












_1076879339.unknown

_1076879348.unknown

_1076879354.unknown

_1076879356.unknown

_1076880988.unknown

_1076879350.unknown

_1076879344.unknown

_1076879346.unknown

_1076879342.unknown

_1076879335.unknown

_1076879337.unknown

_1076879333.unknown


(0.5,0.3,0.2) (0.2,0.5,0.3)

k1 = -0.5,k2 = -0.25 k1 = -0.5,k2 = -0.25

Gandin and Murphy (1992)

ADVANCE \u 29 ADVANCE \d 29 k1 = -0.375,k2 = 0.0 k1 = -0.286,k2 = -0.375

ADVANCE \u 29Gerrity (1992)

ADVANCE \u 29 Equitable scoring matrices for three-category forecasts with two different event probabilities.


(0.33,0.33,0.33)

(0.3,0.4,0.3)

ADVANCE \d 2

ADVANCE \u 2

ADVANCE \d 2

ADVANCE \u 2


_974258285.unknown

_974258289.unknown

_1076879342.unknown

_974258288.unknown

_974258284.unknown

Figure 4.4

Chart5

-1-0.6641666667-0.6666666667-0.601845-1

-0.718-0.5913666667-0.5633666667-0.5432-0.9

-0.599-0.5316166667-0.5034833333-0.48282-0.8

-0.5002-0.4650666667-0.4383333333-0.42591-0.7

-0.4186-0.3961166667-0.3779666667-0.366345-0.6

-0.339-0.3310166667-0.3148333333-0.306665-0.5

-0.271-0.2641666667-0.2554333333-0.24654-0.4

-0.2056-0.1970166667-0.1935-0.18683-0.3

-0.1356-0.1326166667-0.12725-0.123315-0.2

-0.0686-0.0642166667-0.0642333333-0.06417-0.1

-0.00680.0005333333-0.00165-0.0008250

0.06040.07148333330.06606666670.062270.1

0.12540.13803333330.13383333330.132780.2

0.1950.20223333330.206050.1997050.3

0.26340.27388333330.27966666670.2722350.4

0.32880.34698333330.349550.354030.5

0.40920.43248333330.43246666670.434550.6

0.48680.52073333330.52273333330.523690.7

0.58840.62193333330.627250.626990.8

0.72020.73318333330.74953333330.754960.9

10.998333333311.003781

2-class

3-class

4-class

5-class

One-to-one

CORRELATION SCORE

LEPS SKILL SCORE

Figure1

abcd

CorrelationHeidkeHeidkeHeidkeHeidke

-1-0.98-0.32-0.04-0.04-1

-0.9-0.72-0.2-0.16-0.12-0.9

-0.8-0.57-0.16-0.23-0.12-0.8

-0.7-0.49-0.15-0.22-0.11-0.7

-0.6-0.42-0.13-0.2-0.1-0.6

-0.5-0.34-0.115-0.17-0.09-0.5

-0.4-0.27-0.1-0.14-0.08-0.4

-0.3-0.2-0.08-0.1-0.06-0.3

-0.2-0.14-0.06-0.08-0.04-0.2

-0.1-0.08-0.04-0.04-0.02-0.1

000000

0.10.060.030.040.020.1

0.20.120.060.080.040.2

0.30.190.090.130.080.3

0.40.260.120.180.120.4

0.50.330.170.240.160.5

0.60.40.220.30.20.6

0.70.490.30.380.260.7

0.80.590.380.480.340.8

0.90.730.540.620.50.9

10.980.960.960.961

Figure1

2-class

3-class

4-class

5-class

One-to-one

CORRELATION SCORE

NWS SKILL SCORE

Figure2

Leps score

Correlation3-class5-class

-1-0.7-0.64-1-0.06

-0.9-0.62-0.57-0.9-0.05

-0.8-0.54-0.52-0.8-0.02

-0.7-0.48-0.46-0.7-0.02

-0.6-0.42-0.4-0.6-0.02

-0.5-0.34-0.33-0.5-0.01

-0.4-0.28-0.27-0.4-0.01

-0.3-0.2-0.2-0.30

-0.2-0.14-0.14-0.20

-0.1-0.08-0.08-0.10

00000

0.10.060.060.10

0.20.120.120.20

0.30.20.20.30

0.40.280.280.40

0.50.350.350.50

0.60.440.440.60

0.70.520.520.70

0.80.620.620.80

0.90.740.740.90

10.980.9810

Figure2

3-class

5-class

One-to-one

CORRELATION SCORE

LEPS SCORE

Figure2 Jacky

5-class

Difference

One-to-one

LEPS 3-class skore

LEPS 5-class score

Difference (3class - 5class)

Figure 3

r2 classes3 classes4 classes5 classes

-1-1-0.6641666667-0.6666666667-0.601845

-0.9-0.718-0.5913666667-0.5633666667-0.5432

-0.8-0.599-0.5316166667-0.5034833333-0.48282

-0.7-0.5002-0.4650666667-0.4383333333-0.42591

-0.6-0.4186-0.3961166667-0.3779666667-0.366345

-0.5-0.339-0.3310166667-0.3148333333-0.306665

-0.4-0.271-0.2641666667-0.2554333333-0.24654

-0.3-0.2056-0.1970166667-0.1935-0.18683

-0.2-0.1356-0.1326166667-0.12725-0.123315

-0.1-0.0686-0.0642166667-0.0642333333-0.06417

0-0.00680.0005333333-0.00165-0.000825

0.10.06040.07148333330.06606666670.06227

0.20.12540.13803333330.13383333330.13278

0.30.1950.20223333330.206050.199705

0.40.26340.27388333330.27966666670.272235

0.50.32880.34698333330.349550.35403

0.60.40920.43248333330.43246666670.43455

0.70.48680.52073333330.52273333330.52369

0.80.58840.62193333330.627250.62699

0.90.72020.73318333330.74953333330.75496

110.998333333311.00378

Figure 3

2-class

3-class

4-class

5-class

One-to-one

CORRELATION SCORE

LEPS SKILL SCORE

-10.25

-0.5-0.5

00

1-1

-10.250.25-1-0.5-0.5

-0.5-0.500-0.50

1-1-1100

0-0.5

-0.5-0.5

RecommendationsCPC use the Gerrity score for ordinal multi-categorical verificationForecast history is digitized so skill history can be constructedClueless audience remains cluelessScore now equitably accounts for all facets of forecast performanceCPC use actual frequenciesCPC routinely determine confidence limits of scoresReference Jolliffe and Stephenson (2003; Wiley)

a superior alternative to the modified heidke skill score for verification of categorical versions...

Documents