An Experiment to Evaluate the Use of An Experiment to Evaluate the Use of Quantitative Precipitation Forecasts from Quantitative Precipitation Forecasts from

Numerical Guidance by Operational Numerical Guidance by Operational ForecastersForecasters

Joshua M. Boustead and Daniel NietfeldJoshua M. Boustead and Daniel Nietfeld

NOAA/NWS WFO Omaha/Valley, NENOAA/NWS WFO Omaha/Valley, NE

Ray Wolf Ray Wolf

NOAA/NWS WFO Davenport, IANOAA/NWS WFO Davenport, IA

Presentation Overview

• Study purpose and methodology

• Data results– Survey results– Snowfall forecast– Watch/warning statistics – Gridded forecast results

• Forecasting implications, conclusions, and future work

Study Motivation

• Strong interest in the role of the future forecaster– Can we still add value to the everyday

forecast?– How can we better concentrate on high-

impact weather?– How can we better utilize increasingly high-

tech tools into the everyday forecast?• How does this increasingly high-tech information

affect the forecaster?

ExampleNSSL 4km WRF 00Z 8/14/07

Results

Study Purpose

• To evaluate if and how operational forecasters are biased by numerically generated quantitative precipitation forecasts (QPF)

• Use these results to develop an updated methodology for operational forecasters on how to approach a daily forecast and utilize the latest technology, including high resolution model output

Study Methodology

• Utilizing the National Weather Service’s (NWS) Warning Event Simulator (WES) operational forecasters from two NWS offices made two forecasts for two winter weather case– The forecasters first completed the forecast,

including making a warning decision, without the use of model QPF

– The forecasters then went through the same case again with model QPF, again making a snowfall forecast as well as a warning decision

• Once each scenario was completed, the forecasters completed a survey about the specific case

Study Methodology

• Two winter weather cases were chosen from the Central and Northern Plains– December 7-8, 2005

from the Pleasant Hill, MO (EAX) forecast area

– February 28 – March 1, 2004 from the Bismarck, ND (BIS) forecast area

Survey ResultsDistribution of Forecaster Experience

0%

10%

20%

30%

40%

50%

60%

0 to 3 yrs 3 to 10 yrs 10 to 20 yrs 20 + yrs

Years of Operational Forecasting

Perc

en

t o

f F

ore

caste

rs

• Forecaster Demographics:– Forecasters were from the NWS offices in

Omaha/Valley, NE and Davenport, IA– Operational forecasters involved were of a high

experience level

Survey Results

• Forecaster confidence without using model QPF:– Majority of operational forecasters felt confident in making a

forecast without model QPF– Potentially due to the high experience level of the forecasters

Forecaster Confidence Level without Using QPF

0%

10%

20%

30%

40%

50%

60%

Very Confident Confident Neutral Unsure Very Unsure

Survey Results

• Forecaster confidence after seeing QPF:– Most forecasters indicated that seeing QPF either

increased their forecast confidence or it was unchanged

Change in Forecaster Confidence with QPF

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Sig increased Increase Unchanged Decreased Sig decreased

Snowfall Forecast Results

• MAE was computed for each location and then averaged for before and after the use of QPF

• MAE decreased 0.5 inches for both the EAX and BIS case post QPF

Pre and Post QPF Mean Absolute Error for EAX and BIS

00.5

11.5

22.5

3

3.54

4.55

Pre and Post QPF EAX Pre and Post QPF BIS

Inch

es Pre-QPF

Post-QPF

Snowfall Forecast Results

• Majority of the forecasts were unchanged post QPF

• Majority of the forecasts that did change their forecast, increased accuracy

Distribution of Change in Forecast Accuracy with QPF among Forecasters for EAX

34%

29%

37% Improved Degraded

Unchanged

Distribution of Change in Forecast Accuracy with QPF among Forecasters for BIS

38%

26%

36% Improved Degraded

Unchanged

Warning Results

• The probability of detection (POD) and false alarm ratio (FAR) were computed by county for each of the forecast areas

• Forecasters showed improvement in both the POD and in FAR ratio once QPF was used

Warning Statistics

0%

10%

20%

30%

40%

50%

60%

70%

80%

Pre-QPF POD and FAR Post-QPF POD and FAR

POD

FAR

Warning Results

Distribution of Change in Probability of Detection Accuracy with QPF for BIS

30%

13%

57%

Improved Degraded

Unchanged

Distribution of Change in False Alarm Ratio Accuracy with QPF for BIS

29%

25%

46%

Improved Degraded

Unchanged

Distrubution of Change in Probability of Detection Accuracy with QPF for EAX

29%

21%

50%

Improved Degraded

Unchanged

Distrubution of Change in False Alarm Ratio Accuracy with QPF for EAX

33%

22%

45%

Improved Degraded

Unchanged

Gridded Forecast ResultsEAX Case

• EAX Pre and Post QPF MAE– Forecasters had

the most confidence in the northern CWA

– Much better agreement over the southern CWA post QPF

– Also a 2 to 3 inch decrease in MAE over the south

Gridded Forecast ResultsEAX Case

• EAX Pre and Post QPF Standard Deviation– Forecast

differences decreased over the north and south

– Slight increase in differences over the center

Gridded Model ForecastsEAX Case

• Greatest agreement of snow band across central CWA

• Viewing QPF increased the forecast confidence in the southern CWA

Actual SnowfallEAX Case

Gridded Forecast ResultsBismarck Case

• BIS Pre and Post QPF MAE– Good agreement

and low error over the northwest forecast area

– Mean errors of 5 to 6 inches over the southern and eastern forecast area

Gridded Forecast ResultsBismarck Case

• Pre and Post QPF Standard Deviation– Significant

increase in forecaster clustering across the central forecast area

– Greater than 4 inch differences continue over the southern forecast area

Gridded Model ForecastsBIS Case

• Models agree northwest CWA to get least QPF

• Larger uncertainty in the south

• Forecasters tended to pick a model– Led to continued

large MAE in the southern CWA

Actual SnowfallBIS Case

Discussion• Only a slight improvement in snowfall forecasts was

noted once forecasters viewed QPF– When snowfall forecasts were modified, a higher percentage

were improved than degraded• Model QPF seemed best utilized to resolve snow-no

snow areas– This led to improvements in both FAR and POD

• High MAE did not always mean high standard deviation, which can indicate a systematic forecasting error

• Doesn’t clearly answer the question does model QPF bias forecasters– Some evidence in the BIS case where model agreement was

poor• Possible forecast methodology

– Make entire forecast without QPF– Utilize QPF for placement for defining snow-no snow areas

Future Work

• Conduct the study using two warm season convective cases

• Investigate forecaster philosophy from the surveys where standard deviation is low and mean absolute error is higher

• Investigate what, if any, synoptic patterns increase forecaster uncertainty and MAE

• Continue to increase the number of forecasters in the study, and from different areas of the CONUS