ARROWS EVALUATION

WA.RD 608.1

Research ReportFebruary 2005

^-rrf-utt Waehington StatcDopartmcnt of llansportation

Washington State Transportation CommissionPlanning and Capital Program Managementin cooperation with:U.S. DOT - Federal Highway Administration

Final Report

ARROWS EVALUATION

by

Larry Senn Ed BosellyRWIS Engineer Road Weather Program Manager

AdvancedTechnologyBranch MaintenanceandOperationsWashington State Department of Transportation

Washington State Department of TransportationTransportation Building

Olympia, Washington 98504

Prepared for

Washington State Transportation CommissionDepartment of Transportation

and in cooperation withU.S. Department of Transportation

Federal Highway Administration

February2005

TECHNICAL REPORT STANDARD TITLE PAGE 1. REPORT NO. 2. GOVERNMENT ACCESSION NO. 3. RECIPIENT'S CATALOG NO.

WA-RD 608.1

4. TITLE AND SUBTITLE

ARROWS EVALUATION

7, AUTHOR(S)

Larry Senn and Ed Boselly

9, PERFORMING ORGANIZATION NAME AND ADDRESS

Washington State Transportation Center (TRAC) University of Washington, Box 354802

S. REPORT DATE

February 2005 6. PERFORMING OROANIZI\TION com;

8. PERFORMING ORGANIZATION REPORT NO.

IO. WORK UNIT NO.

11. CONTRACT OR GRANT NO.

University District Building; 1107 NE 45th Street, Suite 535 Seattle, Washington 98105-4631 12. SPONSORING AGENCY NAME AND ADDRESS

Research OfficeWashington State Department of TransportationTransportation Building, MS 47372Olympia, Washington 98504-7372Doug Brodin, Project Manager, 360-705-797215. SUPPLEMENTARY NOTES

13. TYPE OF REPORT AND PERIOD COVERED

Research Report

14, SPONSORING AGENCY CODE

This study was conducted in cooperation with the U.S. Department of Transportation, Federal Highway Administration.

I6. ABSTRACT

In a June 2002 meeting, winter maintenance managers at the Washington State Department of

Transportation (WSDOT) said that more accurate forecasts, that told not only where but when inclement

weather was due, would be a valuable tool to aid them in keeping Washington State highways safe and

passable during winter months. In response, the WSDOT ITS Office teamed with the University of

Washington Department of Atmospheric Sciences to build weather information and forecasting Web

pages specifically tailored to the managers' desires and needs.

The result was the development of the ARROWS (Automated Real-time Road Weather System)

Web pages, made available to all WSDOT winter maintenance managers for the 2003-2004 winter

season. WSDOT subsequently conducted an in­house survey to determine what ARROWS users thought

about the usefulness and presentation of information. This report presents the results of that survey,

which have been used as input to the continuing development of ARROWS.

17. KEY WORDS

Roadway maintenance, winter maintenance, road weather information systems, weather forecasts

18. DISTRIBUTION STATEMENT

No restrictions. This document is available to the public through the National Technical Information Service, Springfield, VA 22616

19, SECURITY CL ASS!F. (of this report) 20. SECURITY CLASS1P. (of this page) 21. NO, OF PAGES 22. PRICE

None None

DISCLAIMER

The contents of this report reflect the views of the authors, who are responsible

for the facts and the accuracy of the data presented herein. The contents do not

necessarily reflect the official views or policies of the Washington State Department of

Transportation. This report does not constitute a standard, specification, or regulation.

lll

IV

CONTENTS

The ARROWS Web Pages .......

Sources of ARROWS Forecasts

User Acceptance ........

Sources of Weather Information.....

Summary

I

7

8

ARROWS Features Most Liked

ARROWS Features Most Disliked.............

ARROWS Features Needing Change

Rankings of ARROWS Output Parameters

ARROWS' Usefulness and Continuation...

New Features for Development

Additional Comments

10

12

11

13

t4

15

t6

t7

18

V

Figure

F'IGURES

Sources of weather information other than ARROWS..

Features most liked by ARROWS users..,...

Features of ARROWS disliked by users....

Items users want changed in ARROWS...............

Rankings of ARROWS output parameters

Page

11

l2

13

t4

15

t6

2

J

4

5

6

9

J

4

I Sample ARROWS page showing the statewide precipitation andpavement temp erature forecast

2 State view of "Air Temperature/Wind" page.......

State view of Warnings page

Close up of Org.code 425220 showing areas that have warnings......

5 Doppler radar image .....

6 Number of survey responsos by WSDOT region

7 Number of survey rosponses by job title 10

8

9

10

11

t2

13 Overall ARROWS evaluation from user perspective....

V1

ARROWS EVALUATION

In a June 2002 meeting, winter maintenance managers at the Washington State

Department of Transportation (WSDOT) said that more accurate forecasts, that told not

only where but when inclement weather was due, would be a valuable tool to aid them in

keeping Washington State highways safe and passable during winter months. In response,

the WSDOT ITS Office teamed with the University of Washington Department of

Atmospheric Sciences to build weather information and forecasting Web pages

specifically tailored to the managers' desires and needs.

The result was the development of the ARROWS Web pages, made available to

all WSDOT winter maintenance managers for the 2003-2004 winter season. WSDOT

subsequently conducted an in-house survey to determine what ARROWS users thought

about the usefulness and presentation of information. The results of that survey, which

have been used as input to the continuing development of ARROWS, are presented here.

THE WEB PAGES

ARROWS is a Web-based graphical presentation of weather conditions (see

Figure 1). It also provides links to other useful sites to aid winter maintenance managers

in anticipating weather conditions.

I

Alert Bu fm :Washhaton State

Tfr'- Tfi*jg-sg.tg-

nt0LF.&:gjll!ilI0484:la&!nx

:U!A

Lodel Folffil.

SlcTemo/Recio

AirTeftblwind

Warninos

Curst Data

Obs6ryaiions

Red6r

g.t6llits

Camerds

Olhq Ptoviderr

Nll Wsalh6r Svc

NWAC

N\dWaalherNot

Air Ou;litv

Orqco(rt:Im[: avo rit(]s

@lWashinrlton Slat.l

Wrshinr,{or St.rle D0I;A =R =iR

C -.ilt S

I ore{:asloal {}rdl0re ard

lrir,rr r-.n

r:l.i- I',:-: r lL I

Figure L: sample ARRows page showing the statewide precipitation andpavement temperature forecast

Forecasts are provided in 4-hour increments, and three views of each forecast are

available for each period. The preferred view is selected from the left hand menu, while

the time period is selected from the top menu.

The "surface temperature and precipitation" view indicates the forecast pavement

temperature with color coding, in which green indicates forecast temperatures over 37oF,

yellow between 37oF and 33oF, and red under 33'F. Forecast precipitation is also

indicated by color: various shades of green indicate rain and shades of gray indicate

snow. Deeper shades indicate more intense precipitation. Red is also used to show areas

of pos s ible fr eezing r ain.

2

Similar to the first page, the air temperature and wind page (see Figure 2) reports

forecast air temperatures by color: yellow indicates temperatures between 37'F and 33"F

and red indicates temperatures below 33'F. The standard meteorological symbols are

used to indicate forecast wind direction and speed.

Figure 2: State view of "Air TemperatureAVind" page

The "Warnings" screen (see Figure 3) relays statewide weather information, along

with specific warning notices addressed to the local maintenance unit regarding

conditions that may affect its work.

In winter these warnings are for snow, frost, freezing fog, or other weather that

will adversely affect the highways. In the example in Figure 3, the summer warnings are

3

= SW t5mh

eFe:llflaslsese:$3!tls&Io&:{1!Aslsesdr:{1lgsl2eFo&:{1g1g

:WA

:{A@snffftro&:{5110st?04

e[o{.:gillq

l{odel Fo,ec6li

SfcTemo/Precio

Air TemolWind

Warninos

Culetrt Dda

Obee*alions

Badar

Saiellite

Cameras

Othsr Providor*

Ntl \y'eather Svc

NWAC

NWWeathsrNet

Air Qualitv

;A. n :l?1, C -_,1, SWnihinqton Slntc DOT

lorecdsted Air I re and lrvitrd

Orqcodo

[@I dvoritesrfiffidi@t

Washinqton Stdte

l,i r,r r- | r rji:rrrril,r, :trl:FiL I :i'il lj iir rlr, rilr l,r Fnt t :t'L_,a ! r,n r r ,,t1

for high temperatures, which is important for some paving and pesticide application

tasks.

llodel Forecast8

9fc Teroo/Frecio

Air Temotu/ind

Wouq:

Curent Data

ObseNations

Fadar

gatellite

Other Providerr

tllllWedllrer Svc

NV/AC

Nll WealherNet

Figure 3: State view of Warnings page

Users can narrow the display to their immediate area by entering their WSDOT

organrzation code ("org.code"), which identifies a maintenance group responsible for a

particular area (see Figure 4). The use of the WSDOT organization code for navigation

was desired by the attendees of the 2002 meeting. However, this method of navigation

would make the use of ARROWS difficult for non-WSDOT personnel.

4

rUfol?Prd 12P[tIo4FM 4PMIpgl!4 8ffdTo12ru t?STo4An 4StToBStn rfi [r\lr I t. I I.l

Alert Bu for :Washington State

eFo& :{ts!on28e$.&:ltstDsl?5A 28

s90s2sst 181

5tI0rNTEI

ks..e:t@JRITSRT{

sn?6

sR90

sf, 170

snl?l5\262

ftFo& :!A&l

Orgcode

[@favorites

@WashinUlon State

Univor6ily of Woshington

A. =?:ll. c'-'ll s

S3rnrng: lnf rr'Sjhrndr,n itilrAlE i I g DLr li Ati Tn a!9 '1 I 1l Lr[ in Ftl

Gorrgle-

::. r,lr. La.nr,r_

Qe-r' ,*) i1]

Ilmpdew-ptpaE_t

I precip

=o

IauHa

4n

20

AM8nt

Lr&tlg

oa

!i

IPl\,1

8H

1

Texl Forecast

E1

Forecast for I90 MP 169

,1

rlI

Mlltlort Emdor

168.71 11i 1

7 8 S1011121 2 3 4 5 6 7 IAMsfl

Niu Ey W.rdig.Augll mr00 AItlTotustl l2{0 mI

SRMP Wsn$C

!!9 Hj$rq

Rodc

90

1!! Hidrrqj.C.I tliA]qE! Hjdr rq!!9 lsd|rq.110 HiarqU.! IfiA rry

L*rE re:!AagsR Itsf,n3Rmtl t3tnm5n&rR8ete:@5B It$nilsnrtDn3ntil5R ltt

it5520,-!

:lallS

'!l tabular2 Microsoft lnlerlret txptorera,sldtowarningFrane il

Orqcode

Vali.l tor:Io Au( 31 l2::16:J6

llodel Fqccorlr

Slc Temo/Precio

Air Te m olwi n d

Wanihos

Currcrl Dala

ObseNdlions

Radar

Sat6llit6

Cameras

Olher P.ovilerJ

Ntl W€.lh6r Svc

NU/AC

N\tWeatherNet

Figure 4: Close up of Org.code 425220 showing areas that have warnings (red dots)

Clicking on a location brings up a graphical forecast for air temperature, road

temperature, and dewpoint. The forecast is also available in tabular form.

These tools are intended to give maintenance managers a way to anticipate the

occurrence of freezing pavement or air temperatures, snow or rain, frost, and transitions

between rain and snow. These are all considerations for managers deciding on a course

of treatment. Accurate forecasts have the potential to improve the effectiveness of snow

and ice treatments, which will result in a safer roadway while reducing labor and

materials costs. Reducing material usage also reduces the amount of pollutants released

into the environment.

5

Cument observations from WSDOT weather sites are available, as well as a

composite weather doppler radar showing the entire state (see Figure 5). The composite

image is generated at the UW Department of Atmospheric Sciences. A loop of radar

images covering the last 3 hours is also available, allowing users to see the track of a

storm front or precipitation as it moves across the state.

Figure 5: Doppler radar image

In addition to the forecasts, ARROWS provides useful links to outside sources for

additional information. A link to the local National Weather Service (NWS) gives direct

access to several satellite images from visible light, infrared, and water vapor images.

Several of these products also can be viewed as a loop showing movement over the last

6

l?dMTr4PV 4FliTogPM sFtolrol?ru8tstsAlert Bu for :Washington State

lN*lr:al.-sEeieheaJlblhYEd

hr,ii'iriiit]upruiuiriorus

lPmurlmnt

lN;;Ri.lil*]meucxre roumrr 8:00 AM

A:55 AM

a:OO AM

S:55 AM

8:63 AM

8:{ AM

Si35 AM

8:OO AM

.a:OO AM

S:OO AM i

!,oo eu i

.8:4

A:31

8:15

.8:00

8:00

lpaurqor

lerr"!

lB.inbidebr.nd

]t*je!ruii pms

irt"n&"

Hodel Forecasls

Slc Temo/Precio

ArrTemoAVind

Warninos

Current Ddld

Observaiions

E{d-al

Satellile

Camerds

Olher ProYiders

Nll Wealher Svc

NWAC

!\qld!!ltE4lc!

Air Oualiv

Ws!hington Stato DOT;A=R=?CWS

Currcnt (}bieryatr.'n. I n.

Washrnalon Stak:Washinqton State

Favorites

@

Orqcode

il@

few hours. Other links connect to the Northwest Avalanche Center and to weather

consultants under contract with WSDOT.

SOURCES OF' ARROWS FORECASTS

ARROWS is driven by a group, or ensemble, of MM5 (Mesoscale Model 5)

forecasts, which are run twice a day. Seventeen different forecast runs make up the

ensemble. Each forecast uses initial and boundary conditions from a large-scale global

model. Eight of the forecasts use different large-scale models for initial conditions; one

forecast uses the average of the eight initial conditions (called the "centroid"); and eight

other forecasts use the same eight initial conditions but each.with a different set of

physical parameters within in the MM5 model.

There is a trade-off between running a high resolution MM5 forecast and running

many different forecasts in an ensemble at lower resolution. Insufficient computer power

was available to do both, so the team chose to use the ensemble for ARROWS. This

ensemble of forecasts has several advantages over using one high resolution model. First

of all, it can continue running even if one or more of the global models fails to provide

initial conditions on a given day. Second, the ensemble gives the ability to identify how

likely a forecast is to be accurate. If all the ensemble members predict the same thing,

that forecast is more likely to be true than if the ensemble forecasts vary widely. The

measure of how close the ensemble members are to one another is called "spread." A

high spread indicates low confidence in the forecast, and a low spread equates to greater

confidence. While this assumption does not always prove correct, there is a definite

correlation between spread and forecast skill.

7

USER ACCEPTAI\CE

In general, field personnel reported that they like ARROWS, want continued

access to ARROWS, and would like the development process to continue.

As people began to use ARROWS they identified a few deficiencies in the data

displays. The researchers at the University of Washington quickly installed fixes. These

included adding road detail and geographic references to the map displays for ease in

location registration, adding aradar data intensity scale, and adding date time groups to

surface observation tables.

A training course was developed to help potential users understand ARROWS,

how it was created, and how to use it. The training was conducted in group sessions or as

a hands-on session in a maintenance facility. Although the course was originally an

informal training session, it is now a formal course in WSDOT's training system

ARROWS performed well during the first portion of the 2003-2004 winter. It

correctly forecast an early snowfall in the Cascades Mountains in September. In late

October, ARROWS also correctly forecast a no-snow situation in the Puget Sound area,

which became the basis for a television interview with a WSDOT highway maintenance

crew. The crew touted the benefits of having an accurate forecast to avoid unnecessary

personnel callouts and associated costs. Shortly thereafter the Road Weather Program

Manager issued a mild alert to ARROWS users to not expect the same sort of success all

of the time and reminded them that ARROWS continues to be only one of the "tools in

the toolbox."

Unfortunately, in early January 2004 ARROWS suffered the same problems that

all other meteorological models faced at that time. They are all unable to predict the

8

location and influence of a layer of cold air near the surface. Consequently, the models

did not forecast the extended freezing rain and subsequent ice storms that occurred in

southwest Washington the week of January 5th. Similar problems arose on the east side

of the mountains during mid-January. As a result of these problems, some users reverted

to their previous methods of gathering or obtaining weather information. Of these, a

limited number never returned to using ARROWS.

At the end of the winter season, the Road Weather Program Manager sent a

questionnaire to ARROWS users in all of WSDOT's regions to determine whether

ARROWS was meeting their needs. The rate of return was somewhat underwhelming;

with approximately 150 surveys distributed, 38 responses were returned. Figure 6 shows

the number of responses by region (Northwest, Olympic, Southwest, North Central,

Eastem, and South Central). Figure 7 shows that the largest number of responses came

from supervisors and technicians. Other respondents included superintendents and lead

technicians.

Number of Responses

1B

16

14

12

10B

642

0

NW Oly SW NC E scRegion

Figure 6: Number of survey responses by WSDOT region

9

Number of Responses

14

12

10

I6

4

2

0

Supt Supv Lead Tech

Job Title

Tech Other

Figure 7: Number of survey responses by job title

Below are some of the results from the respondents. Note that all of the extremely

negative comments came from two individuals located at the same maintenance facility.

Sources of Weather Information

Figure 8 shows the various sources of weather information used by the

respondents, including the Weather Channel, the University of Washington Department

of Atmospheric Sciences web site (http://www.atmos.washington.eduldatal), National

Weather Service (NWS), Northwest Weathemet, newspapers, television newscasts, DTN,

radio, NOAA Weather Radio and others. The graph highlights the apparent importance

of television weathercasters, which ranked higher than the NWS in terms of use.

10

25

Wx Chan U Dub NWS

ARROWS Survey Responses

Newspaper TV

Other Weather Souroces

DTN Lmd Radio Wx Radio

20

15

opEI

10

0

Other

Figure 8: Sources of weather information other than ARROWS.

ARROWS Features Most Liked

Figure 9 shows the ARROWS features that users said they liked. The most

popular features were the main distinctive features of ARROWS: the road maps, weather

warnings, and pavement temperature information. Most maintenance people understand

the importance of pavement temperature for their winter maintenance activities. [n

conversations, this was the key forecast parameter. Other features identified by users

were the forecast graphs, icons, ease of use, links, ease of access, and "other," the

composite weather radar images.

11

ARROWS Evaluation

Road maps Fcst Grph lcons Ease Use Warnings Links Access Pvmnt T Other

Figure 9: Features most liked by ARROWS users

ARROWS Features Most Disliked

Figure 10, on the other hand, shows items or features disliked by users. The one

item identified most, albeit by only five respondents, was "ease of use." In talking with

users, most of the people who found ARROWS difficult to use did not have ARROWS

training. The forecast graphs presented some problems because users didn't always

understand the graph process. Training has helped users understand that text forecasts

are available. The two individuals who selected "Links" wanted to see additional links.

The "Other" category was a catch-all for comments such as "not always up," "inaccurate"

(2), "don't like the Vo confidence going to zero with the precipitation forecast,"

"reliability," "too hard to decipher," and "not as good as TV/intemet."

35

30

25

o

oEo

E tcEtz

10

T2

7

iooE

z

4

0

ARROWS Survey Responses

Ease Use Warnings

Lsst Liked

Access Pvmnt T Other

3

Road maps Fcst Grph lcons Llnks

tr'igure 1.0: Features of ARROWS disliked by users

ARROWS Features Needine Chanse

Figure 11 shows what people said they would like to see changed in ARROWS.

The most requested change was to increase the forecast length. Currently ARROWS

forecasts are in six 4-hour time blocks. Most users wanted a minimum of 36 hours; some

at least 48 hours. The issue of map sizes also generated discussion. Some users said they

would like to see the map views larger on the screen. The screens appear to provide

some unused space that could be used for larger maps. Also, users said they wanted the

ability to click on a map to zoom in or out rather than enteringorganization codes to

navigate.

A few others commented on adding or deleting links, changing the wamings, and

pavement temperature. The "Other" category comments included "increase the

13

reliability" [accuracy] (2), "need forecasts out to seven days," "make more user friendly,"

and "make at least as good as TV Web sites"

ARROWS Survey Responses

b

5

4

3

2

2.I5z

0

Add llnks Delete llnks Map slzes Road maps Fcst lenglh Wamlngs Pvmnl T Otherlcons

Want Changed

Figure 11: Items users want changed in ARROWS

Rankines of ARROWS Output Parameters

Figure 12 presents the rankings for various ARROWS output parameters,

including pavement and air temperatures; precipitation amount and time; wind speed and

direction; and snow, frost, and fueezingfog warnings. Not all respondents ranked each

parameter. Most features were subjectively rated 6'OK" or better. Few users rated

parameters "Very Bad." The most dissatisfaction was indicated for snow warnings.

Much of that dissatisfaction derived from users' experiences in the first week of January,

when the computer models had difficulty dealing with cold air trapped near the Earth's

surface west of the Cascade Mountains. Given some computer model adjustments that

t4

have been made since last winter, this forecast feature should improve, and satisfaction

should increase.

Few comments were provided regarding the output parameters. In the "Other"

category, the two Very Bad and the ten Very Good responses were accompanied by no

comments. One verbal comment was, "Freezingrain good."

ARROWS Survey Responses

45

40

35

930oEo

frruEob20oEtz 'tB

lVery Good

lGoodloktr Bad

lVery Bad

10

0

""dis^a.(\'

e" .-""-a.c'

_d.a

e\e-C .u..-

"."." ..J -."" do"'^s

"'-'Parameter Ralings

Figure 12: Rankings of ARROWS output parameters

ARROWS' Usefulness and Conti{ruation

One of the most important components of the evaluation was a question about

whether people were better able to do their jobs because of ARROWS. Figure 13 shows

the results: 2l said "yes" and 9 said "no." A similar question asked whether WSDOT

15

should continue with development of ARROWS. In a resoundingly positive reply, 30

said "yes" and 3 said "no" (see Figure l3).

ARROWS Evaluallon

40

35

30

6

EruEoec20o

3E5i5z

10

5

0

tNoI Yes

Work belter? Contlnue?

Overall Evaluatlon

New Features for Development

Users were also asked for input on what they would like to see developed in

ARROWS. Verbatim responses folloW:

a ARROWS + 5 days, in6-12 hr incr. OK

Waming notification capability, esp when below freezing

Better warnings of freezelsnow events

Snow Techs need info in a hurry, not searching thru graphics ... they can call

NW WeatherNet (a local meteorological forecasting company)

Figure 13: Overall ARROWS evaluation from user perspective

a

o

a

t6

More forecast links, more site estimates

Add more specific sites

a More sites and make it more current

a Links to cameras.

Additional Comments

Respondents were also was asked to provide additional comments. Below is a list

of the comments as written:

o Need RWIS sensors to work

o Not easy to use, not fast enuf

r Little opportunity to use 'cuz of connectivity probs

o Comments based on limited time

o Temps OK early, but from mid Dec on was off, showing snowing when not

o Little or no use, inaccurate, questionnaire just as bad

o As long as ARROWS is another tool and not a tool for scheduling/committing

resources, we'll be fine

o I get better forecasts from TV Wx sites

1 vory useful ... please continue to provide and improve

. only access 3 times. Don't think it was any more useful than the local forecast

o Pavement temp are not very accurate

o Other sources proved much more accurate

o Good tool to use. Only as good as raw data

o Need better data to improve the end product.

t7

Some of the comments showed an understanding of the process; some did not. Some of

the issues mentioned are being resolved. For example, maintenance shed connectivity is

being improved. Also, more computers are being acquired to give more people access.

SUMMARY

One issue buried within the evaluation and the success of ARROWS is training.

Many of the comments about user friendliness related to a lack of proper training. A

formal course now exists. The hope is that more maintenance locations will request the

training and that the training will be conducted at a level below Maintenance Area so that

hands-on training will become the norm

It is apparent from the questionnaire responses, and from conversations with

users, that the initial ARROWS test proved positive and that development and

improvement should continue.

18