recent trends in skill for some leading global nwp …...2019/05/07 · recent trends in skill for...

RecentTrendsinSkillforSomeLeadingGlobalNWPCenters

RossNHoffmana,b,d,e,KrishnaKumarc,d,SidA.Boukabarad,KayoIdee,FanglinYangf,RobertAtlasa

7May2019

(a)NOAA/AtlanRcOceanographicandMeteorologicalLaboratory,Miami,Florida(b)CooperaRveInsRtuteforMarineandAtmosphericStudies,UniversityofMiami,Miami,Florida

(c)RiversideTechnologyInc.,CollegePark,Maryland(d)NOAA/NESDIS/STAR,CollegePark,Maryland

(e)UniversityofMaryland,CollegePark,CollegePark,Maryland(f)NOAA/NCEP/EnvironmentalModelingCenter,CollegePark,Maryland

IntroducRon

•  AlookatthedeterminisRcforecastsofthreeleadingNWPcenters(ECMWF,NCEP,UKMO)fortheyears2015-2017.

•  PAMs(primaryassessmentmetrics)suchasthe500-hPageopotenRalanomalycorrelaRon(AC)orthe250-hPawindRMSEareconvertedtoNAMs(normalizedassessmentmetrics)andthenaveragedintoSAMs(summaryassessmentmetrics).

01/10/19 TrendsinSkill 2

NWPForecasts – PAMs

ReferenceSample

X(�) NAMs Σ SAMs

Verification(Analyses)

Upgrades2015-2017Center i Date Upgrade DeltaECMWF 1 20150512 IFSCycle41r1 2.10

2 20160308 IFSCycle41r2(CubicOctahedral1280) 1.313 20161122 IFSCycle43r1 2.584 20170717 IFSCycle43r3 5.22

NCEP 1 20150114 TIN14-46(T1534) -4.122 20160511 TIN16-11(4DEnVar) 7.373 20170719 SCN17-67(NEMSIO) 0.81

UKMO 1 20161121 PS38(satelliteobs.) 4.752 20170907 PS39(10-kmresoluRon) 2.82



120-h500-hP

aNHX

geo

potenR

alheightR

MSE(m

)

1998 2018

45

65

Context:20yearsofforecastskill


Jan May Sep Jan May Sep Jan May Sep

79

11

ECMWFNCEPUKMO

ECMWFNCEPUKMO

|----------2015--------|---------2016--------|--------2017--------|

Win

d R

MSE

(m s

-1)

120-h500-hPaNHXvectorwindRMSE;MA(365)andMA(31)


ScorecardsofIFSCycle45r1versusIFSCycle43r3.

FromECM

WFNew

leferNo.156

.


ScorecardsofIFSCycle45r1versusIFSCycle43r3.FromECMWFNewleferNo.156.ShowingHRESvs.analysisonly.



ReferenceSample

X(�) NAMs Σ SAMs


PAMstoNAMstoSAMs


ReferenceSample

X(�) NAMs Σ SAMs


PAMstoNAMstoSAMs•  WeogenfocusonafewkeyPAMs,butthismayignoreotherimportantaspects

offorecastskill.TheuseofSAMsincreasesstaRsRcalsignificanceandenablesexploringdifferentaspectsofforecastskill.

•  PAM/NAM/SAMdimension::coordinatevalues–  ForecastRme::24,48,72,96,120,144,168h–  Level::250,500,700,850,1000hPa–  Domain::northernhemisphereextratropics(NHX),southernhemisphereextratropics(SHX),

tropics–  Variable::height(Z),temperature(T),wind(V)–  StaRsRc::anomalycorrelaRon(AC),rootmeansquareerror(RMSE),absolutemeanerror(AME,

theabsolutevalueofbias)–  VerificaRonRme::every24hat0000UTCduring2015-2017–  Center::ECMWF,NCEP,UKMO

•  ReferencesamplefornormalizaRon–  All::(verificaRonRme,center)–  ByCenter::(verificaRonRme)


01/10/19 TrendsinSkillWind RMSE (m s-1)2 5 10 20

0.0

0.2

0.4

0.6

0.8

1.0

PAM

NAM

24 h48 h72 h96 h120 h144 h168 hAllECMWFJan

10

NAM

/ EC

DF

500-hPaNHXvectorwindRMSEECDFnormalizaRonfuncRons

01/10/19 TrendsinSkill 11Wind RMSE (m s-1)6 7 8 9 10 11 12

0.0

0.2

0.4

0.6

0.8

1.0

PAM

NAM

8

8

8

8

8

8

9

9

9

9

9

O

O

O

O

O

O

N

N

N

N

N

D

D

D

D

D

D

1

1

1

1

1

1

2

2

2

2

2

3

3

3

3

3

3

4

4

4

4

4

5

5

5

5

5

5

6

6

6

6

6

7

7

7

7

7

7

1

D

AllECMWFNCEPUKMOJan. . .Dec

NAM

/ EC

DF

120-h500-hPaNHXvectorwindRMSEECDFnormalizaRonfuncRons


Example

0.65 0.70 0.75 0.80 0.85 0.90 0.95

0.0

0.2

0.4

0.6

0.8

1.0

PAM

NAM

2polar3pgps

3polar

cntrl

cntrl3polar3pgps2polar

0.0 0.1 0.2 0.3 0.4 0.5 0.6

01

23

45

6

●

●●●

●●

●

●

●●●

●●

●

●

●●●

●●

●

●●

●

maxavgminstep


x = PAM

s =

n ×

NAM

•  ECDFnormalizaRon– NAM=(rank(PAM)-1/2)/n– RankrelaRvetoref.sample

•  Forminmax,andothernormalizaRons– NAM=aPAM+b– a,bdependonref.sample

ECDFNAMiscalculatedfromtherank



0.0

0.5

1.0

ECMWFNCEPUKMO

ECMWFNCEPUKMO

|----------2015--------|---------2016--------|--------2017--------|

NAM

120-h500-hPaNHXvectorwindRMSENAMs;All;MA(365)andMA(31)


0.0

0.5

1.0

ECMWFNCEPUKMO

ECMWFNCEPUKMO


|----------2015--------|---------2016--------|--------2017--------|

NAM

120-h500-hPaNHXvectorwindRMSENAMs;ByCenter;MA(365)andMA(31)


0.0

0.5

1.0

ECMWFNCEPUKMO

ECMWFNCEPUKMO


|----------2015--------|---------2016--------|--------2017--------|

NAM

120-h500-hPaNHXvectorwindRMSENAMs;ByMonth;MA(365)andMA(31)

020000

60000

100000

0 0.2 0.4 0.6 0.8 1

050000

150000

250000

0 0.2 0.4 0.6 0.8 1

UKMONCEPECMWF

050000

100000

150000

−0.5 −0.1 0.3 0.7 1.1 1.5


Cou

nts

a cb

NAM

020000

60000

100000

0 0.2 0.4 0.6 0.8 1

050000

150000

250000

0 0.2 0.4 0.6 0.8 1

UKMONCEPECMWF

050000

100000

150000

−0.5 −0.1 0.3 0.7 1.1 1.5

HistogramsforallNAMsfor(a)forECDF,(b)forminmax,and(c)forrescaled-minmaxnormalizaRons


SAM

ECMWF NCEP UKMO

0.4

0.5

0.6

ECMWFNCEPUKMO

2015 2016 20170.25

0.50

0.75

ECMWFNCEPUKMO

Center Year

24 48 72 96 120 144 168

0.25

0.50

0.75

ECMWFNCEPUKMO


24 48 72 96 120 144 1680.25

0.50

0.75

ECMWFNCEPUKMO

201520162017

SAM

Forecast time (h)Forecast time (h)


Z T V

0.25

0.50

0.75

ECMWFNCEPUKMO

NHX SHX Tropics0.25

0.50

0.75

ECMWFNCEPUKMO

SAM

Variable Domain

NHX SHX Tropics

0.25

0.50

0.75

ECMWFNCEPUKMO

24 48 72 96 120 144 1680.25

0.50

0.75

ECMWFNCEPUKMO

NHXSHXTropics


SAM

Forecast time (h)Domain

AC RMSE AME0.25

0.50

0.75

ECMWFNCEPUKMO

250 500 700 850 1000

0.25

0.50

0.75

ECMWFNCEPUKMO


SAM

Level (hPa) Statistic

ByCenternormalizaRon•  ReferencesamplefornormalizaRon

–  All::(verificaRonRme,center)–  ByCenter::(verificaRonRme)



ReferenceSample

X(�) NAMs Σ SAMs


24 48 72 96 120 144 1680.4

0.5

0.6

ECMWFNCEPUKMO

201520162017

2015 2016 2017

0.4

0.5

0.6

ECMWFNCEPUKMO


SAM

Forecast time (h)Year






day-by-day


0.3

0.5

0.7

1 2 3 4


SAM

|----------2015--------|---------2016--------|--------2017--------|

ECMWF

day-by-day






day-by-day


0.3

0.5

0.7

1 2 3


SAM

|----------2015--------|---------2016--------|--------2017--------|

NCEP

day-by-day






day-by-day



0.3

0.5

0.7

1 2

SAM

|----------2015--------|---------2016--------|--------2017--------|

UKMO

day-by-day


0.3

0.5

0.7


SAM

|----------2015--------|---------2016--------|--------2017--------|

Average

day-by-day


0.4

0.5

0.6

ECMWFNCEPUKMO

ECMWFNCEPUKMO


SAM

|----------2015--------|---------2016--------|--------2017--------|

MA(365)andMA(31)


SAM

MonthJan Mar May Jul Sep Nov

0.4

0.5

0.6

●

●

● ●

●

●●

●

●

●

●

●

● ECMWFNCEPUKMO

Summary

•  Allthreecentersimproveoverthethreeyearperiod.NCEPshort-termforecastskillsubstanRallyincreasesduringtheperiod.

•  SAMsindicatethatintermsofforecastskillECMWFisbeferthanNCEP,whichisbeferthanbutapproximatelythesameasUKMO.


2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

•  However,theobservedimpactsarewithinthecontextofslowlyimprovingforecastskillforoperaRonalglobalNWPascomparedtoearlieryears.

•  TheuseofSAMsimprovesthesignaltonoiseraRoandclearimprovementsinSAMarerelatedtotheECMWFJuly2017upgradetoIFSCycle43r3,theNCEPMay2016replacementofthe3DEnVarwiththe4DEnVar,andtheUKMONovember2016(PS38)introducRonofimproveduseofsatelliteobservaRons.


ReferenceSample

X(�) NAMs Σ SAMs


Concludingremarks

•  WeogenfocusonafewkeyPAMs,butthismayignoreotherimportantaspectsofforecastskill.TheuseofSAMsincreasesstaRsRcalsignificanceandenablesexploringdifferentaspectsofforecastskill.

•  ClearlythesystemslaggingECMWFcanimprove,andthereisevidencefromSAMsinaddiRontothe4DEnVarexamplethatimprovementsinforecastanddataassimilaRonsystemsaresRllleadingtoforecastskillimprovements.

•  Infuturework,itmightbeinteresRngtoincludeothercentersandtoaddPAMsforrelaRvehumidityandprecipitaRon,forecastvariablesforwhichthereiscurrentlymajorroomforimprovement.



ReferenceSample

X(�) NAMs Σ SAMs


more...•  email:•  [email protected]

•  Dec2018WAFpaper:•  doi:10.1175/WAF-D-18-0117.1

•  AMSpresentaRon:•  hfps://ams.confex.com/ams/2019Annual/meeRngapp.cgi/Paper/

350739


2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

ADetecRveStory•  Webeganwith5centers:

CMC,ECMWF,FNMOC,NCEP,UKMO

•  Wekeptthe3best,partlybecauseFNMOCresultsdidnotmakesense

•  Colorshavechangedfromthe3centerscase.


CMC ECMWF FNMOC NCEP UKMO0.25

0.50

0.75

CMCECMWFFNMOCNCEPUKMO

24 48 72 96 120 144 1680.4

0.5

0.6

● ●● ● ● ● ●

● ● ● ●● ● ●

●

●

●● ● ● ●

●

●●

● ●● ●

●

●

●● ● ●

●

● ● ● ● ● ●●

●

●

●● ● ● ●

●●

● ● ● ● ●

●

●● ● ● ●

●●

● ● ●●

●

●


201520162017

2015 2016 2017

0.4

0.5

0.6



SAM

Forecast time (h)Year


0.3

0.5

0.7

1 2


SAM

|----------2015--------|---------2016--------|--------2017--------|

CMC

day-by-day


0.3

0.5

0.7

1 2


SAM

|----------2015--------|---------2016--------|--------2017--------|

FNMOC

day-by-day

year

SAM

2015 2016 2017

0.4

0.5

0.6

| arraytype = sam | normalization = ecdf | refsample = allNWP | gamma = 0.2652 |SAM[year,exp], only for (var=V;lev=1000), averaged over (stat,ftime,domain,day,month)

p = 0.95 c.i. shaded in grey


outlines are for allNWP rescaled SAM

year

SAM

2015 2016 2017

0.4

0.5

0.6

| arraytype = sam | normalization = ecdf | refsample = allNWP | gamma = 0.1507 |SAM[year,exp], without (lev=1000), only for (var=V), averaged over (stat,ftime,domain,lev,day,month)

p = 0.95 c.i. shaded in grey


outlines are for allNWP rescaled SAM


SAM

Year Year

Wind (250,500,700,850) Wind (1000)

Why?•  Inmid2016,theNAVGEMgridssharedwithNCEPchangedfrom1

degreeto1/2degreelaRtude-longitude.•  Becauseofthewaythefieldsarefiltered,thischangemakesit

seemlikeNAVGEMforecastskillisdegradinginourassessmentsusingtheVSDBstaRsRcs.–  MostoftheNAVGEMfieldsusedinourassessmentsarefilteredwiththe

same2done-passShapirosmootherde-smootherappliedingridpointspaceforboth1and1/2degreefields.

–  Asaresultthe1/2degreefieldshavemoreenergypresentintheinherentlyhardtopredictsmallestscales,resulRnginanapparentdropinforecastskill.

–  DuringVSDBprocessingnofurtherfilteringisapplied.


Butwhataboutlowlevelwinds?•  FilteringisappliedtoallgeopotenRalheights,alltemperatures,andallwindsabove900hPa.– Thusinourstudy,only1000hPawindswerenotaffectedbythischange.

•  ThankstoElizabethSaferfield/NRL-MontereyandRandalPauley/FNMOCforhelpinunravelingthispuzzle.



ReferenceSample

X(�) NAMs Σ SAMs



•  Dec2018WAFpaper:•  doi:10.1175/WAF-D-18-0117.1


350739


2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

2015 2016 2017

0.25

0.50

0.75

ECMWFNCEPUKMO

ApplicaRontoimpactexperiments•  TheoriginoftheSAMworkwastosummarizeOSEresultsforadatagapimactstudy.

•  WerepeatedtheimpactstudyinsimulaRon(OSSEmode)tovalidateourOSSEsystem(CGOP).

•  IdeallyeachOSSEcomponentmustberealisRc,howevertheseOSSEsused–  LowerresoluRonGDAS/GFS– NoobservaRonerrors


ObservingSystemSimulaRonExperiments

•  OSSEshavebeenusedsincethe1950sto–  Evaluateobservingsystemsintermsofaccuracyandcoverage(e.g.,inplanningFGGE)

–  GuidedecisionmakerstoallocateresourcestomiRgatecostsandleadRmeinreality

–  Conducttradestudiesofinstrumentsandsystems,and–  DesignandtestnewDAmethods


OSSE System

NR Simulate obs.

DA Forecasts

Future OSSEs 23 April 22, 2015

Revised: October 23, 2014

CalibrationVerification

ExperimentalSetup:DataGapScenario•  Inter-comparison:OSSEvsOSEforControl,3Polar,2Polar

[OriginalOSEworkbyBoukabaraetal(2016b)]

Period 7Julyto7August,2014

Obssystemconfig.

2014:allconvenRonal+satellitedatagapscenarios

OSSEonly:perfectobs

DAS 3DEnVarwithT670/T254n=80ensemble

Forecast 0000UTCdailyupto168h

VerificaRon OSSE/OSE:owncntrlanalysis

OSSEsystemvalidaRon/calibraRon•  BeforeapplyanOSSEsystemtoanewproposedsensor,wewanttocheckiftheresultsforadatadenialexperimentmatch

•  Ifnot(andogenOSSEresultsareoverlyopRmisRc)wemustcalibrateourOSSEsystemby–  TuningtheobservaRonerrors;and/or–  ChangingsomeparameterizaRonsintheforecastmodel;or–  AdjusRngtheOSSEresultsagerthefact

•  ForthesimulaRonexperimentswith“perfect”observaRons,theOSSEwasoverlyopRmisRc,butnotintermsofSAMs!


Results:ForecastSkill(PAM-AC&RMSE)


•  Alldatagapscenariosresultinpoorforecastskills•  TendencyofimpactmostlyasexpectedalthoughtherearebitofvariabiliResinOSSEvsOSEinter-comparisonresults

00 24 48 72 96 120 144 168

02

46

810

●

●

cntrl−OSSEcntrl−OSE3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

00 24 48 72 96 120 144 168

0.0

0.5

1.0

1.5

2.0

2.5

●

●

3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

●

●

●

● ●● ● ●

●

●●

● ● ● ●●

00 24 48 72 96 120 144 168

0.5

0.6

0.7

0.8

0.9

1.0

●

●


● ● ●●

●

●

●

●

● ●●

●

●

●

●

●

00 24 48 72 96 120 144 168

−0.04

0.00

0.02

0.04

●

●

3polar−OSSE3polar−OSE2polar−OSSE2polar−OSE

● ● ● ● ●●

●

●

● ●●

●

●

●

●

●

Forecast time (h)

V 250 hPa Tropics RMSE (m s-1)Z 500 hPa NHX AC

Forecast time (h)

Results:SAMsglobalandvs.forecastRme


cntrl−OSSE 3polar−OSSE 2polar−OSSE

0.3

0.5

0.7


00 24 48 72 96 120 144 1680.1

0.3

0.5

0.7

0.9

●

●

●●

● ● ● ●

●

●

●● ● ● ● ●

●

●


a bSA

M

Forecast time (h)Experiment

cntrl-OSSE

cntrl-OSE

3polar-OSE

2polar-OSE

3polar-OSSE

2polar-OSSE

Results:SAMsbylevelanddomain


250 500 700 850 1000

0.3

0.5

0.7


NHX SHX Tropics

0.3

0.5

0.7


a bSA

M

DomainLevel (hPa)


•  Oct2018JTECHpaper:•  doi:10.1175/JTECH-D-18-0061.1


353842


cntrl−OSSE 3polar−OSSE 2polar−OSSE

0.3

0.5

0.7


Experiment

cntrl-OSSE

cntrl-OSE

3polar-OSE

2polar-OSE

3polar-OSSE

2polar-OSSE

Control3Polar2Polar

recent trends in skill for some leading global nwp …...2019/05/07 · recent trends in skill for...

Documents