th quarterly progress report for ca noo244-15-2-0005 ... · python, with the pandas, statsmodels...

SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016

david@statisticalsolutionsllc.comhttp://www.statisticalsolutionsllc.com/

5thQuarterlyProgressReportforCANOO244-15-2-0005,IntraseasonalTropicalCycloneForecasting

I. Overview

TheprogressmadebyStatisticalSolutionsLLC(SSL)asoftheendofthefifthquarter(9/30/16)onresearchasdelineatedinthecooperativeagreementN00244-15-2-0005(hereafterreferredtoasCA)isevaluatedasonscheduleandonbudget.Attheendofthequarter,71.0%ofthetotalfundinghasbeenspentover71.0%ofthelifetimeoftheCA.FurtherdetailsonfundingandexpendituresareavailableinSSL’sInterimFinancialReport(SF-425).Forthesakeofbothtransparencyandforachievingthewidestpossibledisseminationofthepublicbenefitresultingfromthisresearch,thisprogressreportwillbemadepublicallyavailableandpostedonSSL’swebsiteat:http://www.statisticalsolutionsllc.com/recentongoing-research.html.Duetothepublicreleaseofthisform,thetotalgovernmentfundingofthecooperativeagreementisomitted,thoughallfinancialdetailswillbefoundintheSF-425,whichalreadyhasbeensenttoDr.TomMurphreeoftheNavalPostgraduateSchool(NPS),theGovernmentSponsor,andtheAdministrativeGrantsOffice.WhencomparingprogresswiththescheduleandmilestonesagreeduponintheCAandasadjustedindiscussionwithDr.TomMurphreeofNPS,progressmadetotheendofthequarterwouldindicatethattheprojectisonscheduleasbothNorthAtlantic(NA)andeasternNorthPacificENPhurricane/tropicalcyclone(TC)forecastingsystemmodelshavebeenconstructedandexperimentationwiththemodelsvs.realworldactivity(formations)isunderway.

II. NorthAtlanticHurricaneModelBuildingA. Background

TheworktobeperformedasdelineatedintheCAincludesthebuildingofastatistical-dynamicalmodelforthepurposeofforecastinghurricaneformationintheNA.AhighlevelviewofhowthesystemfunctionsisoutlinedinFig1.Large-scaleenvironmentalfactor(LSEF)andhurricaneformationdatafortheNAareusedtobuildastatisticalmodelrelating(bymeansoflogisticregression)formationtotheLSEFs.OncethemodelisbuiltitcanbeforcedwithforecastsoftheLSEFstointurngenerateaforecastofthelikelihoodofhurricaneformationatavarietyofdifferentleadtimes.Inordertodotheregression,30years(1985-2014)ofNALSEFandhurricaneformationdatawerecollected,archived,andgridded.TheLSEFsarethoseenvironmentalfactorsthoughttobenecessarybutnotsufficientforTC/hurricaneformationasproposedbyGray[1968,1975]andareessentiallylowshear,warmseasurfacetemperatures,positiveabsolutevorticity,upwardairflow,andhighhumidities).

SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016

david@statisticalsolutionsllc.comhttp://www.statisticalsolutionsllc.com/

Fig.1TheSSL-NPSstatistical-dynamicalhurricaneformationforecastingsystemfortheNorthAtlantic.LSEFdataisavailableormaybederivedfromtheClimatePredictionCenter’sClimateForecastSystemReanalysis(CFSR)dataset,availableat:https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/climate-forecast-system-version2-cfsv2#CFS%20Reanalysis%20(CFSR)fordatathrough2010,andhttp://rda.ucar.edu/datasets/ds094.1/#descriptionfordatafrom2011throughcurrent.HurricaneformationdatausedinthemodelbuildingprocessisfromtheNationalHurricaneCenter’s(NHC)HURDAT2datasetavailableat:http://www.nhc.noaa.gov/data/hurdat/hurdat2-1851-2015-070616.txt.Thelocationanddate/timeofhurricaneformationistakenfromthefirstlineofdatafromthedatasetforeachlistedstorm.TheLSEFdatausedformodelbuilding,whichisavailableata0.5°,6-hourlyresolution,wasthenregriddedontoa1.0°,6-hourlygrid.NotethatwechosetouseashigharesolutionaspossibleconsistentwiththeuncertaintyofthetimingofactualcyclogenesisaswellasthespatialresolutionoftheClimateForecastSystemv.2(CFSv2)forcingdatathatisusedtoforcethestatisticalmodelforforecastingpurposes.

B. ModelRebuildingPython,withthePandas,StatsmodelsandPatsymodules,wasusedtoperformlogisticregressionontheLSEFandhurricaneformationdata.ThenecessarybutnotsufficientLSEFs,Gray[1968,1975],werefoundtobestatisticallysignificantataconfidencelevelofover0.999,thoughthehumiditytermwasdroppedduetomulticolinearity.However,followinginitialmodelbuilding(seeprevioustechnicalreportsforthisCA)itwasdeterminedthattheextendedleadforecastssufferedfrominconsistencyinthevaluesofprobabilitiescreatedbythestatisticalmodel,whenforcedwithCFSforecastsoftheLSEFs.Whiletheshortleadforecastsappearedbothskilledandconsistentinappearance,formandbehaviorwithwhathadbeenexpected(andwasobservedwithourENPshortleadforecasts),thelongleadforecaststendedtooverpredicthurricaneformationintheCaribbean/GulfofMexico/WesternNorthAtlantic,andunderpredictformationintheEasternNorth

1

Producesta%s%cal-dynamicalmodeloutput:ensemble-basedforecastsofhurricaneforma%onprobabili%es:1-dayleadforecaststo2monthleadmonthlyoutlooks

Sta%s%calmodelrelatestheLSEFstoTCforma%on

probability

Buildsta%s%calmodelbasedonrela%onshipsbetweenTCforma%onsandLSEFs(basedonNHCbesttracksandCFSR)

Forcesta%s%calmodelwithdynamical,ensemble-based,CFSv2LSEFforecasts

david.sta*s*cal.solu*ons@gmail.commurphree@nps.edu

SSL-NPSDynamical-Sta*s*calTCForma*onForecas*ngSystem

SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016

david@statisticalsolutionsllc.comhttp://www.statisticalsolutionsllc.com/

Atlantic.Moreover,thereweredifficulttointerpretpeaksofformationpotentialthatseemedbothrandomlyplacedinposition,andnotparticularlygroundedinphysicalplausibility.SSLandDr.Murphreespenttimegoingoverplots,comparingthemtootherbasinforecasts,anddiscussingthevariousbasinmodelsandhoweachperformed.Ultimately,itwasdecidedthatwewouldredotheENPstatisticalmodeltoitsmostbasicpredictors(theLSEFs)andnotincludehigherorderpolynomialandinteractionterms.Thiswasdonesothatwecouldevaluatetheeffectofreducingthenumberofpredictorstosomethingsimplerwhileexploringtheeffectofthereductiononwhatwasconsideredtobeaskilledstatisticalmodel.Dependingontheoutcomeofthemodelsimplification,wewouldthenre-approachmodelingtheNA.TheeffectoftheENPmodelsimplificationwasnegligible.Forecastsatallleadslookednearlyidentical,tothosecreatedusingthemorecomplexmodel,andtherewasnochangeinsystemskill.Emboldenedbytheapparentsuccess(thesimplifiedmodelwasonparwithskillincomparisonwiththemoredetailedmodel)withENPmodelsimplification,SSLcreatedanew,simplifiedstatisticalmodelfortheNA.Thegoalofthesimplificationwastoeliminatepotentialoverfitting,produceamodelthatmoreclearlyagreeswithGray’swork,Gray[1968,1975],andtocreateamodeleasiertoexperimentwithanddiagnose.Withthenew,simplifiedmodel,themagnitudeexcursionsappearedtobeeliminatedintheextendedleadforecasts,butthepreviouslymentionedover/underpredictionproblemwaslittleunchanged.

C. FurtherNAStatisticalModelExplorationAfterconsultationwithDr.Murphree,SSLproceededwithatwo-stepplan.Thefirstwastoexperimentwiththecreationandevaluationofmonthlyclimatologymasks.Thepurposeofthisstepwastoevaluate,onamonthlybasis,theagreement(orlackthereof)oftheNAhurricaneforecastingsystemproducedmonthlyoutlookswithwhatisobservedwithhistoricalNAhurricaneformations.Doingsowouldgiveustheopportunitytobettercomparewheretheforecastingsystemproducedresultsthatwereinroughagreementwithclimatology,andwheretheremightbesubstantialdepartures.ThecreationandusageofmaskshadlittleeffectonNAforecasts,againindicatingthattheproblemismoreoverprediction/underpredictionthanthemodelindicatingformationpotentialexistsinregionswhereitwouldbeunlikelytobetrue.Furthermore,themasksallowfordiscussionaboutthepotentialvalueofcreatingmorethanonestatisticalmodelfortheNA.Usingmorethanonemodel(oratleastparameterizingtheexistingmodel)wasanideathathadbeendiscussedbyDr.MurphreeandSSLimmediatelyaftercreationofthefirstgenerationNAstatisticalmodel.Figure2showsclimatological(mostrecent36yearsforwhichwehavedata)hurricaneformationlocationsfortheNAsuperimposedonourformation

SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016

david@statisticalsolutionsllc.comhttp://www.statisticalsolutionsllc.com/

mask(JuneandJuly).Notethatthetwomasksarenotaggressive.Otherthateliminatingafewonceinagenerationstorms,themasksroughlyencircletheclimatologicalformationlocations.Themasksclearlyshowtheexistenceoftwodistincthurricaneformationdevelopmentregions(onetothenorthwest,theothertothesoutheast,andwhilethemaskscannotindicatethatdifferentmodelsmustbeused,thedistinctlyseparateregionswouldimplythattheideashouldbeatleastinvestigated.

Fig.2ClimatologymasksforJune(Top)andJuly(bottom)fortheNorthAtlantic,with36yearsofactualTCformationsshown(fromtheNHCHURDAT2dataset).Whiletheremaainingmonthsofthehurricaneseasonarenotshown,Octoberistheonlymonthforwhichthereisnoobvioustworegionsourceofformations.Ultimately,basedonthemask/climatologyplots,coupledwiththemonthlyoutlooks(seeFigure3)producedbythecurrentstatisticalmodel,SSLandDrMurphreebothagreedthatthatthenextstepforNAforecastingshouldbethedevelopmentofatworegionmodel.Dr.MurphreeandSSLalsoagreedthatwehavelearnedmuchaboutNAhurricaneforecasting,thataskilledshortleadproductisinplace,butthatfurtherdevelopmentofanextendedleadNAhurricaneforecastingsystemmaynotbeaneffectiveuseoftimeincomparisontotwoothertasks(discussedinsectionIV.

SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016

david@statisticalsolutionsllc.comhttp://www.statisticalsolutionsllc.com/

WayForward),especiallyascurrentlywehavelessthanafullseasonofforecaststoworkwithtoevaluateandfinetuneskill.However,throughSeptemberof2016,ourprobabilityofdetectionforthecurrentmonthmonthlyoutlooksforthe2016hurricaneseasonwas83%(10outof12storms)wherethetwomissescamefromwhereonemightexpect,ourunderpredictivesoutheasternNAregion.

Fig.3ExtendedleadhurricaneformationpotentialforecastvalidforJuly2016,issued30June2016,withmaskapplied.Notethehighestprobabilitiesaretothenorthandwest,andthelowesttothesoutheast,despitetheformationoccurringinthesoutheastregion.

III. PreliminaryENPandNAForecastVerificationIncontrasttothechallengespresentedbytheNAwithrespecttoforecastingthepotentialforhurricaneformation,theENPhasbeenverystraightforward.Thesimplifiedmodel,asdiscussedaboveisinuseforallleads(shortandlong)andisusedasinputforSSL/NPS’weeklyrecommendationsregardingTC/hurricaneformationtotheCenterforEnvironmentalPrediction’sGlobalTropicalHazards/BenefitsTechnicalTeleconference(TheGTHBforecastmaybefoundhere:http://www.cpc.ncep.noaa.gov/products/precip/CWlink/ghazards/).Inadditiontothetechnicalteleconferences,whereSSLprovidesshortlead(48houroutlooks)andintermediatelead(week1andweek2outlooks)inputs,SSLmakes1and4dayleadforecasts,aswellascurrentmonthmonthlyoutlooks,onemonthleadmonthlyoutlooks,andtwomonthleadmonthlyoutlooksfortheWesternNorthPacificandtheENPavailableviaitswebpage(http://www.statisticalsolutionsllc.com/tropical-cyclonehurricane-formation-forecasts-2016.html).The1and4dayNAforecastsarealsomadeavailableinthewebsite.Allextendedleadforecastsarearchivedonthe

SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016

david@statisticalsolutionsllc.comhttp://www.statisticalsolutionsllc.com/

websitesothatusershaveanopportunitytojudgeforthemselvestheskilloftheforecasts.

A. ENPPreliminaryVerificationFigure4showsa1dayleadforecastissued25Aug2016,andvalid26Aug16.Oursynopticallyscaledshortleadforecastsaretypicallyhighlyskilled;somuchsothatapotentialfutureuseofourshortleadforecastscouldbeasasourceofunbiasedformationreanalysis.

Fig.41dayleadleadTCformationpotentialforecastvalidfor26Aug2016,issued25Aug2016,withverifyingTCindicated(dateandlocationasindicatedfromthefirstissuedTropicalCycloneFormationAlert).Notetheregionofelevatedprobabilitiesatroughly18°Nand115°W.Thisregionisactuallythelocationof13E,whichformed2daysearlieronthe24thofAugust.Beyondthehighskilloftheshortleadproductsatforecastingformation,theshortleadforecastsarealsohighlyskilledatforecastingfuturelocationsofTCs.InFigure4weseeata1dayleadtheforecastedpositionof13E,whichhadformedafewdayspriortotheindicatedformationof14E.MoreonanecdotalevidencethattheshortleadforecastsmayhaveuseforforecastingtrackswillbediscussedinSectionIII.BNAPreliminaryVerification.Figure5showsanextendedlead(currentmonthmonthlyoutlook)forecastfortheENP,issuedlateJuly,andvalidforAugust2016.InFigure5weseethat5TCsformed,with1miss(atabout130°W),1thatisjustoutsidethecontouredareabutstilljudgedasahit(weusea2.5°neighborhoodwhenscoringtoaccountforgrid

SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016

david@statisticalsolutionsllc.comhttp://www.statisticalsolutionsllc.com/

errors,CFSspatialerrorsandpositionalerrorsinthelocationoftheformation),and3solidhits.Theforecastshownistypicalfor2016,andasmentionedpreviouslymaybefoundwiththeother2016(and2015)archivedforecastsforuserevaluationofskill.Notshownarethelongerlead(1and2month)forecastsofformationpotential,buttheyareverysimilarinappearanceandskilltothecurrentmonthmonthlyoutlook.Whilenotarchivedonthewebsiteyet,thelongerleadforecastswillbemadeavailabletoosothatpotentialuserscangaintrustinasystemwhoseperformancetheycanjudgeforthemselves.

Fig.5CurrentmonthmonthlyoutlookofTCformationpotentialvalidforAugust2016,issued31July2016.Fromhttp://www.statisticalsolutionsllc.com/eastern-north-pacific-formation-forecasts.html

B. NAPreliminaryVerificationWehavediscussedinSectionII.CsomeoftheissuesregardingtheNAhurricaneformationpotentialforecastingsystem.However,sincetheissuesdidnotappeartoadverselyeffectshortleadforecastperformance,shortleadforecastsweremadeavailableontheSSLwebsite.Figure6showsthe1dayleadforecastvalidfor26Sep2016,thedaythathurricaneMatthew(14L)formedintheNA.Matthew’sformationlocationforthatdayisindicatedwiththepinkdot.

SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016

david@statisticalsolutionsllc.comhttp://www.statisticalsolutionsllc.com/

Fig.61dayleadofhurricaneformationpotentialvalidfor26Sep2016,issued25Sep2016.Thedisturbancecenteredon18°N,70°Wwasalsoofinterest,butenvironmentalconditionsthefollowingdaywerenotfavorabletoformation,andtheformationneveroccurred.Finally,Figure7(a-d)showsaseriesofshort(4day)leadforecastsfortheNAissuedandpostedonthewebsitedailybySSLonceitbecameapparentthatMatthewwouldimpacttheUS.Whiletheshortleadforecastswerenevercreatedtobeusedastrackforecasttools,Figure7correctlyindicatedthegeneraldirectionandspeedofMatthewatleadtimesusefultopeopleinitspath.

SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016

david@statisticalsolutionsllc.comhttp://www.statisticalsolutionsllc.com/

Fig.74dayleadforecastsofhurricaneformationpotentialvalidfor5-8Oct2016,andissued1-4Oct2016respectively.Consideringtheuncertaintyinforecastsavailableatthetimeata4daylead,thefourdayforecastswerequitereasonable.

C. SummarybyBasinSSLhaskeptrunningverificationfilesforbothbasinsthroughouttheTC/hurricaneseason.Theresultsarepreliminary,usingthefirstpositions/datesgivenintheirrespectivetropicalcycloneformationalert(TCFA),butthesepositions/datesfrequentlychangeoncethebesttracksfortheseasonareissued.Note,differencesbetweenpreliminarydataandbesttrackdatahistoricallydonotsignificantlyaltertheverificationresults,andwillendupimprovingresultsattimesaswell.TheENPresultsaregoodenoughthesomemodificationoftheminimumcontourusedmayneedtobeexperimentedwithovertheoff-seasonandoncethebesttrackscomeoutsothatprobabilityofdetect(POD)maybereducedinordertoreduceENPpercentageofcontouredarea(PCA).PCAisanSSLcreatedmeasureoftheamountofoceanareatheforecastsindicateasfavorabletoformationdividedbythatamountoftheoceancapableofsupportingTC/hurricaneformationasindicatedbyclimatology.ThusPCAwouldindicateforecastsystemdiscrimination,withthesmallerthePCAvaluetranslatingtosmallerareasoutofthebasinthatourforecastsindicateasformationsupportive.Figure8showsPODandPCAfortheNAandtheENP.Bothareskilled,evenwiththeissuesidentifiedfortheNA.Ideally,ourforecastswouldhaveroughlyan80%PODanda10%PCA.FortheNA,Figure8indicatesanacceptablePOD,butatoohighvalueofPCA(dominatedbythe

a)

c)

b)

d)

SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016

david@statisticalsolutionsllc.comhttp://www.statisticalsolutionsllc.com/

overpredictionissue).FortheENP,thereispotentialtoreducethePCAtoour10%target,byacceptingareducedPODinreturn.

Fig.8VerificationstatisticsforSSL’sNAandENPcurrentmonthmonthlyoutlooksforthe2016seasontodate.Thelongerleadforecasts(1and2monthleadmonthlyoutlooks)areverysimilarinskill.

IV. PublicBenefitKeyattributesofprojectsfundedbycooperativeagreementsarethattheresultsoftheresearcharetoprovideapublicbenefit,andthattheresultsaretobemadepublicallyavailablesothatthepublic,scientificcommunity,andothersmaytakeadvantageoftheresearchresults.Withoutdoubt,improvedunderstandingofTC/hurricaneformationandtheinfluenceoftheenvironmentonthoseformationsisofgreatbenefittothepublic.Tomakepublictheresearcheffort,andtheongoingresultsofthatresearch,onOctober14th,SSL,withDr.Murphreeasaco-author,presented“VisualizationandVerificationofExtendedLeadForecastsofHurricane/TropicalCycloneFormation”atthe2016Cincinnati-DaytonInformsTechnicalSymposium,outliningourTC/Hurricaneresearchtodate,ourresults,andthewayahead.TheabstractisavailableattheCincinnati-DaytonChapterINFORMSwebsitehttps://www.informs.org/Community/Cincinnati-Dayton-Chapter/Symposium.Additionally,allexperimentalforecastproducts,includingkeys,legends,andbackgroundmaterialsarepostedontheSSLwebsiteat:http://www.statisticalsolutionsllc.com/tropical-cyclonehurricane-formation-forecasts-2016.html.DuringearlyOctober,whileMatthewwasapproachingandimpactingtheUS,SSLupdateditsNApagedaily.Furthermore,SSL,onaweeklybasis,emailsparticipantsintheClimatePredictionCenter’sGlobalTropicalBenefits/Hazardstechnicalteleconferencewithtextualforecastinputandupdates

00.10.20.30.40.50.60.70.80.91

NA ENP

PreliminaryLongLeadVeriVicationbyBasin

POD

PCA

SSLquarterlyprogressreportforquarterending30Sep2016Submitted21Oct2016

david@statisticalsolutionsllc.comhttp://www.statisticalsolutionsllc.com/

theexperimentalforecastsonit’swebsite.SSLparticipatesintheteleconferenceandusestheinteractionwithotherscientistsasanopportunitytolearnmoreaboutformationforecasting,andaswellastheskillofourresearch.Finally,TC/hurricanerelatedresearchperformedtoenableSSL’sgreaterunderstandingofTCactivity(suchasTC/hurricaneformationclimatescience),thoughnotnecessarilyrequiredorfundedbytheCA,isalsomadeavailableontheSSLwebsite(http://www.statisticalsolutionsllc.com/tchurricane-climate-science.html).AmeasureofthepublicbenefitisthatSSL’spageviewsquadrupleeveryMonday(thedayofthetechnicalteleconference)to40pagesperday.Finally,forpurposesofbothtransparencyandpublicawareness,thisquarterlyprogressreport,pastandfuturequarterlyreports,andthefinaltechnicalreportforthiscooperativeagreementwillalsobemadeavailabletothepubliconSSL’swebsite.

V.WayForwardForthenextquarter,SSLwillfocusontwomainefforts.ThefirstistobegintheprocessofresearchingthestateoftheartforTC/hurricanetrackforecastingandoutlineoutanapproachtodevelopastatistical-dynamicalforecastingsystemfortracks.Thisdeviatesfromtheworkoriginallyincludedinthestatementofwork,butwasmutuallyagreeduponbetweenDr.MurphreeandSSLasthenexttasktoworkonfollowingarecommendationfromtheJointTyphoonWarningCenter.Thesecondeffortwillfocusonrobustverificationofthe2016forecasts(allbasins)toquantifytheexperimentalforecastingsystemperformance.Thisisakeycomponenttotheoverallforecastingresearcheffortasadvancesmadeaspartofourresearchwillbeignoredunlessitcanbeshownthatimprovementhavebeenmadetothestateofthescience.

VI.ReferencesGray,W.M.(1968),GlobalViewoftheOriginoftropicalDisturbancesandStorms,MonthlyWeatherReview,96,669-700.Gray,W.M.(1975),TropicalCycloneGenesisintheWesternNorthPacific,NavalAirSystemsCommand,WashingtonDC,20361.