cme arrival time & impact working team...apr 18, 2014 · cme arrival time & impact working...
TRANSCRIPT
![Page 1: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/1.jpg)
CMEArrivalTime&ImpactWorkingTeam
ChristineVerbeke,LeilaMays,SandroTaktakishvilli
![Page 2: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/2.jpg)
CMEArrivalTime&ImpactWorkingTeam
✳ EvaluatewherewestandwithCMEarrivaltimeandimpact prediction✳ Establishcommunity-agreedmetricsandeventsregardingCMEarrivaltimeandimpact.✳ Provideabenchmark againstwhichfuturemodelscanbeassessedagainst✳ Complementarytothe CMEScoreboard (collectanddisplayreal-timeCMEpredictionsandfacilitatethevalidationofreal-timepredictions).
✳ Catalogofmetrics andhowtheyrelatetouserneedsandscienceneeds.✳ Modelassessments withselectedmetricsforselectedtimeintervals.✳ Onlinedatabaseofmodelinputs,outputs,andobservations.✳ Publication describingmodelassessmentresultssummarizingwherewestandwithCMEarrivaltimeandimpactprediction.
Work with Information Architecture for Interactive Archives (IAIA) working teamfor online database.
TeamGoals
TeamDeliverables
![Page 3: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/3.jpg)
CMEArrivalTime&ImpactWorkingTeam
✳ Identifyanddiscussuserneeds✳ Discussandselecttimeintervalstostudy— expandasneeded✳ Discussanddevelopasetofrelevantskillscores,andrelatethemtouserneeds andscienceneeds✳ Identifysourcesofuncertainty✳ Producemodel/techniqueoutputforintervalsofstudy✳ Performmodelassessments withselectedmetrics
✳ Slackchannels(contactleadstobeadded)✳ Mailinglist(contactleadstobeadded)✳ Email✳ Telecons✳ Regularwebsiteupdateshttps://ccmc.gsfc.nasa.gov/assessment/topics/helio-cme-arrival.php
Summaryofteamtasks
Remotecollaboration
![Page 4: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/4.jpg)
CMEArrivalTime&ImpactWorkingTeam
Participants– inviteyourcolleagueshighinterestEricAdamson*·TanjaAmerstorfer ·Anastasios Anastasiadis ·NickArge ·MichaelBalikhin*·DavidBarnes*·Francois-XavierBocquet ·Yaireska Collado-Vega*·PedroCorona-Romero*·JackieDavies·CurtdeKoning*·CraigDeForest*·Manolis K.Georgoulis*·CarlHenney ·BernardJackson*·LanJian·MashaKuznetsova*·Kangjin Lee·Noé Lugaz ·AnthonyMannucci*·Periasamy KManoharan*·DanielMatthiä*·LeilaMays*·MikeMcAleenan*·Slava Merkin*·Marilena Mierla ·JosephMinow*·ChristianMoestl ·KarinMuglach*·TeresaNieves·Nariaki Nitta·MarlonNunez·Dusan Odstrcil*·MathewOwens·Evangelos Paouris ·AthanasiosPapaioannou ·SpirosPatsourakos ·vic pizzo ·PeteRiley·AlexisRouillard·CamillaScolini ·HowardSinger*·RobertSteenburgh*·Aleksandre Taktakishvili*·ManuelaTemmer·W.KentTobiska*·ChristineVerbeke*·Angelos Vourlidas ·KatherineWinters*·AlexandraWold*·KiChang Yoon·Emiliya Yordanova*·Jie Zhang·mediuminterestTarekAl-Ubaidi*·SuzyBingham*·StevenBrown*·BaptisteCecconi ·DavidFalconer·NataliaGanushkina*·LauraGodoy*·BerndHeber·ChristinaKay·AdamKellerman*·BurcuKosar*·AlexanderKosovichev*·YukiKubo·PeterMacNeice*·Chigomezyo Ngwira*·StevePetrinec*·NikolaiPogorelov*·LutzRastaetter*·IanRichardson*·NeelSavani*·BarbaraThompson*·Karlheinz Trattner*·RodneyViereck ·BrianWalsh·Chunming Wang*·DanielWelling*·Yongliang Zhang*·Yihua Zheng*·
*attendingCCMC-LWSworkingmeeting
![Page 5: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/5.jpg)
CMEArrivalTime&ImpactWorkingTeam
Participatingmodels✳ DBM(Vrsnak &Zic)✳ ElEvo (EllipseEvolution)(Moestl)✳ ElEvoHI (EllipseEvolutionbasedonHI) (Amerstorfer)✳ Enhanceddrag-basedmodel(Hess&Zhang) [set1results]✳ EUHFORIA (Pomoell)✳ SARM(Núñez) [set1results]✳ SUSANOO-CME✳ WSA-ENLIL+Cone (Arge,Odstrcil) [set1results]✳ contactustoaddyourmodel
![Page 6: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/6.jpg)
APRIL4•TUESDAY9:00am– 10:15amMetrics•Metricsdiscussionandexamples•Metricsaddressinguserneedsvs.scientificresearch•Discussionquestions•1stsetofeventsforvalidation•Preliminarymodelvalidation/results•AlexWold:Real-timeENLILrunvalidation&discussion•Marilena Mierla (notattending):comparisonsofENLILandEUHFORIA•More2-3slidecontributionsfromparticipants•CMEscoreboarddiscussion•SuzyBingham:InitialCMEscoreboardverificationfromtheUKMetOffice
TeamAgendaMetrics&Impactofbackgroundsolarwind•ContinueitemsremainingfromMetricssession•Scenesettingpresentationonquantifyingtheeffectsofbackgroundsolarwind•Discussiononimpactofbackgroundsolarwind,2-3slidecontributionsfromparticipants•Continueotherdiscussionitemsfromprevioussessions
Summaryandfutureplans•Remainingdiscussionquestions•QuantifyingprogressinthefieldofCMEarrival&impact•Summaryofteamprogress•Futureplans,meetings,remotecollaboration
APRIL5•WEDNESDAY10:45am– 12:00pm
APRIL6•THURSDAY4:45pm– 6:00pm
![Page 7: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/7.jpg)
APRIL4•TUESDAY9:00am– 10:15amMetrics•Metricsdiscussionandexamples•Metricsaddressinguserneedsvs.scientificresearch•Discussionquestions•1stsetofeventsforvalidation•Preliminarymodelvalidation/results•AlexWold:Real-timeENLILrunvalidation&discussion•Marilena Mierla (notattending):comparisonsofENLILandEUHFORIA•More2-3slidecontributionsfromparticipants•CMEscoreboarddiscussion•SuzyBingham:InitialCMEscoreboardverificationfromtheUKMetOffice
TeamAgendaMetrics&Impactofbackgroundsolarwind•ContinueitemsremainingfromMetricssession•Scenesettingpresentationonquantifyingtheeffectsofbackgroundsolarwind•Discussiononimpactofbackgroundsolarwind,2-3slidecontributionsfromparticipants•Continueotherdiscussionitemsfromprevioussessions
Summaryandfutureplans•Remainingdiscussionquestions•QuantifyingprogressinthefieldofCMEarrival&impact•Summaryofteamprogress•Futureplans,meetings,remotecollaboration
APRIL5•WEDNESDAY10:45am– 12:00pm
APRIL6•THURSDAY4:45pm– 6:00pm
![Page 8: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/8.jpg)
CMEArrivalTime&ImpactWorkingTeam
ConsiderationsforEventSelectionConsiderationsforeventselection:✳ SingleCMEevents(fastandslow)✳ MultipleCMEevents(interactingandnon-interacting)✳ CMEeventsthatareexpectedtoarrivebutdonot(falsealarm)✳ FlankimpactCMEevents✳ ConsidereventsfromtheISESTWG4wikipage✳ Overlapsomeeventswiththe IMFBzandL1workingteam
✳ Forsomevalidationmethods,howmanyeventsareneededtobestatisticallysignificant?✳ Eventselection:Shouldwehavea"trainingset","validationset",and"testset"— wherethe"test"setisnotrevealeduntilalaterstage?
![Page 9: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/9.jpg)
CMEArrivalTime&ImpactWorkingTeam
1st setofeventssmallcoreselectiontoexplorechosenmetrics&validationtechniques
• Fourevents:twohits,oneproblematichit,andonefalsealarm.
• Aimfor2hitstowilloverlap withtheIMFBz workingteam’seventlisttoreducetheoverallmodelingburden(forthosemodelsthatpredictbotharrivalandBz).
• Ifdesired,theCMEparametersprovidedonthewebsite(takenfromliterature)canbeused
A)3April201010:33UT(hit)B)15March201307:12UT(hit)C)7January201418:24UT(falsealarm;onlyaweakdiscontinuityarrives)D)15March201501:48UT(hit;problematic,manymodelspredictalatearrival)
Resultsfrom4modelsfortheseeventshavebeenpostedonourwebsite:https://ccmc.gsfc.nasa.gov/assessment/topics/CME/events.php#results
![Page 10: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/10.jpg)
• ThevalidationofCMEforecastingfocusedaroundarrivaltimebecauseitwastheeasiestthingtoagreeon.Longtermgoalistovalidateintensity andduration also.
• Importanttorealizethatquantitiestovalidateforresearchisdifferentoperations.E.g.littleinterestinT or nverificationforoperations.
• Scienceresearchfocusisonhowwellmodelforperformsformostparameters
• Foroperations themostimportantquantitiesareBz andv (couplingfunction)
• Quantitiesimportantinthisorderforoperations:timing(arrival),intensity,andduration
Validation:UserNeedsvsResearchNeedsfeedbackfromSWPC
![Page 11: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/11.jpg)
Validation:UserNeedsfeedbackfromSWPC
Quantitiestovalidate:• timing(arrival),intensity,andduration
• Arrivaltime:powergridisinterestedintiming
• SuggestusinghistoricICMEobservationstodeterminewhattovalidate(e.g.whatintensity,duration,orthresholdisimportanttotestmodelperformance)
• Kp usedbyforecasters,butitmightbeusefultocompareDstpredictionsformodelsfortheirperformanceforecastingofstormintensityandduration.
• Alwaysusefultoprovideameasureofuncertainty
• Abest/worstcasescenarioisusefulforuserstomakedecisions
![Page 12: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/12.jpg)
✳ WhataretheeffectsofthemodelinputsontheCMEarrivaltimeandimpacts✳ modelparameters✳ CMEparameters✳ inputmagnetograms✳ …
✳ Isthereasolarcycledependenceonmodelperformance?
✳ DowewanttofixtheCMEinputparametersandinputmagnetograms(ifapplicable)forallmodels?
Ifso,istheteamcomfortablefortheCMEparameterstobedeterminedbyanexpertthatisnotamodelerintheCMEArrivalTimeWorkingteamtoremovebias?
✳ Whataresomegoodtechniquestodeterminetheuncertainty/confidence ofthearrivaltimeprediction?
Discussionquestions:models
![Page 13: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/13.jpg)
QuantitiestoValidate:someideastostartICME...✳ arrivaltime✳ averagemagneticfieldmagnitude✳ averagetemperature✳ averagespeed✳ duration✳ resultinggeomagneticstormstrengh (Kp,Dst,...)
✳ Arrivaltime:☼ RMSE,meanabsoluteerror(MAE),meanerror(ME),others?
✳ Categorical(yes/no)predictions:☼ skillscoresbasedoncontingencytables☼ probabilisticandcontinuouspredictionscanbeconvertedtocategoricalusingthreshold
✳ Probabilisticpredictions:☼ Reliabilitydiagram,BrierSkillScore,...
Skillscores/metrics
Discussion
![Page 14: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/14.jpg)
✳ Whichcatalog touseforICMEarrival?Takeanaverage?
✳ Overwhatintervalshouldaveragein-situobservationsbederived?Useacatalog?
✳ Directlycomparetimeseriesw/observationsforsomemodels?Time-shiftmodelresults?
✳ Alsovalidatethethemagneticcloudarrivalinadditiontotheshock/discontinuity?
✳ Validatean"impactparameter"extractedfrommodelresults?Comparetoin-situfluxropefitparameters?
✳ Howcanwevalidateandquantifytheeffectofthebackgroundsolarwindpredictiononthearrivaltimeprediction?
✳ HowdointeractingormultipleCMEevents,orSIR+CMEeventsimpactthechosenmetrics?Howtoquantifymodelperformancefortheseevents?
Discussionquestions:Quantities&Observationswhattoquantitiestovalidate,andwhattocomparethemto
![Page 15: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/15.jpg)
✳ Whatisagoodbaselinemodelor climatologytocompareagainst?
✳ Forthehitcalculation:☼ Howtodefineacategoricalyes/nofor"modelpredictedarrival"- human
analysisofmodelresultsoralgorithm?Whatanalysismethod?
☼ IfthemodelpredictedarrivaltimeismorethanxnumberofhoursfromtheobservedCMEarrivaltimeisitahit?Orafalsealarmandmiss?Varythedefinitionofthehitdependingonuserneeds?
✳ Probabilisticprediction:whatthreshold touseforhit/miss?Varyandexplore?
✳ Howbesttoquantifyuncertainty intheskillscoreresultsbasedonvalidationsamplesize,uncertaintiesinobservations,andfromanyothersources.
Discussionquestions:Metrics
![Page 16: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/16.jpg)
Allpredictionmethodsarewelcomeandallareencouragedtoparticipate.Participationfromthecommunity:• Allpredictionmodelsandmethodsarewelcomefromtheworld-wideresearch• community(currently19methodsareregistered)• UserssubmittheirpredictionsforongoingCMEevents,listingtheirmethod• assumptionsandinputparameters• Researcherscanthenviewallofthepredictions,modelingdetails,andthe• ensembleaverageofallpredictedarrivaltimessubmittedbyparticipants
http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard
TheCMEscoreboardisaresearch-based forecastingmethodsvalidationactivitywhichprovidesacentrallocationforthecommunityto:
• submittheirforecastinreal-time• quicklyviewallforecastsatonceinreal-time• compareforecastingmethodswhentheeventhasarrived• viewtheaverageofallforecastsforeachevent(ensemble).
CMEArrivalTimeScoreboard
![Page 17: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/17.jpg)
Communitypredictionsforthe5Nov2016CME
Pleasejoin!Allpredictionmethodsarewelcomeandallareencouragedtoparticipate.
http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard
![Page 18: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/18.jpg)
CommunitypredictionsfortheJanuary7,2014CME(X1.2flare):
http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard
Averageofallsubmissions:12hoursearly,Kp geomagneticindex6to7.615submissions
Pleasejoin!Allpredictionmethodsarewelcomeandallareencouragedtoparticipate.Therearecurrently19registeredmodels.
![Page 19: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/19.jpg)
http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboardAnyonecanviewpredictions,pleaseregistertosubmitpredictions.
![Page 20: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/20.jpg)
BeginbyclickingAddPredictionunderthe"ActiveCMEs"sectionandselectyourforecasting"MethodType"fromthelist.Whileloggedin,ifyoudonotseeanyCMEs listedunderthe"ActiveCMEs"section,clickAddCMEtogetstarted.
http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard
![Page 21: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/21.jpg)
http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard
![Page 22: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/22.jpg)
Suggestedimprovementscomingsoon:• AutomaticforecastsubmissionviaanXMLfile• MailinglistthatnotifiesuserswhenanewCMEhasbeenaddedtothe
scoreboard• SeparategeomagneticstormscoreboardthatcanlinktoCMEscoreboard
Futureplans:• Showingdataintableinplotform• Automaticskillscorecalculations• QualityfactorforconfidenceinobservedICMEassociatedshockarrival• QualityfactorforconfidenceinlinkingobservedICMEarrivalwithCMEin
coronagraph• Yourideas?
CMEArrivalTimeScoreboard
http://kauai.ccmc.gsfc.nasa.gov/CMEscoreboard
![Page 23: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/23.jpg)
Discussion:CMEarrivaltime&impactvalidationtechniques
![Page 24: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/24.jpg)
CMEArrivalTimeErrorValidationExamples
![Page 25: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/25.jpg)
CMEArrivalTimeErrorValidationExamples
![Page 26: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/26.jpg)
0
20
40
60
80
100
0 20 40 60 80 100
Obs
erve
d R
elat
ive
Freq
uenc
y (%
)
Forecast Probability (%)
CME Arrival: Reliability Diagram of 52 Ensembles
line of
perfe
ct rel
iability
(5)
(4)
(4)
(2)
(7) (30)
Assessment:Confidence(likelihood)inCMEarrival
• ExamplereliabilitydiagramforCCMC/SWRCarrivaltimeforecasts
• Underforecasting intheforecastbinsbetween40-80%
• Slightlyoverforecasting inthe80-100%forecastbins
overforecasting
under-forecasting
NeedtoimproveconfidenceinCMEarrivalforecast:• ConsiderbetterwayoftranslatingCME“impactparameter”into
probabilitythattheCMEwillarrivewhichmoreaccuratelyrepresentshead-onvs.grazingimpacts(andtherangesinbetween)
Caution:smallSamplesizesinsomebins!!
![Page 27: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/27.jpg)
LikelihoodofCMEarrivalforecastverification:BrierScore
AmethoddefiningthemeansquaredprobabilityforecasterrorsistheBrierScore:
N=numberofevents,pi =forecastprobabilityofoccurrenceforeventi,oi =1iftheeventwasobservedtooccurand0ifitdidnot.
Rangesfrom0to1,with0beingaperfectforecast.
UsingtheforecastprobabilityaboutthelikelihoodthattheCMEwillarrivesubmittedonthescoreboard.
TheBrierSkillScore(BSS)isthetheBrierscorerelativetoclimatology
Note:confidenceintervalsshouldbecomputedforverificationscores
![Page 28: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/28.jpg)
LikelihoodofCMEarrivalforecastverification:Reliability
0
0.1
0.2
0.3
0.4
0.5
0.6
1 2 3 4 5 6 7 8 9 10Pr
obab
ility
of R
ank
Rank
(b) CME Arrival Time: Rank Histogram of all Ensembles
Howwelldoestheensemblespreadrepresentthetruevariabilityoftheobservations?
TheU-shapedrankhistogramforsuggestsundervariability,indicatingthattheseensemblestonotsampleawideenoughspreadinCMEinputparameters.
![Page 29: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/29.jpg)
EnsembleValidationSummary• Ensemble modeling gives a probabilistic forecast which includes an estimation of arrival time uncertainty from the spread in predictions and a forecast confidence in the likelihood of CME arrival.
• First results for 30 event sample: mean absolute arrival time error of 12.3 hours, RMSE of 13.9 hours, and mean error of -5.8 hours (early bias), comparable with other CME arrival time prediction errors reported in the literature.
• It was found that the correct rejection rate is 62%, and the false-alarm rate is 38%.
• Brier Score of 0.15 shows that the likelihood of CME arrival prediction is fairly accurate.
• However, the reliability diagram shows that the ensemble simulations are underforecastingthe likelihood that the CME will arrive in the forecast bins between 20-80%, and slightly overforecasting in the 1-20% and 80-100% forecast bins.
• For 8 out of 17 of the ensemble runs containing hits, the observed CME arrival was within the spread of ensemble arrival time predictions. The initial distribution of CME input parameters was shown to be an important influence on the accuracy of CME arrival time predictions. The rank histogram suggests undervariability in initial conditions; i.e., these ensembles do not sample a wide enough spread in CME input parameters.• The observed Kp was within ±1 of the predicted mean Kp for 11 out of 17 of the ensembles.
• Kp prediction errors: mean absolute error of 1.4, RMSE of 1.8, and mean error +0.4. • Overall tendency for the overprediction of Kp, for CME input speeds above ~1000 km/s.
![Page 30: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/30.jpg)
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9
Prob
abilit
y of
Occ
urre
nce
(%)
KP
Probabilistic Kp Forecast Distribution (18 April 2014)90˚
135˚180˚
90˚-180˚ Kp isforecastusingENLILpredictedsolarwindquantitiesatEarthasinputtotheNewelletal.(2007)couplingfunction forthreeclockanglescenarios(ΘC=90°,135°,and180°)andallthreeanglescombined,assumingequallikelihood.
18April2014CME:DistributionofKp probabilityforecast
• ObservedKp:5duringperiod12:00-15:00UTon20April.• 84%oftheforecastsfallbetweenKp =5to7.ThemostlikelyforecastisforKp=7at41%,followedbyKp=5at27%andKp=6at16%likelihoodofoccurrence.• UsingthemeanKp forecastofKp=6,thepredictionerrorisKperror=Kppredicted- Kpobserved =1(overprediction)
Kp =5observed
![Page 31: CME Arrival Time & Impact Working Team...Apr 18, 2014 · CME Arrival Time & Impact Working Team Evaluate where we stand with CME arrival time and impactprediction Establish community-agreed](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e93e4bab1d7b25bcb7ead40/html5/thumbnails/31.jpg)
Simulatedvs.ObservedCMEParametersØ Thedifferencefrom
differentobservationdatacanaffecttheresults.Forexample,thedifferenceofVmaxfromOMNIandACEis>200km/sfor3CMEs.ThecorrelationforNpmax isweakerifusingACE
Ø InseveralcaseswheretheCMEVmax isoverestimated,thereareinteractionsofmultipleCMEs
Ø Usingthefixedparameters(a6b1),theVmaxandNpmax areunderestimated.Theyareoverestimatedinthecaseofself-adjustedparameters(ace3b)
Ø SimilartrendsarefoundforthecorrelationsofmeanvaluesofCMEparameters.Themeantemperatureareoverestimatedinbothsettings FromLanJian