ProjectName:HighFrequencyPhysics-BasedEarthquakeSystemSimulations(Year1of2)EndofYearSummary

PINAME:ThomasH.JordanCo-PI(s):JacoboBielak,CarnegieMellonUniversity,

PoChen,UniversityofWyoming,YifengCui,SanDiegoSupercomputerCenter,

PhilipMaechling,SouthernCaliforniaEarthquakeCenter,KimOlsen,SanDiegoStateUniversity,RicardoTaborda,UniversityofMemphis

PROGRAM:INCITEALCFYEAR:2015

PerformancePeriod:January1-December31,2015(Q1,Q2,Q3,Q4)

ALCFProjectName:GMSeismicSimOLCFProjectName:GEO112

ProjectUsagePleasecommentontheuseofyourMiraallocation.Forexample,thiscouldincludeperiodsoflowutilizationduetocode/modeldevelopment,periodsofhighutilizationduetosignificantcampaigns,significantuseofthebackfillqueueortheINCITEoverburnpolicy.Ifmorethan50%ofyourallocationwasusedupbyjobsbelowthecapabilitysize(lessthan8-racks),pleasecommentonthetechnicalbarrierspreventingyourworkloadorcodefromscalingto8-racksandbeyond.Whileour2015Mirausageproducedimportantscientificresults,ourusagepatternwasnotoptimal.Oneissueisthatweusedour2015ALCFallocationinthefirstsixmonthsoftheyear.Then,wecontinuedtorunjobsataslowerrate,usingtheback-fillqueue,duringtheremainderoftheyear.WewillplanforamoreconsistentusageofALCFcomputingtimeduring2016toavoidrunningoutofallocationhoursonMiraduring2016.Anotherissueweareworkingtoaddressisthesizeofourroutinejobs.OurbestresultsfromMirain2015arefromouriterative,data-intensive,full3Dtomographicimprovementsto3DseismicvelocitymodelsforCalifornia.ThestandardcalculationsinthisprocessonMiraarenow1rackinsize.However,in2016weexpecttoruntomographicinversionsvaliduptohigherfrequencies,whichwillrequireatleastafactorof8increaseincomputationalscalefortheseroutinejobs.However,thedatastoragerequirementsforourinversionswillalsoincreasesubstantiallywhenwemovetohigherfrequencies.WearecurrentlyevaluatingdatacompressiontechniquesforuseonourALCFdatathatwillreducetherequirementsofthesehigherfrequencyinversions.WehaveidentifieddatacompressionevaluationasprioritywithourALCFCatalystfor2016.

ReportonProjectMilestones• Pleaseinsertacopyofthemilestonesasoutlinedintheproposal• Provideastatusoneachofthemilestones• Listanyadditionalmajoraccomplishmentsthusfarthisyear.Pleaseinclude

scientificandcomputationaldetailsofsimulationsundertaken,includingimagesifpossible

• Listanycommentstoclarifythestatusoftheprojecto Forexample,fundingchallengesslowedwork,changeofsciencescope

withanexplanation,etc DuringthisINCITEallocation,researchersfromtheSouthernCaliforniaEarthquakeCenter(SCEC)usedcomputingtimeandstoragespaceonALCFandOLCFINCITEcomputationalresourcestocarryoutresearchonhigh-frequencyphysics-basedearthquakesystemsimulations.Whilethis2015end-of-yearreportisrequestedbyALCF,wedescribetheoverallprogressinourINCITE-supportedresearchprogramusingbothOLCFandALCFresources.Wherepossible,weidentifywhichINCITEresourceswereusedtoproducespecificresults.In2015,SCECusedINCITEresourcestoinvestigateabroadrangeofscientificissues.Our2015-2016INCITEproposaldefinedthefollowingfourobjectives:

• O1:Improvetheresolutionofdynamicrupturesimulationsbyanorderofmagnitudeandinvestigatetheeffectsofrealisticfrictionlaws,geologicheterogeneity,andnear-faultstressstatesonseismicradiation.

• O2:Extenddeterministicsimulationsofstronggroundmotionsto10Hzforinvestigatingtheupperfrequencylimitofdeterministicground-motionprediction.

• O3:Computephysics-basedProbabilisticSeismicHazardAttenuation(PSHA)mapsandvalidatethoseusingseismicandpaleo-seismicdata.

• O4:Improve3Dearthstructuremodelsthroughfull3Dtomographyusingobservedseismicityandambientnoise.

Duringthefirstyearofourcurrentallocation,SCECresearchershaveprogressedtowardseachoftheseobjectives:WedefinedandimprovedanewcentralCalifornia3Dvelocitymodelusingfull3Dtomography(O4).Weproducedacomprehensive,physics-basedhazardmodelfortheLosAngelesregionvaliduptoseismicfrequenciesof1Hz(O3),weperformedhigh-frequencysimulations(upto8Hz)onTitanusingGPU-optimizedfinite-differencecodes(O2)whileincorporatingnewaspectsofearthquakephysics(O1).Here,weprovideaprogressupdateonthemilestoneswedefinedinour2015INCITEallocationrequest,followedbyasummaryofourmostsignificantaccomplishments.Theprogressmadein2015highlightsonceagainhowthispartnershipallowsustoperformthemostscientificallyadvancedearthquakegroundmotionandseismichazardcomputationsconductedworldwide.

SummaryofMilestonesProgress:Weidentifiedthefollowingeightmilestones[M1–M8]inouroriginalINCITEallocationrequest.Abriefsummaryofprogressispresentedbelow.

Year1MilestoneDescriptions MilestoneAchievementStatusM1 Usefull3Dtomographyandcomparative

validationsusingtoimproveexistingCaliforniavelocitymodelsforuseinhighfrequencywavepropagationsimulationsat0.2Hz

Achieved.UsedMiratocalculatesixiterationsofCentralCaliforniaModel.Theimproved3DmodelisnowavailabletogroundmotionmodelersthroughSCECdistributionsoftwarecalledUCVM.

M2 Runhighfrequencyforwardsimulationsusingalternativematerialattenuation(Q)andseismicvelocitymodels(CVMs).Comparetheimpactofmaterialproperties,topography,andmodelsincludingspatialvariability(heterogeneities)andsoft-soildeposits(orgeotechnicallayers)on4Hz+simulationsbysimulatingforwardeventsusingalternativemodelsandcomparingresultsamongsyntheticsandwithdata.

Started,notcompleted.AWP-ODCandHerculescodebrancheshavetestedthesephysics.Currentlyrunningbaseline4Hzsimulationswithoutthesephysicswithgoodagreementat4Hzamong3wavepropagationcodeswithasimplevelocitystructure.

M3 Runhighfrequencyforwardsimulationsusingalternativeapproachestoincludetheeffectsofoff-faultandnear-surfaceplasticdeformation.Comparetheimpactofalternativeplasticitymodels(linear-equivalent,3D+1Dhybrid,full3Dplastic)on4Hz+simulationsbysimulatingforwardeventsandcomparingtheresultsamongsyntheticsandwithempiricalrelationshipsanddata.

Started,notcompleted.AWP-ODCandHerculescodebrancheshavetestedthesephysics.Currentlyrunningbaseline4Hzsimulationswithoutthesephysicswithgoodagreementat4Hzamong3wavepropagationcodeswithasimplevelocitystructure.

M4 Calculatea1.0HzCyberShakeHazardcurve.UseupdatedCVMs,sourcemodels,andcodestocalculateahigherfrequencyCyberShakehazardcurve

Achieved.UsedTitanandBlueWaterstocalculateaCyberShake1HzLosAngelesareaprobabilisticseismichazardmodelbasedon336site-specifichazardcurves.

Year2MilestoneDescriptions ObjectiveM5 Usefull3Dtomographyandcomparative

validationsusingtoimproveexistingCaliforniavelocitymodelsforuseinhighfrequencywavepropagationsimulationsat0.5Hz

Notstarted

M6 Runhighfrequencyforwardsimulationsusingalternativematerialattenuation(Q)andseismicvelocitymodels(CVMs).Comparetheimpactofmaterialproperties,topography,andmodels

Notstarted

includingspatialvariability(heterogeneities)andsoft-soildeposits(orgeotechnicallayers)on8Hz+simulationsbysimulatingforwardeventsusingalternativevelocitymodelsandcomparingtheresults.

M7 Runhighfrequencyforwardsimulationsusingalternativeapproachestoincludetheeffectsofoff-faultandnear-surfaceplasticdeformation.Comparetheimpactofalternativeplasticitymodels(linear-equivalent,3D+1Dhybrid,full3Dplastic)on8Hz+simulationsbysimulatingforwardeventsandcomparingtheresultsamongsyntheticsandwithempiricalrelationshipsanddata.

Notstarted

M8 Calculatea1.5HzCyberShakeHazardcurve.UseupdatedCVMs,sourcemodels,andcodestocalculateahigherfrequencyCyberShakehazardcurve

Notstarted

MajorProjectAccomplishments:1)Wecompleteda1-HzurbanseismichazardmodelfortheLosAngelesregion(OLCFwithproductsfromALCF).SCEC'sresearchteamusedtheOLCFTitanandNCSABlueWaterssupercomputerstoperformCyberShakeStudy15.4.ThiscomputationdoubledthemaximumseismicfrequencyrepresentedintheLosAngelesurbanseismichazardmodel,from0.5Hzto1Hz(Figure1).Seismichazardcurveswerederivedfromlargeensemblesofseismogramsatfrequenciesbelowthismaximumfor336surfacesitesdistributedacrosstheLosAngelesregion.Thisnewprobabilisticmodelusesrefinedearthquakerupturedescriptionsthroughrevisionstotheconditionalhypocenterdistributionsandtheconditionalslipdistributions.ThisseismichazardcalculationusedtheCVM-S4.26.GTL3Dvelocitymodel,whichwasvalidatedandimprovedusingALCFMira,asthebestavailablesouthernCalifornia3Dvelocitymodel.TheCS15.4modelprovidesnewseismichazardinformationofinteresttobroadimpactcustomersofCyberShake,includingseismologists,utilitycompanies,andcivilengineersresponsibleforCaliforniabuildingcodes.Thenewmodel,whichsamplesthecompleteUniformCaliforniaEarthquakeRuptureForecast,willberegisteredintotheUSGSUrbanSeismicHazardMappingProject(http://earthquake.usgs.gov/hazards/products/urban/).

TheGPU-basedanelasticwavepropagationAWP-ODCsoftwarewascoupledwiththeCPU-basedpost-processingcalculationsthatsynthesizedover300millionseismograms.InStudy15.4,SCECutilizedapproximately200pilotjobstorunCyberShaketasksonTitanresources.Over80%ofthenode-hoursburnedonTitanwerefromjobswhichranon25%ormoreofthemachine.Approximately200TBofSGTdatawastransferredfromTitantoBlueWatersautomaticallyaspartoftheworkflow.OnTitan,theacceleratedcalculationsoftheGPUStrainGreenTensor(SGT)implementationis6.3timesmoreefficientthantheCPUimplementation,comparedtheperformanceonXK7toXE6atnode-to-nodelevel,whichsavedusmorethansixtymillionofcore-hoursoverthecourseofthestudy.OurGPUdevelopmentwasrecognizedwithNVIDIA’s2015GlobalImpactAward.“Thefullthree-dimensionaltreatmentofseismic-wavepropagationhasthepotentialtoimproveseismichazardanalysismodelsconsiderably,andthatiswheretheacceleratingtechnologyisparticularlyhelpfulatthismoment,”saidThomasJordan,directorofSCEC.“WithGPUcomputingpowerwe’regaininginsightastohowthegroundwillmoveinhigh-riskareas,andhowwecanbetterplanfortheaftermathofamajorevent.”http://www.hpcwire.com/off-the-wire/sdsc-researchers-awarded-nvidia-2015-global-impact-award/2)WerefinedthecentralCalifornia3Dvelocitymodel,thatwecallCentralCaliforniaArea(CCA),andusedfull3Dtomographycomputationalmethodsto

validateandimprovethe3Dseismicvelocitymodelusingbothobservedmoderateearthquakesandambientseismicnoiseobservations(ALCF).Wehavefurtherimprovedthecomputationalefficiencyofourfull3Dtomography(F3DT)workflowonALCFMiraandarenowapplyingittoCentralCaliforniaandstatewide.Anaccurate3Dvelocitymodelisessentialinputneededforaccuratedeterministicearthquakewavepropagationsimulations.GradualimprovementsinourvelocitymodelshaveallowedustoincorporateanincreasingvolumeofobservedseismogramsintoourF3DTworkflow,whichisallowingustoresolvefinerstructuraldetailswithhigheraccuracy.Figure2belowshowsthestudyareaandresultsfromthe6thiteration.Thestartingmodelisdefinedusinga500mgridspacingandweusetrilinearinterpolationinbetweenthegridpointswhenconstructingmeshes.Themodelcoversdepthsdownto50km.Wehavealsointegratedalossy,onlinecompressioncodezfpintothefull-3Dtomographybasedonthescattering-integralmethod(F3DT-SI)workflow.ThelastF3DTimplementation,inparticularthescattering-integralimplementation(F3DT-SI),requiredhighdiskstoragecostandtheassociatedI/Ooverheadofarchivingthe4Dspace-timewavefieldsofthereceiver-orsource-sidestraintensors.WehavesuccessfullyintegratedalossycompressionalgorithmintoourF3DT-SIworkflowtosignificantlyreducethediskspaceforstoringthesedata.

Figure2.Shearwave(Swave)velocityat(top)2km,(middle)10km,and(bottom)20kmdepthsin(left)theinitialmodelCCA00,(middle)the6thiterationmodelCCA06,and(right)theperturbations.ThecolorbaronthelowerleftcornerofeachplotshowstherangeofthecolorscalewithredindicatingrelativelyslowSwavevelocitiesandblueindicatingrelativelyfastSwavevelocities.Blacksolidlinesshowmajorfaultsinourstudyarea.3)Weperformedhigh-frequency(upto8Hz)simulations(OLCFTitan)TheSCECresearchersranhighfrequencygroundmotionsimulationsonTitanusingGPU-enabledfinitedifferentwavepropagationsolvers,andafinite-elementwavepropagationsolver,Hercules,whichintegratesanefficientoctree-basedhexahedralmeshgeneratorwithanexplicitFEformulation.Thesecodeswereusedinaverificationandvalidationstudiesforthe2014Mw5.1LaHabraearthquakeonTitan,totesttheaccuracyofthecodes,andtoexaminehowclosethepredictedgroundmotionsaretoobservations.Wemodelgroundmotionvariabilityforlargestrike-slipandblindthrustearthquakesincludingfrequenciesupto8Hz.Theearthquakesourceisobtained

−122˚

−122˚

−120˚

−120˚

−118˚

−118˚

−116˚

−116˚

34˚

36˚

38˚

(a) CCA00 Vs 2.0 km−122˚ −120˚ −118˚ −116˚

34˚

36˚

38˚

2.42.62.83.03.2

−122˚

−122˚

−120˚

−120˚

−118˚

−118˚

−116˚

−116˚

(b) CCA06 Vs 2.0 km−122˚ −120˚ −118˚ −116˚

2.42.62.83.03.2

−122˚

−122˚

−120˚

−120˚

−118˚

−118˚

−116˚

−116˚

34˚

36˚

38˚

(c) dVs 2.0 km−122˚ −120˚ −118˚ −116˚

34˚

36˚

38˚

−20−16−12−8−4

048

121620

%

−122˚

−122˚

−120˚

−120˚

−118˚

−118˚

−116˚

−116˚

34˚

36˚

38˚

(d) CCA00 Vs 10.0 km

34˚

36˚

38˚

3.0

3.2

3.4

3.6

3.8

4.0

−122˚

−122˚

−120˚

−120˚

−118˚

−118˚

−116˚

−116˚

(e) CCA06 Vs 10.0 km−122˚ −120˚ −118˚ −116˚

3.0

3.2

3.4

3.6

3.8

4.0

−122˚

−122˚

−120˚

−120˚

−118˚

−118˚

−116˚

−116˚

34˚

36˚

38˚

(f) dVs 10.0 km−122˚ −120˚ −118˚ −116˚

34˚

36˚

38˚

−15−12−9−6−3

0369

1215

%

−122˚

−122˚

−120˚

−120˚

−118˚

−118˚

−116˚

−116˚

34˚

36˚

38˚

(g) CCA00 Vs 20.0 km

34˚

36˚

38˚

3.6

3.8

4.0

−122˚

−122˚

−120˚

−120˚

−118˚

−118˚

−116˚

−116˚

(h) CCA06 Vs 20.0 km−122˚ −120˚ −118˚ −116˚

3.6

3.8

4.0

−122˚

−122˚

−120˚

−120˚

−118˚

−118˚

−116˚

−116˚

34˚

36˚

38˚

(i) dVs 20.0 km−122˚ −120˚ −118˚ −116˚

34˚

36˚

38˚

−10−8−6−4−2

02468

10%

throughdynamicrupturepropagationusingSORDalongseveralrealizationsofroughfaulttopographies.Anensembleofsourcesismodeledbyvaryingthehypocenterlocationthatresultsinsimilarmomentmagnitudes.Theslip-ratedatafromtherupturemodelingisconvertedtoakinematicsourceandinputintothewavepropagationcodeAWP-ODC,whichincorporatesfrequency-dependentattenuation(Q(f),Withersetal.,2015)aswellasDrucker-Pragerplasticity.Wealsoincludesmall-scalemediumcomplexityinbotha1D-layeredmodelanda3DmediumextractedfromSCECCVM-S4includingasurfacegeotechnicallayer.ThemediangroundmotionfollowsasimilardecayascomparedtoGMPEswhenusingaQ(f)power-lawexponentintherange0.6-0.8.Nonlineareffectsareneededtoreducenear-fieldgroundmotiontoobservablelevelsinregionsofnear-surfacelowvelocitylayers.Small-scalemediacomplexityisobservedtodecreasethepolarizationratiotothatofsimilartoobservations.Theintra-eventvariabilityforthelayeredmodelsimulationsisnearobservedvaluesofsingle-stationstandarddeviation.Small-scaleheterogeneitycansignificantlyaffecttheintra-eventvariabilityatfrequenciesgreaterthan~1Hz,becomingincreasinglyimportantatlargerdistancesfromthesource.Theintra-eventvariabilityofoursimulationsintheCVMistypicallylargerthanthatfortheobservationsatfrequencies>1Hz.However,thisdiscrepancytendstodecreasewhensmall-scaleheterogeneitiesareincluded,suggestingtheneedforahighlycomplexvelocitymodeltofitgroundmotionvariability.Plasticeffectsinthemediumalsoreducethevariability,particularlyatdistancesclosetothesource.Weuseanon-associatedDrucker-PragernonlinearrheologyfollowingthereturnmapalgorithmintheAWP-ODCcodetomodelgroundmotionsfromtheM7.8ShakeOutscenariosourcedescription(M3)includingpermanentrockdeformation.Toaccountfornonlinearityinthefaultdamagezoneandinnear-surfacesediments,weusedatwo-stepmethod.Inthefirststep,thedynamicruptureprocessincludingplasticitywassimulatedinasmallcomputationaldomainusingtheCPUversionoftheAWP-ODCfinitedifferencecode.ThesedynamicsimulationswereperformedonNCSABlueWatersorTACCStampede,whereakinematicsourcewasalsogeneratedfromthedynamicsolution.Thesourcepartitions,whichrequireupto4Tbofdiskspace,werethentransferredtoOLCFTitanusingGlobusGridFTP.ThewavepropagationresultingfromthekinematicsourcewassimulatedonTitanwithAWP-ODC,usingalargervelocitymeshthatincludesmajorurbanareasintheregion.Thesesimulationsareperformedusing700GPUsduring4hoursforamaximumfrequencyof2Hzor5,600GPUsduring7hoursforamaximumfrequencyof2Hz.ResultsconfirmthatnonlineardeformationwouldoccurinawideareaaroundthefaultandintheSanBernardinoandLosAngelesbasins,reducinglong-periodgroundmotionsintheLosAngelesbasinbyupto50%withrespecttothelinearsolution(Figure3).Theimportanceofnonlinearityincreasesassimulationsareperformedathigherfrequencies.Weexploretheeffectsoffaultzonenonlinearityonpeakgroundvelocities(PGVs)bysimulatingasuiteofsurfacerupturingearthquakesinavisco-plasticmedium.Oursimulations,covermagnitudesfrom6.5to8.0,withseveral

realizationsofthestochasticstressdropforagivenmagnitude.Wetestthreedifferentmodelsofrockstrength,withfrictionanglesandcohesionsbasedoncriteriawhicharefrequentlyappliedtofracturedrockmassesincivilengineeringandmining.Weuseaminimumshear-wavevelocityof500m/sandamaximumfrequencyof1Hz.Inrupturescenarioswithaveragestressdrop(~3.5MPa),plasticyieldingreducesnear-faultPGVsby15to30%inpre-fractured,low-strengthrock,butlessthan1%inmassive,highqualityrock.Thesereductionsarealmostinsensitivetothescenarioearthquakemagnitude.Inthecaseofhighstressdrop(~7MPa),however,plasticityreducesnear-faultPGVsby38to45%inrocksoflowstrengthandby5to15%inrocksofhighstrength.Becauseplasticityreducesslipratesandstaticslipnearthesurface,theseeffectscanpartiallybecapturedbydefiningashallowvelocity-strengtheninglayer.Thesesimulationresultssuggestthatnonlineareffectsmayberelevantevenatlongperiods,especiallyforearthquakeswithhighstressdrop.

ProjectProductivity• ListPapersfromworkatALCFandincludeasentenceabouthowthisisrelated

totheproject.Examplesare:o “Thisworkwasdonewithaone-monthcampaignon32racksofMira.”o “TheseresultsweredeterminedusingdatageneratedfromanALCC

project.”o “TheseresultsarefromcollaboratorswhodidnothaveALCCtimebut

usedtheresultsofourwork.”o Etc.

• ListPresentationsworkatALCFhttp://www.alcf.anl.gov/projects/high-frequency-ground-motion-simulation-seismic-hazard-analysis

• Listanyotherawards,honors,mediacoverage,etc

Papers

Lee,E.-J.,P.Chen,T.H.Jordan,P.J.Maechling,M.A.M.Denolle,andG.C.Beroza(2014),Full-3-DtomographyforcrustalstructureinSouthernCaliforniabasedonthescattering-integralandtheadjoint-wavefieldmethods,J.Geophys.Res.SolidEarth,119(8),6421–6451,doi:10.1002/2014JB011346.Thispapersummarizesthefull3DtomographyresultsfromcomputationsattheArgonneLeadershipComputingFacility.

Xu,Z.,P.Chen,andY.Chen(2013),SensitivityKernelfortheWeightedNormoftheFrequency-DependentPhaseCorrelation,PureAppl.Geophys.,170(3),353–371,doi:10.1007/s00024-012-0507-3.

Thesis

Donovan,J.(2015),ForecastingDirectivityinLargeEarthquakesinTermsoftheConditionalHypocenterDistribution,PhDThesis,UniversityofSouthernCalifornia,154pp.

Presentations

SeismologicalSocietyofAmerica,AnnualMeeting2015(SSAAM)-Lee,E.,Thomas,H.J.,Chen,P.,Maechling,J.P.,Boué,P.,Denolle,M.,Beroza,G.,&Eymold,W.K.(2015)Full-3DTomographyofCrustalStructureinCentralCalifornia.Abstractandpresentation,2015SSAAnnualMeeting.BlueWaters2015Symposium-SCECpresentedtheresultsattheannualBlueWaterssymposium,includetheCyberShakecalculation,anexampleofaSCEC,NSFBlueWaters,andINCITEresearchcollaborativeeffort.LinkstothepresentationarepostedonaSCECwikiat:

http://scec.usc.edu/scecpedia/Blue_Waters_Symposium_2015NSFSoftwareInfrastructureforSustainedInnovation2015PIMeeting:-SCECmemberspresentedsoftwaredescriptionsandresearchresultsrelatedtoourFull3Dtomography(F3DT)andUnifiedCommunityVelocityModel(UCVM)workusingINCITEresourceataFeb2015NSFSoftwareInfrastructureforSustainedInnovation(SI2)meetingJan2015.TheNSFSI2programcurrentlyprovidesresearchfundingforsoftwareinfrastructureincludingF3DT,UCVM,AWP-ODC,andHerculesusedonourSCECINCITEresearchactivities.Moredetailsareavailableviathefollowinglinktothemeetingwebsite:https://share.renci.org/SI2PI2015/Lists/SI2PI2015Posters/View_01.aspx

NSCICoordinationMeeting(RestonVirginia)-SCECDirectoryThomasH.Jordanparticipatedinaninvitation-onlymeetinginOct2015concerningtheNationalStrategicComputingInitiative(NCSI)wherehediscussedSCECearthquakesystemscienceresearchbeingconductedusingALCF,OLCF,andNSFcomputingresources.

ExascaleComputingRFI:-SCEC’scomputationalscienceresearchgroupdevelopedadescriptionofourexascalecomputingneedsandcontributedittoaNSF/NIHHPCcommunity.SCEC’sdescriptionofourexascalecomputingneedsispostedhere:http://hypocenter.usc.edu/research/Exascale/SCEC_NCSI_Exascale.pdf

Supercomputing2015(SC15):SCECDirectoryThomasH.Jordanwasfeaturedasthe“InvitedTalkSpotlight”withhispresentationonSocietalImpactofEarthquakeSimulationsatExtremeScale.http://sc15blog.blogspot.com/2015/10/sc15-invited-talk-spotlight-societal.html

CodeDescription Pleasedescribeanymajordevelopmentorperformanceimprovementsthattookplacetoyourcode.Ifthedescriptionofthecodeissubstantiallydifferentfromearlierintheproject,pleasedescribe.SCECHPCcodeshavebeenfurtheroptimizedonINCITEresourcesin2015.Amajoreffortwastoportandintegratethenewnumericalalgorithmkernelplasticityandfrequency-dependentattenuationfeaturesintotheGPU-basedversionofAWP-ODC(OLCF).Simulationofnonlinearmaterialbehaviorrequiresanumberofadditionalvariables compared to linear computations. The added stencil kernel plasticityresultsincreasedghostcellregionfrom4to8layersinAWP-ODC,and17additionalvariablesaddedtotheoriginalcode.TheGPUcodeisrevisited,incollaborationwithDr. Peng Wang of NVIDIA. The combined new code moves advanced numericalalgorithms to a new level producing realistic seismograms at high frequencies,

whichhasbeentestedandvalidatedfromsciencerunswiththeversioncalledAWP-ODC-PLA-QF-v2.1.0.5.ThiseffortissupportedbyNCSABlueWatersPAIDprogram.Theplasticitykernelwasabletoachieveeffectivememorythroughputof223GB/safter the tuning, or 89% of the peak, the tuned code has seen 13% overallimprovement. In addition, we have implemented and validated CUDA multi-streamingonKeplerforexecutingmultiplekernelsconcurrentlyonOLCFTitanandNCSA Blue Waters. Multistreaming was available on Fermi with 16-wayconcurrency,butCUDAstreamsmultiplexintoasinglequeue.OnKeplerthe32-wayconcurrency is allowed at full stream level, with one work queue per stream,without inter-stream dependencies. The new implementation of multi-streamingcodingisvalidatedandresults inaspeedupof17%intermsofwallclocktimeonKeplerXK7.Weplantoapplythismulti-streamingfeaturetothebaselineAWP-ODCandalsothenewplasticitycodetotakethefullperformanceadvantage.The near future plan is to complete the optimization for the velocity kernel insimilar way like performed for plasticity and stress kernel, also add parallel filewriting capability to thenewcode.Weplan to startmediumsize science runsonINCITEresources later inSpring2016,after thenewAWP-ODC implementation iscompleteandvalidated.SCECmakesextensiveuseofscientificworkflowstorunourCyberShakemodelcalculation.CyberShakeproductionrunsareourlargestheterogeneousensemblesimulationsthatdependheavilyonworkflowtechnology.ThescaleandcomplexityoftheCyberShakecalculationrequirestheautomationandreproducibilityprovidedbyworkflows.WewouldliketorunourworkflowsonbothMiraandTitan.SCEC’sexistingCyberShakeworkflowimplementationisbasedonaNSF-developedsoftwarestackthatincludesCondor,CondorDAGManager,andPegasus-WMS.Inthepast,SCEChasnotbeenabletorunourworkflowsonINCITEresourcesprimarilybecauseourworkflowsolutionrequiresremotejobsubmission.Inourcurrentworkflow-computingmodel,anexternalcomputer,atSCEC,runsaCondorDAGManagerjobqueue.Jobsaresubmittedtotheresourceprovider’sjobqueueatareasonablerate.HPCproviders,includingTACCStampede,andNCSABlueWatershavesupportedthisworkflow-computingmodelwithGlobusGRAM-basedjobsubmission.Thisyear,SCECworkedwithUSCPegasus-WMSworkflowtooldevelopers,andTitan’stechnicalgrouptodevelopasolutionthatwillenableSCECtorunourworkflowsonTitan.TitanprovidesthelargeGPUcountneededforthecomputeintensivepartofourCyberShakecalculations.Weareworkingtodeveloptechnicalsolutionthatwillenableustoliketorunthehigh-throughputpartoftheworkflowonTitan.OnceworkingonTitan,wewillworktoimplementthissolutiononMiraaswell.

1. WhatisthecontributionmadebyLCFstaffand/orprograms(liaison/catalyst

collaboration,vendorsupport,userassistance,otherstaff,training,etc.)SCECinteractedwithALCFCatalystsandOLCFLiaison’sduring2015onstart-upcalls,onprogressreports,andintechnicaldiscussions.SCEC’sALCFCatalysthelpedusmigratelargedatasetsatALCFbetweenINCITEprojectswithoutinterruptingongoingactiveresearchusingthedata.SCEC’sOLCFliaisonshelpedcoordinatetechnicaldiscussionbetweenourworkflowdevelopmentgroupandtechnicalgroupsinsupportofourworkflowdevelopment.

NextSteps• Pleaselisttheexpectednextstepsforthiswork.Willyoubeworkingon

analysis?Ifso,where?Doyouneedadditionalallocations?Arethereupcomingpapersorpresentationsofthematerial?

• Aretherechangestocomingyearworkifyourprojectiscontinuing?Weareontracktocontinueourprogressforyear2milestones.Ourfull3DtomographyworkatALCFwillbemodifiedtorunathigherfrequencies.Thiswillrequirelargerparalleljobs.WewillworkwithALCFstaffsothatasweincreasethefrequencyofourtomographyresearch,ourroutinetomographycalculationswillscaleuptouse8racks,ormore,ofMira.

OtherCommentsPleaseanswerasapplicable:Hasthesupportreceivedfromthefollowingbeenbeneficialtoyourprojectteam?Citeexamplesifpossible• UserAssistanceCenter• CatalystsorPerformanceEngineering• VisualizationandAnalysisTeam• Operations

AnyadditionalfeedbackfromyourprojectteamfortheALCF?Pleaseincludeanyothercommentsorrequestsyouhaveaboutthefacility,allocationprogram,etc.