climate applications support at nersc
Post on 11-Dec-2021
1 Views
Preview:
TRANSCRIPT
NERSC Workload Diversity
6,000users850projects
-3-
Applica4onsonHopperandEdisonbyhoursusedJanuary2014–December2014
Top5ScienceCategoriesbyalloca4on(2014)
MaterialsScience 20%
FusionEnergy 18%
Chemistry 12%
ClimateResearch 11%
LaDceQCD 11%
CESM,WRF,NCAR-LESamongshown
70 Climate Projects at NERSC (AY2016)
• Awardsarepublishedat:– hKps://www.nersc.gov/users/accounts/awarded-projects/2016-allocaTon-awards/
• Seethedetailedlistofprojectsin“ExtraSlides”(orderedbyalloca4onhours).
• 30projectsuseCESM/ACMEorCESMcomponents.254ac4veusers
• 17projectsuseWRF.85users(36ac4ve).
-4-
AY16 Climate Allocations by PI’s Institutions
-5-
LBNLNCARPNNLU.ColoradoBoulderORNLLLNLUIUCUCIrvineScrippsInstUSEPAAMADU.WiscMadisonNOAAANLBrookhavenLabNorthCarolinaStateUmassAmherstUCDavisColoradoStateU.WashingtonVeturnPartnersHebrewUnivU.SussexUKUCBerkeleyU.MichiganU.MarylandNASAGISSCADeptofWaterResourceIGES-COLAUCMercedTerafluxCorpYaleUnivUCSanDiego
Climate Projects at NERSC (AY2016)-1
Repo ProjectTitle PI Organiza4on Codes
acme AcceleratedClimateforEnergy BillCollins LBNL ACME
k_2 Cloud-SystemSimulaTonswithaMulTscaleNonhydrostaTcGlobalAtmosphericModel
BillCollins LBNL ARAM
mp9 ClimateChangeSimulaTonswithCESM:ModerateandHighResoluTonStudies
GeraldMeehl NCAR CESM
m958 OceanAtmosphereReanalysesforClimateApplicaTons(OARCA)1831-2015
GilCompo UColorado,Boulder
ForecastModel,EnsembleFilter,…
m1704 MulTscaleMethodsforAccurate,Efficient,andScale-AwareModelsoftheEarthSystem
BillCollins LBNL CESM,ChomboAMRDycore
m1867 WaterCycleandClimateExtremesModeling(WACCEM) RubyLeung PNNL MPAS,WRF-CLM
m1199 High-LaTtudeApplicaTonandTesTng(HiLAT)ofGlobalandRegioanlClimateModels
PhilipRasch PNNL CESM
m2467 QuanTfyingandReducingBiogeochemicalandAerosolFeedbackUncertainTes
ForrestHoffman
ORNL ACME,CESM
m1041 PredicTngIceSheetandClimateEvoluTonatExtremeScales EsmondNg LBNL FELIX,CISM,BISICLES,POP2x
mp193 ProgramforClimateModelDiagnosisandIntercomparison His-YenMa LLNL CESM,CAM5,ACME,SAM
-6-
Thelast“Codes”columnisobtainedfromanswerstoERCAPrequestQuesTon20.
CESM Support • Proac4velyworkwithNCARtoprepareCESMportand
configura4onchangesforupcomingNERSCsystems,andProgrammingEnvironmentupdatesonexis4ngsystems.
• Sendto240+ac4veCESMusersmul4ple4mesayearonrecommendedconfigura4onstorunonNERSCsystems(Hopper,Edison,Cori)
• Inves4gatebuildandrunissuesforindividualusers.HelpuserstoadapttonewNERSCsystemenvironment.
• Helptoimproveworkflowforbestthroughput.Bundlejobs,dependencyjobs.Op4mizejobsizesandwall4merequests.
-8-
CESM on Cori P1 and Edison (1) • TheCESM/1.2.2versionportedbyNCAR/NERSCtoCoriP1
(newmachine)andEdisonwithSLURMbatchscheduler.• Sourcecodedistribu4onandscripts:/project/projectdirs/
ccsm1/collec4ons/cesm1_2_2/.Usesimilarscriptsasbeforetocreatenewcasesbyusingthe"-machcorip1”or“-machedison”op4on.
• Func4onalportonCoriP1only,nottunedforop4malpe-layoutsorperformance.Basedonpopulardemandandresources,NCAR/NERSCcanhelptotunepe-layoutsforafewspecificmodelconfigura4ons.
• Youcantunethesefngsin"env_mach_pes.xml"yourselfbyadjus4ngtheNTASKS_xxxandNTHRDS_xxxvaluesandthenredothecesm_setupstepsaierwards.
-9-
CESM on Cori P1 and Edison (2) • OlderCESMversionsarenotportedtoCoriP1orEdison.
UsingEdisonasanexample,youcanget“mkbatch.edison”,“env_mach_specific.edison”fromthecesm1_2_2Machinesdirectoryalongwiththeedisonentryin"config_machines.xml"andcopythemintoyourversion.
• UserswithDevelopersaccesscanusecesm1_5_beta04ornewer.Morecasestherewithpe-layouttuned.
• Severalproblemsrelatedtomodelruncrasheswithspecificmodelconfigura4onshavebeenworkedoutwithusersandNCAR.CESMsourcecodesinNERSC/projecthavebeenmodified.
• Workwithusersoncompila4onandrun4meissueswithspecificmodelconfigura4onsoncasebycasebasis.
-10-
CESM Survey, May 2014 • SurveyconductedtohelpNERSCandNCARunderstandCESM
usagemodelforsystemprocurement.• BiggestprojectsareatNCARandLBNL;othersincludePNNL,LLNL.
Widerangeofsciencegoals,includingIPCCproduc4onruns,extremeweathereventspredic4on,modeldevelopment,abruptclimatechangesimula4on,andmore
• Jobprofileasafunc4onofmodelcomponentvaries:CAM~50-85%.
• Threadparallelismalsovaries,using1,2,3,4,and6threadsperMPIperrunforCAMandPOP
• Parallelismtypicallydrivenbymodelconfigura4on,componentefficiencies,queue4me,alloca4onsize,andotherconsidera4ons
• Keypoint:duetothelarge#ofprojectsanddiversesciencegoals,itisimpossibletodis4nguishanyonesimulatorthatisdominantatNERSC
-11-
Typical CESM User Jobs, May 2014 (1/6) • Repomp9,useraaa,IPCCAR5
– Typical104-nodejobonHopper:• aprun-n828-N8-d3./cesm.exe• 2603MB/task• #cplntasks=128nthreads=1rootpe=0ninst=1• #camntasks=768nthreads=3rootpe=0ninst=1• #clmntasks=128nthreads=3rootpe=0ninst=1• #cicentasks=640nthreads=3rootpe=128ninst=1• #pop2ntasks=60nthreads=3rootpe=768ninst=1• #sglcntasks=768nthreads=3rootpe=0ninst=1• #swavntasks=768nthreads=3rootpe=0ninst=1• #rtmntasks=128nthreads=3rootpe=0ninst=1
-12-
TOT Run Time: 10413.674seconds 28.531seconds/mday 8.3myears/wdayLND Run Time: 422.124seconds 1.157seconds/mday 204.68myears/wdayICE Run Time: 756.239seconds 2.072seconds/mday 114.25myears/wdayATM Run Time: 8673.825seconds 23.764seconds/mday 9.96myears/wdayOCN Run Time: 2268.196seconds 6.214seconds/mday 38.09myears/wdayGLC Run Time: 0seconds 0seconds/mday 0myears/wdayCPL Run Time: 1379.231seconds 3.779seconds/mday 62.64myears/wday
TorunonEdisonwithSLURMnow:#SBATCH-N104…setenvOMP_NUM_THREADS3srun-n828-c3./cesm.exe
TaskdistribuTonreflectsopTmalloadbalanceamongmodelcomponents.
NESAP Climate Projects
• ThegoalforNESAPistoprepareDOEOfficeofScienceusercommunityforCoriPhase2,theIntelXeonPhiKnightsLanding(KNL)manycorearchitecture.
• SelectedClimateNESAPProjectsare:– ACME(PI:HansJohansen,LBNL)– CESM(PI:JohnDennis,NCAR)– MPAS(PI:ToddRingler,LANL)– WRF(PI:JohnMichalakes,NOAA)
• Wehavepostdocposi4onsavailabletoworkonNESAPprojects.Pleasehelptospreadtheword.– PosiTondescripTonat:hKp://cs.lbl.gov/careers-and-fellowshipsChoose“NERSCOnly”,RequisiTonIDis81356.
-14-
CESM NESAP Case Study
-15-
CESMNESAPTeamMembersNCAR:JohnDennis(PI),ChrisKerr,SeanSantosCray:MarcusWagnerIntel:NadezhdaPlotnikova,MartynCordenNERSCLiaison:HelenHe
MG2 Kernel
• MG2isakernelforCESMthatrepresentsversion2oftheMorrison-Geslemanmicro-physicspackage.Typicallyconsumesabout10%ofCESMrun4me.
• Kerneliscorebound– Notbandwidthlimitedatall– ShowsveryliKlevectorizaTon
• Someloopboundsareshort(e.g.10)• DependentsequenceofinstrucTons
– Heavyuseofmathinstrinsicsthatdonotvectorize• Kernelhaslongcomplexloopswithinterleavedcondi4onalsandelementalfunc4oncalls.
-16-
MG2 Vectorization Prototype • Usecompilerreporttocheckandmakesurekey
func4onsarevectorized(andallfunc4onsonthecallstackarevectorizedtoo)
• Add!$OMPDECLARESIMDand!DIR$ATTRIBUTEFORCEINLINEwhenneeded.
-17-
MG2 Optimization Steps
-18-
Op4miza4onStepsVersion1• Simplifyexpressionstominimize#operaTons• UseinternalGAMMAfuncTonVersion2• Remove“elemental”aKribute,moveloopinside.• InlinesubrouTnes.Divide,fuse,exchangeloops.• Replaceassumedshapedarrayswithloops• ReplacedivisionwithinversionofmulTplicaTon• RemoveiniTalizaTonofloopstobeoverwriKenlater• Usemoreaggressivecompilerflags• Useprofile-guidedopTmizaTon(PGO)Version3(Intelcompileronly)• Use!$OMPSIMDALIGNEDtoforcevectorizaTon
Op4miza4onStepsVersion1• Simplifyexpressionstominimize#operaTons• UseinternalGAMMAfuncTonVersion2• Remove“elemental”aKribute,moveloopinside.• InlinesubrouTnes.Divide,fuse,exchangeloops.• Replaceassumedshapedarrayswithloops• ReplacedivisionwithinversionofmulTplicaTon• RemoveiniTalizaTonofloopstobeoverwriKenlater• Usemoreaggressivecompilerflags• Useprofile-guidedopTmizaTon(PGO)Version3(Intelcompileronly)• Use!$OMPSIMDALIGNEDtoforcevectorizaTon
Op4miza4onStepsVersion1• Simplifyexpressionstominimize#operaTons• UseinternalGAMMAfuncTonVersion2• Remove“elemental”aKribute,moveloopinside.• InlinesubrouTnes.Divide,fuse,exchangeloops.• Replaceassumedshapedarrayswithloops• ReplacedivisionwithinversionofmulTplicaTon• RemoveiniTalizaTonofloopstobeoverwriKenlater• Usemoreaggressivecompilerflags• Useprofile-guidedopTmizaTon(PGO)Version3(Intelcompileronly)• Use!$OMPSIMDALIGNEDtoforcevectorizaTon
Version1• Simplifyexpressionstominimize#opera4ons• UseinternalGAMMAfunc4onVersion2• Remove“elemental”asribute,moveloopinside.• Inlinesubrou4nes.Divide,fuse,exchangeloops.• Replaceassumedshapedarrayswithloops• Replacedivisionwithinversionofmul4plica4on• Removeini4aliza4onofloopstobeoverwrisen
later• Usemoreaggressivecompilerflags.Trydifferent
compilers.• Useprofile-guidedop4miza4on(PGO)Version3(Intelcompileronly)• Use!$OMPSIMDALIGNEDtoforcevectoriza4on
0
200
400
600
800
1000
1200
Original Ver1 Ver2 Ver3
Tim
e pe
r ite
ratio
n (u
sec)
SandyBridge/Intel
IvyBridge/Intel
IvyBridge/CCE
MG2 Summary • Direc4vesandflagscanbehelpful,howevernota
replacementforprogrammers’workoncodemodifica4ons.• Breakuploopsandpushloopsintofunc4onswhere
vectoriza4oncanbedealtwithdirectlyandcanexposelogictocompiler.
• IncrementalimprovementsnotnecessaryaBIGwinfromanyonething.Accumula4veresultsmaser.
• Performanceandportabilityisamajorgoal:use!$OMPSIMDprovestobebeneficialbutveryhardtouseregardingtheneedofprovidingthealignedlist.
• Requestedop4onalalignmentdeclara4oninFortranlanguagestandard.
-19-
WRF Support
• UsersurveyconductedinOct2015.14responses(~40%fromac4veusers).Mostusersuseversion3.5.1andup,somehasowncustomiza4ons,100to500coreswithpureMPI.2usersuseMPI/OpenMP.
• HelpedindividualuserstobuilddifferentWRFversionswithdifferentcompilersonHopperandEdison.
• WRFisaTier3NESAPapplica4on.• WRFteamhasconfigura4onforKNC;WillreleaseaKNL
versiontouseforCori.• TheKNLversionwillbeop4malwithhybridMPI/OpenMP.• Encourageuserstoexplorehybrid(dm+sm)onEdisonand
Corinow.
-21-
We can offer help on • Por4ngandop4mizingapplica4ons• Workflowop4miza4onandqueueanalysis,jobbundleadviceetc.forbeserjobturnaround
• Computereserva4ons– hKps://www.nersc.gov/users/queues/reservaTon-form/
• Queueboostandusespecialqueue• Networktuningforlargedatatransfer• Sciencegateway(portals)forclimatedata• LargedatastorageatNERSCHPSSarchivesystem
-23-
20th Century Reanalysis Project
-24-
• PI:GilCompo,UniversityofColorado,Boulder
• NERSChelpedon: – ProvideopTonsfordatatransfer
betweenNERSCandNCAR.– Tuninganddebuggingthe
applicaTon– AddingmulT-levelparallelismto
bundleseveralassociatedparalleljobstogethertoobtainhighthroughputandsimplifydatahandling
– Providespecialqueueaccessandincreasemaxjoblimitonmachines.
GlobalTroposphericCircula4onMaps
Global Ensemble Reforecast Project
• PI:TomHamill,NOAA.• NERSChelpedon:
– DatatransferfromNERSCtoNOAA– PorTngfromIBMtoCraysystems– OpTmizinganMPIreducTonfuncTon(runTmereducedfrom24min
toafewseconds)– SuggesTngpost-processingworkflowopTmizaTons– Detailsat
hKps://www.nersc.gov/news-publicaTons/nersc-news/nersc-center-news/2012/end-to-end-network-tuning-sends-data-screaming-from-nersc-to-noaa/
– WebGatewayforGlobalEnsembleReforecastData• hKp://portal.nersc.gov/project/refcst/v2/• ThisportalallowsuserstodownloadselecteddaysofthefullReforecastmodeloutput
-25-
Do You Have a “realtime” Use Case? • Exampleusercasesare:
– RealTmedataprocessingwithexperimentsorevents.– RealTmedataanalysisfromusersofsciencegatewayportals.Owen
shortjobsthatneedanimmediateresponse– Sowwaredevelopment/testofworkflowsystemswithrealTmequeue– VisualizaTonjobs-thiswouldreplacethefuncTonalitytradiTonally
offeredbyavisnodeandallowsforinteracTvevisualizaTontasks
• SLURMonCorinowsupportsa“real4me”par44on.– Userscanrequestasmallnumberofon-demandnodesiftheirjobs
havespecialneedsthatcannotbeaccommodatedthroughtheregularbatchsystem.
– RequestFormat:hKps://www.nersc.gov/users/queues/realTme-queue-request-form/
– RequiresDOEapprovaltouse“realTme”onCori
-26-
We Welcome Science Highlights
• Samplesciencehighlightscanbefoundat:– hKps://www.nersc.gov/news-publicaTons/publicaTons-reports/science-highlights-presentaTons/
– Acoupleofexamplesalsoshownin“ExtraSlides”
• Anewwebformtosubmityourresearchtousat:– hKps://www.nersc.gov/science/share-your-research/
-27-
Available Tools and Libraries for Climate Applications
• nco• cdo• ncview• ncl(ncar)• nccmp• netcdf4-python• cray-netcdf,cray-parallel-netcdf• cray-hdf5,cray-parallel-hdf5• esmf
-29-
Summary
• Facilitateinforma4onexchangeamongNERSCclimateusers.
• Keepthecommunica4onchannelsopenbetweenusersandNERSC/DOE.
• Pleaseletusknow:– yourscienceandcomputaTonalaccomplishment– yourneedforrunningclimateapplicaTonsandperformingclimatedataanalysisatNERSC
-30-
Climate Projects at NERSC (AY2016)-2 Repo ProjectTitle PI Organiza4on Codes
m1517 CalibratedandSystemaTcCharacterizaTonAKribuTonandDetecTonofExtremes
TravisO’Brien LBNL CESM
m1355 AnalysisofGlobalCoupled0.1-degPOP/CICEintheCESMFramework
JulieMcclean ScrippsInst CESM
m1795 ProjecTonsofIceSheetandOceanEvoluTon StephenPrice LANL POPCICLES,CISM,BICICLES
m2071
DeterminingimpactsofsmallicecrystalsonbulkscaKeringproperTesoficecloudstoimproverepresentaTonsoficecloudsonsatelliteretrievalalgorithmsandnumericalmodels.
JunshikUm U.IllinoisU-C AmsterdamdiscretedipoleapproximaTon
m2222 ExpandingthecomputaTonalfronTerofmulT-scaleatmosphericsimulaTontoadvanceunderstandingoflowcloud/climatefeedbacks
MichaelPrTchard
UCIrvine SPCESM
sobl SouthernOceanUptakeintheMPAS-OceanModel EdwardPaKon
NCAR NCAR-LES
m2098
CalculaTonsofsingle-scaKeringproperTesofsmallicecrystalstoimprovein-situmeasurements,satelliteretrievalalgorithms,andnumericalmodels
GregMcfarquhar
U.IllinoisU-C ADDA
m1657 EnvironmentaleffectsinthelifecycleofconvecTveclouds MikhailOvchinnikov
PNNL WRF,SAM
m2410 DevelopmentoftheLESARMSymbioTcSimulaTonandObservaTon(LASSO)Workflow
WilliamGustafson
PNNL SAM,WRF
-32-
Climate Projects at NERSC (AY2016)-3 Repo ProjectTitle PI Organiza4on Codes
m1204 CenteratLBNLforIntegraTveModelingoftheEarthSystem(CLIMES)
BillCollins LBNL CESM
m1231
EvaluaTonoftheInteracTonsAmongTroposphericAerosolLoading,RadiaTveBalance,Clouds,andPrecipitaTon
JonathanPleim
USEPAAMAD WRF-CMAQ,WRF
m2320 Consequencesofcloudsuper-parameterizaTonforland-atmospherecouplingphysics
MichaelPritchard
UCIrvine SPCAM,UPCAM
m1642 EvoluToninCloudPopulaTonStaTsTcsoftheMJO SamsonHagos
PNNL ARWRF
m2402 Next-GeneraTonIntensity-DuraTon-FrequencyCurvesConsideringSpaTotemporalNon-StaTonarityinClimate,IntensePrecipitaTonEvents,andSnowmelt
RubyLeung PNNL WRF
m1196 InteracTonsofClouds,ConvecTon,andClimate DavidRomps LBNL DAM
m2067
EvaluaTonofthelarge-scaleandregionalclimaTcresponseacrossNorthAfricatonaturalvariabilityinoceanicmodesandterrestrialvegetaTonamongtheCMIP5models
MichaelNotaro
U.Wisc.Madison
CESM
m1660 RegionalModelingofLand-Ocean-AtmosphereInteracTons LarryBeng PNNL WRF,WRF-Chem,CLM
m1602
InvesTgaTngtheRoleofBiogeochemicalProcessesintheNorthernHighLaTtudesUsinganEarthSystemModel
AtulJain U.IllinoisU-C IntegratedScienceAssessmentModel
-33-
Climate Projects at NERSC (AY2016)-4 Repo ProjectTitle PI Organiza4on Codes
m2190 NextGeneraTonGlobalPredicTonSystem(NGGPS)Benchmarking JohnMichalales
NOAA NGGPSbenchmarksuite
m2136 EvaluaTonandimprovementofConvecTveParameterizaTonsinACMEmodel
WuyinLin BNL ACME
m1929 AMulTscaleReduced-OrderMethodforIntegratedEarthSystemModeling
GeorgePau LBNL CESM-PFLOTRAN-ROM,PROMEs
m1637 ReducingtheUncertaintyinSimulaTngAerosolChemistryoverMulTpleSpaTalScales
JeromeFast PNNL WRF
m1372 HighresoluTonmodeldevelopmenttoquanTfytheimpactoficebergsonthestabilityoftheAtlanTcMeridionalOverturningCirculaTon
AlanCondron UMassAmherst
MITgcm
m1439 EarthSystemModeling YangZhang NorthCarolinaState
WRF/CAM5,WRF/CMAQ,WRF/Chem,WRF/Chem-ROMS
m2250 MajorimprovementsonthelongwaveradiaTveinteracTonsbetweensurfaceandcloudsinthePolarRegionsinatmosphericglobalcirculaTonmodel(GCM)
DanielFeldman
LBNL CESM,LBLRTM-DISORT
m1540 DynamicDownscalingofClimateProjecTonsforDoDInstallaTons RaoKotamarthi
ANL NRCM(WRF),RegCM4
m2249 IntegraTngClimateChangeintoAirQualityModeling AjithKaduwela
UCDavis CMAQ,WRF
-34-
Climate Projects at NERSC (AY2016)-5 Repo ProjectTitle PI Organiza4on Codes
m328 Globalcloudmodeling DavidRandall ColoradoState
CESM,GCRM,SAM,Super-CAM
m1822 UsingAMIEdatatostudycloudprocesseswithintheMadden-JulianOscillaTon
RobertHouze U.Washington
WRF
m2451 IntegratedPredicTveSystemsforSolarEnergywithModeling,PostProcessingandMachineLearning
CameronWhiteman
VertumPartners
WRF
m2478 LinkedRainfallandRunoffIntensity-DuraTon-FrequencyinfaceofClimateChangeandUncertainty
EugeneYan ANL WRF
m1481 ImprovementofRepresentaTonoftheCloudAerosolInteracToninLarge-ScaleModels
AlexanderKhain
HebrewUnivJerusalem
WRF
m726 SimulaTngthetropicalcloud-climateinteracTonsandMJOduringtheAMIEandDYNAMOfieldexperimentsusingtheNCARWRFmodel
YiWang U.SussexUK AMIE-WRF,CESM-UM
m2076 RoleofForestEcosystemsinMiTgaTngClimateChangeImpactsthroughLand-SurfaceEnergyandWaterProcesses
GuangshanChen
U.Wisc.Madison
RegCM4-CNDV
m1865 Predictabilityofthecarbon-climatesystemonseasonaltodecadalTmescales
InezFung UCBerkeley CCSM
mp79 Parallelmodelingofclimate,chemistry,andphysicsaffecTngtheglobalatmosphere
DonaldWuebbles
U.IllinoisU-C CESM,CAM-Chem,WACCM
-35-
Climate Projects at NERSC (AY2016)-6 Repo ProjectTitle PI Organiza4on Codes
mp231 Three-dimensionalGlobalAtmosphericAerosolandChemistryModeling
JoycePenner U.Michigan IMPACT,GCEcloudmodel
m2352 MarineboundarylayercloudssimulatedbySPCAM5-CLUBB StevenGhan PNNL SPCAM5
m1590 RegionalAerosolEmissionandItsDirectandCloud-RelatedForcingEsTmates
TamiBond U.IllinoisU-C
CESM
m1006 DimenTonReducTonoftheCloud-Aerosol-RadiaTon(CAR)EnsembleModelingSystem
Xin-ZhongLiang
U.Maryland CWRF/CAR
m411 InteracTonofAtmosphericChemistryandAerosolswithClimate PhilipCameron-Smith
LLNL CESM,IMPACT
m843 IntegraTngobservaTonsandsimulaTonsofshallowwarmcloudstoadvanceunderstandingofdrizzleformaTonandevoluTon
AnnFridlind NASAGISS DHARMA
m640 InfluencesoftheBoundaryLayerFlowonVegetaTon-AirExchangesofEnergy,WaterandGreenhouseGases
XuhuiLee NCAR NCAR-LES
m1576 ReducingUncertaintyofClimateSimulaTonsUsingtheSuper-ParameterizaTon
CrisTanaStan IGES-COLA SP-CCSM
m2319 UsingcarbonylsulfidetoquanTfyregionalterrestrialbiologicalcarbonfluxes
JohnCampbell
UCMerced STEM
-36-
Climate Projects at NERSC (AY2016)-7 Repo ProjectTitle PI Organiza4on Codes
m997 UsingtheCaliforniaCentralValleyGroundwater-SurfaceWaterSimulaTonModeltostudyimpactsofextendeddroughtongroundwaterlevels,surfacewaterflows,andagriculturalmanagementpracTces.
CharlesBrush CADeptofWaterResource
IWFM,PEST
refcst AmulT-decadalreforecastdatasettoimproveweatherforecastsforrenewableenergyapplicaTons
ThomasHamill
NOAA GFS
m2492 Paleo-megadroughtsandAbruptClimateChangesintheSpeleothemRecords
InezFung UCBerkeley CAM
m2420 NextGeneraTonEcosystemExperiments--Tropics CharlesKoven LBNL CLM,CESM,ACME
m184 High-SpaTalResoluTonModelsofAtmosphericChemistry,AerosolsforRegionalScaleClimateSimulaTons
RaoKotamarthi
ANL CESM,CLM,CMAQ,GEOS-Chem
m1178 DevelopmentofFrameworksforRobustRegionalClimateModeling
RubyLeung PNNL WRF-CLM-ROMS,MPAS-A/MPAS-O,HOMME-POP-ROMS
m1357 CAM5ParameterSensiTvityandUncertaintyQuanTficaTonforCSSEF
YunQian PNNL CAM5.1
m612 ConTnuousDynamicGridAdaptaToninRegionalandGlobalAtmosphericModels
JosephPrusa TerafluxCorp CEU,EULAG
m1269 AmulT-decadalreforecastdatasettoimproveweatherforecastsforrenewableenergyapplicaTons
NigelQuinn LBNL HydroGeosphere
-37-
Climate Projects at NERSC (AY2016)-8 Repo ProjectTitle PI Organiza4on Codes
m2095 InfluenceoftheBoundaryLayerFlowonVegetaTon-AirExchangeofEnergy,WaterandCarbonDioxide
JianpingHuang
YaleUniv NCAR-LES
m1861 TowardsparameterizaTonofroot-rockhydrologicinteracTonsintheEarthSystemModel
InezFung UCBerkeley CLM4/CLM4RR
m1374
MulTscaleModelingofAerosolIndirectEffectsonDecadalTimescales
StevenGhan PNNL CESM,SPCAM
m543 RiverTransportandHydrologyinCCSM MarciaBransteKer
ORNL CESM
m238 TheoreTcalCalculaTonofWaterVaporConTnuumAbsorpTon AndrewLacis NASAGISS atm
m1060 NorthPacificMesoscaleCoupledAir-OceanSimulaTonsComparedwithObservaTons
IvanaCerovecki
UCSD POP2,WRF
-38-
Typical CESM User Jobs, May 2014 (2/6)
• Repom1040,ccc,IPCCruns• 304jobsonEdison,6to104nodes,upto6hrs• Typical52-nodejob:
– aprun-j2-n828-N8-d3./cesm.exe– 1214MB/task– #cplntasks=128nthreads=1rootpe=0ninst=1– #camntasks=768nthreads=3rootpe=0ninst=1– #clmntasks=128nthreads=3rootpe=0ninst=1– #cicentasks=640nthreads=3rootpe=0ninst=1– #pop2ntasks=60nthreads=3rootpe=768ninst=1– #sglcntasks=1nthreads=1rootpe=0ninst=1– #swavntasks=128nthreads=1rootpe=0ninst=1– #rtmntasks=128nthreads=3rootpe=0ninst=1
• Typical21-nodejob:
– aprun-n504-N24-d1./cesm.exe– #cplntasks=480nthreads=1rootpe=0ninst=1– #datmntasks=24nthreads=1rootpe=480ninst=1– #clmntasks=480nthreads=1rootpe=0ninst=1– #sicentasks=480nthreads=1rootpe=0ninst=1– #socnntasks=480nthreads=1rootpe=0ninst=1– #sglcntasks=480nthreads=1rootpe=0ninst=1– #swavntasks=480nthreads=1rootpe=0ninst=1– #rtmntasks=480nthreads=1rootpe=0ninst=1
-39-
TorunonEdisonwithSLURM:#SBATCH-N52…setenvOMP_NUM_THREADS3srun-n828-c3./cesm.exe
TorunonEdisonwithSLURM:#SBATCH-N21…setenvOMP_NUM_THREADS1srun-n504./cesm.exe
Typical CESM User Jobs, May 2014 (3/6) • Repomp9,useraaa,IPCCAR5
– 116jobsonHopper,86to104nodes,upto36hrs– Typical86-nodejob:
• aprun-n1680-N24-d1./ccsm.exe:-n192-N12-d2./ccsm.exe• 579MB/task• #totalnumberoftasks=1872• #maximumthreadspertask=2• #cplntasks=1680nthreads=1rootpe=0• #camntasks=1664nthreads=1rootpe=0• #clmntasks=400nthreads=1rootpe=1280• #cicentasks=1280nthreads=1rootpe=0• #pop2ntasks=192nthreads=2rootpe=1680• #sglcntasks=1nthreads=1rootpe=0Timingprofileforonerun:
-40-
TOT Run Time: 104327.625seconds 11.433seconds/mday 20.7myears/wdayLND Run Time: 5020.151seconds 0.55seconds/mday 430.27myears/wdayICE Run Time: 10427.512seconds 1.143seconds/mday 207.14myears/wdayATM Run Time: 55907.355seconds 6.127seconds/mday 38.64myears/wdayOCN Run Time: 80272.484seconds 8.797seconds/mday 26.91myears/wdayGLC Run Time: 0seconds 0seconds/mday 0myears/wdayCPL Run Time: 0seconds 0seconds/mday 0myears/wday
MPMDusingdifferentnum_threadsfordifferentexecutablesisnotyetsupportedinSLURM.
Typical CESM User Jobs, May 2014 (4/6) • Repomp9,useraaa,IPCCAR5
– 9jobsonEdison,16to84nodes,upto30hrs– Typical84-nodejob:
• aprun-j2-S6--ccnuma_node-n1008-N12-d4./ccsm.exe• 1188MB/task• #cplntasks=960nthreads=4rootpe=0ninst=1• #camntasks=960nthreads=4rootpe=0ninst=1• #clmntasks=48nthreads=4rootpe=0ninst=1• #cicentasks=912nthreads=4rootpe=48ninst=1• #pop2ntasks=48nthreads=4rootpe=960ninst=1• #sglcntasks=1nthreads=4rootpe=0ninst=1• #swavntasks=1nthreads=4rootpe=0ninst=1• #rtmntasks=48nthreads=4rootpe=0ninst=1
-41-
TorunonEdisonwithSLURM:#SBATCH-N84…setenvOMP_NUM_THREADS4srun-n1008-c4./cesm.exe
Typical CESM User Jobs, May 2014 (5/6) • Reposcascade,mp193,usermmm• 26jobsonHopperforcascade,18jobsformp193,1to854nodes,upto48hrs
– Typical854-nodejob: • aprun-n20480./tstorms.exefilelist16
– Typical320-nodejob• aprun-n1280-N4-d6./ccsm.exe• 708MB/task• #cplntasks=1280nthreads=6rootpe=0• #camntasks=1280nthreads=6rootpe=0• #clmntasks=1280nthreads=6rootpe=0• #cicentasks=1280nthreads=6rootpe=0• #docnntasks=1280nthreads=6rootpe=0• #sglcntasks=1280nthreads=6rootpe=0
– 160jobsonEdison,84nodes• aprun-j2-n1008-N12-d4-S6--ccnuma_node./cesm.exe• 1315MB/task
-42-
TOT Run Time: 16555.161seconds 22.678seconds/mday 10.44myears/wdayLND Run Time: 2476.78seconds 3.393seconds/mday 69.77myears/wdayROF Run Time: 15.698seconds 0.022seconds/mday 11007.77myears/wdayICE Run Time: 1492.282seconds 2.044seconds/mday 115.8myears/wdayATM Run Time: 11063.755seconds 15.156seconds/mday 15.62myears/wdayOCN Run Time: 10997.98seconds 15.066seconds/mday 15.71myears/wdayGLC Run Time: 0seconds 0seconds/mday 0myears/wdayWAV Run Time: 0seconds 0seconds/mday 0myears/wdayCPL Run Time: 1806.178seconds 2.474seconds/mday 95.67myears/wdayCPL COMM Time: 5557.865seconds 7.614seconds/mday 31.09myears/wday
SampleEdisonTimingProfile
TorunonEdisonwithSLURM:#SBATCH-N320…setenvOMP_NUM_THREADS6srun-n1280-c6./cesm.exe
Typical CESM User Jobs, May 2014 (6/6)
• Repom1199,userjjj• 1203ccsm/cesmjobs,1826atmjobs,1to400nodes• Typical400-nodejob:
– aprun-n4800-N12-d2./cesm.exe– 668MB/task– WallTme:6to8hrs– #cplntasks=4800nthreads=2rootpe=0ninst=1– #camntasks=4800nthreads=2rootpe=0ninst=1– #clmntasks=4800nthreads=2rootpe=0ninst=1– #cicentasks=4800nthreads=2rootpe=0ninst=1– #docnntasks=4800nthreads=2rootpe=0ninst=1– #sglcntasks=4800nthreads=2rootpe=0ninst=1– #swavntasks=1024nthreads=1rootpe=0ninst=1– #rtmntasks=1024nthreads=1rootpe=0ninst=1
-43-
TorunonEdisonwithSLURM:#SBATCH-N400…setenvOMP_NUM_THREADS2srun-n4800-c2./cesm.exe
Scientific Achievement Successinsimula4onstoassessimpactofcomplexterrainonwindturbineefficiencyisnowleadingtoassessmentofclimatechangeandlighteningeffectsonwindpowerproduc4onandrevenue.
Significance and Impact Electricu4li4esarekeenlyinterestedinevalua4ngtheeffectsofclimatechangeandextremeweatheronwindenergy.
Research Details – UsestheWRFcodetosolvea4Dmodeloftheatmosphere,includingallbasicdynamicsandenergyequaTons,whileallowingfortargetedparameterizaTonsandinteracTonwiththelandandwaterbodies.
– NERSCresourcesallowevaluaTonofthesensiTvityofthemodelresultstodifferentlandsurfacetypes,atmosphericdatasetsandotherforcingsbothinternalandexternaltothemodel.
Supercomputers and the Future of Wind Energy
CappsandWhiteman,15thAnnualWRFUsers'Workshop
Thisworkhasbeenusing4-Dnext-genera4onmesoscalenumericalweathersimula4onsthatmodelcomplexatmosphericdataresolvedtolocalwindfarmscales.ThiswillallowforopTmalturbinesiteselecTonandevaluaTngthetrueclimatechangeeffectstowindenergy. WorkwasperformedatLBNLusingNERSC
SBIR -44-
Impact on Climate Simulations • M.E.MaltrudandE.C.Hunke,LosAlamosNa7onalLaboratory;J.L.
McClean,NavalPostgraduateSchool
• High-Resolu4onGlobalCoupledOcean/SeaIceModelingusingPOP• NERSCprovides
§ Environmentforhighlyparallelruns
§ Serviceandstability§ Specialqueuesupport
-45-
top related