an automated performance evaluation framework for the ...hanzt/talks/gamm_gpe.pdf · continuous...
TRANSCRIPT
KIT – The Research University in the Helmholtz Association
Hartwig Anzt, Terry Cojean, Goran Flegar, Thomas Grűtzmacher, Pratik Nayak, Tobias RibizelSteinbuch Centre for Computing (SCC)
www.kit.edu
AnAutomatedPerformanceEvaluationFrameworkfortheGinkgoSoftwareEcosystem
GAMM,February18th- 22nd,2019
MovingtowardsContinuousBenchmarking(CB)
GAMM20192 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
Developer
A Healthy Software Development Cycle
GAMM20193 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
Developer
A Healthy Software Development Cycle
SourceCodeRepository
Push
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
GAMM20194 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
Push
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
• CIBuildsystemcontinuously checksthecompilationsuccessondifferentarchitectures.
GAMM20195 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
• CIBuildsystemcontinuously checksthecompilationsuccessondifferentarchitectures.
• Unittestsensurefunctionalityandvalidityofallbuildingblocks.
GAMM20196 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
• CIBuildsystemcontinuously checksthecompilationsuccessondifferentarchitectures.
• Unittestsensurefunctionalityandvalidityofallbuildingblocks.
• Goal:Automatedbenchmarkrunseverytimenewcodeisadded.
GAMM20197 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
• CIBuildsystemcontinuously checksthecompilationsuccessondifferentarchitectures.
• Unittestsensurefunctionalityandvalidityofallbuildingblocks.
• Goal:Automatedbenchmarkrunseverytimenewcodeisadded.
GAMM20198 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
SecurityThrust!• ExternalcodeisrunonHPCsystem• Possiblepointofattack
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
• CIBuildsystemcontinuously checksthecompilationsuccessondifferentarchitectures.
• Unittestsensurefunctionalityandvalidityofallbuildingblocks.
• Goal:Automatedbenchmarkrunseverytimenewcodeisadded.
GAMM20199 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
CodeReview
TrustedReviewer
MergeintoMasterBranch
• ManualcodereviewsbytrustedreviewerswhotakeresponsibilityforexecutingcodeontheHPCsystem.
GAMM201910 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
CodeReview
TrustedReviewer
MergeintoMasterBranch
• ManualcodereviewsbytrustedreviewerswhotakeresponsibilityforexecutingcodeontheHPCsystem.
• Howtoarchivetheresultsandmakethemaccessible?
GAMM201911 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
CodeReview
TrustedReviewer
MergeintoMasterBranch
PerformanceDataRepository
• Storedatainseparaterepository.• UseJSONdataexchangeformat.
GAMM201912 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
CodeReview
TrustedReviewer
MergeintoMasterBranch
PerformanceDataRepository
Users
• Storedatainseparaterepository.• UseJSONdataexchangeformat.• Cloningahugerepository is
expensive.• ”Interestinmyexperiments,only.”
GAMM201913 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
CITest
PerformanceDataRepository
ContinuousIntegration(CI)
DeveloperCodeReview
CIBuildSourceCodeRepository
Push
Schedule inBatchSystem
Web-Application
HPCSystem
TrustedReviewer
Users
MergeintoMasterBranch
CIBenchmarkTests
A Healthy Software Development Cycle
• Storedatainseparaterepository.• UseJSONdataexchangeformat.• Cloningahugerepository is
expensive.• ”Interestinmyexperiments,only.”• Interactiveweb-basedGinkgo
PerformanceExplorer(GPE).
GAMM201914 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
CITest
PerformanceDataRepository
ContinuousIntegration(CI)
DeveloperCodeReview
CIBuildSourceCodeRepository
Push
Schedule inBatchSystem
Web-Application
HPCSystem
TrustedReviewer
Users
MergeintoMasterBranch
CIBenchmarkTests
A Healthy Software Development Cycle
GPEGinkgoPerformanceExplorer
GAMM201915 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
Ginkgo Performance Explorer (GPE)
DataSelectionTab
TransformationScriptEditor
DataandPlotViewer
1.SelectDatainGitrepository.
GPE
GAMM201916 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
Ginkgo Performance Explorer (GPE)
DataSelectionTab
TransformationScriptEditor
DataandPlotViewer
1.SelectDatainGitrepository.
2.WriteJSONatascripttovisualizedata(examplesareprovided).
GPE
GAMM201917 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
Ginkgo Performance Explorer (GPE)
DataSelectionTab
TransformationScriptEditor
DataandPlotViewer
1.SelectDatainGitrepository.
2.WriteJSONatascripttovisualizedata(examplesareprovided).
3.Analyzedatavisually.
GPE
GAMM201918 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
Ginkgo Performance Explorer (GPE)
GPE
GAMM201919 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
Ginkgo Performance Explorer (GPE)
GPE
GAMM201920 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
Ginkgo Performance Explorer (GPE)
GPE
GAMM201921 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
Ginkgo’s Performance Evaluation Framework
GPE
ContinuousBenchmarkingBenefits
• Archivingperformancedataalongwithexecutionparametersensuresfullbenchmarkreproducibility.
• Comparing theperformanceresultsoverthecodelifetimeidentifiesperformancedegradations.
• Easeofuse:thesetupallowstolaunchbenchmarkwithfewclicks.
GAMM201922 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
Ginkgo’s Performance Evaluation Framework
GPE
ContinuousBenchmarkingBenefits
• Archivingperformancedataalongwithexecutionparametersensuresfullbenchmarkreproducibility.
• Comparing theperformanceresultsoverthecodelifetimeidentifiesperformancedegradations.
• Easeofuse:thesetupallowstolaunchbenchmarkwithfewclicks.
GinkgoPerformanceExplorer(GPE)Benefits• ThedesignofGPEefficientlyrealizestheanalysisaswebservice,removingtheneedfordownloading
performancedatatolocaldiskorinstallingadditionalsoftware.
• ExternaldeveloperswithoutaccesstoHPCsystemscantestandengineertheircodesonHPCresources.
• Extensibility:Optiontocompareperformancewithothersoftware libraries.
GAMM201923 02/19/2019 Hartwig Anzt:AnautomatedperformanceevaluationframeworkfortheGinkgosoftwareecosystem
• Open-sourceC++frameworkforsparselinearalgebra.
• Sparselinearsolvers,preconditioners,SpMV etc.
• Genericalgorithmimplementation:+referencekernelsforcheckingcorrectness;+architecture-specifichighlyoptimizedkernels.
• FocusedonGPUaccelerators(i.e.NVIDIAGPUs).
• SoftwarequalityandsustainabilityeffortsguidedbyxSDK communitypolicies:
•
•
•
•
https://xsdk.info/
PratikNayakTerryCojean GoranFlegar ThomasGrützmacher
TobiasRibizel
https://ginkgo-project.github.io/
Learn More about Ginkgo