productive & sustainable: more effective cse · §clever ideas: david keyes talk from monday....

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.

Productive & Sustainable:

More Effective CSEMike HerouxSenior ScientistCenter for Computing ResearchSandia National Laboratories

Scientist in ResidenceSt. John’s University, MN

MichaelHerouxSIAMCSE2017 1

16th SIAMConferenceonParallelProcessingforScientificComputing(PP18)March7-10,2018,Tokyo,Japan§ Venue

§ Nishi-Waseda Campus,Waseda University§ OrganizingCommitteeCo-Chairs’s

§ SatoshiMatsuoka(TokyoInstituteofTechnology,Japan)§ KengoNakajima(TheUniversityofTokyo,Japan)§ OlafSchenk(Universita della Svizzera italiana,Switzerland)

§ Contact§ KengoNakajima,nakajima(at)cc.u-tokyo.ac.jp§ http://nkl.cc.u-tokyo.ac.jp/SIAMPP18/§ http://www.siam.org/meetings/pp18/


TEAMS,COLLABORATORS,INSPIRATIONS

3MichaelHerouxSIAMCSE2017

DOEASCRIDEASProductivityProject


Michael Heroux (SNL), Co-Lead PILois Curfman McInnes (ANL), Co-Lead PIDavid Bernholdt (ORNL), Co-PI, Outreach LeadTodd Gamblin (LLNL), Co-PIOsni Marques (LBNL), Co-PIDavid Moulton (LANL), Co-PIBoyana Norris (Univ of Oregon), Co-PISatish Balay (ANL)Roscoe Bartlett (SNL)Anshu Dubey (ANL)Rinku Gupta (ANL)Christoph Junghans (LANL)Alicia Klinvex (SNL)

Reed Milewicz (SNL)Mark Miller (LLNL)Elaine Raybourn (SNL)Barry Smith (ANL)Louis Vernon (LANL)Greg Watson (ORNL)James Willenbring (SNL)Lisa Childers (ALCF)Rebecca Hartman-Baker (NERSC)Judy Hill (OLCF)Hai Ah Nam (LANL)Jean Shuler (LLNL)

Facilities Liaisons

IDEAS-ECPProductivityProject

+xSDK-ECPProjectMichaelHerouxSIAMCSE2017

InspirationsandCollaborations


Outline

§ Concepts&Definitions.§ WhyIcareaboutproductivity&sustainability.§ Incentives:Betterproductivity&sustainability.§ Completenessphilosophy.§ SomePracticalStrategies.§ Productivity++andBetterScientificSoftware

7 MichaelHerouxSIAMCSE2017 7

MyRoles

§ Regardingobservationsonopportunitiestoimprove:§Morelikeapsychologistthanexpert.

§ Regardingsoftwaretools,processes,practicesimprovements:§Morelikeacarpenterthanexpert.


CONCEPTS&DEFINITIONS


Productivity&Sustainability

§ Productivity– Outputperunitinput.§ Specifically:DeveloperProductivity.

§ Sustainability– Thefuturecostofusability.§ Specifically: SoftwareSustainability.

§ Strategy:Focusonimprovement.§ Optometristapproach.


Scenario: Youarebehindindevelopingasophisticatednewmodelinyoursoftwarethatyouwanttouseforresultsinanupcomingpaper.Whichofthesecouldbereasonablechoices?

§ Developasimplermodelforthepaper.§ Setotherworkasideandspendmoretimeondevelopment.§ Askforanextensiononthepaperdeadline.§ Developsophisticatedmodel,butdon’ttestitscorrectness.§ Developsophisticatedmodel,butdon’tdocumentitorcheckitin.

Better,Faster,Cheaper:Picktwoofthethree.


ImprovedDeveloperProductivity

“Better,Faster,Cheaper:Pickallthree.”– Nearterm.

Scenario:(6monthslater)Afterinvestingindeveloperproductivityimprovements,youareontimeindevelopingasophisticatednewmodelinyoursoftwarethatyouwanttouseforresultsinanupcomingpaper.

Investindevelopertools,processes,practices.


ImprovedSoftwareSustainability

“Better,Faster,Cheaper:Pickallthree.”– Longterm.

Scenario:(3yearslater)Afterinvestinginsoftwaresustainabilityimprovements,youareontimeindevelopingseveral sophisticatednewmodelsinyoursoftwarethatyouwanttouseforresultsinupcomingpapers.

Investintesting,documentation,integrationforlong-termsoftwareusability.


PRODUCTIVITY&SUSTAINABILITY:WHYNOW?


Reproducibility• NY Times highlights

“problems”.• Only one of many

cited examples.• CSE has been spared

this “spotlight” (so far).

http://www.nytimes.com/2015/08/28/science/many-social-science-findings-not-as-strong-as-claimed-study-says.html?_r=0


FutureHighPerformanceComputing:§ ISNOTprimarilyaboutHW.§ ISprimarilyaboutAlgorithms.§ ISprimarilyaboutsoftware:

§ΔA,ΔS>>ΔH§Algs:AdequateR&Dfocus.§SW:Opportunitiesforimprovement.


“Easy”WorkinProgress§ Newparallelalgorithms:

§ Turningouttobefeasible.§ Cleverideas:DavidKeyestalkfromMonday.§ Lotstodo,butsteadyprogress§ Muchevidenceinthisconference.

§ CurrentThreadProgrammingEnvironments:§ C++,OpenMP,others:Working.§ Runtimes:Stillalotofwork,butprogress.

§ Lotstodo,butcommunityisfocused.


“Hard”Work

§ BillionsSLOCofencodedscience&engineering.

§ Challenge:§ Transfer,refactor,rewriteformodernsystems.§ Dosowithmodestinvestmentbumpup.§ Workacrossdistinctanddiverseteams.§ Deliverscienceatthesametime.§ Makethenextdisruptioneasiertoaddress.


TOWARDREPRODUCIBLESCIENCE:CREATINGINCENTIVES


IncentivesToChange

• Reproducibility • SW Quality Requirements• Employer Recognition

Productivity & Sustainability Investments

Demand

EnableCommon statement: “I would love to do a better job, but I need to:• Get this paper submitted.• Complete this project task.• Do something my employer values more.Goal: Change incentives to include value of better software.


ReproducibilityTerminology

§ ReviewableResearch.Thedescriptionsoftheresearchmethodscanbeindependentlyassessedandtheresultsjudgedcredible.(Thisincludesbothtraditionalpeerreviewandcommunityreview,anddoesnotnecessarilyimplyreproducibility.)

§ ReplicableResearch.Toolsaremadeavailablethatwouldallowonetoduplicatetheresultsoftheresearch,forexamplebyrunningtheauthors’codetoproducetheplotsshowninthepublication.(Heretoolsmightbelimitedinscope,e.g.,onlyessentialdataorexecutables,andmightonlybemadeavailabletorefereesoronlyuponrequest.)

§ ConfirmableResearch.Themainconclusionsoftheresearchcanbeattainedindependentlywithouttheuseofsoftwareprovidedbytheauthor.(Butusingthecompletedescriptionofalgorithmsandmethodologyprovidedinthepublicationandanysupplementarymaterials.)

§ AuditableResearch.Sufficientrecords(includingdataandsoftware)havebeenarchivedsothattheresearchcanbedefendedlaterifnecessaryordifferencesbetweenindependentconfirmationsresolved.Thearchivemightbeprivate,aswithtraditionallaboratorynotebooks.

§ OpenorReproducibleResearch.Auditableresearchmadeopenlyavailable.Thiscomprisedwell-documentedandfullyopencodeanddatathatarepubliclyavailablethatwouldallowoneto(a)fullyauditthecomputationalprocedure,(b)replicateandalsoindependentlyreproducetheresultsoftheresearch,and(c)extendtheresultsorapplythemethodtonewproblems.

V. Stodden, D. H. Bailey, J. Borwein, R. J. LeVeque, W. Rider, and W. Stein. 2013. Setting the Default to Reproducible: Reproducibility in

Computational and Experimental Mathematics. (2013). https://icerm.brown.edu/tw12-5-rcem/icerm_report.pdf


§ TOMSRCRInitiative:RefereeData.§ WhyTOMS?

§ Traditionofrealsoftwarethatothersuse.

ACMTOMS

22MichaelHerouxSIAMCSE2017 22

ACMTOMSReplicatedComputationalResults(RCR)

§ Standardreviewerassignment:§ Nothingchanges.

§ RCRreviewerassignment:§ Concurrentwithstandardreviews.

§ RCRprocess:§ Multi-facetedapproach,Bottomline:Trustthereviewer.

§ Publication:§ ReplicatedComputationalResultsDesignation.§ Reviewreportappearswithpublishedmanuscript.


RCRProcess:TwoBasicApproaches

1. Independentreplication:§ Openreview.

2. Reviewofcomputationalresultsartifacts:§ ”Boutique”computingsystem.§ Inthissituation:

§ Carefuldocumentationoftheprocess.§ SoftwareshouldhaveitsownsubstantialV&Vprocess.


BigPictureofTOMSRCR

§ Improvescience.

§ Notgoodnow:§ Trustcomesfrom“cloud”ofpaperswithsimilarresults.§ Whichcouldstillbewrong.§ Replicability:Firststeptowardimprovement.

§ Engagea“darkportion”oftheR&Dcommunity.§ Reviewersnotamongtypicalreviewerpool.§ Practitioners,users.ExpertatuseofMathSW.

Thank you for taking the time to consider our paper for your journal.

XXX has agreed to undergo the RCR process should the paper proceed far enough in the review process to qualify. To make this easier we have preserved the exact copy of the code used for the results (including additional code for generating detailed statistics that is not in the library version of the code).


ComingtoYourWorldSoon:ReproducibilityRequirements

§ Theseconferencesexpectartifactevaluationappendices(mostoptionally):§ CGO,PPoPP,PACT,RTSSandSC.§ http://fursin.net/reproducibility.html

§ ACMReplicatedComputationalResults(RCR).§ ACMTOMS,TOMACS.§ http://toms.acm.org/replicated-computational-results.cfm

§ ACMBadging.§ https://www.acm.org/publications/policies/artifact-review-badging


SC17ReproducibilityInitiative§ Twoappendices:

§ Artifactdescription(AD).§ Schematicofyourcomputingenvironment.§Makesiteasiertoreruncomputationsinfuture.§ ADrequiredforbestpaper&studentpaper.

§ ComputationalResultsAnalysis(CRA).§ Targets”boutique”environments.§ Improvestrustworthiness.

§ Details:http://sc17.supercomputing.org/submitters/technical-papers/reproducibility-initiatives-for-technical-papers/


SourcesforCRAmetrics§ Syntheticoperatorswithknown:

§ Spectrum(Hugediagonals).§ Rank(byconstructions).

§ Invariantsubspaces:§ Example:Positional/rotationalinvariance(structures).

§ Conservationprinciples:§ Example:Fluxthroughafinitevolume.

§ General:§ Pre-conditions,post-conditions,invariants.


Incentives&Productivity/Sustainability

• Reproducibility • SW Quality Requirements

• Employer Recognition

SW Productivity & Sustainability Investments

Demand

Enable

New funding proposal element:SW Productivity and Sustainability Plan


DOESWProductivityandSustainabilityPlan(SWPSP).

§ KeyEntities:§ DOEBiologicalandEnvironmentalResearch(BER).§ DOEAdvancedScientificComputingResearch(ASCR)§ IDEASProject

§ Milestone:§ First-of-a-kindSWProductivityandSustainabilityPlan.


DOEBERSWPSPRequirements

§ DescribeoverallSWdevelopmentprocess.§ Softwarelifecycle,testing, documentationandtraining.

§ Developmenttoolsandprocesses:§ sourcemanagement,issuetracking,regressiontesting,SWdistribution.

§ Trainingandtransition:§ Newanddepartingteammembers.

§ Continuousprocessimprovement:§ Gettingbetteratproductivityandsustainability.


Employers/Community&Productivity/Sustainability

• Reproducibility • SW Quality Requirements• Employer Recognition

SW Productivity & Sustainability Investments

Demand

Enable

Next focus: • Work with DOE labs to recognize SW as base research.• Pressing for Scientific SW Productivity R&D base funding.


Community

HowtheFuturewillbeDifferent

§ Publishers:§ Willexpectreproduciblecomputationalresults.

§ Fundingagencies:§ Willexpectimprovedproductivity,sustainablesoftware.

§ Employers:§ Willrewardstaff,facultyproducinggoodsoftware.

§ Impact:High-qualityCSEsoftwarewillmatteralotmore.


PRODUCTIVITYIMPROVEMENT


CSE&FormalSoftwareMethodologies:TroubledHistory

35

§ Cray(1990):§ FormalWaterfallMethod.

§ DOEASCI(2000):§ CMMI

§ Failedtofollowownprocess:Elicitrequirements.

MichaelHerouxSIAMCSE2017

CSEComplete:Useful“Overhead”§ CodeComplete:Ultimatevalueiscode.

§ Shouldweonlywritecode?§ Somenon-codingactivitiesimprovecode.

§ CSEComplete:UltimatevalueisCSE.§ WhatactivitiesimproveCSE?

36

“Plans are worthless, but planning is everything.”Dwight D. Eisenhower

“Give me six hours to chop down a tree and I will spend the first four sharpening the axe.”

Abraham Lincoln


PRODUCTIVITYIMPROVEMENT:PLANNING


38

Visible Progress(Writing code,computing results)

100 %

0 %Planning

Recoding and Portingto new Platforms

PercentEffort

TimeMidlife Effort Profile

Endlife Effort Profile

Adapted from Software Project Survival Guide, Steve McConnell

Code-and-Fix Development Approach

Early Effort Profile


39

Visible Progress(Writing code,

computing results)

100 %

0 %

Planning

Recoding and Portingto new Platforms

PercentEffort

Time

Simple Planned Development Approach

Midlife Effort Profile

Ongoing Effort Profile

Early Effort Profile


Planningtools:Usewhatyouknow

40

KokkosKernels - add design note for discussion.This design note is very informal and working note for discussion.…

For typeset, "make"KokkosKernels - add more algorithm variants.In this algorithm design, I have a few assumptions.…

Commit log messagesLatex Planning Document

KokkosKernels:Micro&BatchedBLASDesignDocument§ 6weeks:DesignbyLaTeX.

§ Reviewbydiverseexperts.§ Significantdesignchanges:Intextonly.

§ 2weeks:Writecode.

Message:Usethetoolsyouknow.

Courtesy: KokkosKernels Development Team


PRODUCTIVITYIMPROVEMENT:ISSUETRACKING


Managingissues:Fundamentalsoftwareprocess

42

§ Issue:Bugreport,featurerequest§ Approaches:

§ Short-termmemory,officenotepad§ ToDo.txt oncomputerdesktop(1person)§ Issues.txt inrepositoryroot(smallco-locatedteam)§ …§ Web-basedtool+Kanban (distributed,largerteam)§ Web-basedtool+Scrum(full-timedevteam)

Informal, lesstraining

Formal, moretraining

Continual improvement


Kanban principles

43

§ Limitnumberof“InProgress”tasks§ Productivityimprovement:

§ Optimize“flexibility vs swapoverhead”balance.Noovercommitting.

§ Productivityweaknessexposedasbottleneck.Teammustidentifyandfixthebottleneck.

§ EffectiveinR&Dsetting.Avoidsadeadline-basedapproach. Deadlinesaredealtwithinadifferentway.

§ Providesaboardforviewingandmanagingissues


BasicKanbanBacklog Ready In Progress Done• Any

task idea

• Task + how to do it.

• Could be pulled when slot opens.

• Task you are working on right now.

• The only Kanban rule: Can have only so many “In Progress” tasks.

• Limit is based on experience, calibration.

• Completed tasks.

• Record of your lifeactivities.

• Rate of completion is your “velocity”.


Kanban Iterative Develop

ment

XLoop

KanbanX

X=Sustainability,Performance

TOWARDAREPEATABLEPROCESS


SummarizeProjectPractices

SetGoals ConstructPTC

RecordCurrentPTC

Values

CreateImprovement

Plan

Repeat

AssessProgress

PSIPWorkflow

ExecutePlan

Start

SWProductivity&SustainabilityImprovementPlans


ProgressTrackingCardExample


Practice: Test Coverage Score (0 – 5):Score Descriptions0 Little or no independent testing. Functional testing via users.1 Independent functional testing of primary capabilities.2 Primary functional testing, some unit test coverage.3 Comprehensive unit testing, primary functional testing.4 Comprehensive unit testing, functional testing for documented use cases.5 Comprehensive unit, use case functional testing; test coverage commitment.

49

BSS Software PlatformComponentTechnology

Backend FrontendGoogle Docs GitHub Ruby on Rails

Location Google Drive betterscientificsoftwareorganization

betterscientificsoftware.info

Purpose • Rapid collaborative content development.

• Multi-user typing, suggested edits, comments.

• Content creation, refinement, management (from Google Drive).

• Content packaging for use with BSS.info

• User-facing portal • Polished backend

content• Blogs, forums• Mailing lists.

Contributors Community content experts

Community content experts. BSS staff.

BSS staff. Web dev experts.

Consumers BSS GitHub Backend BSS Frontend CSE community

Content Notes

Content migrates to GitHub after it stabilizes.

Content is managed in git repos, markdown.

Content from backend, community.

New community portal: ScientificSoftware Improvement Exchangehttps://betterscientificsoftware.io May

2017


Summary§ CSEisnotcompletewithoutincorporatingbetter:

§ Reproducibility:Canyourresultsbetrusted?§ Sustainability:Willyourecosystemlivelongandprosper?§ Productivity:Willyoursoftwaredevelopmenteffortsbecompetitive?

§ ProductivityeffortmustenhanceCSEresults:§ Highquality,lowvaluesoftwarecannotbetheoutcome.§ Qualityimprovementsmustbemeasured.

§ Changesinpublishers,funders,employersexpectations:§ Couldviewasbothersome:More“overhead”impedingprogress.§ Couldbeopportunity:Beproactive,partoftheinitiative.


Productivity++InitiativeAsk:IsMyWork_______?

https://github.com/trilinos/Trilinos/wiki/Productivity---Initiative

Productivity++ Traceable In Progress Sustainable Improved

Version(1.2(


Contribute!

§ https://github.com/betterscientificsoftware/betterscientificsoftware.github.io/blob/master/README.md

§ Orsearch“github betterscientificsoftware”.


Acknowledgments53

§ SandiaNationalLaboratoriesisamulti-programlaboratorymanagedandoperatedbySandiaCorporation,awhollyownedsubsidiaryofLockheedMartinCorporation,fortheU.S.DepartmentofEnergy’sNationalNuclearSecurityAdministrationundercontractDE-AC04-94AL85000.SANDNO.2016-8466C.


PRODUCTIVITYIMPROVEMENT:APPLICATIONCOMPOSITIONANDSOFTWAREECO-SYSTEMS


Extreme-ScaleScientificSoftwareEcosystem

Libraries• Solvers,etc.• Interoperable.

Frameworks&tools• Docgenerators.• Test,buildframework.

Extreme-ScaleScientificSoftwareDevelopmentKit(xSDK)

SWengineering• Productivitytools.• Models,processes.

Domaincomponents• Reactingflow,etc.• Reusable.

Documentationcontent• Sourcemarkup.• Embeddedexamples.

Testingcontent• Unittests.• Testfixtures.

Buildcontent• Rules.• Parameters.

Libraryinterfaces• Parameterlists.• Interfaceadapters.• Functioncalls.

Shareddataobjects• Meshes.• Matrices,vectors.

Nativecode&dataobjects• Singleusecode.• Coordinatedcomponentuse.• Applicationspecific.

Extreme-scaleScienceApplications

Domaincomponentinterfaces• Datamediatorinteractions.• Hierarchicalorganization.• Multiscale/multiphysics coupling.


xSDK focus

56§ Commonconfigureandlinkcapabilities

§ xSDK usersneedfullandconsistentaccesstoallxSDK capabilities§ Namespaceandversionconflictsmakesimultaneousbuild/linkof

xSDK difficult§ Determininganapproachthatcanbeadoptedbyanylibraryor

componentsdevelopmentteamforstandardizedconfigure/linkprocesses

§ Libraryinteroperability§ Designingforperformanceportability

Libraries• Solvers,etc.• Interoperable.

Frameworks&tools• Docgenerators.• Test,buildframework.

Extreme-ScaleScientificSoftwareDevelopmentKit(xSDK)

SWengineering• Productivitytools.• Models,processes.

Domaincomponents• Reactingflow,etc.• Reusable.


StandardxSDK packageinstallationinterface

57

Motivation: Obtaining, configuring, and installing multiple independent software packages is tedious and error prone. • Need consistency of compiler (+version,

options), 3rd-party packages, etc.Approach: Define a standard xSDK package installation interface to which all xSDKpackages will subscribe and be tested Accomplishments: • Work on implementations of the standard by

the hypre, PETSc, SuperLU, and Trilinosdevelopers

• PETSc can now use the “scriptable” feature of the installers to simultaneously install hypre, PETSc, SuperLU, Trilinos with consistent compilers and ‘helper’ libraries. Impact: Foundational step toward

seamless combined use of xSDKlibraries, as needed by BER use cases and other multiphysics apps

xSDK Build Example


xSDKMinimumComplianceRequirements:

58

§ M1.EachxSDKcompliantpackagemust supportthethestandardxSDKcmake/configureoptions.

§ M2.EachxSDKpackagemustprovideacomprehensivetestsuitethatcanberunbyusersanddoesnotrequirethepurchaseofcommercialsoftware

§ M3.EachxSDKcompliantpackagethatutilizesMPImustrestrictitsMPIoperationstoMPIcommunicatorsthatareprovidedtoitandnotusedirectlyMPI_COMM_WORLD.

§ M4.Eachpackageteammustdoa‘besteffort’atportabilitytokeyarchitectures,includingstandardLinuxdistributions,GNU,Clang,vendorcompilers,andtargetmachinesatALCF,NERSC,OLCF.AppleMacOSandMicrosoftWindowssupportarerecommended.

§ M5.Eachpackageteammustprovideadocumented,reliablewaytocontact thedevelopmentteam;thismaybebyemailorawebsite.Thepackageteamsshouldnotrequireuserstojoinageneric mailinglist(andhencereceiveirrelevantemailtheymustwadethrough)inordertoreportbugsorrequestassistance.

§ M6– 11…

https://ideas-productivity.org/resources/xsdk-docsMichaelHerouxSIAMCSE2017

xSDKRecommendedComplianceRequirements:

59

§ R1.Itisrecommendedthateachpackagehaveapublicrepository,forexampleatgithub orbitbucket,wherethedevelopmentversionofthepackageisavailable.Supportfortakingpullrequestsisalsorecommended.

§ R2.Itisrecommendthatalllibrariesbetestedwithvalgrindformemorycorruptionissueswhilethetestsuiteisrun.

§ R3.Itisrecommendedthateachpackageadoptanddocumentaconsistentsystemforpropagating/returningerrorconditions/exceptionsandprovideanAPIforchangingthebehavior.

§ R4.Itisrecommendedthateachpackagefreeallsystemresourcesithasacquiredassoonastheyarenolongerneeded.


MessagetoThisAudience

Considerwhatsoftwareecosystem(s)youwantyoursoftwaretobepartofanduse.


SUSTAINABILITYIMPROVEMENT


“Alwayscodeasiftheguywhoendsupmaintainingyourcodewillbeaviolentpsychopathwhoknowswhereyoulive.”

- JohnWood


Sustainable?Pullrequestscenario

• Pull request.• Competent external user.• Useful feature.• No conflicts.• Accept: Yes, if OK.• What must happen?

• Vet commit:– Comes with tests.– Comes with docs.– Follows style rules.

• Test: – No regressions.– Portable.


MessagetoThisAudience

Writetestsnow,while(orbefore)writingyourintendedproductionsoftware.


productive & sustainable: more effective cse · §clever ideas: david keyes talk from monday....

Documents