@lisacrispin

TheWholeTeamApproachtoTestinginContinuousDelivery

LisaCrispinWithmaterialfromAbbyBangser,AshleyHunsberger,Lisi Hocke &more

It takes a village…

Copyright2018LisaCrispin

@lisacrispin

Alittleaboutme...

@lisacrispin

Howaboutyou?Who’salreadydoingcontinuousdelivery?Who’sworkingtowardsit?Whatcompetenciesdoyouhave?● Coding● Testing● Operations/sysadmin● Datascience/database● Coaching● ????

@lisacrispin

Learningintentions● WaystoengagethewholeteaminaDevOps(DevTestOps)culture● Sometoolstohelpshortenfeedbackloops&mitigaterisks● Howtofitallnecessarytestingactivitiesintothecontinuousworld

@lisacrispin

Expectations

● Pleaseaskquestionsandcontributeyourstories!

● Thereareno“bestpractices”,thereare“leadingpractices”.

● Readmore,andexperimentwiththeseideastoreallylearnthem.

● DevOpsisabigarea.We’llfocusonpipelines.

@lisacrispin

Whatdoyouthinkwhenyouhear“DevOps”?

Howabout“ContinuousDelivery”?

@lisacrispin

Whatdoyouthinkwhenyouhear“Continuoustesting”?

@lisacrispin

Wholeteamresponsibility forquality

● Commitmenttoalevelofconfidence○ Bugpreventionoverbugdetection○ Learningfromproductionuse,errors○ …andrespondingfast○ Focusonwhat’svaluabletocustomers

● Diverseperspectives,skillsets,biases

@lisacrispin

Let’sagreeonsomecommonterminology

@lisacrispin

“DevOps”

● Termcoinedin2009,buttheconceptgoesbacktoearlydays● Devs,testers,ops,otherscollaborate

○ Create,test, maintaininfrastructureforCI,deployments,test&prodenvironments

○ Supportcontinuousdelivery&testing○ Makeourcustomers’dayabitbetter

@lisacrispin

It’sallabout:

● Collaboration

● Continuousimprovement

● Continuouslearning

TestingistheheartofDevOps

@lisacrispin

Measuring our flow of work

Work identified

Work started

Work completed

Lead Time

Process / Cycle Time

Cycletime:howlongfromstarttodelivery?● Re-workslowsusdown● Sharedunderstandingspeedsusup

@lisacrispin

ContinuousIntegration

● Integrate code into a shared repository multiple times per day

● Preferably on trunk/master, but on branches too● Typically the start of a pipeline● Each check-in can be verified by an automated build with

automated regression tests

@lisacrispin

Deploymentpipeline● Breakthebuildintostagestospeedupfeedback● Eachstagetakesextratime&providesmoreconfidence● Earlystagescanfindmostproblems->fasterfeedback● Laterstagesprobemorethoroughly● Automateddeploymentpipelinesarecentraltocontinuous

delivery

From A Practical Guide to Testing in DevOps, Katrina Clokie

@lisacrispin

ContinuousDelivery(CD)

● Ability to get many types of changes into production safely, quickly and sustainably (Jez Humble)○ eg. new features, configuration changes, bug fixes, experiments

● Heavily benefits from, but not dependent on, automated regression tests

● Each commit is independently verified as a deployable release candidate

● A deployable release candidate is always available

@lisacrispin

Compile & Unit Testing

Static Analysis

Integration Testing

GUI &End to End Testing

Exploratory Testing

User Acceptance Testing (UAT)

Perf Testing

Penetration Testing

Deploy to Production

Deploy to QA Env

Deploy to StagingEnv

Deploy to Stress Env

Deploy to Security Env

ContinuousDeliveryExample

Steps outlined in blue in this example are manual

@lisacrispin

Manualstepsinthepipelinemightbe…

● Deploys● Exploratorytesting● Visualchecking● …whatelsecanyouthinkof?● Moreonthislater

@lisacrispin

PrinciplesofcontinuousdeliveryFromJezHumbleandDavidFarley,continuousdelivery.com:

● Buildqualityin● Workinsmallbatches● Computersperformrepetitivetasks,peoplesolve

problems● Relentlesslypursuecontinuousimprovement● Everyoneisresponsible

@lisacrispin

ContinuousDeployment(alsoCD:-/)● Deploymentsoccuroneverysuccessfullyverifiedcommit.

Oftenmanyaday.● Heavily fromautomatedtestingandContinuousDelivery

environment,butdoesnotactuallyrequireeither

Image: www.squirrelpicnic.com

@lisacrispin

Compile & Unit Testing

Static Analysis

Integration Testing

GUI &End to End Testing

Exploratory Testing

User Acceptance Testing (UAT)

Perf Testing

Penetration Testing

Deploy to Production

Deploy to QA Env

Deploy to StagingEnv

Deploy to Stress Env

Deploy to Security Env

ContinuousDeploymentExample

Steps outlined in blue in this example are manual

@lisacrispin

CD(eitherone)withoutautomation…

Likedrivingatnightwithoutyourheadlights.It’spossible… butheadlightsreducetherisk.

Thewholeteamapproachisoneheadlight…Automatedregressiontestsaretheother.

Props to Ashley Hunsberger for the analogy

@lisacrispin

To sum up the terms again…

DeploymentPipeline

DeveloperCommit

DeployedtoProduction

Stage Stage Stage Stage Stage Stage

DeploymentPipeline

DeveloperCommit

DeployedtoProduction

Codeismerged(master/trunk)

Stage Stage Stage

ContinuousIntegration(CI)

Stage Stage

Businessapproval

Compilation/unittesting

Createdeploymentartifact

Testingstage1

Testingstage2

DeploymentPipeline

DeveloperCommit

DeployedtoProduction

Codeismerged(master/trunk)

ContinuousIntegration(CI)

ContinuousDelivery(CD)

ContinuousDeployment(confusingly,thisisalsoCD)

Businessapproval

Compilation/unittesting

Createdeploymentartifact

Testingstage1

Testingstage2

DeploymentPipeline

DeveloperCommit

DeployedtoProduction

Codeismerged(master/trunk)

ContinuousIntegration(CI)

ContinuousDelivery(CD)

@lisacrispin

Visualizeyourpipeline

● Trygettingacross-functionalgroupofteammemberstogether● Devs,testers,productfolks,ops

● Writeyourpipelinestepsonbigstickies (realorvirtual)● Arrangethemonatable,wall,virtualwhiteboard● Talkaboutit!

@lisacrispin

Howcouldyoudeliverfaster?

● Parallelizingsteps?● Newtools?Moreautomation?Movingtothecloud?● Whatregressiontestsdoyouneedtorunforconfidence?

@lisacrispin

TheTestSuite Canvas (fromAshleyHunsberger,inspiredbyKatrinaClokie)

https://github.com/ahunsberger/TestSuiteDesign

@lisacrispin

Let’sexploreafewcanvasdiscussionpoints

Confidence

@lisacrispin

Agoodplacetostartis…

dowedoeachstepinourcurrentpipeline?

@lisacrispin

Whatdowewanttolearnfromeachstep?

Whatbusinessquestionscaneachstepinourpipelineanswer?● Integrationandbuild● Staticcodeanalysis● Automatedtestsuites● ManualtestingWhocanbenefitfromtheinformation?Howshouldtheybeinformed?Whatriskscanwemitigatewitheachstep?

@lisacrispin

Afewrealworldexamples…

APITestSuite● AmIgettingproperresponsesthatwarrantUItesting?StaticCodeAnalysis● Arewemeetingaccessibilitystandards?BuildInstallerTesting● Doesthebuildinstallwithouterrorsothatitisworth

furthertesting?

@lisacrispin

Dependencies

Whatneedstobeinplaceforagivensteptorunsuccessfully?○ Othersystems○ Tools○ Data,environments...

@lisacrispin

Constraints● What’spreventingusfromoptimizingagivenstep?● Forexample:isitamanualstepwecouldautomate?● Dowehaveautomation,butit’sslowandflaky?● Whatareourknownworkarounds?

@lisacrispin

Triggeringeachstepinyourpipeline

• Whatkicksoffeachstepinyourpipeline?

• Canyouparallelizetoshortenyourfeedbackloop?

• Ifoneprerequisitestepfailsbutanotherpasses– doyourunthesharednextstep?Orstop?

@lisacrispin

Gates

● Automaticallystopdefectsfrommakingitanyfurtherdownstream – fastfeedback

● Trustyourtests– flakytestsareuseless● Usenewtechnologytohelpmaketeststrustworthy

@lisacrispin

Testdata

Howdowemanagetestdata?● Tradeoffofspeedvs.simulatingproduction● Unittestsusetestdoubles- fakes,stubs,mocks● Higherleveltestsusefixtureorcanonicaldata

○ whichsimulatesproddata● Setupandteardownforeachtest

@lisacrispin

There’smoretothecanvas

• Useittogenerateconversations• Makesureyouaddresseverythingimportanttogiveyou

confidenceforcontinuousdelivery

@lisacrispin

BuildingaDevOpsCulture

DevOpsisn’taroleorateam

It’scollaborationbetweenthesoftwaredeliveryteam(includingtesters)andthesystemadministrationandoperationsteam

@lisacrispin

Buildingrelationships

Thiswholeteamapproachsoundsnice,but…● Howcanweengageotherstocollaborate?

Whatareyourideas?

@lisacrispin

We’rehumans!(orpossiblydragons,donkeys,unicorns…)

● Startwithcasual,friendlyconversations● Dofood● Sharesomethinguseful● Askpeopleinotherroles/teamsto

participate,sharetheirknowledgeKatrinaClokie hasexcellenttipsinAPracticalGuidetoTestinginDevOps

@lisacrispin

Cross-disciplinepairing,mobbing

Picture from Mob Programming Guidebook, MaaretPyhäjärvi and Llewellyn Falco

@lisacrispin

“Stoptheline”mentality- fromToyota

● Everyemployeeontheassemblylinehasaresponsibilityto“stoptheline”whentheyseeadefect● Benefitofwholeteamapproach

● Pushingthe“bigredbutton”isaninvestmentthatleadstoimprovements:○ Knowledgesharing○ Cost,speed○ Reliability

@lisacrispin

Learningfromproductionuse

@lisacrispin

Monitoring,observing

● Testinginproductionisanecessity● Bigdataandthetoolstomonitoritarehere● ANI,MLallowustoprocessthedata● Needabilitytorespondquicklytopainpoints● Teamdisciplinetorespondtoalerts● Usagetrendscaninspirenewfeatures● “Learningreleases”aka“MVP”● A/B,betatesting

@lisacrispin

Exposurecontrol

● Yourteamneedstomasterfeaturetoggles!● Dogfooding,canaryrelease,stagedrollout,darklaunch

@lisacrispin

HowDOwefitinallthosedifferenttypesoftesting?● Thosefeaturetoggles● Developersexploratorytest&moreatstorylevel● Storiesforalltypesoftestingatfeature/epiclevelgointo

thebacklogwithfeaturestories● ExploratorytestchartersAND● A11y,i18n,security,reliability,performance… storiesgo

intothebacklogwithfeaturestories● Testerspair&mobwithdevs,designers,productpeople…● Anyonecanpickupatestingtask

@lisacrispin

ImportantDevOpstopicsI’mnotcoveringtoday

● Infrastructureascode● Configurationmanagement● Containers● Cloud● Environmentmanagement● Infrastructuretesting… SeeContinuousDeliveryandAPracticalGuidetoTestinginDevOps

@lisacrispin

● Collaboratingtocontinuouslyimprovepipelines,feedback● Wholeteamcommitment,engagement● Visualizetogether,experiment● Babysteps- it’saprocess● Not“shiftingleftorright”– it’sinfinite!

Succeeding with the whole team approach

Ellen Gottesdiener and Mary Gorman, Discover to Deliver

@lisacrispin

Anymorequestions?

Whatwillyoutrywithyourownteam?

@lisacrispin

Afewresources

• AgileTesting,MoreAgileTesting(LisaandJanet)• ContinuousDelivery:ReliableSoftwareReleasesThroughBuild,Test,and

DeploymentAutomation (JezHumble andDavidFarley)• APracticalGuidetoTestinginDevOps(KatrinaClokie)• Accelerate:TheScienceofLeanSoftwareandDevOps:BuildingandScaling

HighPerformingTechnologyOrganizations(NicoleForsgren,JezHumble,GeneKim)

• https://github.com/ahunsberger/TestSuiteDesign• CharityMajors’blog,lotsonmonitoring:https://charity.wtf/

@lisacrispin