tech talk: the continuous delivery metrics that will make your pipeline fast and your apps better

World®’16

TechTalk:TheContinuousDeliveryMetricsThatWillMakeYourPipelineFastandYourAppsBetter

DanZentgraf – Sr.PrincipalConsultant– CATechnologies

DO5T20T

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation

3 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

Abstract

Akeyprincipleofcontinuousdeliveryiscontinuouslyoptimizingyourapplicationdeliverypipelineforqualityandspeed.Thisreliesonadeepunderstandingofthevariedprocessesatdifferentstagesofyourpipelineandtheirpotentialtokilldeliverytimesandproductivity.

Inthissession,learnwhyyouneedbothresultsandperformancemetricstouncoverdeepinsightsthatdrivesubstantiallongtermperformanceincreases,notjustincrementalchange.Understandhowapplyingmetricsthroughoutthepipelinefundamentallyenablesfeedbackloopsacrossyourcontinuousdeliverytoolchain.ExploretheintegralroleofCAReleaseAutomationContinuousDeliveryEditionintrackingandapplyingsoftwaredevelopmentlifecyclemetricstouniquelyoptimizeyourprocessesandenvironments.

DanZentgraf

CATechnologiesSr.PrincipalConsultant,Presales

4 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

TopicsforToday

INTRODUCTION

METRICS

RELATEDCAWORLDSESSIONS

INSTRUMENTATION

VALUESTREAMMAPPINGCONCEPTS

KEYMEASUREMENTITEMS

1

2

3

4

5

6

5 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

CoreValuesofDevOpsMovement– “CAMS”

CultureCAMS

Automation

Measurement

Sharing

6 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

AutomationIsAwesome

THEREARE,HOWEVER,AFEWPRE-REQUISITESTOASUCCESSFULEFFORT

§ Wheredowestart?§ Howdoweknowwearemakingprogress?§ Howdoweknowwearegettingvaluefromthe

effort?Arewedoingtherightthings?

Let’sAutomateALLTHETHINGS!!!

7 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

WE

“IfYouCan'tMeasureIt,YouCan'tImproveIt. ”

–PeterDrucker

“Ineverguess.Itisacapitalmistaketotheorizebeforeonehasdata.Insensiblyonebeginstotwistfactstosuittheories,insteadoftheoriestosuitfacts.”

– Sir.ArthurConanDoyle

“Ifthestatisticsareboring,you’vegotthewrongnumbers.” –EdwardTufte

“Therearelies,damnlies,andstatistics.”

–MarkTwain

8 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

ResultsMetricsandPerformanceMetrics

Myth:ResultmetricswillimproveperformanceReality:Youneedboth&theymustbealigned

RESULTMETRICS

ShowtheRESULTofanActionorProcess

PERFORMANCEMETRICS

TrackandMeasuretheActionsandProcessesthatLEADTOtheResult

9 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

ContinuousDeliveryMetrics

THECDCLASSIC:“CycleTime”Actualworktimeonaniteminaprocess

DEVOPS:“LeadTime”Elapsedtimefromwhenrequestwasmadetodelivery

CycleTimeandLeadTimeareRESULTmetrics

PERFORMANCEmetricsarelesswelldefined§ Muchvariancebasedonindividualteamor

businesssituation§ Organizationalbarrierscreatechallenges

10 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

Where/WhattoInstrument

VALUESTREAMMAPPINGAleanmanufacturingorleanenterprisetechniqueusedtodocument,analyzeandimprovetheflowofinformationormaterialsrequiredtoproduceaproductorserviceforacustomer.

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

11 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

EnterpriseDeliveryProcessesAreComplex…Anygivenenvironmenthasmultipleinvolvedstreams/subprocesses

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

Validate

Deploy

Configure

Provision

PROMOTIONDECISION

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

12 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

AndTheyMultiplyThroughthePipeline…

Dev/Int QA Prod

RequiresvisibilitytosecondleveltounderstandactualmixofCycle/Waittimetofindbottlenecks.§ Needthecoreautomationdriver+oversightinordertotrack

manualprocesseswiththeautomationacrossthewholepipeline§ THIS isthemetricsengineeringallofusmustdo!

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

Validate

Deploy

Configure

Provision

PROMOTIONDECISION

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

Validate

Deploy

Configure

Provision

PROMOTIONDECISION

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

Validate

Deploy

Configure

Provision

PROMOTIONDECISION

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

13 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

KeyOperationalMetricsatEachPipelinePhase

Howlongtillyouhaveadeployablepackage?

TimeframesforkeyTasksperenvironment

§ Howlongtoprovision/configure/deploy/validate?

§ Failurerateateachstep

ReductioninmanualTasks

Reductioninmaintenanceofautomation

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

Validate

Deploy

Configure

Provision

PROMOTIONDECISION

STEP STEP STEP STEP

CycleTime CycleTime CycleTime CycleTime

LeadTime

WaitTime

WaitTime

WaitTime

14 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

KeyOperationalMetricsforQualityintheFlow

CodequalitymetricsFeaturecompleteness

§ Howfardoesachangegobeforebeingkickedoutofprocess§ Rateoffeedback-basedrework

Deliveryquality§ “SmokeTest”passagerateperenvironment§ Decreaseinunintendedbreakage/brokendependencyevents

ApplicationHealth/Usage§ Monitoringsystem/reliability§ Appfeaturetracking

YoucanonlyBUILDqualityin.Youcannottestqualityin.

Dev

QA

Deploy

Run

15 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

PipelineDeliversStatisticstoReleaseManagementDiscipline-specificANDOverallFlowExamples

AgileMetrics§ Velocity§ Burndown/burnup§ Leadtime

Requirements&Defects

ConcurrentTesting

Dev&Build

Pre-production

Production&Monitoring

CAReleaseAutomation+Continuous

DeliveryEdition

AgileRequirements

Designer

ServiceVirtualization

TestDataManagement

CodeQuality§ Coverage§ UnapprovedOpenSource§ AppSec Remediation§ CycleTime

Testing§ Dependencybreakagerate§ Stagetimeperphase§ Smoketestpassage§ CycleTime

ProductionMetrics§ MTTR§ Supportincidentrate§ Usagedata

UAT§ Rejection/Reworkrate§ Stagetimeperphase§ Smoketestpassage§ CycleTime

16 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

ApplyingMetrics->AnalyticsIdeasfromTheLab– KPIDashboard/ReportCard

17 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

YourPipelineMetricGoalsAre…§ UnderstandhowyourPerformanceandResultMetricsrelatetoeachother§ AnalyzeyourValueStream§ IdentifyyourInstrumentationpoints§ TrackbothEfficiencyandQualitymetrics§ Listentoyourmetricsandapplythelessons

SummaryAFewWordstoReview

18 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

RecommendedSessions

SESSION# TITLE DATE/TIME

DO5X07SJeweler’sMutualInsuranceStreamlinesComplexAppDeployw/CARA

11/17/2016at12:45pm

DO5X12SEchoStarTransformsComplexSoftwareReleasestoCustomerNetworkSetTopBoxeswithCARACDE

11/17/2016at01:45pm

DO5T19SGMFinancialBuildsaSustainable,Holistic,ContinuousDeliveryPractice

11/17/2016at04:30pm

19 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

MustSeeDemos

ModernizeAppDeliveryIntegratedCDTheater5- DOV501P

DeliverTestDataFasterTestDataManagerTheater5- DOV511P

DeliverBetterAppsServiceVirtualizationTheater5- DOV507P

OrchestrateYourReleaseReleaseAutomationTheater5- DOV513P

20 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

Stayconnectedatcommunities.ca.com

Thankyou.

21 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

DevOps– ContinuousDelivery

FormoreinformationonDevOps– ContinuousDelivery,pleasevisit:http://cainc.to/PiTFpu