Build & Deployment InitiativePerformance Analysis Summary

Agenda• Overview

• Performance Analysis

▫ Data Gathering

▫ Root Cause Analysis

• Interventions

▫ Selection & Evaluation

▫ Justification

▫ Impact

• Analysis Summary

• Q&A

Overview

Opportunities

• Enable continuous integration builds

• Automatic bug creation for failures

• Triage team to perform root cause analysis on recurring failures

Benefits

• Developer productivity increased

• Test productivity increased

Key DefinitionsBuilds: custom code that is written by our developers and compiled into packages that can be deployed.

Deploys: the installation of compiled code packages onto specific servers within the websites infrastructure.

LOB: Line of Business (different business/product groups).

Problem

• Average 70% Build & Deploy success rate over 6 months prior to Intervention

• All Corp and LOB applications are below the 95% target success rate

Impact

• Developer productivity decreased;

• Tester productivity decreased

Performance Analysis

Goal

• Determine root cause of low build and deploy success rates.

Steps

• Data Gathering

• Root Cause Analysis

Data GatheringData from Tools

• Build success metrics

• Bugs logged against projects

• Tracked impacting issues

Data from Employees

• Processes and common issues

• Schedule and workload impacts

• Improvement ideas and suggestions

RAID

•Tracks build success and failure rates, duration, group, project, and release.

•Can create reports from this data, showing success rates across all projects and frequency of failures.

Anthill Pro

•Tracks all bugs found during a project, including build bugs.

•All bugs are tracking here and can be queried for build failure bugs. Typically includes a great level of detail.

SharePoint

•Tracks the top issues within the entire organization across all disciplines and groups.

•May contain non-bug related problems with the build process or higher level trends resulting from low build success rates.

Build Engineers

•Responsible for running the builds and troubleshooting any failures. Firsthand knowledge of common issues and trends across the projects.

Developers

•Write the code that is compiled and familiar with code related build failures and pre-code-checkinprocesses.

Dev Leads

•Responsible for code quality and processes developers must follow to deliver code. Will be familiar with code quality trends and the impact on development from low build success rates.

Project Managers

•Run the projects and track all issues and risks related to the project. Will be familiar with schedule impacts as a result of low build success rates.

Testers

•Test the builds once they have been deployed to test environments and will be familiar with issues blocking testing that may be related to the build and deploy process.

Data Sources: Tools

Data Sources: Employees

Methods Used

• Database queries

• Surveys

• Interviews

Root Cause Analysis

Problem

• Average 70% Build & Deploy success rate over last 6 months

• All Corp and LOB applications are below the 95% target success rate

Impact

• Developer productivity decreased

• Tester productivity decreased

Ro

ot

Ca

use

Disparate Processes: *Different LOB applications use different build and deployment

processes, bug creation standards methods, and issue analysis.

LOB Specific Tools: *40% of teams use non-standard build and deploy tools, increasing

tool support and licensing costs.

No Pre-build Validation: *Modified code is not validated for build errors prior to the scheduled

full build.

There are 3 main root causes that have contributed to the low build and deploy success rate:

Interventions

Goal

• Identify and implement specific actions to improve the low build and deploy success rate

Steps

• Selection & Evaluation

• Justification

• Impact

Selection & Evaluation

There are three main areas of improvement that each have a targeted intervention.

Interventions are evaluated based on five criteria:

Evaluation Criteria

• Specific

• Measureable

• Action-oriented

• Results-oriented

• Trackable

Intervention Justification

Organizational Goals

Quick delivery of new

features

High quality releases

Intervention Impact

Immediate identification of build breaks

Improves tracking of bugs and issues

Systemic issues identified and resolved

CI Builds

Automatic Bug Creation

Root Cause Analysis

Direct link between interventions and organizational goals.

One-time changes for build and deployment technology and processes.

Improvement in build and deploy success rate improves time to market and quality of releases.

Intervention Impact

Improvements

• Average 91% Build & Deploy success rate over 2 months following the Intervention

• All Corp and LOB applications are trending upwards towards 95% build and deploy success rate

• Bugs are created automatically upon any build or deploy failures

• Root cause for systemic problems is reducing problem recurrence, improving build and deploy success rates

Analysis Summary

December*Builds and deploys at a 70% success rate

January*Interventions implemented*Immediate improvement seen*Success rates at 82% and trending upwards

February*Success rates at 91% and trending upwards

March*Expected build and deploy success rates at or above 95% goal

Problem: Low build and deploy success rate

Interventions:

• Continuous integration builds

• Root cause analysis of failures

• Auto bug creation

Q&A