a tale of two pipelines: to devops or not to devops

A Tale of Two Pipelines: To DevOps or Not To DevOps

Brett HoferGlobal DevOpsPractice Lead

25+ years of IT experience

Dynatrace• Recognized thought leader DevOps & Big Data• Featured speaker, presenter, writer - DevOps.com,

DevOpsSummit Journal, InfoWorld and many more• Author eBook – “The Art of DevOps”

Prior to Dynatrace• Senior app manager for 80M+ budget, staff of 70+

including onshore/offshore• Lead designer and developer of multi-platform

apps, multi-channel intranet/internet web • Senior Solutions Architect on 60M+ initiative(s)

Welcome to“A Tale of Two

(very different) Pipelines”

It’s about people,

processes and technologies!

Are you sinking, surfing, or just

surviving?

Our idea here will be to help identify bad situations by

relating to all or part of the two stories …

We’ll aim to ignite ideas of positive

change by sharing experiences through

these stories.

A story about one fictional organization struggling

with all too familiar pipeline

challenges and how it took a corporate catastrophe to

finally decide on transitioning to solid DevOps practices.

Which pipeline will look like yours?Maybe somewhere in the middle?

What can you do to influence change?

What exactly are we moving through the pipeline?

Servicing requests from business ideas, non functional needs, fixes, functionality adjustments

Not every change is

created equal!

Success equals the RIGHT change at the RIGHT time

• Precision in priority ensures all efforts are focused on what team has agreed is success!

• Getting people to agree can be tough.

• Key: invest in “best-of-breed” technology to track, collaborate, and continually monitor FACTS versus OPINIONS when prioritizing

Typical top priority factors

• Value• Cost• Risk• Impact• Time

Add continuous monitoring to deliver continuous improvements

• Find opportunities and improve value

• Identify / manage cost factors

• Reduce and mitigate risk

• Identify and manage impacts

• Find problems fast and reduce time, costs and risk

All the right people know the health status of a change and build as it moves through the pipe

• Test and monitor early and often• Find and fix problems fast and early

To DevOps or not to DevOps

Fingerscrossed…

Welcome to easyTravel

Debute latest online booking system

• Development over a year• Lots of internal jokes

“neverTravel”• Ops “hope they load tested”• Late and over budget• App manager changed twice• Tension between business

and dev stalled budget twice

Mark SmoltenChief Marketing Officer

• 2 years at easyTravel• Other successful roles driving biz up

30%• Challenged by

• channels against fierce competition• convincing exec management to

invest in new, unproven outlets• not enough info on current

platform use and state• not being tied into new system in

• Mark contacts TravelVision and lines up a spot on their wildly popular live TV Show “Book Me Now!”

• Mark offers viewers an amazing deal on a live TV promo code in exchange for debuting the new easyTravel site

• Live show November 7th, 8:00 p.m.

• Yay, Mark!

TravelVision’s BookMeNow!

Tina AntonVP, eCommerce

• 8 years at easyTravel, 15 years industry• Promoted to head of eCommerce

two years ago• More experienced in the travel

business than eCommerce• Oversees the overall vision of every

easyTravel eBusiness function• Maintains online budgeting decisions• Very friendly, but long history of

battling it out with IT

Mark: Enthusiastic | Tina: Concerned

Mark speaks with Tinaabout his Live Promo Deal with TravelVision.

He explains the code should affect pricing only for the duration of the show and only for those who provide the code issued during live broadcasting.

Tina mentions that they don’t have such a feature on the site and wishes he had consulted with her first. She asks him “when is this?”

“We air on live TV to show off the new site and the promo on November 7th at 8pm! It’s the only time until next fall that they will have this event.”

Tina, very apprehensive, explains “but that’s less than two months away! We don’t even have a promo feature on the site since all our travel packages are already discounted.”

Mark states “This is the business nowadays. We have to adapt quickly to capture market opportunities. The discounts will come from the marketing budget. We need to make this happen!”

Tina, continues to voice her concerns trying to explain that there is a big change process for something like this.

“Mark, things are also more complicated these days due to all the channels we support. This would need to be available on Web, Web Mobile and in our native Mobile apps.”

Rick TrentonSenior Project Manager

• Consultant with the company one year• Ex-military very structured and organized• Assigned to the project management of

easyTravel Web and mobile • Brokers budgets and costs between

business and IT for major deliverables • Challenged with producing weekly

project statuses to management in PowerPoint

• Regularly maintains high level tasks in his Microsoft Project Plan

Tim FolkDirector of Digital Services

• 5 years at easyTravel• Former Senior Solutions Architect • Inherited responsibility 4 months ago • Not a fan of his Management Admin

Tasks• Manages all the Development of Web

& mobile channels for easyTravel• Challenged with lack of time, too many

meetings, putting out fires• He manages a local smaller IT staff but

required to use at least 70% offshore staffing for his dev to keep costs down

Tina now contacts both Rick and Tim to explain this new Feature that must be ready and tested in all channels by November 7th. Less than two months away.

Rick’s PM contract is funded by the business so he is very appeasing to Tina while Tim is drowning in numerous problems. Tim explains “I’m worried about the viability of the entire new site by November never mind this new feature.”

“Well you guys know the marching orders. I have confidence you will get this done. Keep me posted on the progress and take care.”

Meet Kelly RochellSenior Business Analyst easyTravel

- Works for Tina

Meet Scott DuttonDigital Services Dev LeadeasyTravel

- Works for Tim

Meet Frank BozmanDigital Services Operations LeadeasyTravel

- Works for Tim

Tim contacts three key members

critical to getting this

- Works for Tina

Kelly explains her understanding of the feature

and it’s urgency.

- Works for Tina

“Ok Kelly, we are going to need to get this fully documented and exactly how it should work.”

- Works for Tina

- Works for Tim

“Hey Tim, I’m going to need to put this on a completely

separate code branch and we

will need another environment. It’s going to impact too many things to put in the Oct

release.”

To understand Tim’s concerns,let’s look at the app pipeline he owns …

Fingerscrossed…

Tale of Two Pipelinesby Brett Hofer

Senior Business Analyst easyTravel

- Works for Tina

- Works for Tim

separate branch and we will need

another environment. It’s going to impact too many things

Scott’s Traditional Reasoning on incorporating this change PROMOFEATURECHANGE

Source Control

easyTravel Master CODE Branch v2.0

easyTravel CODE Branch v2.2 (Oct)

easyTravel CODE Branch v2.5 (Jan)

easyTravel CODE Branch v3.0 (Mar)

Managed VM Infrastructure

DEV1vm

DEV2vm

DEV3vm

DEV4 (ER)vm

QA4 (ER)vm

PERFvm

PROD Avms

PROD Bvms

mergeHosts 2.2 build

Hosts 2.5 build

Hosts 3.0 build

Hosts Fix Builds

Hosts 2.2 build

Hosts 2.5 build

Hosts 3.0 build

Hosts Fix BuildseasyTravel CODE Branch v2.3 (Nov)

promotions

PROMO Only Dev related Code

FEATURECHANGE

PROMOMeet Kelly RochellSenior Business Analyst easyTravel

- Works for Tina

- Works for Tim

“Haha, we have about as much

chance of getting new machines provisioned in

that timeframe as I have in getting a decent build from Scott.”

FEATURECHANGE

- Works for Tina

- Works for Tim

“Hey Frank, it might be easier if the machines

you get me weren’t as slow as you. I’m

tempted to have our guys bring in

their home machines and

deploy to those.”

FEATURECHANGE

- Works for Tina

- Works for Tim

“Honestly, if I put in a ticket to provision new VMs with the infrastructure department it

will have mold on it by November.”

Senior Business Analyst easyTravel

- Works for Tina

- Works for Tim

Frank’s Traditional Ops Reasoning on procuring new VMsPROMOFEATURECHANGE

Managed VM Infrastructure

DEV1vm

DEV2vm

DEV3vm

DEV4 (ER)vm

QA4 (ER)vm

PERFvm

PROD Avms

PROD Bvms

Hosts 2.2 build

Hosts 2.5 build

Hosts 3.0 build

Hosts Fix Builds

Hosts 2.2 build

Hosts 2.5 build

Hosts 3.0 build

Hosts Fix Builds

promotions

Enter a Dev 5 Provisioning Ticket

Enter a QA 5 Provisioning Ticket

Outsourced Company Managing Environments locates frame & Builds Hostaccording to security, OS, CPU, Memory.

Franks staff will manually setupthe DEV environment for deploying Nov

Franks staff will manually setupthe QA environment for deploying Nov

DEV5vm

Hosts Fix Builds

Hosts 2.3 PromoLead Time to identify, provision and get cost contract – 30 to 60 days

FEATURECHANGE

- Works for Tina

- Works for Tim

“Guys, maybe we should just

concentrate on finding a

solution that will get us

there. This needs to get done.

Kelly let us know when you have our requirements.”

FEATURECHANGE

- Works for Tina

- Works for Tim

“Wow, it sounds like you guys have your work cut out and on

that note, thanks Tim, we will have your requirements in a few days”

Now let’s evaluate

this feature as it moves

through their

pipeline.

The good news – the pipeline is somewhat agile

Design

DevelopTest

Operate

The bad news – all the cycles are long!3 month cycles now require an interim

releasePlan

Design

DevelopTest

Operate

ScoreC-

Communication spreadsheets/meetings/phone/emailsBi

Plan DesignFEATURECHANGE

Creates Need

ValidatesSets

Urgent

IdentifiesBiz Details

High LevelEstimate

InitiatesTracking

Creates Need

ValidatesSets

Urgent

High LevelEstimate

InitiatesTracking

CMO Recognizes Major Opportunity Initiates Need

FEATURECHANGE

• Industry is fast past and always fluid.

Creates Need

ValidatesSets

Urgent

High LevelEstimate

InitiatesTracking

High Level Business Functionality

• All Channel must change• Actually 3 Pipelines affected

• Mechanics of how it will work

Creates Need

ValidatesSets

Urgent

High LevelEstimate

InitiatesTracking

Separately Feature tracking ++Risk, +Time, +Cost • Dev uses tracking software, but no

other groups do

• Frank adds the task to his trackingspreadsheet

Plan Design FEATURECHANGE

nIdentify FeatureImpacts

AppArchitect

CapacityDesign

StoryBoard

Identify FeatureImpacts

DesignTesting

Strategy

GatherTime &

Track on Overall

AppArchitect

CapacityDesign

StoryBoard

DesignTesting

Strategy

GatherTime &

Track on Overall

Meet Raheeb Patel Senior Quality Assurance LeadeasyTravel Contractor

- His company contracted by QA department- Follows his QA dept and corporate guidelines- Needs very specific detailed technical specs

to put his team in motion- Runs a large team of offshore manual testers

AppArchitect

CapacityDesign

StoryBoard

DesignTesting

Strategy

GatherTime &

Track on Overall

Design Feature Impact Assessment - +++RISK, +++TIME, +++COST• Done conversationally among

a few technical resources• Frequent inaccuracy and guessing.

Has major cascading effects on capacity, architecture, scheduling

AppArchitect

CapacityDesign

StoryBoard

DesignTesting

Strategy

GatherTime &

Track on Overall

Architecture of Feature - +++RISK, +++TIME, +++COST• Done conversationally among

architects • Many know the general architecture

but aren’t up to speed on all of the detailed changes over each release.

AppArchitect

StoryBoard

DesignTesting

Strategy

GatherTime &

Track on Overall

Capacity Planning - +++RISK, +++TIME, +++COST• Lacks comprehensive understanding

of site traffic and visits• Lacks comprehensive understanding

of scalability concerns

CapacityDesign

Communication spreadsheets/meetings/phone/emails

FEATURECHANGE

Code Check-In RunBuild

Manual Dev Env Config

ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

DocumentTest Scripts

Manual Stage Env

Config

Manual Prod Env

Config

FEATURECHANGE

ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

Manual Stage Env

Config

Manual Prod Env

Config

Diagnose

Using Offshore Development - +RISK, ++TIME, +COST • Timezone diff often = long response

times for fix & feedback cycles• Lack of Backend System Support

FEATURECHANGE

ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

Manual Stage Env

Config

Manual Prod Env

Config

Diagnose

Developers on varying Platforms - +RISK, +TIME • Not developing on the same

technical stack as intended deploytargets - introduces unnecessaryrisks

FEATURECHANGE

ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

Manual Stage Env

Config

Manual Prod Env

Config

Diagnose

Overuse of Merging - +RISK, +TIME, +COST • Increased manual effort on changes

FEATURECHANGE

ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

Manual Stage Env

Config

Manual Prod Env

Config

Diagnose

Too many premature Check-Ins - ++RISK, ++TIME, ++COST

• No static policy checking on dev code • No runtime policy checking on dev code• No code review on critical functions • No architectural validation

FEATURECHANGE

ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

Manual Stage Env

Config

Manual Prod Env

Config

Diagnose

Manual Run Builds - +RISK, +TIME

• Often requires contacting one person• Limited Visibility and Feedback of

build results

FEATURECHANGE

ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

Manual Stage Env

Config

Manual Prod Env

Config

Diagnose

Slow Build Time - +TIME • Build is taking over 50 minutes

FEATURECHANGE

ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

Manual Stage Env

Config

Manual Prod Env

Config

Diagnose

9% Low and Limited Unit Coverage - ++RISK, -VALUE • Limited Visibility and Feedback

• Places high/all reliance on functional testing – Which is manual!!!

FEATURECHANGE

ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

Manual Stage Env

Config

Manual Prod Env

Config

Limited Result Visibility - +RISK, +TIME • Only one or two people

• Manual skimming of build logs • Often too much promoted “thrown

over the wall”

Diagnose

FEATURECHANGE

ManualDeploy

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

Manual Stage Env

Config

Manual Prod Env

Config

Manual Changes to Environments - +RISK, -VALUE• Prone to cross environment errors• Manual adjustments forgotten and

often not repeated as necessary• Static environments get dirty• Kills innovation speed

CheckResults

Diagnose

FEATURECHANGE

ManualDeploy

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

Manual Stage Env

Config

Manual Prod Env

Config

Manual Deployments in Development - +TIME, -VALUE• More manual delays in promotion

CheckResults

Diagnose

FEATURECHANGE

ManualDeploy

CheckResults

ManualMerge

Unit Test

Test Prod

ManuallyPrepare

Test Data

Manual Stage Env

Config

Manual Prod Env

Config

Dev4Dev4

Diagnose

Scott’s Dev Visibility & Status Sharing

• Few actually know the build health in Dev• Few actually know common Dev problems • Too little screened • Lacks macro level metrics

spreadsheets/meetings/phone/emails

Test Prod

DetermineTest Suite

Manual DeployQA 5

FEATURECHANGE

Manual Tests

EnterProblems

Manual Deploy

Perform

Manual Tests

ReportProblems

Validate Build

DiagnoseRoot

Check Infrastruct

Manual Deploy

Test Prod

DetermineTest Suite

Manual DeployQA 4

FEATURECHANGE

Manual Tests

EnterProblems

Manual Deploy

Perform

Manual Tests

ReportProblems

Validate Build

DiagnoseRoot

Check Infrastruct

Manual Deploy

Manual Test Selection - ++TIME, --VALUE, ++COST

• Selecting a lot more manual testing than what the change requires stifles new changes and costs time

• Manual Testing Greatly slows feedback time.

Test Prod

DetermineTest Suite

Manual DeployQA 4

FEATURECHANGE

Manual Tests

EnterProblems

Manual Deploy

Perform

Manual Tests

ReportProblems

Validate Build

DiagnoseRoot

Check Infrastruct

Manual Deploy

MTTR SLOW - ++TIME, --VALUE, ++COST • Test usually reported as

just failed with confusing description

• Finding exact issue is hard• Wasted Time - Finger

Pointing• Unnecessary people

involved in failure

Validate Build

Almost 2 months later

Now it’s prime time

November 7 at 8:10 p.m.

Tale of Two Pipelinesby Brett Hofer Before DevOps

Hi everyone, I’m Lisa Goodheart, your host of

Book Me Now!

Today is our Big Deal day where once a year we

showcase a select travel service live allowing them to put their best foot forward

demonstrating great deals to our viewers.

Here with me is Eric Wallace the VP of Sales

from easyTravel. He has a special treat for you

today.

Thanks Lisa, I’m honored to be here and to show your audience what we

believe is the most exciting travel site on the

market.

Our business negotiates some of the most

incredible packages for the cheapest prices and

then bundles them.

Today we will demonstrate an absolute exclusive for this

show. If your audience jumps on our site either web or on their phone and uses this one time PROMO code.

Your Book Me Bo

Your Book Me Now Exclusive Live Promo

It will randomly discount our packages between 5% to 95% upon searching and

if you begin to book it upon that search you can

book that deal.

Your Book Me Now Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Bo

Thank you Eric. This is really exciting. Our

audience will now notice the Promo Code scrolling

on the bottom of their screen.

Let’s demonstrate

Exclusive Live Promo Code is ROLLTHEDICE for cheap travel.. Your Book Me Now Exclusi

We put in the Promo

ROLLTHEDICE

Where we want to take our Journey

ROLLTHEDICE

Belize

When we want to go…

ROLLTHEDICE

Belize

12/18/2016

1/5/2017

Now click search…

ROLLTHEDICE

Belize

12/18/2016

1/5/2017

20 seconds later . . .

Um this is unusually long…

ROLLTHEDICE

Belize

12/18/2016

1/5/2017

ROLLTHEDICE

Belize

12/18/2016

1/5/2017

Wow, your site must be working pretty

hard on those deals.

ROLLTHEDICE

Belize

12/18/2016

1/5/2017

Yes, it’s chugging away to find your

viewers the deals… is it

possible it’s your internet here?

ROLLTHEDICE

And… that doesn’t look good. Maybe a

commercial?

ROLLTHEDICE

Well folks we are sorry but it looks like easyTravel is running

into a technical difficulty. We will be

back after these messages…

ROLLTHEDICE

Belize

12/18/2016

1/5/2017

Off Air

ROLLTHEDICE

Belize

12/18/2016

1/5/2017

Well that was a complete waste of

time and money. You people need to get your act together.

Off Air

ROLLTHEDICE

Belize

12/18/2016

1/5/2017

Off AirYou’re not kidding, heads will roll for this one. That

was embarrassing and expensive.

ROLLTHEDICE

Belize

12/18/2016

1/5/2017

And that had nothing to do with our

internet.

Off Air

Tina’s house . . .

Are you kidding me!

And the War Room begins . . .

• Not only was all the money spent on the feature a loss but it turned out the 60% booking growth was a long term growth strategy for Mark the CMO.

• The actual event traffic was 1600% of peak.

• An estimated $4M in new net bookings was lost.

• Losses due to brand damage unquantifiable.

Now let’s walk through this same scenario in a DevOps culture . . .

Let’s see the story when the promo

feature is sent through their new DevOps

pipeline.

The good news – the pipeline IS Agile

2 Week Sprints Plan

Design

DevelopTest

Operate

The new Digital Services EnvironmentPipeline Transparency, Proximity & Availability Matters

Development Test Production

- Works for Tina

- Works for Tim

The DevOps Reasoning on incorporating this change (Use a feature toggle)

Source Control

easyTravel Release

Ops Infrastructure As Code

Performance and Dashboards

Infrastructure Cloud ServicesAvailable in Dev/Test &

SECURED Prod

Dev Source Code

Test Scripts

CI_CD Configs

LOAD BALANCE POLICY

easyTravelInstance

Tech StackImage for 2.0

PROMOFEATURECHANGE

Cloud Image Library

- Works for Tina

- Works for Tim

The DevOps Reasoning on incorporating this change (Use a feature toggle)

Source Control

easyTravel Release

Ops Infrastructure As Code

Performance and Dashboards

Infrastructure Cloud ServicesAvailable in Dev/Test &

SECURED Prod

Dev Source Code

Test Scripts

CI_CD Configs

LOAD BALANCE POLICY

easyTravelInstance

PROMOFEATURECHANGE

Cloud Image Library

Aligning All Release Source Code, Automation Scripts, Database Scripting,

Performance and Operational Assets in Source Control Is critical.

It is the PIPE!

Build Once, Deploy and Continuously Monitor in any environment

- Works for Tina“Hey Tim, I’m going to need to put this on a completely

The DevOps General FlowPROMOPROMOFEATURECHANGE

Test EnvironmentDev Environment

DeveloperLocal

Environments

Source Control

Production Environment

Dynatrace in Dev Dynatrace in Stage Dynatrace Prod

Load, Int, Func Tests

Report Results

Env Automation

Deploy Automation

Build ServerPerform Build

Run Unit Tests

Report Results

Environment Setup

Deploy Automation

FEEDBACK LOOP dashboards/emails/tracking technology/group chatBi

Creates Need

ValidatesSets

Urgent

High LevelEstimate

InitiatesTracking

Creates Need

ValidatesSets

Urgent

High LevelEstimate

InitiatesTracking

CMO Recognizes Major Opportunity Initiates Need

FEATURECHANGE

• Industry is fast past and always fluid.

Creates Need

ValidatesSets

Urgent

High LevelEstimate

InitiatesTracking

High Level Business Functionality

• All Channel must change• Actually 3 Pipelines affected

• Mechanics of how it will work

Creates Need

ValidatesSets

Urgent

High LevelEstimate

InitiatesTracking

Aligned all Release Tracks and Sprint Tasks • BA

• Dev• Ops• Test• Data• Security

• Automation• XA

AppArchitect

CapacityDesign

StoryBoard

GatherTime &

Track on Overall

TestDesign

XADesign

PerfArchitectReview

AutomArchitect

Design

FEEDBACK LOOP dashboards/emails/tracking technology/group chat

AppArchitect

CapacityDesign

StoryBoard

GatherTime &

Track on Overall

TestDesign

XADesign

PerfArchitectReview

AutomArchitect

Design

Design Feature Impact Assessment

• Leveraging Dynatrace runtime Visualizations on current relationships creates high accuracy on Impact decisions.

AppArchitect

CapacityDesign

StoryBoard

GatherTime &

Track on Overall

TestDesign

XADesign

PerfArchitectReview

AutomArchitect

Design

Design - Feature Impact Assessment

• Leveraging Dynatrace Actions provides high accuracy on page Impact decisions.

AppArchitect

CapacityDesign

StoryBoard

GatherTime &

Track on Overall

TestDesign

XADesign

PerfArchitectReview

AutomArchitect

Design

Architecture of Feature

• Leveraging Dynatrace runtime Visualizations on current relationships creates high accuracy on architectural decisions.

AppArchitect

CapacityDesign

StoryBoard

GatherTime &

Track on Overall

TestDesign

XADesign

PerfArchitectReview

AutomArchitect

Design

Architecture of Feature

• Leveraging Dynatrace end-to-end transactional visibility creates high accuracy on architectural decisions.

AppArchitect

CapacityDesign

StoryBoard

GatherTime &

Track on Overall

TestDesign

XADesign

PerfArchitectReview

AutomationArchitect

Design

eXperience Assurance Decisions

• Leveraging all of the Dynatrace User Experience Management Data

AppArchitect

CapacityDesign

StoryBoard

GatherTime &

Track on Overall

TestDesign

XADesign

PerfArchitectReview

AutomationArchitect

Design

Automation Architecture - ++Time, ++Cost, ++Risk • Role in designing and managing all

automation to move the feature through the pipeline.

• Leveraging Technologies to Automate Processes & Orchestration in CI & CD

AppArchitect

CapacityDesign

StoryBoard

GatherTime &

Track on Overall

TestDesign

XADesign

PerfArchitectReview

AutomationArchitect

Design

Performance Architecture • Performance Architect Reviews

Performance & Scalability Concerns of the Feature

• Designs Dashboarding Strategy forMonitoring Feature & Health

FEATURECHANGE

Code Check-In

Setup Release Blueprint

CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

Dev4Setup Developer Blueprint

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

InstrumentApp

Unit TestCI Server

CheckResults

FEATURECHANGE

Code Check-In

CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

InstrumentApp

Unit TestCI Server

CheckResultsBIG GOALS

• How fast can we determine a check-in will fail and get feedback to the right resource• How far can we progress a healthy build/app without manually touching it• How fast can we identify the problem and turnaround the fix

FEATURECHANGE

Code Check-In

CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

Using Onsite & Localized Development - -RISK, --TIME, -COST • Collaborate Often

• No Black Box Development • Culture

CheckResults

FEATURECHANGE

Code Check-In

CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI ServerDevelopers use Cloud Instances - -RISK, -TIME

• Developer Instances match targetdeployment stack.

• Easy to onboard other developers• Allows switching technology stacks

CheckResults

FEATURECHANGE

Code Check-In

CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

CheckResults

Architectural Validation - --RISK, --TIME, +Value

• Developers, Leads and Solution Architects use Dynatrace to check transaction validity prior to check-in. Great on critical transactions and faster than code reviews.

FEATURECHANGE

Code Check-In

CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

CreatePerf

Dashboard

CreatePerf

Dashboard

CheckResults

Unit TestCI Server

Using a feature toggle - -TIME, -COST • Increased manual effort on changes

FEATURECHANGE

Code Check-In

CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

CreatePerf

Dashboard

CreatePerf

Dashboard

Unit Test

CI Server

CheckResults

Code Coverage Static Code Checking - -RISK, -TIME, -COST • Manage Testing Code Coverage

• Govern Coding Practices • Code Coverage is on Main

Dashboards

JaCoCo

FEATURECHANGE

Code Check-In

CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit Test

CI Server

Source Control Respository - --RISK, -TIME, -COST • Aligning ALL Release assets

is critical

CheckResults

FEATURECHANGE

Code Check-In

CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

Build Automation - -RISK, -TIME

• Comprehensive Build Scripts• Include Dependency Resources • Leverage Profiles (Build script

is host aware)• Includes Monitoring of Tests

CheckResults

FEATURECHANGE

Code Check-In

CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

Automated Builds- -RISK, -TIME

• Scheduled off a check-in trigger• Orchestrate tests and deployments• Steps Exec Build Automation Goal

CheckResults

FEATURECHANGE

Code Check-In

CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

Fast Build Time - -TIME • Build takes only 15 minutes

CheckResults

FEATURECHANGE

Code Check-In

CI-CDProv Env

CheckResults

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit TestCI Server

Unit & Unit Integration Testing - -RISK, -TIME

• 78% Coverage • All Unit Tests are monitored by

Dynatrace with Results maintainedby Build

CheckResults

FEATURECHANGE

Code Check-In CI Server

CI-CDProv Env

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Unit Test

CheckResults

• dynaTrace Monitored Unit Results - --RISK, --TIME

• Regression Violations generate change actions in tracking software

FEATURECHANGE

Code Check-In

ReleaseCloud

Blueprint

CI-CDProv Env

Unit Test

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

Dev4DeveloperCloud

Blueprint

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

CI Server

CheckResults

Configuration Management --RISK, --TIME

• All ops/infrastructure configs are represented in code and

aligned with the release• EnvForRelease = Cloud/Container Blueprint + ConfigCode

FEATURECHANGE

Code Check-In

CI-CDProv Env

Unit Test

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

Automated Cloud/Container Provisioning - --RISK, --TIME

• Establish Instance to Deploy

CI Server

CheckResults

FEATURECHANGE

Code Check-In

CI-CDProv Env

Unit Test

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

CI Server

CheckResults

Automated Test Development- --RISK, --TIME

• All testers are test automation engineers.• All tests are integrated with Dynatrace

Silk Performer

Selenium

FEATURECHANGE

Code Check-In

CI-CDProv Env

Unit Test

Test Prod

Eng Manually

PrepareTest Data

CreateAutomation

Scripts

Diagnose

ManualMerge

Code ConfigMgmnt

Eng Create

Perf Dashboard

CreatePerf

Dashboard

CI Server

CheckResults

Performance Dashboards are Deliverables --RISK, ++VALUE

• Major Features need Visibility• The dashboard indicates if the PROMO feature

is being used, response times, inter-dependency health or any errors related. Watched throughout lifecycle.

Test Prod

Auto RunFull

Test Suite

EnvReady

FEATURECHANGE

Manual Tests

ReportProblems

Validate Build

Validate Results

DiagnoseRoot

Check Infrastruct

SelectiveLoadTest

Test Prod

Auto RunFull

Test Suite

EnvReady

FEATURECHANGE

Manual Tests

ReportProblems

Validate Build

Validate Results

DiagnoseRoot

Check Infrastruct

SelectiveLoadTest

Finding Root Cause –TIME, ++VALUE

The Team under DevOps was able

to do four iterations and a full load test

three weeks ahead of schedule.

The feature has been in.

Now click search…

ROLLTHEDICE

Belize

12/18/2016

1/5/2017

ROLLTHEDICE

11/29/2016

12/5/2016

And there you go an 80% discount

on a trip to Belize…

ROLLTHEDICE

11/29/2016

12/5/2017

Wow, that is fast and incredible. I’m sure our viewers want to thank easyTravel. I’m being

told this is our highest viewed show to date!

The DevOps Xcelerator

Confidential, Dynatrace LLC

Master Performance

Architect

Confidential, Dynatrace LLC

Assess

• Identify Target Application• Interview Key Teams with Set Topics

BluePrint

• Record Gaps & Recommendations• Written Roadmap to Drive Implementation Plan

Implement

• Integrate into Automation Process• Build Dashboards• Educate on Process

What We Do

Confidential, Dynatrace, LLC

Questions or to Inquire about the DevOps Engagement email: brett.hofer@dynatrace.com

Please put TALES WEBINAR in the Subject Line

a tale of two pipelines: to devops or not to devops

Technology

agile at scale @ ing belgium...release train devops high...

devops perspectives [decoding devops conference -...

continuous delivery with jenkins pipelines (incl. …...©...

orasi devops strategy · 2019-11-21 · salesforce)...

online kickoff lecture will take place on wednesday,...

accelerating devops pipelines with aws

devops devops devops

a ux tale: devops guide to empathy

building continuous delivery pipelines...some teams use the...

devops oxford- devops + bigdata @ realtime

tdc 2016 |trilha devops - dissecando e entendendo pipelines...

devops devops devops, at froscon

openstack and devops - devops meetup

knightmare_ a devops cautionary tale – doug seven

enterprise devops - bjss...

devops, platforms and devops platforms

devops(1) : what's devops - (mosg)

monitoring attack surface to secure devops pipelines

driving secure collaborationand e2e transparency … ·...

rory combe fischbachauer str. 6, 81539 munich web: combe.io...