why everyone needs devops now: my fourteen …high performing devops teams they’re more agile 30x...

Session ID:

@RealGeneKim, [email protected]

Why Everyone Needs DevOps Now:

My Fourteen Year Journey Studying High Performing IT OrganizationsGene Kim

@RealGeneKim

Where Did The High Performers Come From?

@RealGeneKim

Visible Ops: Playbook of High Performers

The IT Process Institute has been studying high-performing organizations since 1999

What is common to all the high performers?

What is different between them and average and low performers?

How did they become great?

www.ITPI.org

@RealGeneKim

Act I: IT Ops Fixing Fragile Artifacts

@RealGeneKim

@RealGeneKim

The Product Managers

@RealGeneKim

Act 2: The Developers

@RealGeneKim

@RealGeneKim

IT Ops And Dev At War

10

@RealGeneKim

Nothing Left For Infosec

@RealGeneKim

@RealGeneKim

13

The DownwardSpiral…

@RealGeneKim16

The IT Core Chronic Conflict

Every IT organization is pressured to simultaneously:

Respond more quickly to urgent business needs Provide stable, secure and predictable IT service

Source: The authors acknowledge Dr. Eliyahu Goldratt, creator of the Theory of Constraints and author of The Goal, has written extensively on the theory and practice of identifying and resolving core, chronic conflicts.

@RealGeneKim

Every Company Is An IT Company…

95% of all capital projects have an IT component…

50% of all capital spending is technology-related

We are here…

Where we need to be…

IT is always in the way(again…)

Act 3:There Must Be A Better Way…

21

@RealGeneKimSource: John Allspaw

@RealGeneKim

@RealGeneKimSource: Theo Schlossnagle

@RealGeneKimSource: John Jenkins, Amazon.com

@RealGeneKim

Who Is Doing DevOps?

Google, Amazon, Netflix, Etsy, Twitter, Facebook, Pinterest …

BNY Mellon, Bank of America, World Bank, Paychex, Intuit…

The Gap, Nordstrom, REI, Macy’s, GameStop, Target …

Portland State University, Seton Hill University, Kansas State University…

Who else?

33

@RealGeneKim

High Performing DevOps Teams

They’re more agile

30x more frequent deployments 8,000x faster lead time than their peers

They’re more reliable

2x the change success rate 12x faster MTTR

Source: Puppet Labs 2012 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic

@RealGeneKim

35

@RealGeneKim

@RealGeneKim

46

“This book will have a profound effect on IT, just as The Goal did for manufacturing.” --Jez Humble, co-author Continuous Delivery

“This is the IT swamp draining manual for anyone who is neck deep in alligators.” --Adrian Cockroft, Cloud Architect at Netflix

“This is The Goal for our decade, and is for any IT professional who wants their life back.” --Charles Betz, IT architect, author “Architecture and Patterns for IT”

@RealGeneKim

The First Way: Flow

@RealGeneKim

The First Way: Flow

Understand the flow of work

Always seek to increase flow

Never unconsciously pass defects downstream

Never allow local optimization to cause global degradation

Achieve profound understanding of the system

@RealGeneKim

Questions

What is your lead time for changes? (i.e., how long does it take to go from “code committed” to “code successfully running in production”)

How much of that is queue time vs. run time?

51

@RealGeneKim

Day 2: PMO Meeting

@RealGeneKim

Create One Step Environment Creation Process

Make environments available early in the Development process

Make sure Dev builds the code and environment at the same time

Create a common Dev, QA and Production environment creation process

@RealGeneKim

If I had a magic wand, I’d change the Agile sprints and definition of “done”:

“At the end of each sprint, we must have working and shippable code, demonstrated in an environment that resembles production.”

@RealGeneKim

Deploy Smaller Changes, More Frequently *

Decouple feature releases from code deployments

Deploy features in a disabled state, using feature flags

Require all developers check code into trunk daily (at least)

Practice deploying smaller changes, which dramatically reduces risk and improves MTTR

56

@RealGeneKim

Breaking The Bottlenecks In The Flow

Environment creation

Code deployment

Test setup and run

Overly tight architecture

Development

Product management

57

58

How organizations achieve high performance

• 89% are using infrastructure version control• 82% are using automated code deployments


@RealGeneKim

Blackboard Learn: 2005-Present

62

Source: David Ashman, Chief Architect, Blackboard, Inc.

@RealGeneKim

Blackboard Learn Building Blocks

63Source: David Ashman, Chief Architect, Blackboard, Inc.

@RealGeneKim

The First Way:Outcomes

Creating single repository for code and environments

Determinism in the release process

Consistent Dev, Test and Production environments, all properly built before deployment begins

A continuous delivery pipeline that can be relied upon and daily Dev code commits

Free ourselves from the learned behavior of catastrophic deployments

Decreased lead time

Reduce deployment times from 6 hours to 45 minutes Refactor deployment process that had 1300+ steps spanning 4 weeks

Faster cycle time and release cadence

@RealGeneKim

The Second Way: Feedback

@RealGeneKim

The Second Way: Feedback

Understand and respond to the needs of all customers, internal and external

Shorten and amplify all feedback loops: stop the line when necessary

Create quality at the source

Create and embed knowledge where we need it

@RealGeneKim

“We found that when we woke up developers at 2am, defects got fixed faster than ever”

-Patrick Lightbody, CEO, BrowserMob

@RealGeneKim

Require That Devs Manage Their Own Code For 6+ Months

69

Source: Tom Limoncelli, Google

@RealGeneKim

Test Whether Developers Qualify For IT Operations Resources

Types/frequency of pager alerts

Maturity of monitoring

System architecture review

Release process

Defect counts and severity

Production hygiene

70


@RealGeneKim

Return Fragile Services Back To Dev

71


@RealGeneKim

Feedback And Situational Awareness

“Having a developer add a monitoring metric shouldn’t feel like a schema change.”

-- John Allspaw, SVP Tech Ops, Etsy

72

@RealGeneKim

73

@RealGeneKim

74

@RealGeneKim

Integrating Into Continuous Delivery

The days of reviewing RFCs in Word docs in change management meetings are over

Failures must result in automated tests in the continuous deployment pipeline (Release, Config, Change)

Invite or embed Ops into Dev standups and the scrum teams (“hey, we can sprint and scrums, too!”)

@RealGeneKim

The Second Way:Outcomes

Defects and security issues getting fixed faster than ever

Standardized and reusable Ops and Infosec user stories now part of the Agile process

All groups communicating and coordinating better

Everybody is getting more work done

@RealGeneKim

The Third Way:Continual Experimentation And Learning

@RealGeneKim

The Third Way:Continual Experimentation And Learning

Foster a culture that rewards: Experimentation (taking risks) and learning from

failure Repetition is the prerequisite to mastery

Why? You need a culture that keeps pushing into the danger

zone And have the habits that enable you to survive in the

danger zone

@RealGeneKim

Break Things Early And Often

“Do painful things more frequently, so you can make it less painful… We don’t get pushback from Dev, because they know it makes rollouts smoother.”

-- Adrian Cockcroft, Architect, Netflix

@RealGeneKim

82

@RealGeneKim

Inject Failures Often

@RealGeneKim

You Don’t Choose Chaos Monkey…Chaos Monkey Chooses You

@RealGeneKim

Break Things Before Production

Enforce consistency in code, environments and configurations across the environments

Add your ASSERTs to find misconfigurations, enforce https, etc.

Add static code analysis to automated continuous integration and testing process

@RealGeneKim

Reduce Technical Debt

“The deal with engineering goes like this. Product management takes 20% of the capacity right off the top and gives this to engineering to spend as they see fit. Whatever is required to avoid, ‘we need to stop features to rewrite code.

“If you’re in really bad shape today, you might need to make this 30% or even more of the resources. I get nervous when I find teams that think they can get away with much less than 20%.”

-- Marty Cagan, Inspired

@RealGeneKim

Allocate 20% Of Cycles To Technical Debt Reduction

@RealGeneKim

Recognize Compounding Technical Debt…

@RealGeneKim

That Gets Worse…

@RealGeneKim

And Fixing It…

Source: Pingdom

@RealGeneKim

An Innovation Culture

“By installing a rampant innovation culture, they now do 165 experiments in the three months of tax season.

Our business result? Conversion rate of the website is up 50 percent. Employee result? Everyone loves it, because now their ideas can make it to market.”

--Scott Cook, Intuit Founder

91

93

Performance by DevOps maturity

Organizations that implemented DevOps practices over 12

months ago were 5x more likely to be high performing than

organizations that weren’t implementing DevOps at all.


Why Do I Think This IsImportant?

94

@RealGeneKim

95

The DownwardSpiral…

@RealGeneKim

@RealGeneKim

97

@RealGeneKim

If I Could Wave A Magic Wand, Everyone Will…

See the suffering downstream, and have confidence that your intuitions and skills can make a profound and positive difference…

Become conversant with DevOps and recognize the practices when you see them

Be energized about how practitioners can contribute in this organizational journey

Leave with some concrete steps to get some great outcomes

Help create a team that starts putting DevOps practices into place

98

@RealGeneKim

100

“Some books you give to friends, for the joy of sharing a great novel.

“Some books you recommend to your colleagues and employees, to create common ground.

“Some books you share with your boss, to plant the seeds of a big idea.

“The Phoenix Project is all three.”

--Jeremiah Shirk, Integration & Infrastructure Manager at Kansas State University

@RealGeneKim

Our Mission: Positively Impact The Lives Of One Million IT Workers By 2017

Free 170 page excerpt:http://itrevolution.com/the-phoenix-project-excerpt/

http://slideshare.net/realgenekim

DevOps Defensive Audit Toolkit

Enterprise DevOps Case Studies

Early draft of upcoming “DevOps Cookbook” (Allspaw, DeBois, Edwards, Humble, Kim, Orzen)

Email me at [email protected]

why everyone needs devops now: my fourteen …high performing devops teams they’re more agile 30x...

Documents