puppetconf 2017: the five dirty words of ci- j. paul reed, release engineering approach

The Five Dirty Words of CI

J. Paul Reed Release Engineering Approaches PuppetConf, 2017

“CI”

@jpaulreed #PuppetConf

“CI”

@jpaulreed #PuppetConf

@jpaulreed #PuppetConf

J. Paul Reed@jpaulreed on

Alum of The Ship Show

15+ Years in Build/Release Engineering

Now, a DevOps™ Consultant™

Master of Science candidate in Human Factors & System Safety

@jpaulreed #PuppetConf

Root Cause Analysis

Dirty Word #1

@jpaulreed #PuppetConf

@jpaulreed #PuppetConf

Root Cause AnalysisA method of problem solving used for

identifying the root causes of faults or problems.

A factor is considered a root cause if removal thereof from the problem-fault-sequence prevents the final undesirable

event from recurring.— Wikipedia

@jpaulreed #PuppetConf

Our Perception

@jpaulreed #PuppetConf

Our Reality@jpaulreed #PuppetConf

“We found the Root Cause!”

@jpaulreed #PuppetConf

“Root” “Cause”Cause is something

you construct. What you call “root cause”

is simply the place where you stop looking any further.

— Sidney Dekker@jpaulreed #PuppetConf

A Better Choice: “Root Cause Analysis”

Proximate Cause(s)

But…@jpaulreed #PuppetConf

The “Five Whys”

Dirty Word #2

@jpaulreed #PuppetConf

The “Five Whys”Five Whys is an iterative interrogative

technique used to explore the cause-and-effect relationships

underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or

problem by repeating the question "Why?" — Wikipedia@jpaulreed #PuppetConf

What “Five Whys” Always Feels Like to Me@jpaulreed #PuppetConf

The Perception: Incidents are deterministic, like code: same inputs, same outputs.

Every time.@jpaulreed #PuppetConf

A Crash Where?!

int foo(object& r) { r.Blah(); return 1; }

@jpaulreed #PuppetConf

The Operational Reality@jpaulreed #PuppetConf

A Better Choice: “Five Whys”

Just… no. “Swiss Cheese” model

Systemic model

@jpaulreed #PuppetConf

Human Error

Dirty Word #3

@jpaulreed #PuppetConf

Human ErrorHuman error has been cited as a primary cause or contributing factor in disasters and accidents

in industries as diverse as nuclear power, aviation, space exploration, and medicine.

Prevention of human error is generally seen as a major contributor to reliability and safety of

(complex) systems.— Wikipedia@jpaulreed #PuppetConf

But Really: What Is Human Error?

— James Reason’s conception@jpaulreed #PuppetConf

Who gets to draw “the line?” What incentives/interests do they have in putting that “line” where it is? It ignores other stories or even the possibility of entertaining other explanations…

Isues with “Human” “Error”

@jpaulreed #PuppetConf

“Human” “Error”Human error is not the cause of

failure, but the effect.

So, human error… can never be the conclusion of your investigation.

It is the starting point.— Sidney Dekker@jpaulreed #PuppetConf

Human Error often a prelude to a constraint on learning: “Well, just fire the dumb, bad apples… problem solved!”

@jpaulreed #PuppetConf

Were the World So Simple…

@jpaulreed #PuppetConf

A Tiny Problem in “The Cloud”

@jpaulreed #PuppetConf

A Different Take on Failure

@jpaulreed #PuppetConf

A Huge Opportunity to LearnOther operational tools with no input sanity checks The Service Health Dashboard’s real dependencies Indexing Subsystem’s insufficient partitioning Indexing Subsystem hadn’t been fully restarted for years

Had Amazon “just fired” this engineer, they would have never learned these critical details

about their system or how to operate it@jpaulreed #PuppetConf

A Better Choice: Human Error

Stop Saying It. Then, Keep

Not-Saying It.@jpaulreed #PuppetConf

“Why didn’t you?”/ “You should

have…”

Dirty Word #4

@jpaulreed #PuppetConf

Counterfactuals

Dirty Word #4

@jpaulreed #PuppetConf

CounterfactualsCounterfactual thinking is a concept in psychology that

involves the human tendency to create possible alternatives to life events that have already occurred.…

Counterfactual thinking is, as it states, "counter to the facts." These thoughts consist of the "What if?" and the "If I had only..." that occur when thinking of how

things could have turned out differently.— Wikipedia@jpaulreed #PuppetConf

A Waste of Time

@jpaulreed #PuppetConf

Discussing a reality that does not exist.

@jpaulreed #PuppetConf

A Better Choice: Counterfactuals

Don’t.

@jpaulreed #PuppetConf

Best Practice

Dirty Word #5

@jpaulreed #PuppetConf

@jpaulreed #PuppetConf

Best PracticeA best practice is a method or technique

that has been generally accepted as superior to any alternatives because it produces results that are superior to

those achieved by other means or because it has become a standard way of

doing things.— Wikipedia@jpaulreed #PuppetConf

Only the Best of Practices

@jpaulreed #PuppetConf

Only the Best, Artisanal of Practices“Best” is superlative Best practices in complex systems often ignore context Best practices are often not completely defined (especially in complex systems)

@jpaulreed #PuppetConf

Only the Best, Artisanal of Practices“Best practice,” by definition, leaves no space for innovation / discovery Relatedly (and maybe worst of all) it discourages experimentation

@jpaulreed #PuppetConf

Best Practicin’ Ourselves Outta Business

@jpaulreed #PuppetConf

Only the Best, Artisanal of Practices“Best practice,” by definition, leaves no space for innovation / discovery Relatedly (and maybe worst of all) it discourages experimentation Best practice applies to a domain little of our work exists in

@jpaulreed #PuppetConf

Obvious

ComplicatedComplex

Chaotic

Disorder

@jpaulreed #PuppetConf

Obvious

ComplicatedComplex

Chaotic

Governing Constraints

Good Practice

Rigid Constraints Best Practice

Lack of Constraints Novel Practice

Enabling Constraints Emergent Practice

Disorder

@jpaulreed #PuppetConf

Obvious

ComplicatedComplex

Chaotic

Governing Constraints

Good Practice

Rigid Constraints Best Practice

Lack of Constraints Novel Practice

Enabling Constraints Emergent Practice

Disorder

@jpaulreed #PuppetConf

@jpaulreed #PuppetConf

A Better Choice: Best Practice

“Good Practice” Or ensure you apply

“best practice” in the correct domain

@jpaulreed #PuppetConf

“Yeeahhh... so what kind of… Continuous Improvement...

would you... say ya do here?”@jpaulreed #PuppetConf

The Path of Continuous Improvement

is not “linear” (Nor is it “one-and-done”)

Takeaway I

@jpaulreed #PuppetConf

Respect Reality

Takeaway II

@jpaulreed #PuppetConf

Treat People Like the Professionals

They Are

Takeaway III

@jpaulreed #PuppetConf

Go Forth and Continuously Improve

J. Paul Reed www.jpaulreed.com

@jpaulreed

www.release-approaches.comSimply Ship. Every Time.@jpaulreed #PuppetConf

@jpaulreed #PuppetConf

Broken Build

Dirty Word #1

@jpaulreed #PuppetConf

“Flappers”

Dirty Word #2

@jpaulreed #PuppetConf

bobs-mac-mini. local

Dirty Word #3

@jpaulreed #PuppetConf

Merge Window

Dirty Word #4

@jpaulreed #PuppetConf

Jenkins Build Number

Dirty Word #5

@jpaulreed #PuppetConf