data-driven software mastery @open mastery austin

Janelle Kleinopenmastery.org @janellekz

Tour Speaker

Data-Driven Software Mastery

newiron.com

, Developer, Consultant, CTO @New IronSpecialized in Statistical Process Control (SPC) and Supply Chain Optimization from Lean Manufacturing (I <3 cool data)

Continuous Delivery infrastructure, automation strategy & technical mentorship

Janelle Klein

Who Am I?

I’m also a hobbyist Cognitive Scientist

How to Measure the PAIN in Software Development

Janelle Klein

Author of “Idea Flow”

leanpub.com/ideaflow

LEARN YOUR WAY TO AWESOME.

Community of Practice for Data-Driven Software Mastery

Why Are We Here?

Why Should You Care?

Problem #1: Organizational Dysfunction

Why Should You Care?

Problem #2: Solving the Wrong Problem

Why Should You Care?

This talk is about a REALISTIC STRATEGY

to solve these problems.

I’m going to show you:

POINT A

(I’ve been working on this for 5 years)

I Need Your Help to Get to:

POINT AWESOME

Point A Point AWESOME?

Iterate!

LEARN YOUR WAY TO AWESOME.

Why Are We Here?

Learning Takes Work.

Let’s Make the PAIN Visible!

#OpenDX

Our Campaign:

This isn’t About Me.

This is About ALL OF US.

Five Years Ago I Read This Book:

How to Build a Learning Organization

Five Disciplines of a Learning Organization

Personal Mastery

Mental Models

Shared Vision

Team Learning

Systems Thinking

These disciplines were emergent practice on our team.

Five Disciplines of a Learning Organization

Personal Mastery

Mental Models

Shared Vision

Team Learning

Systems Thinking

These disciplines are emergent practice in software development

What’s the Gap?

What was different about our team?

My Day Job is Technical Assessment

GeorgeMe

What does “better quality” mean to you?

What does “better” really mean?

What does “better” really mean?

Where does my “better” come from?

“Better”

Fifth Discipline: “Shared Vision” = “Shared Better”

The Key to Team Learning:

“Better”

Five Disciplines of a Learning Organization

Personal Mastery

Mental Models

Team Learning

Systems Thinking

This book is about the art of group problem-solving.

Shared LearningShared Better

Five Years to Put “Better” into Words

How to Measure the PAIN in Software Development

Janelle Klein

This is the my story.

We were trying to do all the “right” things.

About 8 Years Ago…

We were building a factory automation system...

We shipped to production...

We shipped to production... (again)

We couldn’t reproduce the problem.

We shipped to production... (AGAIN)

The rollback failed!

Project FAILURE

The Retrospective

“What are we going to do?!”

Our biggest problem

“Well, we know we’ve got a quality problem right?”

The Retrospective

“What are we going to do?!”

Our biggest problem

“The problem is we don’t have enough test automation!”

So the Test Automation Began…

Our regression testing took 3-4 weeks…

Let’s automate the tests!

20%

Percent Automated:

So the Test Automation Began…

50%

Maintenance started getting PAINFUL…

Let’s refactor the tests!

Percent Automated:

So the Test Automation Began…

80%

It was still really PAINFUL…

“Well, at least our regression cycle is faster, right?”

Our regression cycle still took 3-4 weeks!

Percent Automated:

So the Test Automation Began…

ApplicationTests

Our Test Suite

ApplicationTests

Our Test Suite

Any ideas?

The Retrospective

“What are we supposed to do?!”

Our biggest problem

“Well, if we don’t understand a problem, we should

collect data.”

The Retrospective

Our biggest problem

“What data would help us understand the problem?”

“What are we supposed to do?!”

Technical Debt Mistakes

I thought the main obstacle was Technical Debt

?Most of our mistakes were in the

most well-written parts of the code.

Mistakes

We made significantly more mistakes in code that we didn’t write ourselves.

Lower Familiarity

More Mistakes=

There had to be more to the story...

Complex(So*ware(

PAIN

This is what I knew...

What made development feel painful?

Unexpected Behavior

Problem Resolved

Tracking Painful Interaction with the Code (Friction)

Troubleshooting

Progress

5 hours and 18 minutes of troubleshooting...

PAINFUL

The amount of PAIN was caused by…

Likeliness(of((Unexpected(Behavior(

Cost(to(Troubleshoot(and(Repair(

High(Frequency(Low(Impact(

Low(Frequency(Low(Impact(

Low(Frequency(High(Impact(

PAIN(

What Causes Unexpected Behavior (likeliness)?

What Makes Troubleshooting Time-Consuming (impact)?

Semantic Mistakes

Stale Memory Mistakes

Association Mistakes

Bad Input Assumption

Tedious Change Mistakes

Copy-Edit Mistakes

Transposition Mistakes

Failed Refactor Mistakes

False Alarm

Non-Deterministic Behavior

Ambiguous Clues

Lots of Code Changes

Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

Most of the pain was caused by human factors.

What causes PAIN?

What Causes Unexpected Behavior (likeliness)?

What Makes Troubleshooting Time-Consuming (impact)?

Non-Deterministic Behavior

Ambiguous Clues

Lots of Code Changes

Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

What causes PAIN?

Most of the pain was caused by human factors.

Semantic Mistakes

Stale Memory Mistakes

Association Mistakes

Bad Input Assumption

Tedious Change Mistakes

Copy-Edit Mistakes

Transposition Mistakes

Failed Refactor Mistakes

False Alarm

What Causes Unexpected Behavior (likeliness)?

What Makes Troubleshooting Time-Consuming (impact)?

Non-Deterministic Behavior

Ambiguous Clues

Lots of Code Changes

Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

What causes PAIN?

Semantic Mistakes

Stale Memory Mistakes

Association Mistakes

Bad Input Assumption

Tedious Change Mistakes

Copy-Edit Mistakes

Transposition Mistakes

Failed Refactor Mistakes

False Alarm

Most of the pain was caused by human factors.

What Causes Unexpected Behavior (likeliness)?

What Makes Troubleshooting Time-Consuming (impact)?

Non-Deterministic Behavior

Ambiguous Clues

Lots of Code Changes

Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

What causes PAIN?

PAIN is a consequence of how we interact with the code.

Semantic Mistakes

Stale Memory Mistakes

Association Mistakes

Bad Input Assumption

Tedious Change Mistakes

Copy-Edit Mistakes

Transposition Mistakes

Failed Refactor Mistakes

False Alarm

PAIN occurs during the process of understanding and extending the software

Complex(So*ware(

PAIN

Not the Code.

Optimize “Idea Flow”

Analyzing the Types of Mistakes

#1 Cause of Mistakes: Misunderstandings of how the system worked

ApplicationTests

The Bugs were in Our Conceptual Models

Fix: We built a Data Dictionary

The “Coat of Armor” Principle

Software

Coat of Armor

“Alarm Pain”

Six hours of troubleshooting and repairing tests when nothing is broken.

The Waxy Coating Principle

Optimize the signal to noise ratio.

Software (before)

Software (after)

Tests are like a waxy coating poured over the code.

We deleted everything…

The automation wasn’t solving the problem!

What are the specific risks in this release?

How can we mitigate the risks?

Handful of manual tests

Custom Tools

+

Release Size

(weeks)

Time

Defect Rates

We Could See the Effects…

Effective Risk Management Looks Like This

My team spent tons of time working on improvements that didn’t make much difference.

We had tons of automation, but the automation didn’t catch our bugs.

My team spent tons of time working on improvements that didn’t make much difference.

We had well-modularized code,

but it was still extremely time-consuming to troubleshoot defects.

The hard part isn’t solving the problems it’s identifying the right problems to solve.

“What are the specific problems that are causing the team’s pain?”

“Better”

Fifth Discipline: “Shared Vision” = “Shared Better”

The Key to Team Learning:

“Better”

“Better” following best practices

(solution-focused) (problem-focused)

“Better” solving the problems

What does “better” really mean?

How do we identify the right problems to solve?

Cost & Risk are a Function of Increased Difficulty

Cost &

Risk

Difficulty of Work

The Difficulty of Doing Our Jobs

Human Limitations

Likelihood)of))Unexpected)Behavior)

Cost)to)Troubleshoot)and)Repair)

High)Frequency)Low)Impact)

Low)Frequency)Low)Impact)

Low)Frequency)High)Impact)

PAIN)

“Better” = Reduce the Risk

Reduce Likeliness =

Mistake Proofing

Reduce Recovery Cost = Mistake Tolerance

What Exactly does Idea Flow Measure?

Complex(So*ware(

PAIN

Not the Code.

Optimize “Idea Flow”

FRICTION is a Function of Increased Difficulty

Cost &

Risk

Difficulty of Work

The Difficulty of Doing Our Jobs

Human Limitations

The Rhythm of “Idea Flow”

Write a little code.

Work out the kinks.

Write a little code.

Work out the kinks.

Write a little code.

Work out the kinks.

Conflict Confirm

Rework'

Learn'

Validate(

Modify'

Progress Loop Conflict Loop

Troubleshoot'

Write a little code. Work out the kinks.

The Rhythm of “Idea Flow”

On Intelligence Jeff Hawkins

Confirmed Expectations

Our brain works like a giant prediction engine…

Conflicting Expectations

Our brain works like a giant prediction engine…

Conflict Confirm

Rework'

Learn'

Validate(

Modify'

Progress Loop Conflict Loop

Troubleshoot'

The Rhythm of “Idea Flow”

“Friction” in Idea Flow

Conflict Confirm

Rework'

Learn'

Validate(

Modify'

Progress Loop Conflict Loop

Troubleshoot'

“Idea Flow Map”

“Friction”

A “conflict” starts with an unexpected observation and ends when understanding is resolved.

Idea Flow Mapping Tools

(Open Source, Supported GA ~June 2016)github.com/ideaflow/tools

Typical Idea Flow Maps

Single Problem

Multi-Problem

Reading Visual Indicators in Idea Flow Maps

Le#$Atrium$

Le#$Ventricle$

Right$Ventricle$

Right$Atrium$

What’s$causing$this$pa7ern?$

Similar to how an EKG helps doctors diagnose heart problems...

...Idea Flow Maps help developers diagnose software problems.

Problem-Solving Machine

Reading Visual Indicators in Idea Flow Maps

= Solution Strategy

The “Heart” of Software Development(the problem-solving machine)

= Solution Strategy

The “Heart” of Software Development(the problem-solving machine)

Choose a general strategy

= Solution Strategy

The “Heart” of Software Development(the problem-solving machine)

Understand the system

= Solution Strategy

The “Heart” of Software Development(the problem-solving machine)

Code & work out the kinks

= Solution Strategy

The “Heart” of Software Development(the problem-solving machine)

Back to the drawing board

The symptoms we experience depend on where the disruptions are in the problem-solving process

Trial and Error

The Friction is Here

Troubleshooting

Progress

Learning

Rework

Assumption Risk (Rework)

Likelihood)of))making)a))

Bad)Assump4on)

Cost)to)Correct)Decisions)

High)Uncertainty)Low)Delay)

Low)Uncertainty)Low)Delay)

Low)Uncertainty)High)Delay)

PAIN)

Struggling with Understanding the Code

The Friction is Here

Troubleshooting

Progress

Learning

Rework

Familiarity Risk (Learning)

Likelihood)of))working)with)Unfamiliar)

Code)

Cost)to)Learn)

High)Frequency)Easy)to)Learn)

Low)Frequency)Easy)to)Learn)

Low)Frequency)Hard)to)Learn)

PAIN)

Struggling with Feedback

The Friction is Here

Troubleshooting

Progress

Learning

Rework

Quality Risk (Troubleshooting)

Likelihood)of))Unexpected)Behavior)

Cost)to)Troubleshoot)and)Repair)

High)Frequency)Low)Impact)

Low)Frequency)Low)Impact)

Low)Frequency)High)Impact)

PAIN)

measures the time spent on:

Idea Flow

x

Troubleshooting

x

Learning

x

Rework

Quality Risk Familiarity Risk Assumption Risk

What percentage of time do you think your team spends on friction?

Troubleshooting

Progress

Learning

7:070:00

0:00 19:52

12 year old project after all original developers left.

Case Study: Huge Mess with Great Team

70-90% of dev capacity on “friction”

The Team’s Improvement Focus: Increasing unit test coverage by 5%

Case Study: Huge Mess with Great Team

1. Test Data Generation

2. Merging Problems

3. Repairing Tests

1000 hours/month

The Biggest Problem: ~700 hours/month generating test data

The Retrospective

“What’s the biggest opportunity for improvement?”

Our biggest problem

“What’s the biggest opportunity for improvement?”

“The awful email template engine code!”

Our biggest problem

The Retrospective

“What’s the biggest opportunity for improvement?”

“Fill in missing unit tests!”

Our biggest problem

The Retrospective

“What’s the biggest opportunity for improvement?”

“I know how to improve database performance!”

Our biggest problem

The Retrospective

“What’s the biggest opportunity for improvement?”

“Let’s improve maintainability of our test framework!”

Our biggest problem

The Retrospective

“What’s the biggest opportunity for improvement?”

Just because a problem comes to mind, doesn’t mean it’s an important problem to solve.

Our biggest problem

The Retrospective

“What’s the biggest opportunity for improvement?”

Our biggest problem

What do I feel the most intensely about?

Daniel Kahneman Thinking Fast and Slow

The Retrospective

“What’s the biggest opportunity for improvement?”

“The awful email template engine code!”

Recency Bias

Our biggest problem

The Retrospective

“What’s the biggest opportunity for improvement?”

Guilt Bias

“Fill in missing unit tests!”

Our biggest problem

The Retrospective

“What’s the biggest opportunity for improvement?”

“I know how to improve database performance!”

Known Solution Bias

Our biggest problem

The Retrospective

“What’s the biggest opportunity for improvement?”

Sunk Cost Bias

“Let’s improve maintainability of our test framework!”

Our biggest problem

The Retrospective

Relying on Gut Feel:

18 months after a Micro-Services/Continuous Delivery rewrite.

Troubleshooting

Progress

Learning40-60% of dev capacity on “friction”

0:00 28:15

12:230:00

Case Study: From Monolith to Microservices

The Architecture Looked Good on Paper

Team A Team B Team C

Complexity Moved Here

We Don’t Have TIME To Fix It!

PAIN

The Classic Story of Project Failure

Problems get deferred

Builds start breaking

Releases get chaotic

Productivity slows to a crawl

Developers begging for time

It’s never enough

Project Meltdown

The Cost of Escalating Risk

0%

100%

Release 1 Release 2 Release 3

Troubleshooting

Progress

Learning

Percentage Capacity spent on Troubleshooting (red) and Learning (blue)

(extrapolated from samples)

0%

100%

Release 1 Release 2 Release 3

Percentage Capacity spent on Troubleshooting (red) and Learning (blue)

Figure out what to do Learning is front-loaded

Troubleshooting

Progress

Learning

The Cost of Escalating Risk

0%

100%

Release 1 Release 2 Release 3

Percentage Capacity spent on Troubleshooting (red) and Learning (blue)

Rush Before the Deadline Validation is Deferred

Troubleshooting

Progress

Learning

The Cost of Escalating Risk

0%

100%

Release 1 Release 2 Release 3

Percentage Capacity spent on Troubleshooting (red) and Learning (blue)

Pain Builds Baseline friction keeps rising

Troubleshooting

Progress

Learning

The Cost of Escalating Risk

0%

100%

Release 1 Release 2 Release 3

Percentage Capacity spent on Troubleshooting (red) and Learning (blue)

Chaos Reigns Unpredictable work stops

fitting in the timebox

Troubleshooting

Progress

Learning

The Cost of Escalating Risk

This is the Biggest Cause of FAILURE In Our Industry:

Have you struggled with these problems in your organization?

“Better”“Better”

Fifth Discipline: “Shared Vision” = “Shared Better”

Developer

Developer

Manager Manager

The Key to Organizational Learning:

Lessons Learned

How can we reduce the impact of bad architecture assumptions?

(Assumption Risk)

The cost of bad architecture decisions in the microservices world

is EXTREMELY HIGH.

If you’ve got beautiful code

Because you pushed all the PAIN to your clients…

Your code SUCKS.

How do we know what the client’s experience will be like?

Lots of Friction in BRAND NEW code:

27:15

“WHY?”

Application

3rd Party Dependencies

Local Components

Usability problems cascade through dependencies

Usability Problems

This is Painful!

Application

3rd Party Dependencies

Local Components

Usability Problems

This is Painful!

90% of our Software is HERE

“The Idea Flow Factory” (supply chain model)

Optimize the Rate of Idea Flow

Across ANY Software Supply Chain

What if we learned our way to “better”, together?

CompanyCompany

Company

Company

Company

Company Company

LEARN YOUR WAY TO AWESOME.

Start with Why Simon Sinek

Why - Your Purpose

How - Your Process

What - Your Solution

Humans Align Around Shared Purpose

Let’s Make the PAIN Visible!

#OpenDX

Why?

#OpenDX

Why?

Because This Really SUCKS.

Why?

And So Does This.

#OpenDXMeasure the PAIN that you CARE About

LEARN YOUR WAY TO AWESOME.

We BELIEVE That:

Even though we may not have all the answers, what matters is that we know how to learn.

How Do We Learn Our Way There?

Point A Point AWESOME?

Iterate!

Because we’re part of a community with other smart people…

We BELIEVE That:

We will get better, faster, by working together.

#OpenDXHow?

Community Analytics Platform

(Long-Term Roadmap)

Community Problem #1:

Solve This Pain:

Team Visibility Platform

IFM Data Collection

Team Visibility

Raw Data

Timeline Abstraction

1. Record an Idea Flow Map 2. Review/Annotate an Idea Flow Map 3. Identify the Biggest Pains

Project Abstraction

#Private Taxonomy

Community Analytics Platform

Anonymizer Anonymizer Anonymizer Anonymizer

#Private Taxonomy

Team Visibility

#Private Taxonomy

Team Visibility

#Private Taxonomy

Team Visibility

#Private Taxonomy

Team Visibility

Community Analytics

#Shared Taxonomy

Community Analytics Platform

Community Analytics

Shared Taxonomy

Tiger

Bear

Requirement: Analyze Patterns & Codify Lessons Learned

Community Focus

Community Analytics

#Shared Taxonomy

Community Problem #2:

Solve This Pain:

Team Server

Team Server

“Idea Flow Factory” Supply Chain Cost Model

Supply Chain Optimization

Team Server

“Customer Factory” Revenue Model (Ash Maurya)

Throughput Accounting (Effort vs Revenue)

Organizational Mastery Platform

Community Problem #3:

Solve This Pain with Lack of Skills:

CompanyNope.Nope.Nope.Nope.Nope.

Nope.Nope.Nope.Nope.

Nope.Nope.

Nope.Nope.Nope.

Nope.Nope.Nope.Nope.Nope.Nope.Nope.Nope.

Nope.Yay!

We need aSr Developer!

Open Certification Platform (Crowd-Sourced Peer Mentorship)

Decision Assessment

Archetype Examples & Decision-Making Tests

Mentorship Training

M1M2

M3

C1C2

C3

Crowd-Sourced Peer Video Assessment

#Shared Taxonomy

Community Problem #4:

Solve This Pain with Upstream Usability:

#OpenDXBring visibility up to the highest levels of our

organizations and across the industry

The “Open Source” Roadmap

Point A Point AWESOME?

Iterate!

Get together over lunch and share lessons learned.

Leaders Circle

Developers Circle

Open Mastery Circle Meetings

Circle Leader

Circle Member

Focus: What’s the problem to solve?

What: Ask questions about the facts

Why: Breakdown the causes

How: Strategies to reduce the pain

Codify: Lessons Learned into #Patterns.

Observation Questions

The Explicit Mastery Loop

The Explicit Mastery Loop

Input: Constraints

Target: Optimize the Rate of Idea Flow

short-term loop long-term loop

1. What 2. Why 3. How

F ocus!

Output: Pain Signal

Improve Quality of Decisions

Input: Constraints

Target: Optimize the Rate of Idea Flow

short-term loop long-term loop

1. What 2. Why 3. How

F ocus!

Output: Pain Signal

Target - The direction of “better”

Target: Optimize the Rate of Idea Flow

The Explicit Mastery Loop

Input: Constraints

Target: Optimize the Rate of Idea Flow

short-term loop long-term loop

1. What 2. Why 3. How

F ocus!

Output: Pain Signal

Input - The constraints that limit our short-term choices…

The Explicit Mastery Loop

Input: Constraints

Target: Optimize the Rate of Idea Flow

short-term loop long-term loop

1. What 2. Why 3. How

F ocus!

Output: Pain Signal

Output - The pain signal we’re trying to improve

The Explicit Mastery Loop

Input: Constraints

Target: Optimize the Rate of Idea Flow

short-term loop long-term loop

2. What 3. Why 4. How

F ocus!

Output: Pain Signal

1. !

1. Focus on the biggest pain

F ocus!

The Explicit Mastery Loop

Input: Constraints

Target: Optimize the Rate of Idea Flow

short-term loop long-term loop

2. What 3. Why 4. How

F ocus!

Output: Pain Signal

1. !

2. What - Identify the specific symptoms

2. What

The Explicit Mastery Loop

Input: Constraints

Target: Optimize the Rate of Idea Flow

short-term loop long-term loop

2. What 3. Why 4. How

F ocus!

Output: Pain Signal

1. !

3. Why - Breakdown cause and effect

3. Why

The Explicit Mastery Loop

Input: Constraints

Target: Optimize the Rate of Idea Flow

short-term loop long-term loop

2. What 3. Why 4. How

F ocus!

Output: Pain Signal

1. !

4. How

4. How - Run Experiments to learn what works

The Explicit Mastery Loop

Input: Constraints

Target: Optimize the Rate of Idea Flow

short-term loop long-term loop

2. What 3. Why 4. How

F ocus!

Output: Pain Signal

1. !

5. Codify Lessons Learned and Modify Behavior

The Explicit Mastery Loop

#OpenDXWhat?

Companies

Community Groups

HQ in Austin

Open Mastery Austin

WE ARE a “Pay it Forward” Industry Peer Mentorship Network

LEARN YOUR WAY TO AWESOME.

Five Years Ago I Read This Book:

How to Build a Learning Organization

—>

2.0

Visibility Changes Everything

How to Measure the PAIN in Software Development

Janelle Klein

Visibility

+

Lean Manufacturing

—>+

Everyday Science (Lean Startup)

We get a design template for how to build a learning organization.

Visibility Changes Everything

2.0

Focus: Quality of Execution Decisions

Focus Traction

$$$ $$

Focus: Quality of Investment Decisions

Risk Translator (interface)

Management(Coordination)

Engineering(Execution)

Open Mastery Learning Framework

Idea Flow Pain Sensor

is a Learning Organization

Visibility Starts a Chain Reaction

#OpenDX

Next Steps:

openmastery.slack.com