why #opendx?

Gradle Summit 2016

How to Measure Your #GradleDX

Janelle Klein

, Developer, Consultant, CTO @Specialized in Statistical Process Control (SPC) and Supply Chain Optimization from Lean Manufacturing (data geek)

Continuous Delivery infrastructure, automation strategy & technical mentorship

Janelle Klein

Who Am I?

How to Measure the PAIN in Software Development

Janelle Klein

Author of “Idea Flow”

leanpub.com/ideaflow

Yay!

Tour SpeakerFounder of

newiron.com

http://newiron.com

What is This Talk About?

#OpenDXAn Open Standard for Measuring PAIN

Universal definition of “Better”

(Specification for Data Collection)

Developer Experience

#GradleDX is a Pilot Project

#OpenDX

within

#GradleDX

Feedback LoopCommunity Pain Sensor

Tools

Pilot: Supplier Partnership

#GradleDX Community Project

Plugin

Plugin

Plugin

Plugin

Why does partnership matter in a software supply-chain?

My Codeis AWESOME!

100% Code Coverage

Low Cyclomatic Complexity

Usability Problems

This is PAINFUL!

Depend On API

WTF?!

WTF?!

WTF?!

WTF?!

WTF?! WTF?!

My Codeis AWESOME!

100% Code Coverage


This is PAINFUL!

Depend On API

WTF?!

WTF?!

WTF?!

WTF?!

WTF?! WTF?!

We can both be looking at the exact same code, and have a totally different experience.

100% Code Coverage


Usability Problems

Depend On API

WTF?!

WTF?!

WTF?!

WTF?!

WTF?! WTF?!

Here’s what caused me PAIN

Painful Experience

Oh…I can fix that!

100% Code Coverage


Depends On API

WTF?!

WTF?!

WTF?!

WTF?!

WTF?! WTF?!


100% Code Coverage


Depend On API


No More Pain

Thisis AWESOME!

100% Code Coverage


Depend On API

Yay! It’s AWESOME!

No More Pain

This is AWESOME!

Visibility Facilitates the Conversation

Application

3rd Party Dependencies

Local Components

Usability Problems

90% of our Software is HERE

Application


Local Components

Usability Problems


This is PAINFUL!

Bad Software has become the new Environment Pollution

Cost &

Risk

Human Limitations

We are HERE:

Difficulty of Work

What’s Causing our Pain? Cumulative Usability Problems in the Software Supply Chain

Alert!

Alert!

Our Application

PRESSURE for better quality parts PAIN

What if we started measuring PAIN across the software supply chain?

This is PAINFUL!

Everything is AWESOME!

Our Application

PRESSURE for better quality parts

What if we started measuring PAIN across the software supply chain?

Feedback LoopCommunity Pain Sensor

Tools

#GradleDX: Pilot for Supply-Chain Feedback Loop

Plugin

Plugin

Plugin

Plugin

Pilot: Supplier Partnership

Why Should YOU Care?

source: Domics - “What’s a Shrug?”

Every few years we rewrite our software…

Start%Over%

Unmaintainable%So0ware%

Across the Industry

We Start with the Best of Intentions

High Quality Code

Low Technical Debt

Automated Deployment

Good Code Coverage

Then This Happens!

PAIN

The Classic Story of Project Failure

Problems get deferred

Builds start breaking

Releases get chaotic

Productivity slows to a crawl

Developers begging for time

It’s never enough

Project Meltdown

Our Solution…

Until we can’t take it anymore.

On to greener pastures…

Has anyone struggled with the challenge of business pressure?

Has anyone ever tried to fix this problem within their organization?

Has anyone ever succeeded?

This is the biggest cause of FAILURE in our industry:

Because PAIN Doesn’t Show Up

on a Balance Sheet

If we don’t MEASURE it,

Why?

they don’t SEE it.

“Better”“Better”

What if you could get managers and developers all pulling the same direction in your company?

ManagersDevelopers

“Better”“Better”

What if we could solve this problem in ALL of our companies?

ManagersDevelopers

Two Options:

Option 1

Stay the Course

Here’s the Catch:

In order for this to work, we have to start working together as a community.

Option 2

Build Happiness

or

Let’s Do This

LEARN YOUR WAY TO AWESOME.

Free to Join Industry Peer Mentorship Network

1. Make the PAIN visible

2. Breakdown the causes into patterns

3. Help each other to learn & improve

We are working together to INVENT solutions for:

Community Analytics Platform

Process Control, Supply-Chain Optimization &

Throughput Accounting for

Software Organizations

#OpenDXAn Open Standard for Measuring PAIN

Based on:

The Goal Eliyahu M. Goldratt

“Throughput Accounting”

System Productivity =Rate of Revenue

Rate of Investment

Theory of Constraints Optimizing Flow in a Supply Chain System

“Throughput Accounting”

System Productivity =Rate of Revenue

Rate of Investment

Throughput Accounting measures how efficiently we turn effort into money.

Our PAIN Matters

How Do We Measure PAIN?

Great Team Disciplined with Best Practices Constantly Working on Improvements+

Project FAILURE

About 8 Years Ago…

ApplicationTests

Our Test Suite

ApplicationTests

Our Test Suite

“What are we supposed to do?!”

The Retrospective


Our biggest problem

“Well, if we don’t understand a problem, we should

collect data.”

The Retrospective

Our biggest problem

“What data would help us understand the problem?”


Technical Debt Mistakes

I thought the main obstacle was Technical Debt

?Most of our mistakes were in the

most well-written parts of the code.

Mistakes

We made significantly more mistakes in code that we didn’t write ourselves.

Lower Familiarity

More Mistakes=

There had to be more to the story...

Complex(So*ware(

PAIN

This is what I knew...

What made development feel painful?

Unexpected Behavior

Problem Resolved

Tracking Painful Interaction with the Code (Friction)

Troubleshooting

Progress

5 hours and 18 minutes of troubleshooting...

PAINFUL

The amount of PAIN was caused by…

Likeliness(of((Unexpected(Behavior(

Cost(to(Troubleshoot(and(Repair(

High(Frequency(Low(Impact(

Low(Frequency(Low(Impact(

Low(Frequency(High(Impact(

PAIN(

What Causes Unexpected Behavior (likeliness)?

What Makes Troubleshooting Time-Consuming (impact)?

Semantic Mistakes

Stale Memory Mistakes

Association Mistakes

Bad Input Assumption

Tedious Change Mistakes

Copy-Edit Mistakes

Transposition Mistakes

Failed Refactor Mistakes

False Alarm

Non-Deterministic Behavior

Ambiguous Clues

Lots of Code Changes

Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

Most of the pain was caused by human factors.

What causes PAIN?




Ambiguous Clues


Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

What causes PAIN?


Semantic Mistakes





Copy-Edit Mistakes



False Alarm




Ambiguous Clues


Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

What causes PAIN?

Semantic Mistakes





Copy-Edit Mistakes



False Alarm





Ambiguous Clues


Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

What causes PAIN?

PAIN is a consequence of how we interact with the code.

Semantic Mistakes





Copy-Edit Mistakes



False Alarm

PAIN occurs during the process of understanding and extending the software

Complex(So*ware(

PAIN

Not the Code.

Optimize “Idea Flow”

We deleted everything…

The automation wasn’t solving the problem!

What are the specific risks in this release?

How can we mitigate the risks?

Handful of manual tests

Custom Tools

+

Time-Bound Releases

RISK-Bound Releases

Release Size

(weeks)

Time

Defect Rates

RISK-Bound Releases

Continuous Delivery is about learning how to effectively manage risk

My team spent tons of time working on improvements that didn’t make much difference.

We had tons of automation, but the automation didn’t catch our bugs.

My team spent tons of time working on improvements that didn’t make much difference.

We had well-modularized code,

but it was still extremely time-consuming to troubleshoot defects.

The hard part isn’t solving the problems it’s identifying the right problems to solve.

“What are the specific problems that are causing the team’s pain?”

Complex(So*ware(

PAIN

Measure Friction in “Idea Flow”

OpenDX measures the “friction” that occurs during a developer’s experience

The Rhythm of “Idea Flow”

Write a little code.

Work out the kinks.


Work out the kinks.


Work out the kinks.

Conflict Confirm

Rework'

Learn'

Validate(

Modify'

Progress Loop Conflict Loop

Troubleshoot'

Write a little code. Work out the kinks.


On Intelligence Jeff Hawkins

Confirmed Expectations

Our brain works like a giant prediction engine…

Violated Expectations

Our brain works like a giant prediction engine…

Conflict Confirm

Rework'

Learn'

Validate(

Modify'


Troubleshoot'


“Friction” in Idea Flow

Conflict Confirm

Rework'

Learn'

Validate(

Modify'


Troubleshoot'

“Idea Flow Map”

“Friction”

Idea Flow Mapping Tools

(Open Source, Supported GA ~July 2016)github.com/ideaflow/tools

http://ideaflow.org/tools

Typical Idea Flow Maps

Single Problem

Multi-Problem

We can read Visual Indicators in Idea Flow Maps

Le#$Atrium$

Le#$Ventricle$

Right$Ventricle$

Right$Atrium$

What’s$causing$this$pa7ern?$

Similar to how an EKG helps doctors diagnose heart problems...

...Idea Flow Maps help developers diagnose software problems.

Problem-Solving Machine

We can read Visual Indicators in Idea Flow Maps

= Solution Strategy

The “Heart” of Software Development(the problem-solving machine)

Based on: Grandon Gill and Richard Hicks models of Task Complexity and John Boyd’s OODA Loop

= Solution Strategy


Choose a general strategy

= Solution Strategy


Understand the system

= Solution Strategy


Code & work out the kinks

= Solution Strategy


Back to the drawing board

The symptoms we experience depend on where the disruptions are in the problem-solving process

Trial and Error

The Friction is Here

Troubleshooting

Progress

Learning

Rework

Assumption Risk (Rework)

Likelihood)of))making)a))

Bad)Assump4on)

Cost)to)Correct)Decisions)

High)Uncertainty)Low)Delay)

Low)Uncertainty)Low)Delay)

Low)Uncertainty)High)Delay)

PAIN)

Struggling with Understanding the Code


Troubleshooting

Progress

Learning

Rework

Familiarity Risk (Learning)

Likelihood)of))working)with)Unfamiliar)

Code)

Cost)to)Learn)

High)Frequency)Easy)to)Learn)

Low)Frequency)Easy)to)Learn)

Low)Frequency)Hard)to)Learn)

PAIN)

Struggling with Feedback


Troubleshooting

Progress

Learning

Rework

Quality Risk (Troubleshooting)

Likelihood)of))Unexpected)Behavior)

Cost)to)Troubleshoot)and)Repair)

High)Frequency)Low)Impact)

Low)Frequency)Low)Impact)

Low)Frequency)High)Impact)

PAIN)

”Friction” is the time spent on:

x

Troubleshooting

x

Learning

x

Rework

Quality Risk Familiarity Risk Assumption Risk

“What caused the pain in this case?”

Categorize the Problems with #HashTags

#ReportingEngine

#Hibernate

#MergeHell

1. Problem A

2. Problem B

3. Problem C

1000 hours/month

Add up the Pain by Category

What’s the biggest problem to solve?

Case studies.

Troubleshooting

Progress

Learning

7:070:00

0:00 19:52

12 year old project after all original developers left.

Case Study: Huge Mess with Great Team

70-90% of dev capacity on “friction”

The Team’s Improvement Focus: Increasing unit test coverage by 5%

Case Study: Huge Mess with Great Team

1. Test Data Generation

2. Merging Problems

3. Repairing Tests

1000 hours/month

The Biggest Problem: ~700 hours/month generating test data

18 months after a Micro-Services/Continuous Delivery rewrite.

Troubleshooting

Progress

Learning40-60% of dev capacity on “friction”

0:00 28:15

12:230:00

Case Study: From Monolith to Microservices

The Architecture Looked Good on Paper

Team A Team B Team C

Complexity Moved Here

We Don’t Have TIME To Fix It!

The Cost of Escalating Risk

0%

100%

Release 1 Release 2 Release 3

Troubleshooting

Progress

Learning

Percentage Capacity spent on Troubleshooting (red) and Learning (blue)

(extrapolated from samples)

0%

100%



Figure out what to do Learning is front-loaded

Troubleshooting

Progress

Learning


0%

100%



Rush Before the Deadline Validation is Deferred

Troubleshooting

Progress

Learning


0%

100%



Pain Builds Baseline friction keeps rising

Troubleshooting

Progress

Learning


0%

100%



Chaos Reigns Unpredictable work stops

fitting in the timebox

Troubleshooting

Progress

Learning


Over 1000 Hours/Month in Downtime!

They tried to solve their design problems with more test automation.

25 developers down for 2 days

(It didn’t work)

Lots of Friction in BRAND NEW code:

27:15

“WHY?”

Application


Local Components

Usability Problems


This is PAINFUL!

“The Idea Flow Factory” (supply chain model)

Optimize the Rate of Idea Flow

Across ANY Software Supply Chain

Build Test Deploy

Typical manufacturing metaphor mapping:

Customer

The Problem: Focus on short-term effects

Software Supply Chain as the Factory Floor

Customer

Software Supply Chain as the Factory Floor

Customer

Dev = Operator “flows” the components

Architecture Changes Create Bottlenecks

Customer

Lack of Familiarity Creates Bottlenecks

WTF?!

Customer: Big Kahuna

Customers: Seg 2 Seg 3

Customers: Seg 2 Seg 4 Seg 5

Connect Maintenance Effort to Revenue through Software Dependencies

“Big Kahuna”

Effort

Revenue

“Throughput Accounting” Metrics

Cost of Support vs Revenue per Customer Segment (%)

Effort

Revenue


Cost of Support vs Revenue per Customer Segment (%)

Breakdown of Development Capacity

If you were an investor: wouldn’t YOU want to reduce friction?


With #OpenDX, we can apply the entire suite of Risk Management Tools from Lean Manufacturing

Statistical Process Control (SPC)

Optimal Friction

Upper Control Limit X"

“Out of Control”

20min

0m

Average Pain per Incident

Target

Control Limit

Supply Chain Optimization

Throughput Accounting

We have the power to solve this problem:

If we choose to do it.

Application


Local Components

Usability Problems

This is PAINFUL!

We have the power to solve this problem:

Two Options:

Option 1

Stay the Course

Option 2

Build Happiness

or

Let’s Do This

Miko Matsumura: “Build Happiness” is a philosophy that happiness must be built.

Miko: “That’s not a happy elephant!”

Focus on the PAIN & Build Happiness

Let’s Talk About It.

openmastery.slack.com

#OpenDX #GradleDX

http://openmastery.slack.com

Janelle Kleinopenmastery.org @janellekz

Check out openmastery.org for details.

Read my Book.

Think About It.

FREE with MembershipBuy It

How to Measure the PAIN in Software Development

Janelle Klein

http://@openmastery

http://openmastery.org

Learn more at www.gradle.org