slay the dragons of agile measurement

AW4Session6/8/1611:30AM

SlaytheDragonsofAgileMeasurement

Presentedby:

LarryMaccherone

AgileCraft

Broughttoyouby:

350CorporateWay,Suite400,OrangePark,FL32073888---268---8770··904---278---0524-info@techwell.com-http://www.techwell.com/

LarryMaccheroneAgileCraftAnindustry-recognizedleaderinagile,metrics,andvisualization,LarryMaccheronecurrentlyhelpsanumberofcompanieswiththedesignoftheiranalyticsproductsincludingAgileCraftandPendo.io.Previously,LarryledtheinsightsproductlineatRallySoftwarewhichenabledbetterdecisionswithdata,leveragedbigdatatechniquestoconductgroundbreakingresearch,andofferedthefirst-everagileperformancebenchmarkingcapability.BeforeRally,LarryworkedattheSoftwareEngineeringInstituteforsevenyearsconductingresearchonsoftwareengineeringmetricswithaparticularfocusonreintroducingquantitativeinsightbackintotheagilecommunities.

https://LinkedIn.com/in/LarryMaccherone

The Eight

Dragons of

Measurement

Topic presented by:

Larry Maccherone LinkedIn.com/in/LarryMaccherone

Tabaka She taught us how

to collaborate. She

made us all better.

She made us want

to be better. The

agile community

has lost a most

insightful, caring,

inspiring person.

Larry Maccherone LinkedIn.com/in/LarryMaccherone

Using measurement in

an agile environment

can be dangerous

The Dragons of Agile Measurement If you do metrics wrong, you will harm your agile transformation

1. Dragon: Measurement as a lever

Slayer: Measurement as feedback

2. Dragon: Unbalanced metrics

Slayer: 1 each for Do it

fast/right/on-time, and Keep doing it

3. Dragon: Metrics can replace

thinking

Slayer: Metrics compliment thinking

4. Dragon: Expensive metrics

Slayer: 1st work with the data you

are already passively gathering

5. Dragon: Using a convenient

metric

Slayer: Outcomes Decisions

Insight Metric (ODIM)

6. Dragon: Bad analysis

Slayer: Simple stats and

simulation

7. Dragon: Single outcome forecasts

Slayer: Forecasts w/ probability

8. Dragon: Human emotion and bias

Slayer: Tricks to avoid your own

biases and overcome those of

others

Dragon #1

Using measurement as a

lever to directly drive

behavior

Manipulating

Others

Dragon #1

Using metrics as a

lever to drive

someone else’s

behavior

Improvement

Dragon slayer #1

Using metrics to

reflect on your own

performance

Dragon #2

Unbalanced Metrics

Do it fast

Do it on

Do it right

Keep doing

Balanced scorecard of metrics

at least one from each quadrant

Dragon #3

Metrics can replace thinking

Dragon #4

Expensive metrics

Favor metrics derived from

existing records

Dragon #5

Using a convenient metric

aka “Lamp post metrics”

Good players?

Monta Ellis

9th highest scorer

(8th last season)

Carmelo Anthony (Melo)

8th highest scorer

(3rd last season)

Not everything

that counts

can be counted,

not everything

that can be

counted,

counts.

Dragon slayer #5

O U T C O M E

D E C I S I O N

I N S I G H T

M E A S U R E

EFFECT

like Vic Basili’s

Goal-Question-Metric (GQM)

but without

ISO/IEC 15939 baggage

Dragon #6

Bad analysis and

visualization

Top 10 criteria for great visualization

1. Answers the question,

"Compared with what?”

(SO What?)

2. Shows causality, or is at least

informed by it.

(NOW WHAT?)

3. Tells a story with whatever it

takes.

4. Is credible.

5. Has business value or impact in

its social context.

6. Shows

differences

easily.

7. Allows you to see the forest

AND the trees.

8. Informs along multiple

dimensions.

9. Leaves in the numbers where

possible.

10. Leaves out glitter.

Credits:

• Edward Tufte

• Stephen Few

• Gestalt (School of Psychology)

11. Uses good visual grammar

Time check?

Skip the next 8 slides if it’s

now past half the allotted time.

They go over the elements of

good visual grammar.

11. Good visual grammar (1)

When to use a straight line series?

1. Cumulative data

(later data points include

items from earlier data points)

example: Burn or Scope

series

2. Benchmark for column series

3. Connected by “ordinal inertia”

(ordinal usually = temporal)

However, spline is usually

what you want in this

instance.

When to use a dashed line?

1. “Ideal” or reference

2. Forecast

3. Regression or fit

When to use a column (vertical bar)

series?

1. Counts/sums independent of

neighbors

example: Throughput or

velocity chart

When to use a bar (horizontal)

series?

(Rarely)

1. The x-axis variable is causal?

2. In a table

3. Stylistic reasons

• Length of labels

• Two-variable display

When to use stacked area?

1. Multiple cumulative series

(the ordinal sum means

something)

example: a cumulative

flow diagram (CFD)

When to use a single area series or

multiple (non-cumulative) area

series?

NEVER?

When to use a single spline series?

1. Filling in for missing granularity

2. Rolling average

When to use a single scatter series?

1. Two variables that are

imperfectly

correlated

Dragon #7

Single outcome forecasts

Monte Carlo Forecasting What it looks like

Live demo: http://lumenize.com (use Chrome)

Seek to

change the nature of

the conversation

Every decision is a

forecast!

You are forecasting that

your choice will have better

outcomes than the other

alternatives

quality of decision depends upon:

1. alternatives considered, and

2. models used to forecast the

outcome of those alternatives.

Probabilistic models are superior

For a given alternative, let:

Pg = Probability of good thing happening

Vg = “Value” of good thing happening

Value of the alternative = Pg × Vg

lean/agile product

management example

Best case

Likely case

Worst case

$2M $2M $1M

Which strategy is best…

…for your company?

PW × VW = .25 × -$1.00M = -$0.25M

PL × VL = .50 × $1.00M = $0.50M

PB × VB = .25 × $8.00M = $2.00M

-----------

$2.25M

…for your career?

PW × VW = .25 × $1.00M = $0.25M

PL × VL = .50 × $2.00M = $1.00M

PB × VB = .25 × $2.00M = $0.50M

-----------

$1.75M

If you get only 1 project then

strategy 2 is better

75% of the time

If you get ∞ projects then

strategy 1 is better

100% of the time

How many projects do you need for

strategy 1 to be better

more often than not?

Play with it yourself at:

http://jsfiddle.net/lmaccherone/j3wh61r7/

Dragon #8

Human emotion and bias

https://LinkedIn.com/in/LarryMaccherone @LMaccherone @TheAgileCraft

We don't see things

the way they are.

We see things

the way we are.

~The Talmud

Men's Airo Outdoor Shoes

Getting

probability

input you

can trust

Trained/Calibrated

Untrained/Uncalibrated

Statistical Error

“Ideal” Confidence

50% 60% 80% 90% 100%

75 71 65 58

68 152

Assessed Chance Of Being Correct

99 # of Responses

We are inaccurate when assessing probabilities

dwhubbard@hubbardresearch.com

But, training can “calibrate” people so that of all the times they

say they are X% confident, they will be right X% of the time

Equivalent Bet calibration

What year did Newton published the Universal Laws of

Gravitation?

Pick year range that you are 90% certain it would fall within.

Win $1,000:

1. It is within your range; or

2. You spin this wheel and it lands green

Adjust your range until 1 and 2 seem equal.

Even pretending to bet money works.

Now what?

• Questions?

• Day-long seminar on agile metrics

• Workshop to design your own

metrics regimen

• AgileCraft Demo

• Make contact with me on LinkedIn

https://linkedin.com/in/larrymaccherone

Now what?

• Meet me at the AgileCraft booth to

answer questions

“They” say…

Nobody knows what’s gonna happen

next: not on a freeway, not in an

airplane, not inside our own bodies

and certainly not on a racetrack with

40 other infantile egomaniacs.

– Days of Thunder

Trying to predict the future is like

trying to drive down a country road

at night with no lights while looking

out the back window.

– Peter Drucker

Never make predictions, especially

about the future.

– Casey Stengel

When you come to a

fork in the road…

take it!

~Yogi Berra

Now what?

• Questions?

• Day-long seminar on agile metrics

• Workshop to design your own

metrics regimen

• AgileCraft Demo

https://linkedin.com/in/larrymaccherone

Now what?

• Meet me at the AgileCraft booth to

answer questions

• Share webinar recording of this

content to co-workers who couldn’t

be here https://attendee.gotowebinar.com/register

/5432450926944227330

slay the dragons of agile measurement

Software

drink slay love

sistas leadership for african americanyouth slay … ·...

we slay internet dragons. · local business owners can gain...

don't slay the dragon

slay the dragon 50 page pdf

slay 2.0 game changer 8 week challenge€¦ · 8 week...

slay 1 - lexington selected

slay your competitors on social media m

how to slay the customer support beast

connecting with chinese culture dragons and kites…kites...

tornadoes in the moldavian plateau between …...

slay with me

dragons that i made for dragons world

shepherds who slay the sheep

young australian art & writers’ awards · a turncoat and...

online reputation threats that can slay a brand

even though he slay me

holiday slay ride

10 outstanding tips to slay content marketing

ap european history: - how to “slay the beast!!!”