agile metrics - pbworksagileconsortium.pbworks.com/f/metricsnotjustformanagers2.pdfgoal of scrum is...

89
© Joe Little 2009 AGILE METRICS For the Team, for the Managers For the customers and shareholders 1

Upload: phungkien

Post on 13-May-2018

214 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

AGILE METRICSFor the Team, for the Managers

For the customers and shareholders

1

Page 2: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

WIIFM

What does a team cost, per year?

What does a team bring back, per year? Net present value (return on investment).

If we can double team productivity, would that be interesting?

2

Page 3: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Design goals of Scrum

Increase team productivity 5x-10x

3

Page 4: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009Henrik Kniberg

XP

LeanAgile

Scrum

Trifork CrispXebia

Queue theory

Game theory

History

Research

Philosophy

Topic: Introduction

Chaos theory Principles

Practices

Implementation

More theory...

4

Page 5: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

With help from...

With help from Accenture, Alliance Global Services, American Greetings Interactive, AOL, Applied Physics Laboratory, Argonaut Group, Asurion,

Avid Technology, Booz Allen Hamilton, CA, CAE, Canada Post, Capital One, Charles Schwab, Citigroup, CNN/Turner, Comcast, Compuware, Cornell,

Crisp, Dell, DST, EDR, Exigen Services, FedEx, GE Power Systems, Georgia Institute of Technology, Gilbarco, Google, HP, Huawei, IBM, iContact, INM, Intersect, J Ray McDermott, Mantech, McKesson, McKinsey & Co, Medco, Microsoft, Morrison Management, Motley Fool, MySpace, Nationwide, NC

State University, NEA, Nortel, Northrop Grumman, NYSE Euronext, Ontario Legislative Assembly, Pearson, Philips, Polycom, Rally, RealTravel, Red Hat, REMITData, S1, SAIC, Scripps Network Interactive, Scrum Training Institute, Siemens, SirsiDynix, Smart Online, SolutionsIQ, Sonic Boom

Media, SteamTheWorld, Sungard, Systematic Software Engineering, The Hartford, The Library Company, The New Teacher Project, Tradeware,

Travelocity, Trifork, Ultimate Software, Vanguard, Version One, Vignette, Wake Forest University, Wells Fargo/Wachovia, Wireless Generation, Xebia,

and others.

5

Page 6: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Attributions

Jim YorkJeff SutherlandHenrik KnibergMike CohnKen SchwaberKert PetersonMany others

6

Page 7: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Joe Little

• Agile Coach & Trainer• 20+ years in senior level consulting to well-known firms in New York, London and

Charlotte• Focus on delivery of Business Value; interest in Lean • CST, CSP, CSM; MBA• Was Senior Manager in Big 6 consulting• Head of Kitty Hawk Consulting, Inc. since 1991• Head of LeanAgileTraining.com• Started trying to do [Agile] before reading The Mythical Man-Month

– http://agileconsortium.blogspot.com

[email protected]

7

Page 8: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Start-up

IntroductionsStand in lineTeam normsBreaks & Lunch“Biggest question Now” card

8

Page 9: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

6 Blindmen and an Elephant

9

Page 10: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009Henrik Kniberg

XP

LeanAgile

Scrum

Trifork CrispXebia

Queue theory

Game theory

History

Research

Philosophy

Topic: Nokia Test

Chaos theory Principles

Practices

Implementation

More theory...

10

Page 11: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

ScrumButtKudos to Eric Gunnerson for coining the term ScrumBut in 2006.

11

Page 12: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Avoiding ScrumButt - Nokia Test OriginsNokia Siemens Networks

In 2005, Bas Vodde started training and coaching teams at Nokia Networks in Finland. The first Nokia test focused on Agile practices

jeffsutherland.com/scrum/basvodde2006_nokia_agile.pdf

By 2007, Siemens joined Nokia Networks to form Nokia Siemens Networks with over 60,000 employees and 15 billion Euro in revenue. Bas Vodde moved to China to train Nokia Siemens Networks staff on Scrum and updated the Nokia Test to include Scrum practices.

In 2007, Jeff Sutherland tuned the Nokia Test for Scrum Certification and in 2008 developed a scoring system

agileconsortium.blogspot.com/2007/12/nokia-test.html

jeffsutherland.com/scrum/Agile2008MoneyforNothing.pdf

Each person on the team takes a sheet of paper and prepares to score eight questions on a scale of 1-10.

12

Page 13: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

One way to measure ScrumButt

Excellent Scrum - annual revenue up 400%

PatientKeeperOthers companies I work with in Scandinavia

Good Scrum - revenue up 300%

Companies in Scandinavia I can’t talk aboutPretty Good Scrum - revenue up 150% - 200%

Systematic Software Engineering - 200%Google - 160%

ScrumButt - revenue up 0-35%

Yahoo, most companies

13

Page 14: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

The Nokia Test1. Are you iterative?

Sprints are 4 weeks or less

Features are tested and working by the end of the Sprint

Sprints start with an Agile Specification2. Are you doing Scrum?

You know who the Product Owner is

There is a Product Backlog prioritized by Business Value

The Product Backlog has estimates created by the Team

The Team generates burndown charts and knows their velocity

There are no project managers (or anyone else) disrupting the Team

14

Page 15: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Question 1 - Iterations

No iterations - 0Interations > 6 weeks - 1Variable length < 6 weeks - 2Fixed iteration length 6 weeks - 3Fixed iteration length 5 weeks - 4Fixed iteration 4 weeks or less - 10

15

Page 16: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Question 2 - Testing

No dedicated QA - 0Unit tested - 1Feature tested - 5Features tested as soon as completed - 7Software passes acceptance testing - 8Software is deployed - 10

16

Page 17: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Question 3 - Agile Specification

No requirements - 0Big requirements documents - 1Poor user stories - 4Good requirements - 5Good user stories - 7Just enough, just in time specifications - 8Good user stories tied to specifications as needed - 10

17

Page 18: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Question 4 - Product Owner

No Product Owner - 0Product Owner who doesn’t understand Scrum - 1

Product Owner who disrupts team - 2Product Owner not involved with team - 2

Product owner with clear product backlog estimated by team before Sprint Planning meeting (READY) - 5Product owner with release roadmap with dates based on team velocity - 8Product owner who motivates team - 10

18

Page 19: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Question 5 - Product Backlog

No Product Backlog - 0Multiple Product Backlogs - 1

Single Product Backlog - 3Product Backlog clearly specified and prioritized by ROI before Sprint Planning (READY) - 5

Product Owner has release plan based on Product Backlog - 7Product Owner can measure ROI based on real revenue, cost per story point, or other metrics - 10

19

Page 20: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Question 6 - Estimates

Product Backlog not estimated - 0Estimates not produced by team - 1Estimates not produced by planning poker - 5Estimates produced by planning poker by team - 8Estimate error < 10% - 10

20

Page 21: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Question 7 - Burndown Chart

No burndown chart - 0Burndown chart not updated by team - 1Burndown chart in hours/days not accounting for work in progress (partial tasks burn down) - 2Burndown chart only burns down when task in done (TrackDone pattern) - 4Burndown only burns down when story is done - 5Add 3 points if team knows velocityAdd two point if Product Owner release plan based on known velocity

21

Page 22: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Track Done - Scrum pattern by Jim Coplien

… you have a burn-down chart that you are using to track remaining work. The burn-down chart is a visible picture of the project state, and serves as a team motivator and sanity check.

It is easy to interpret the burn-down chart as a good portrayal of estimated remaining time, and to use that portrayal to develop confidence in meeting the Sprint’s actual business goals of done functionality.

Usually, team members update the burn-down chart daily to reflect adjustments to the amount of remaining work. Such updates reflect a desire to have as good knowledge as is possible about the effort remaining. These estimates are made in mid-stream and reflect increases that arise from emergent requirements. However, given that one emergent requirement has been discovered in a task doesn’t imply that no others remain. While the confidence in an estimate usually improves with each revision and with continued work on the task, unusually wicked problems seem never to converge.

On the other hand the Product Owner is not centrally interested in partially completed work, only in items that are done and potentially shippable. Since the goal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk, the focus should be on done. Emergent requirements increase risk, and the Product Owner is certainly interested if estimates expand. Because there may always be emergent requirements, any estimate of remaining time based on work mid-stream in a task has a higher degree of uncertainty than the relatively risk-free estimate of zero remaining time for done items.

In theory, it is possible for the remaining time on a burn-down chart to be quite near zero, yet to have few (or perhaps zero!) tasks in the done state.

Therefore:

Update the Product Backlog in only two cases: reducing the amount of remaining known work if the task is done; and increasing the amount of known work if the task grows in size due to emergent requirements or other insights gained during the Sprint. Do not reduce the amount of remaining work that arises from progress on partially completed tasks.

* * *

The team and Product Owner have a better picture of the state of the Sprint with respect to the Sprint’s business goals of delivering done functionality. The team can revise estimations in the middle of a Sprint with more confidence because they are not dependent on the unknown remaining time for partially completed tasks. Yet, the risks incurred by the “surprises” of emergent requirements are embraced and made visible.

It is impossible, using this approach, to come near the end of a Sprint with a burn-down chart that projects success even if the Sprint only ends with 90% of the tasks 90% done.

* * *

There is a chance that a completed task can become “un-completed” by emerging requirements in some other task during the sprint. For such cases, see the pattern Domino Effect.

This pattern was suggested by Jeff Sutherland, co-founder of Scrum, and he reports that it is widely used by his clients.

James O. CoplienWednesday, September 24, 2008 22

Page 23: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Question 8 - Team Disruption

Manager or Project Leader disrupts team - 0Product Owner disrupts team - 1Managers, Project Leaders or Team leaders assigning tasks - 3Have Project Leader and Scrum roles - 5No one disrupting team, only Scrum roles - 10

0

20

40

60

80

100

120

0 20 40 60 80

Number of Roles

% S

atur

atio

n

Organizational Patterns of Agile Software Development by Coplien and Harrison (2004)

23

Page 24: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

XP

CompanyC

LeanAgile

Scrum

CompanyB Company

C

CompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: Design goals of good metrics

Chaos theory Principles

Practices

Implementation

More theory...

24

Page 25: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Good metrics should...

be accurate enough to enable better decision-makingenable better actions and serious improvementnot be seriously gamed (inaccurate); ideally “gaming” is better behaviorchange behavior of all members of team and related managersmotivate the team (or at least not de-motivate)be simple enough that they are done, and used wellenable optimizing the whole

25

Page 26: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

XP

CompanyC

LeanAgile

Scrum

CompanyB Company

C

CompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: Current problems

Chaos theory Principles

Practices

Implementation

More theory...

26

Page 27: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

I find metrics are...

de-motivating to the teamintrusive in search of a false precisionsubverted and/or gamed by the teamcauses of sub-optimizationleading people to view reality as something to avoid (“just follow the plan”)creating uselessly conflicting goals

27

Page 28: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

...and are...

not used and acted upon effectively“help” managers but have little impact on behavior of the teamoften complex and overlappingleading to only minor improvementleading to hiding from the truth

28

Page 29: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Examples of waterfall metrics

On timeOn budgetOn scopeNumber of bugs (proxy for quality)

29

Page 30: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

XP

CompanyC

LeanAgile

Scrum

CompanyB Company

C

CompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: Why do we have metrics?

Chaos theory Principles

Practices

Implementation

More theory...

30

Page 31: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Reason #1

We have toSelf-defense

31

Page 32: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Reason #2

To make business-decisions

Decision-making frequency increases multi-fold

Such as:

Should I start this effortWhich team needs the most help nowWhen do I stop doing this product backlogDo I understand the customer better

Did it actually help to remove that “impediment”

32

Page 33: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Reason #3

To get feedback, so that forward-looking guesses have a higher probability of being right

We make a guess (aka estimate), and then we check later how good the guess was

If it is off a lot...maybe: ”gee, we need to learn how to estimate better”

33

Page 34: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Reason #4

To change behavior...

Not just the key business-decisionsBut all the behavior on a day-to-day basis

34

Page 35: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

XP

CompanyC

LeanAgile

Scrum

CompanyB Company

C

CompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: Some ideas about metrics

Chaos theory Principles

Practices

Implementation

More theory...

35

Page 36: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Lean principles

Eliminate wasteBuild quality inCreate knowledgeDefer commitmentDeliver fastRespect peopleOptimize the whole

36Source: Mary & Tom Poppendieck

Page 37: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Toyota Way: Learn by DoingFujio Cho, Board Chairman

• We place the highest value on actual implementation and taking action. Agile Principle #1

• There are many things one doesn’t understand, and therefore we ask them why don’t you just go ahead and take action; try to do something? Agile Principle #3, #11

• You realize how little you know and you face your own failures and redo it again and at the second trial you realize another mistake … so you can redo it once again. Agile Principle #11, #12

• So by constant improvement … one can rise to the higher level of practice and knowledge. Agile Principle #3

"Anyone who has never made a mistake has never tried anything new." Albert Einstein

Page 38: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009Source: Henrik Kniberg

XP

LeanAgile

Scrum

CompanyB Company

CCompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: Recommendation

Chaos theory Principles

Practices

Implementation

More theory...

38

Page 39: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Focus on two metrics

At the team level, or some aggregation...

BV produced per month or quarter

Velocity improvement per team (acceleration)

39

Page 40: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Goals behind the scenes

Lean: Process cycle efficiency (eg, value-added time over total time for a feature)

Just-in-time knowledge creation

Minimizing knowledge decay

40

Page 41: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Are there other agile metrics?

Yes, of course....... and we will survey them later

...and mostly they are not for optimizing the whole

41

Page 42: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009Source: Henrik Kniberg

XP

LeanAgile

Scrum

CompanyB Company

CCompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: Team wants metrics

Chaos theory Principles

Practices

Implementation

More theory...

42

Page 43: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Metrics are not forced

The managers cannot force the metrics down on the TeamsThe Team must have fun.

43

Page 44: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

The Team wants metrics. Why?

To help them see their work

To plan with

To determine success

To push back on magical thinking managers

To challenge themselves

44

Page 45: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009Source: Henrik Kniberg

XP

LeanAgile

Scrum

CompanyB Company

CCompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: BV Engineering

Chaos theory Principles

Practices

Implementation

More theory...

45

Page 46: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

A Start

“You’ve got to be very careful if you don’t know where you’re going, because you might not get there.” Yogi Berra

46

Page 47: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

What are the costs and benefits of your teams?

Cost per year, all-in. Assume 8 people, FT, including SM and PO.

Net Present Value produced annually (the return on that investment in the team)

How many of you know these numbers, or a serious semblances of them?

47

Page 48: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

Is BV Engineering Important?

We make the stories 20% betterWe use “Pareto’s” 85-33 rule to get the most important stuff done in less timeWe identify more high value epicsMaybe: We motivate the team, so that they are more productiveMaybe: We hit the mark of what the customer really wants more.

What’s that worth?

48

Page 49: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

Let’s do a thought experiment...

Assume team costs $1,000,000 per yearAssume normal multiple is 3x (ie, team delivers $3,000,000 in BV)Assume the “real work” itself does NOT get any faster

49

Page 50: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

One version....

50

Year 1 Year 2 Year 3

Cost of Team $1,000,000 $1,000,000 $1,000,000

Orig Value Delivered per Year $3,000,000 $3,000,000 $3,000,000

NPV $7,460,556

ID Better Stories (+20%) $3,600,000

Deliver Top 33% (85% of BV) $3,060,000

Deliver Top 33% again $3,060,000

Deliver Top 33% again $3,060,000

TOTAL FIRST YEAR $9,180,000 $9,180,000 $9,180,000

Better NPV $22,829,301

Better/Original 3.1

Page 51: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

And what if I said...

No more lyingMore satisfaction working with a teamMore fun!More satisfied customers

51

Page 52: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

Content © Joseph Little 2009

What the Product Owner doesBV Engineering

CustomersExternal

&

Internal

The BusinessCustomer facing

people

Internal groups (Firm oriented)

The Team

Page 53: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

Content © Joseph Little 2009

Is it better this way?

CustomersExternal

&

Internal

The BusinessCustomer facing

people

Internal groups (Firm oriented)

The Team

Page 54: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

Some axioms

1. A “technical success” is no success at all2. The most important thing is satisfying the

customer; making money is only a constraint3. You win by learning faster than the next firm4. You win with small “scientific” experiments;

frequent and fast5. The numbers never get precise, but that does

not mean ‘use no numbers’6. Numbers can be useful, but that does not mean

‘human judgment is no longer needed’7. There is no one best approach to BV

engineering54

Page 55: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

Theories & Practices

BV Engineering is based upon theories of what BV is (for your firm), how it changes, and how we should deliver it

BV Engineering is instantiated in a wide set of practices that possibly operate in a kind of virtuous cycle...each practice building on the other

55

Page 56: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

What is BV?Varies by situationIn whose opinion? (Oh, humans assign value?)Things to consider:

ROI/NPV

Risk

Customer satisfaction

Cost reduction

Increased revenues

Customers want many things (see Lean, Six Sigma, others)

Etc, etc, etc Recommendation: Discuss “what is BV for us” with the Team

56

Page 57: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Make BV of each story clear

Why?To whom?

How much time is discussing BV worth?

Depending how we make BV clear, can there be side benefits that are very valuable?

57

Page 58: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Ways to identify relative BV

Bubble sort

Compare new story to othersDollars (euros, etc)

On each storyBV Points (via Priority Poker)

Gather “experts” and size versus some standard using Fibonacci scale

Distribute set number of “points”

How many dots from my pool of 1,000 does this story get?

Life-boat strategy

58

Page 59: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Minimum on BV

In Scrum, PO must have a rank-ordered list of PB Items...ranked by BV (as she defines it)No two items are “equal” on this list.

59

Page 60: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Release burndown

100

200

300

400

Work remaining(story points)

Sprint1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Source: Henrik Kniberg60

Page 61: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

-40

-20

0

20

40

60

80

100

120

1 2 3 4 5

Work Added

Work Remaining

© Joe Little 2009

Release burndown with “unexpected” scope additions

Release date

61

Burndown idealized

Page 62: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

3 roles• Product owner• Scrum master• Team

3 artifacts• Product backlog• Sprint backlog• Sprint burndown

4 activities• Sprint planning• Daily scrum• Sprint review• Retrospective

© Joe Little 2009

V V V

V

2010

Q42009

Q32009

June2009

May2009

2010

2011

2012

2013

Apr2009

Backlog maintenance

62

Page 63: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

XP

CompanyC

LeanAgile

Scrum

CompanyB Company

C

CompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: BV Engineering (cont)

Chaos theory Principles

Practices

Implementation

More theory...

Source: Henrik Kniberg

63

Page 65: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

Hallmarks of real BV Engineering!!

1. The process is visible and articulated & improved

2. Failures in BV communication are identified and corrected frequently, quickly

3. There is a theory, and a concerted attempt to prove out the theory

4. There is appropriate dynamism and change5. Business & Technology are partners6. Success is forecast and also measured after

the fact7. Human judgment is involved (it’s not just the

numbers)65

Page 66: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

The BV process is visible and articulated

Do you understand your’s, end-to-end?

66

CustomersExternal

&

Internal

The BusinessCustomer facing

people

Internal groups (Firm oriented)

The Team

Page 67: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

A theory, that is being proved out

Is the theory stated as such, or is it assumed to be right?How it is being proved out?What happens when (not if) it is (somewhat) wrong?

67

Page 68: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

Success is measured

1 to 3 key “end” metrics. Identified. Forecast.Then the real results are obtained.

Perhaps not perfectly, but reasonablyAnd learned from. (Was the product wrong? Was the theory wrong?)And communicated back to the Team

68

Page 69: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

Human judgment

Yes, stuff happens that makes one question whether the ‘scientific” experiment was fairYes, one can still have a hunch that the product will succeed later (if not now)

So, metrics do not absolve managers from tough human judgment about the actuals and other information they get back

69

Page 70: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

The unbearable lightness of metrics

We use metrics (about the past) to take forward-looking actionMetrics help us see how bad we are at predicting the futureMetrics help us learn (perhaps first, by helping us see how much we don’t know)

70

Page 71: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

XP

CompanyC

LeanAgile

Scrum

CompanyB Company

C

CompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: MetricsChaos theory Principles

Practices

Implementation

More theory...

Source: Henrik Kniberg

71

Page 72: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Scrum Information1. Velocity history2. Working Software (and related benefits)3. Stories Completed (done, done)4. Number of Passing Unit or Functional Tests (today or

with growth trend

5. Bugs open today6. Sprint Burndown chart7. Scrum Board8. Release Burndown chart9. Stories/Sprints to next Release (Release Plan)10. Product Roadmap

72

Page 73: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

More Scrum Information

1. Full Product Backlog (remaining stories)2. Impediments List (current impediments)3. % BV completed (if use BV points or similar)4. % Change in Velocity since (inception, last year)5. Number of story points completed to date; % of total.6. Bugs that escaped the Sprint7. Oldest bug open (with Sev level)8. Sprints with stories incomplete9. Sprints with added stories10. Unplanned tasks (in the X Sprint); related hours

73

Page 74: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Yet More Scrum Information

1. Stories added to / subtracted from the Release2. Age of each story to done, done; average age (not

commonly done, easy to do)

3. Impediments removed to date4. Builds that passed/failed initially, to date5. Defects identified after done, done6. Defects identified after release

74

Page 75: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Additional metrics

75

1. If start with big bug listBugs added (old features) (per time)Old Bugs resolved / closed (per time)Old Bugs remaining (over time)

2. If starting with minimal automated testsNumber of automated tests (unit, functional, etc)Number of manual tests (that could be automated)Effort on manual testing

3. Metrics around quality of builds and regression tests4. Metrics around quality of code (eg, cyclomatic complexity)5. Code coverage by automated tests (unit, functional, etc.)

Page 76: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

Some metrics I like (re BV)

1. NPV (net present value)2. ROI (return on investment)3. Faster end-to-end cycle time4. Increased sales5. Increased market share6. More eyeballs (on a webpage)7. Improved eyeball demographics8. Reduced costs

76

Page 77: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

More metrics I like (BV)

1. Reduced risk (although I prefer if this is made more concrete by being monetized...see underwriting)

2. Net promoter score3. Any specific metric showing higher customer

satisfaction4. Others??

77

Page 78: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joseph Little 2009

Lies, damn lies & statistics

It is not having numbers...It is making good use of numbers (that are reasonably accurate)

78

Page 79: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

XP

CompanyC

LeanAgile

Scrum

CompanyB Company

C

CompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: Some important issuesChaos theory Principles

Practices

Implementation

More theory...

Source: Henrik Kniberg

79

Page 80: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Some important issues...

Not addressed...

The Team must invent metrics (as they need them)What if...[impediment X exists that keeps me from doing metrics right], what do I do?Transition from old metrics to newThe business guys won’t estimate $ per large effort or BV points for storiesThe Team won’t do a decent velocity (using story points)How many metrics become too much

80

Page 81: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

XP

CompanyC

LeanAgile

Scrum

CompanyB Company

C

CompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: A real caseChaos theory Principles

Practices

Implementation

More theory...

Source: Henrik Kniberg

81

Page 82: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

This is where we were (Jan 2008?)

82

Page 83: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

And this is where we were (Mar 2009?)

83

Page 84: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

We got serious success

...and you can too.

84

Page 85: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Our business partners...

...hated Agile at the start

...want all projects Agile now

85

Page 86: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

We also have metrics...

...that prove our success

86

Page 87: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Current metrics vs Future metrics

For us...

Current MetricsAre Ok.

We got to success

Our main problems are...

Future metricsWill be hard to implement

Will look like...

We expect them to enable yet greater success

87

Page 88: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

XP

CompanyC

LeanAgile

Scrum

CompanyB Company

C

CompanyA

Queue theory

Game theory

History

Research

Philosophy

Topic: Why you must have themChaos theory Principles

Practices

Implementation

More theory...

Source: Henrik Kniberg

88

Page 89: AGILE METRICS - PBworksagileconsortium.pbworks.com/f/MetricsNotJustForManagers2.pdfgoal of Scrum is to achieve the Sprint target agreed with the Product Owner, and to reduce risk,

© Joe Little 2009

Why?

Team members can rest easierTeam members can be proudThe bastards can’t justify getting rid of Agile (or at least, you have more to fight with)The metrics can cause an increased in satisfaction for your customers -- not a bit, but a LOT.

89