prioritization according to estimated business value and cost · production elements requirements...

Jo Hannay © 2011 Fluffy Tufa Involvement

Prioritization according to Estimated

Business Value and Cost in Large Agile IT Development Projects

INF5890

Jo Hannay

Avdeling for ledelsessystemerhttp://smidigeprosjekter.no/itpp


VisionRoadmap

Business Case

ReleasePlanning

Sprint planning

Daily Stand-up

Release Release ReleaseProject

Retrospective

Sprint Sprint SprintRelease

Retrospective

Daily Work Daily Work Daily WorkSprint

retrospective

TaskCompletion

TaskCompletion

TaskCompletion

Progress Update

Sliger and Broderick: The Software Project Manager's Bridge to Agility, 2008

Agile Fractal

• Side 2


Agile Fractal and Production Elements

Epic

User Story

User Story

User Story

VisionRoadmap

Business Case

ReleasePlanning

Sprint planning

Daily Stand-up


Retrospective


Retrospective


retrospective

TaskCompletion

TaskCompletion

TaskCompletion

Progress Update

Epic

User Story

User Story

User Story

Epic

User Story

User Story

User Story

TaskTaskTask

CodeCodeCode

• Side 3


Production Elements

Requirements specs at appropriate levels of detail

according to knowledge in the project at the current time.

according to pieces of functionality which can be reasoned about in terms of business (benefit to business)

according to pieces of functionality which can viably be put into production

• Side 4


Essence of Agile Project Management

VisionRoadmap

Business Case

ReleasePlanning

Sprint planning

Daily Stand-up


Retrospective


Retrospective


retrospective

TaskCompletion

TaskCompletion

TaskCompletion

Progress Update

Epic

Task

EpicEpic Epic

Epic

User Story

User Story

User Story User

Story

User Story

User Story User

Story

User Story

TaskTaskTask Task

User Story User

Story

Epic

When to do what What not to doBut by which criteria?

• Side 5


Business Value

Essence of Agile Business value is the most important criterion for establishing common

goals and quality!

The right solution which meets the needs of the enterprise!

Continuous prioritization such that unnecessary functionalty is prioritizedaway!

Early return on investment where most important functionality is deployedearly!

PRINCE (Projects in Controlled Environments) 2 Principle no. 1:

Continued business justification!

• Side 6


Food for Thought

Business value is the foremost goal!

But:

A lot of effort goes into structured cost estimation: planning poker, reestimation, speed, burndown charts

Productivity according to cost

What about productivity according to business value (quality)!

Hardly anyone engages in structured business value estimation.

Trap: business value (quality) = cost

• Side 7


Estimated cost / Cost estimationPredict cost to produce system

Relative size-based measure: Story points

Estimated business value/ Business value estimationPredict benefit of system

Relative size-based measure: Benefit points

Estimation

• Side 8


Epic

•5

story points

benefit points

User Storystory points

benefit points

•5User Story

story points

benefit points

The driving artefacts (production elements) in the project should have both cost estimates and business value estimates attached

Both Benefit Points and Story Points

• Side 9


That’s it?

Yes

But, no

• Side 10


High-level backlog

Business Case Analysis

Inception fase

Solutions fase

Construction

Impact Goals

Epic

User Story User Story

Task Task Task Task

Road map

Backlog

Sprint backlog:

Business value/Cost estimation based on Business Case

• Side 11


Societal GoalsEnterprise Goals

Need an IT modernization project

The project has impact goals; for example:- IG1: Reduce case processing time by 30%- IG2: Reduce number of erroneous case evaluations by 90%- IG3: Reduce number of erroneously filed requests from public by 95%.

Modernized system

Functional and non-functional requirementsEpic s and og user stories realize impact goals

Epic User Story

ROI ,for example:- ROI1: saved man hours (100MNOK)- ROI2. Better service for users (?MNOK)

cost

ROI due to project to business

ROI due to project to society

• * Se KSII for offentlige prosjekter. NB: DIFI gir retningslinjer for gevinstanalyse med et samfunnsøkonomisk perspektiv.

cost

benefitbenefit

Impa

ctgo

als

cont

ribut

eto

RO

I

Business Case: Relevant Notions

• Side 12


High-level backlog


Inception fase

Solutions fase

Construction

Epic

User Story User Story

Task Task Task Task

Road map

Backlog

Sprint backlog:


EffektmålCost

Impact GoalsStakeholders

ROI

Societal GoalsBusiness Goals

• Side 13


High-level backlog


Inception fase

Solutions fase

ConstructionTask Task Task Task

Road map

Backlog

Sprint backlog:


EffektmålCost


ROI


EpicEpicEpicEpiccost

benefit

EpicEpicEpicUser Storycost

benefitEpicEpicEpicUser Storycost

benefit

• Side 14


High-level backlog


Inception fase

Solutions fase


Road map

Backlog

Sprint backlog:


EffektmålCost


ROI



benefit



benefit

• Side 15


Estimate story points for epics with relative sizes (Fibonacci).

Estimate benefit points for epics with relative sizes (Fibonacci) according to howmuch each epic contributes to realizing each impact goal.

Technique 1 – Estimation: Epics’ contribution to Impact Goals

Epic8

Epic5

83

Epic2

5

CostStakeholders

ROI

Impact Goal 3Impact Goal 2

• Side 16

Impact Goal 1


Benefit Poker with Fibonacci sequence Familiar from Planning poker

Fibonacci sequnce stimulates differentiation

Ratio scale: 13 benefit points signifies as much benefit as 8 and 5


• Side 17


Ratio scale

Ratio

Interval

Ordinal

Nominal13 ≠ 8

13 > 8 > 5

20-13 = 8 -1

13 = 5 + 80 element

Equidistant

Categories with names


Ordered

• Side 18



Each epic on each impact goal

• Side 19



Each epic on all impact goals at once

• Side 20



Field experiment showed substantially higher agreement between peoples’ estimates when explicitly considering each impact goal compared to when considering an unspecific impact goal.

Spread in estimates is a common problem in human judgment.

• Side 21



Field experiment: 16 user stories were estimted for business value by experts in a large development project.

Low agreement (large spread) between the experts (Kendall W=0.33)

Rater 1 Rater 2 Rater 3 Rater 4 Rater 5

Story 1 9 1 6 2 14

Story 2 10 3 5 4 12

Story 3 12 4 1 13 15

Story 4 15 2 3 10 9

Story 5 2 5 4 5 3

Story 6 11 11 13 14 7

Story 7 14 6 7 8 8

Story 8 1 9 8 3 1

Story 9 3 7 9 9 13

• Side 22


• Representatives for all stakeholders meet.

• Prior to meeting up, each representative gathers information and produces estimates from his/her perspective.

• Each representative has a responsibility to describe the epics’ impact from the perspective of their professional role.

• To make decisions, the members of the group should think in terms of the benefit for the enterprise’s total busines.

• The Chief Product Owner makes the final decision.

• The meeting may be run as a planning poker session.

Understand benefit from

pure perspectives

Make timely good enough

descisions


• Side 23


It’s difficult for one person to perceive how a large group ofpeople with diverse roles think.

Information gathered from many people (sources) gives a better picture of central tendency and spread.

Elicit a large body of estimates with tool support.

Does not replace the Product Owner Forum, but is included as background information.


• Side 24


High-level backlog


Inception fase

Solutions fase


Road map

Backlog

Sprint backlog:


EffektmålCost


ROI



benefit



benefit

• Side 25


story points for user stories can be (re-)estimated by using planning poker (again)

benefit points for user stories should not be re-estimated according to theircontribution to impact goals. Instead, benefit for user stories should be estimated as shares of the benefit points of their respective epics.

Epic

•5

8

0.75*8 story points

5

0.3*5 benefit points

User Story User Story0.3

0.75

0.7

0.250.25*8 story points


Technique 2 – Estimation: User Stories’ contribution

• Side 26


100-dollar test


• Side 27


Analytical Hierarchy Process gives%-shares


Pairwise comparisons

• Side 28



16 user stories were estimated for benefit in a large agile project. Each expert used one of 4 alternative techniques.

1. Pulldown menue

2. Physical grouping

3. Drag & drop + sorting

4. Parwise compasrison

10 weeks later, they redid the estimation with the same techniques.

They didn’t agree with themselves

But those who used pairwise comparisons and drag & drop were more consistent.

• Side 29



Pairwise comparison is the central cognitive process in all judgment

• Comparison is domain specific and the effect is the result of available knowledge.

Have enough functional knowledge

• There are a large number of unconscious cognitive processes which disturb human judgment.

• Important to å support processes that stimulate the use of conscious processes.

• Side 30


High-level backlog


Inception fase

Solutions fase


Road map

Backlog

Sprint backlog:

Prioritization of Product Elements

EffektmålCost


ROI



benefit



benefit

prio

ritiz

e

• Side 31


Teknikk 4 – Prioritering: Poengbasert

benefit/cost = benefit points/story points

User Story5*0.8 benefit/cost = 0.60

8*0.3

User Story5*0.8

benefit/cost = 1.00User Story

20*0.5

benefit/cost = 4.00

8*0.5

User Story20*0.75

benefit/cost = 6.00

5*0.5

User Story13*0.75

3*0.5

benefit/cost = 7.50

benefit/cost = 6.50

5*0.5

User Story20*0.75

8*0.25

Technique 3 – Prioritization

• Side 32


Realized benefit points

Realized story points

Planned realization curve

benefit/cost = benefit points/story points

Teknikk 4 – Prioritering: Poengbasert produktkø

User Story20*0.75

User Story13*0.75

3* 0.5

8*0.25

User Story5*0.8

8*0.5

benefit/cost = 7.50

benefit/cost = 6.50

User Story5*0.8

8*0.3

benefit/cost = 1.00

benefit/cost = 0.60

User Story20*0.75

5* 0.5 benefit/cost = 6.00

User Story20*0.5

5* 0.5 benefit/cost = 4.00

• Side 33

Technique 3 – Prioritization


Benefit/cost-prioritization is the ideal

But what if two user stories have equal benefit/cost?- One is more costly but also equally more benficial?

Dependencies which go counter to the prioritization- Unfortunate partitioning into user stories?- Unfortunate system architecture?- Unfortunate enterprise architecture?- Didn’t think in terms of SOA?- Perhaps all of the above

But remember: The benefit/cost prioritization is (merely) an input to release planning.

• Side 34

The Ideal versus Reality


Actual value for epic story points is calulated from velocity or negotiations (with subcontractors), and gives cost estimates for epics. This is used to (re-)calculate Estimated cost and Budgetted cost for the total system.

Actual value fo epic benefit points is calculated from assumed ROI in the busniess case and gives busines value estimates for epics.

Epic8

Epic5

83

Epic2

5

CostImpact goals

Stakeholders

ROI

Actual value: 1 story point= 120 hours* 1000 kr/hour= 120000 kr

(5+8+3)*120000= NOK 1,92 mill. ( + admin ).

NOK 3 mill.

0,4 mill.

1 mill.1,6 mill.

Actual value:1 benefit points=1/15 * 3 mill. = 0,2 mill.

Side 35

Actual Values for Epics


Cost estimates for user stories is then calculated from their shares of their respective epic’s estimated cost.

Business value estimates for user stories may be calculated from their shares of their respective epic’s estimated buiness value.

Epic

5

8

0.75*8 story points

5


User Story User Story0.3

0.75

0.7 0.25

0.25*8 story points


1 story point = 120 hours1 benefit point=1/15*3 mill.

• Side 36

Actual Values for User Stories


Realized actual benefit

Realized actual cost


Teknikk 5 – Realverdi-estimering

User Story20*0.75

User Story13*0.75

3* 0.5

8*0.25

User Story5*0.8

8*0.5

Actual benefit/cost = 7.50*3/2 = 11.25


User Story5*0.8

8*0.3



User Story20*0.75

5* 0.5 Actual benefit/cost = 6.00*3/2 = 9.00

User Story20*0.5

5* 0.5 Actual benefit/cost = 4.00*3/2 = 6.00 1 benefit point = 300000 kr.1 story point = 200000 krbenefit point value/story point value = 3/2

User Stories with benefit points/story points < 2/3 cost more than they benefit.

Side 37

Technique 4 – Project Management by Prioritization

Actual benefit/cost= (benefit points*benefit point value) / (story points*story point value)= benefit points/story points * benefit point value/story point value


Realized actualbenefit

Realized actual cost


Teknikk 5 – Realverdi-estimering

User Story20*0.75

User Story13*0.75

3* 0.5

8*0.25

User Story5*0.8

8*0.5



User Story5*0.8

8*0.3



User Story20*0.75


User Story20*0.5


Actual benefit/cost= (benefit points*benefit point value) / (story points*story point value)= benefit points/story points * benefit point value/story point value

1 benefit point = 300000 kr.1 story point = 200000 krbenefit point value/story point value = 3/2

User Stories with benefit points/story points < 2/3 cost more than they benefit.

Side 38

When Should You Cease Production?


High-level backlog


Inception fase

Solutions fase

ConstructionTask Task

Road map

Backlog

Sprint backlog:

Earning

EffektmålCost


ROI


prio

ritiz

e

• Side 39

EarnEarned Value

Earned Business Value

TestTestTest

EpicEpicEpicTask

TestTestTest

EpicEpicEpicTask

TestTestTest

EpicEpicEpicTaskTestTestTest

EpicEpicEpicTask

Godkjenning

Godkjenning

Godkjenning

Godkjenning

Godkjenning

GodkjenningAcceptance Acceptance

EpicEpicEpicEpicEpicEpicUser Story User Story

Godkjenning

Epic

Godkjenning

Epic

Godkjenning

Epic

Acceptance

Epicbenefit

cost

benefit

cost

benefit

cost



Realizedactual cost

Planned curve of realization

Status at time t (for example at the end of a release)

User Story

User Story

User Story

User Story

User Story

User Story

User Story

User Story

PVEV

PBV

EBV = AB

User Story

User Story

User Story

User Story

User Story

User Story

AC

Actual curve of realization

Planned value (PV) 8 user stories: estimated cost 3.8 MNOKEarned value (EV) 6 user stories: estimated cost 2.4 MNOKActual Cost (AC) 6 user stories: 4.1 MNOKCost Performance Index (CPI): EV/AC = 2.4 / 4.1

Side 40

Earned Business Value Management



Realizedactual cost



User Story

User Story

User Story

User Story

User Story

User Story

User Story

User Story

PVEV

PBV

EBV = AB

User Story

User Story

User Story

User Story

User Story

User Story

AC



Planned Business Value (PBV) 8 user stories: estimated benefit 5,6 MNOKEarned Business Value (EBV) 6 user stories: estimated benefit 4,3 MNOKBenefit Performance Index (BPI): EBV/AC = 4.3 / 4.1.

Side 41

Technique 5–Project Management by Earned [Business] Value

Here: Even if the project is over planned cost, the benefit/cost is still OK


Earned Value: amount of functionality realized

Earned Business Value: quality of functionality realized

CPI = EV/AC

BPI = EBV/AC

Is it OK to just manage by Earned Value?

• Side 42



For story points: Past velocity (observed hours per story point) is a recommended predictor for future velocity

- Easy to gather data- Evidence based. For example, experience on things like project

overhead which is hard to predict ahead of time.

For benefit points: No direct experience in project. Must still be based primarily on judgements since actual ROI lies after thesystem is deployed. However:

- User experience at acceptance tests and increment control points maygive rise to revised judgments of how benefit is distributed on epicsand user stories, and/or the actual value of benefit points.

- Check if critical assumptions in the business case still hold; e.g., legislation, market, state of technology.

- Initiate meaurements of ROI as early as possible. Are the impact goals being met

• Side 43

Technique 6 – Update Business Case


User Story

User Story

User Story

User Story

User Story

User Story

User Story

User Story

•PV•EV

•PBV

•EBV

User Story

User Story

User Story

User Story

User Story

User Story

•AC

•AB

•

• Side 44



Realizedactual cost





Planned Business Value (PBV) 8 user stories: estimated benefit 5,6 MNOKEarned Business Value (EBV) 6 user stories: estimated benefit 4,3 MNOKBenefit Performance Index (BPI): EBV/AC = 4.3 / 4.1.Adjusted Benefit (AB): due to updated value of benefit points:3,9 MNOK

Adjusted benefit entails down-estimation of business value. Now, thebenefit/cost-ratio is not that good anymore


Summary

Estimation of benefit according to impactgoals ensures continuous business case justification.

Keep backlog ordered according to benefit points/story points. But rememberthat final order is done in releaseplanning.

Estimates of actual monetary value for story points and benefit points enablesyou to

- Manage production according to quality(earned business value) in addition to amount (earned value).

- Update business case in a proper way for all releases.

• Side 45

The End

prioritization according to estimated business value and cost · production elements requirements...

Documents