quantitative software management

© Quantitative Software Management, Inc. All rights reserved. Slide #1

Quantitative Software Management

Using Metrics to Develop aSoftware Project Strategy

Donald M. BeckettQSM, Inc.

2000 Corporate Ridge, Suite 900McLean, VA 22102

(360) 697-2640, fax: (703) 749-3795e-mail: [email protected]

http://www.qsm.com

http://www.pdfcomplete.com/cms/hppl/tabid/108/Default.aspx?r=q8b3uige22

© Quantitative Software Management, Inc. #2

OutlineOutline

•• OverviewOverview

•• Measurement, Expense or InvestmentMeasurement, Expense or Investment

•• State of the Industry: Project EstimationState of the Industry: Project Estimation

•• Staffing and ScheduleStaffing and Schedule

•• Understanding TradeUnderstanding Trade--offsoffs

•• ConclusionConclusion

•• Questions?Questions?



OverviewOverview

Does this sound familiar?



Measurement: Expense or InvestmentMeasurement: Expense or Investment

•• Software measurement (and process Software measurement (and process improvement) are viewed as improvement) are viewed as expensesexpenses: : OverheadOverhead–– Lean, agile organizations want to reduce overheadLean, agile organizations want to reduce overhead

–– But, how do organizations become “lean & agile”?But, how do organizations become “lean & agile”?

•• Part of cost of doing business Part of cost of doing business –– 3 3 –– 5% on average5% on average

–– Project management averages 14%Project management averages 14%



Measurement: Expense or InvestmentMeasurement: Expense or Investment

•• What does software measurement provide?What does software measurement provide?1.1. Knowledge of an organization’s capabilitiesKnowledge of an organization’s capabilities2.2. Identifies patterns and trends (Strengths to leverage and Identifies patterns and trends (Strengths to leverage and

weaknesses to correct)weaknesses to correct)3.3. Insight into projects in time to make effective midInsight into projects in time to make effective mid--stream stream

correctionscorrections4.4. Ability to benchmark against competition or “the industry” in Ability to benchmark against competition or “the industry” in

quality, productivity, and time to marketquality, productivity, and time to market5.5. Quantitative basis for evaluating project and organizational Quantitative basis for evaluating project and organizational

performanceperformance

•• Improves ability to meet commitments, avoid Improves ability to meet commitments, avoid pitfalls, and evaluate tradepitfalls, and evaluate trade--offsoffs



State of the Industry: Project State of the Industry: Project EstimationEstimation

•• Software estimates are Software estimates are notnot project plansproject plans

•• Estimates contain uncertainty about two key Estimates contain uncertainty about two key components:components:–– Scope of the requirements (project size)Scope of the requirements (project size)

–– Team productivityTeam productivity



The Cone of UncertaintyThe Cone of Uncertainty

• Not enough information is available early in the development lifecycle to make accurate estimates

• Precision is not accuracy

Low level requirements

Project Definition



Actual vs. Estimated EffortActual vs. Estimated Effort

Effort Growth

0

10

20

30

40

50

60

Smaller At Size Larger

Actual vs. Estimated Effort

Perc

ent



Actual vs. Estimated ScheduleActual vs. Estimated Schedule

Schedule Growth

0

10

20

30

40

50

60


Actual vs. Estimated Schedule

Perc

ent

Percent



Actual vs. Estimated SizeActual vs. Estimated Size

Size Growth

0

10

20

30

40

50

60

70

80

90

100


Actual vs. Estimated Size

Perc

ent



In SummaryIn Summary

•• Average schedule growth is 8%Average schedule growth is 8%

•• Average cost/effort growth is 16%Average cost/effort growth is 16%

•• Average size growth is 15%Average size growth is 15%

•• So how can we use this information to create more So how can we use this information to create more accurate estimates?accurate estimates?



Modeling Increased SizeModeling Increased Size

•• Create best project estimate based on proposed Create best project estimate based on proposed sizesize–– Use historically based productivityUse historically based productivity

–– Account for project constraints (staff, effort, schedule)Account for project constraints (staff, effort, schedule)

•• Create estimate based on 15% size growthCreate estimate based on 15% size growth–– Does this account for projected schedule & effort growth?Does this account for projected schedule & effort growth?



S taffin g & P rob ab ility Ana lys is

Avg S ta ff L i fe C ycle (p e o p le )< 500 F P projec t>

1 2 3 4 5 6 7 8 9 10 11O c t'07

N ov Dec Jan'08

Feb M ar A pr M ay Jun Jul A ug S ep

0

1

2

3

4

5

6

7

Avg S

taff Life Cycle (people)

87654321

M iles tones 0 - C SR 1 - S R R 2 - H LD R 3 - LLDR 4 - C U T 5 - IC 6 - S T C 7 - U AT 8 - F C R 9 - 99R 10 - 99 .9R

M iles tones 0 - C SR 1 - S R R 2 - H LD R 3 - LLDR 4 - C U T 5 - IC 6 - S T C 7 - U AT 8 - F C R 9 - 99R 10 - 99 .9R

S O L U TI O N PAN EL - <500 F P p ro ject >

D ura tionEffo r tC ost

Pea k S ta ffM T T D

Star t D a te

C & T6 .729

493 .45 .7

1 .82312 /23 /2007

L if e C yc le9 .437

643 .65 .7

1 .82 310 /1 /2007

M o nth sP M$ ( K)pe op leD ay s

PI =16 .5 M B I =3 .6 E ff F P =500

9.4 months duration

37 person months effort

50% probability

500 FP Project



Ev a lu a te Pr o b a b ility o f C u rr e n t Es tim a te

L ife Du r at io n (M o n th s ) Ris k Pr o f ile<500 FP p rojec t>

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0

A s s uranc e Lev e l (% )

7

8

9

1 0

1 1

1 2

1 3

Life Duration (M

onths)

L ife Ef fo r t (PM ) Ris k Pr o file<500 FP p rojec t>

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0

A s s uranc e Lev e l (% )

0

2 0

4 0

6 0

8 0

1 0 0

Life Effort (P

M)

Life Dur ation (Months ) R is k P r ofi le - P robabi l ity de mo<5 0 0 FP pr oje ct>

A s s uranceLeve l(% )

15

10152025303540455055606570758085

Life Dur ation(Months )

7.267.908.258.488.668.828.969.099.219.339.459.579.699.819.94

10.0810.2410.42

Life Effort (P M) R is k P r ofile - P r obabi li ty de mo<5 0 0 FP pr oje ct>

A s s uranceLeve l(% )

15

10152025303540455055606570758085

Life Effor t(P M)

0.005.11

12.2016.9920.7924.0526.9829.6932.2734.7537.2039.6542.1344.7147.4250.3553.6157.41

P ro jec t: P robab i l i ty dem o

Likely outcomes 10.2 months schedule, 43 effort months

500 FP Project



S ta ffin g & P ro b a b ility An a ly s is

Avg S ta ff L i fe Cycle (pe o p le )< 15% s ize grow th>

1 2 3 4 5 6 7 8 9 10 11O ct'07

N ov D ec J an'08

F eb M ar A pr M ay J un J ul A ug S ep

0

2

4

6

8

Avg S

taff Life Cycle (people)

87654321

M iles tones 0 - C SR 1 - SR R 2 - HLDR 3 - LLDR 4 - C U T 5 - IC 6 - ST C 7 - U AT 8 - F C R 9 - 99R 10 - 99 .9R

M iles tones 0 - C SR 1 - SR R 2 - HLDR 3 - LLDR 4 - C U T 5 - IC 6 - ST C 7 - U AT 8 - F C R 9 - 99R 10 - 99 .9R

S OL U TI O N P AN E L - <1 5% size g ro wt h >

D u r ationE ffor tC o st

P ea k S ta ffM T T D

S tar t D a te

C & T7.335

60 3.76.5

1.6 8112 /28 /2 00 7

L if e C y cle1 0.24 6

7 87 .56 .5

1 .68 11 0/1 /2 00 7

M on th sP M$ (K )pe op leD a y s

P I =16 .5 M B I =3 .4 E ff F P =57 5

15% Growth (575 FP)

10.2 months duration

46 person months effort



E v a lu a te P r o b a b il i t y o f C u r r e n t E s t im a te

L if e Du r a t io n (M o n t h s ) R is k P r o f ile<1 5 % s iz e g r o w th >

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0

A s s u r a n c e L e v e l (% )

7

8

9

1 0

1 1

1 2

1 3

1 4

Life Duration (M

onths)

L if e Ef fo r t ( P M ) R is k P r o f ile<1 5 % s iz e g r o w th >

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0

A s s u r a n c e L e v e l (% )

0

2 0

4 0

6 0

8 0

1 0 0

1 2 0

Life Effort (P

M)

L i fe D u r ati on (M on th s ) R i s k P r o fi l e - P r o ba bi l i ty de m o< 1 5 % s i z e g r owth >

A s s u r a n c eL e ve l(% )

15

10152025303540455055606570758085

L i fe D u r ati on(M on th s )

7.838.518.889.129.329.489.639.779.91

10 .0 310 .1 610 .2 910 .4 110 .5 510 .6 910 .8 411 .0 011 .2 0

L ife Effo r t (P M ) R is k P r ofi l e - P r obabi l i ty de m o< 1 5 % s i z e g r owth >

A s s u r a n c eL e ve l(% )

15

10152025303540455055606570758085

L i fe Effor t(P M )

0.008.61

16 .7 622 .2 726 .6 430 .3 933 .7 636 .8 939 .8 542 .7 145 .5 248 .3 351 .1 954 .1 557 .2 860 .6 564 .4 068 .7 7

P ro j ec t: P ro b ab i l i ty d em o

15% Growth (575 FP)

Averages close to numbers predicted for effort and schedule growth (10.2 duration and 43 staff months of effort)



Staffing & ScheduleStaffing & ScheduleValidate Estimate with History

C&T Duration (Months) vs Effective SLOC

100 1,000Effective SLOC (thousands)

1

10

100

C&

T Duration (M

onths)

C&T Effort (PM) vs Effective SLOC

100 1,000Effective SLOC (thousands)

10

100

1,000

10,000

C&

T Effort (P

M)

Current Solution Logged Solutions Historical Projec ts QSM 2005 Business Av g. Line Sty le 1 Sigm a Line Sty leProjec t: Staffing & Schedule Demo

Schedule varies by a factor of 3.5 from -1σ to +1 σ

Effort varies by a factor of 8 from -1σ to +1 σ

What is “normal” variability?



How Should Project Effort Be ExpendedHow Should Project Effort Be ExpendedA Case StudyA Case Study

•• 838 projects that had data reported for Analysis/Design as well 838 projects that had data reported for Analysis/Design as well as Construction and Test phasesas Construction and Test phases

•• Average Effort applied to Analysis/Design = 20%Average Effort applied to Analysis/Design = 20%

•• 474 projects in the sample used <= 20% design effort474 projects in the sample used <= 20% design effort–– Average Analysis/Design Effort = 11%Average Analysis/Design Effort = 11%

•• 364 projects in the sample used > 20% design effort364 projects in the sample used > 20% design effort–– Average Analysis/Design Effort = 33%Average Analysis/Design Effort = 33%

•• Size profiles of samples very similarSize profiles of samples very similar



ObservationsObservations

•• Projects with <20% effort in Requirements and Projects with <20% effort in Requirements and Design Design –– Took 12% longer to completeTook 12% longer to complete

–– Averaged 5.6% more effort (median 24.4% greater)Averaged 5.6% more effort (median 24.4% greater)

–– Had an average staff 14.6% higherHad an average staff 14.6% higher

•• But these projects did excel at one thing:But these projects did excel at one thing:–– Found 63.7% more defects in systems testFound 63.7% more defects in systems test

–– Had 127% more defects in the first 12 months after deliveryHad 127% more defects in the first 12 months after delivery



Understanding TradeUnderstanding Trade--offsoffs

1 43 3

a b

a b= =

× ×where and

Size = Effort Time Productivity

Additional schedule has a much larger impact on a software project than increased effort



The Estimating ConundrumThe Estimating ConundrumSchedule / Effort TradeoffSchedule / Effort Tradeoff

Duration

Effo

rt

Impossible Zone

Impractical ZoneFeasible Solutions

Uncertainty about Size and Productivity creates uncertainty about the Duration-Effort curve



The Estimating ConundrumThe Estimating ConundrumSometimes no Solution WorksSometimes no Solution Works

Duration

Effo

rt

Impractical Zone

Impossible Zone

Feasible Solutions



The Estimating ConundrumThe Estimating ConundrumRelax the ScheduleRelax the Schedule

Duration

Effo

rt

Impossible Zone




The Estimating ConundrumThe Estimating ConundrumIncrease EffortIncrease Effort

Duration

Effo

rt

Impossible Zone




The Estimating ConundrumThe Estimating ConundrumReduce Size (Functionality)Reduce Size (Functionality)

Duration

Effo

rt

Impossible Zone




The Estimating ConundrumThe Estimating ConundrumAssume Higher ProductivityAssume Higher Productivity

Duration

Effo

rt

Impossible Zone




ConclusionsConclusions

•• Measurement is an integral part of managementMeasurement is an integral part of management

•• Information required to make precise estimates is Information required to make precise estimates is unavailableunavailable at project startat project start--upup–– Estimate uncertainty decreases rapidly with more informationEstimate uncertainty decreases rapidly with more information

•• Project estimates understate effort, schedule, & Project estimates understate effort, schedule, & sizesize–– Estimating based on a larger size or at a higher assurance levelEstimating based on a larger size or at a higher assurance level

can account for thiscan account for this

•• The tradeThe trade--off between schedule & cost/effort is off between schedule & cost/effort is nonnon--linearlinear



ConclusionsConclusions

•• Effort spent in Analysis & Design pays Effort spent in Analysis & Design pays bigbigdividendsdividends–– Reduces overall project effort (cost$$$$)Reduces overall project effort (cost$$$$)

–– Reduces overall project scheduleReduces overall project schedule

–– Improves project qualityImproves project quality



QuestionsQuestions??


quantitative software management

Documents