Software Estimation and

Performance Measurement

Harold van Heeringen

Sogeti Sizing, Estimating & Control (SEC)

Sogeti Nederland B.V.

Stockholm, April 18 2013 @haroldveendam

2

Topics

• Software projects

• Software estimation

• Expert vs. Parametric Estimates

• Challenge for parametric estimates

• E&PM process

• Accuracy of the different estimation types

• The Swedish software industry?

3

Software projects

• Software project industry : low maturity − Low estimation maturity

− No or little formal estimation processes

− No or little use of historical data

• Lots of schedule and cost overruns − Standish Chaos reports: Most projects fail or are at least

unsuccessful

• Low customer satisfaction rates − In Europe: only slightly higher than the financial sector

4

Software project estimation

• Most of the projects are estimated by ‘experts’ − Bottom up, task by task effort estimation

• Usually very optimistic (>30%) − Experts estimate, but other people (juniors) do the job

− Forgotten activities (e.g. testscript reviews)

− No feedback loop with past projects: experts don’t learn from past estimates and actuals

− No scenario’s: duration, team size, etc.

− Not objective, transparent, verifiable or repeatable

• Not defendable! − ‘Easy’ to push back by stakeholders

• No risk assessment (distribution of the estimate)

5

Expert Estimation Pro

bability

Effort

50/50 median result

90% . . . . . . . . . . . . . . . . 24 hours

75% . . . . . . . . . . . . . . . . 22 hours

50% . . . . . . . . . . . . . . . . 20 hours

10% . . . . . . . . . . . . . . . . 18 hours

0% . . . . . . . . . . . . . . . . 16 hours

. . . . . . . . 14 hours

. . . . . . . . 12 hours

. . . . . . . . 10 hours

. . . . . . . . 8 hours

. . . . . . . . 6 hours

. . . . . . . . 4 hours

. . . . . . . . 2 hours

Expert Estimate: 1st possibility of success

0%

100%

Task– Code and Unit test of module XYZ

Expert estimate

Realistic estimate

6

Project Estimates

• Two types of project estimation: − Expert estimation

− Parametric estimation

• Expert − Knowledge and experience of experts

− Assign effort hours to tasks (bottom-up)

− Subjective, but always applicable

• Parametric − Size measurement, historical data and tooling

− Size measurement methods: NESMA FPA, COSMIC, IFPUG

− Objective, but well documented specifications required

7

Comparing the two types

• Expert Estimates − Bottom-up estimation

− Usually optimistic (up to 30% under estimation is common)

− Forgotten activities

− Hard to defend

− The expert is not going to do all the work

− The expert may not be an expert on the new project

− Are the real experts available?

• Parametric Estimates − Top-down estimation

− Estimating & Performance Measurement process needed

− Effort = size * Productivity

◦ Size is objectively measureable (COSMIC, FPA)

◦ Productivity from historical data (organization / ISBSG)

◦ scenario’s through tools (QSM / SEER-SEM / ISBSG)

8

Eestimate & Performance Measurement

PLAN

Estimate

Administrate Evaluate

Adjust & Report

Size measurement: FPA Historical data Estimation tools

Finetune Estimation model Analyse productivity, Report productivity

Start: Estimate request

Start: Project completed

Results: - Parametric Estimation - Expert Estimation

Result: -Management report, -Adjusted model

Result: -Growing project DB, -Performance measurement -Updated expert knowledge

ACT

CHECK DO

Start: Project start

Continuous data collection • effort hours registration • defect registration • change measurement • project characteristics

Result: Project data

Data collection and administration • Collect project data • Measure size • Benchmark the project

Start: Periodically

Expert Estimate

9

Challenge – ‘sell parametric estimates’

• Project/Bid management still believe experts more than parametrics − ‘More detail must mean more accurate, right’?

− ‘This project is very different from past projects’

− ‘I see that we don’t get any function points for module XYZ, but we have to do a lot of work for that!’

− ‘I think the project is quite easy and I think that the parametric estimate overestimates the effort’

− But what about: team size, duration, forgotten activities, past performance.

• How can we convince project management ??

10

Software equation (Putnam)

Size/productivity

= Effort 1/3 * duration4/3

Effort

Duration

Plan A: 6 months, 4.500 hours

Plan B: 7 months, 2.400 hours

11

Project at different durations E

ffort

(ho

urs

)

Duration

Plan A

Duration: 6 months

Effort: 4.500 hours

Max. team size: 5,8 fte

MTTD: 1,764 days

Plan B

Duration: 7 months

Effort: 2.400 hours

Max. team size: 2,7 fte

MTTD: 2,816 days

Size and productivity being constant

Which duration have the experts in mind??

12

Sogeti SEC

• Sizing, Estimating & Control − Certified (COSMIC) Function Point Analysts

− Metrics consultants

• Responsible for metrics part of a quotation. − Size: FPA/COSMIC

− Estimation: SEER-SEM / QSM / Sogeti tool / ISBSG

− Product: Methodical Estimation Report (scenario’s)

− Pricing: EUR/FP

− Quality: Defects/FP

• Centers of Excellence: MS.Net, Java, Oracle, mobile, drupal, sharepoint, BI, etc.

13

Assignment 2005

• Build estimation instrument − Gain time and effort in estimating bids

− Accurate enough to depend and rely on

− Flexible:

◦ Estimate onshore / offshore and hybrids

◦ Calculate different test strategies

◦ Take into account complexity

◦ Implement Deming cycle (PDCA)

• Give scenario’s for duration !!!

14

First version Estimating Wizard (2005)

• Try to grasp the duration / effort tradeoff in a model − Tuned with experience data

Hour/FP: Average Complexity

Duration in months 3½ 4 4½ 5 5½ 6 6½ 7 7½ 8

0-250 FP 10,1 8,9 8,1 7,7 6,9

250-500 FP 9,1 8,0 7,3 6,9 6,2

500-750 FP 8,6 7,6 6,9 6,5 5,9

750-1000 FP 8,3 7,3 6,6 6,3 5,6

1000 -1250 FP 8,1 7,1 6,5 6,2 5,5

1250 - 1500 FP 7,9 6,9 6,3 6,0 5,4

15

Estimating Wizard 2013

InputFunctional design parameters

Functional Design Yes

Overlap Yes, manually 5

Language English

Availability key users Low

Location On site

Build and test parameters

Development tool Java

Onshore Offshore

Construction 30% 70%

Translation FD required No

System test approach TMap Heavy

System test strategy

Tools/methodologies 5 - Average

Complexity 6

Development team 5 - Average

Reuse 0 - None 5%

General parameters

Size 367 FP

Start date 01-05-13

Project control 2 hrs/week

Risk surcharge (%) 10 %

Warranty (%) 4 %

Organization type Banking

Quality documentation 6

Non functional req.

Scenario interval 2,0 weeks

m20v20120224

Scripting and design NL, execution in India

Average (0)

16

EW - output

• Functional Design – no schedule scenarios

Functional design phase

Duration in weeks 17,6

Design complete 4-05-11

Total effort 1.975

Effort per FP 2,53

Effort cost € 208.531

Additional cost € 14.815

Totaal cost € 223.346

Cost per FP € 286

Average team size 2,80

Data altered due to company security reasons

17

EW - output

• Main build – 7 schedule scenarios

Build and test phase

Duration in weeks 20,0 22,0 24,0 26,0 28,0 30,0 32,0

Start phase 18-02-11 18-02-11 18-02-11 18-02-11 18-02-11 18-02-11 18-02-11

Effort 9.794 6.690 4.723 3.429 2.550 1.935 1.495

Effort per FP 28,06 19,17 13,53 9,83 7,31 5,54 4,28

Effort cost € 550.720 € 376.152 € 265.590 € 192.826 € 143.360 € 108.787 € 84.036

Additional cost € 56.446 € 41.090 € 31.364 € 24.963 € 20.611 € 17.570 € 15.393

Totaal cost € 607.167 € 417.242 € 296.954 € 217.789 € 163.972 € 126.357 € 99.428

Cost per FP € 1.740 € 1.196 € 851 € 624 € 470 € 362 € 285

Average team size 12,24 7,60 4,92 3,30 2,28 1,61 1,17

Risk and warranty

Risk hours 834 586 429 325 255 205 170

Risk cost € 54.301 € 39.107 € 29.484 € 23.150 € 18.845 € 15.836 € 13.682

Warranty hours 209 146 107 81 64 51 43

Warranty cost € 13.575 € 9.777 € 7.371 € 5.788 € 4.711 € 3.959 € 3.420

Total

Duration in weeks 27,0 29,0 31,0 33,0 35,0 37,0 39,0

Delivery for acceptance 8-07-11 22-07-11 5-08-11 19-08-11 2-09-11 16-09-11 30-09-11

Total effort 11.470 8.055 5.892 4.469 3.501 2.825 2.340

Effort per FP 32,87 23,08 16,88 12,80 10,03 8,09 6,71

Totaal cost € 746.634 € 537.717 € 405.400 € 318.319 € 259.120 € 217.744 € 188.122

Cost per FP € 2.139 € 1.541 € 1.162 € 912 € 742 € 624 € 539

Average team size 10,64 6,96 4,76 3,39 2,50 1,91 1,50

Average hourly rate € 65 € 67 € 69 € 71 € 74 € 77 € 80

Data altered due to company security reasons

18

Calibrating the model

• Collect historical data

• Use ISBSG data

19

Calibrating the model

• Collect historical data

• Use ISBSG data

20

Estimation accuracy

• Study of 10 completed projects, Different sizes, programming languages, etc − Expert estimate

− Estimating Wizard estimate

− Estimates vs. Actual results

• Calculate − Effort accuracy (Effort estimate / Actual effort)

− Duration Accuracy (Duration Estimate / Actual Duration)

− Cost Accuracy (Cost Estimate / Actual Cost)

21

Results

• Calculate − Effort accuracy (Effort estimate / Actual effort)

− Duration Accuracy (Duration Estimate / Actual Duration)

− Cost Accuracy (Cost Estimate / Actual Cost)

Expert Expert Expert Expert Est. Wizard Est. Wizard Est. Wizard Est. Wizard

Project Size (FP)

Effort

Accuracy

Duration

Accuracy

Cost

Accuracy Time Spent

Effort

Accuracy

Duration

Accuracy

Cost

Accuracy

Time

spent

Project 1 277 0,675 1,545 0,467 30 0,477 1,204 0,501 17

Project 2 359 0,579 0,951 1,139 35 0,707 0,775 1,170 26

Project 3 347 0,589 0,142 0,615 40 1,067 0,996 1,283 14

Project 4 1.178 0,414 0,557 0,312 60 0,774 0,590 0,862 55

Project 5 951 1,430 0,997 0,946 34 1,067 0,877 1,718 24

Project 6 295 0,763 0,857 0,619 26 0,881 1,200 0,845 6

Project 7 790 0,717 0,850 0,976 34 0,926 0,865 1,132 27

Project 8 350 1,258 0,800 1,309 28 1,203 1,096 1,318 20

Project 9 746 0,586 0,296 0,545 34 0,826 0,385 1,953 22

Project 10 2.293 0,766 0,421 0,797 40 0,931 0,632 1,058 14

22

Estimation Accuracy results

• Average time spent − Expert: 36,1 hours

− EW (including FPA / COSMIC measurement): 22,5 hours

Expert Estimate Est. Wizard Estimate

Effort Accuracy

Average 0,778 0,886

St.Dev. 0,319 0,207

Median 0,696 0,904

Duration Accuracy

Average 0,742 0,862

St.Dev. 0,405 0,272

Median 0,825 0,871

Cost Accuracy

Average 0,772 1,184

St.Dev. 0,316 0,423

Median 0,708 1,151

23

Results

• Expert estimates take a lot of time too! − on average 60% more than Parametric Estimates

− More than 1 expert has to read through all the documentation

− Discussions and agreements take time

• Parametric estimates are more acurate! − Effort and duration estimates on average still optimistic, but

less optimistic than expert estimates

− Cost estimate pessimistic, but still closer to actuals than Expert estimate

• Experts might win the project, but the result will be overruns!

24

Cost of high and low estimates

Non-linear extra costs

- Planning errors

- Team enlargement more expensive, not faster

- Extra management attention / overhead

- Stress: More defects, lower maintainability !!

Linear extra costs

Extra hours will be used

25

Conclusions of the study

• Estimating wizard − Higher effort estimation accuracy

− Higher duration estimation accuracy

− Higher cost estimation accuracy (although >1)

− Less hours spent than expert estimates

− Standard WBS, historical data collection and parameter calibration improve the maturity of the process

• Next steps − Analyze why costs are overestimated

− Try to identify the projects where EW estimate is enough

− Use the results to convince project management and bid management to use parametric estimating more

26

Do we have time for some other thoughts?

• What about the maturity in the Swedish market?

27

ISBSG ‘Country Analysis Report’

• ISBSG repository (>6000 projects)

28

ISBSG ‘Country Analysis Report’

29

Where are the Swedish projects??

• Please submit project data to ISBSG !

• Independent, anonymous

• Free benchmark report

• Flyer ISBSG available

www.isbsg.org

30

Thank you for your attention !!

@haroldveendam

President ISBSG (International Software Benchmarking Standards Group (www.isbsg.org))

Board member NESMA (Netherlands Software Metrics Association (www.nesma.nl))

IAC member COSMIC (www.cosmicon.com)

Harold.van.heeringen@sogeti.nl

Harold van Heeringen

Senior Consultant Software Metrics /Software Cost Engineer

Sogeti Sizing, Estimating & Control (SEC)