Harold van Heeringen, ISBSG president

Harold van HeeringenPresident ISBSG International Software Benchmarking Standards Group (www.isbsg.org)

Board member NESMA Netherlands Software Metrics Association (www.nesma.nl)

IAC member COSMIC Common Software Measurement International Consortium (www.cosmicon.com)

Senior Cost Engineer, Sogeti Nederland B.V. (www.sogeti.com)

@haroldveendamharold.van.heeringen@sogeti.nl

OverviewISBSG introductionIFPUG and ISBSG in practice: 3 cases

Reality check of an expert estimate;Competitiveness assessment of an IT

department;Selecting the right (sole) supplier to do all the

IT and manage the supplier based on productivity.

ISBSG: other useful purposesHow does Spain compare to the industry

We measured 350 FP, so... What is the estimate? Effort, duration, quality, risk? Price per FP?

Why are we losing most of the bids we do? Competitive advantage? Or not? What is our productivity compared to the industry?

We are going to send out a request for bid based on price/FP How can we compare the proposals? How can we exclude unrealistic (overoptimistic) ones? What should the market average bid approximately look like?

We did a project and the client thinks it was too expensive How did we perform against similar projects?

Size does matter, but….

Historical data of completed projects!!

The answer…Without software metrics based on an objective unit of measure, there is little we can do. To be able to analyze and to make decisions, we need:

International Software Benchmarking Standards GroupIndependent and not-for-profit;Full Members are non-profit organizations, like IFPUG, NESMA,

GUFPI-ISMA, FiSMA, QESP, DASMA, JFPUG, Swiss-ICT and CESI;Associate members: AEMES (Asociacion Espanola de Metricas de

Software), ASSEMI (France);Grows and exploits two repositories of software data:

New development projects and enhancements (> 6000 projects); Maintenance and support (> 1200 applications).

Everybody can submit project data DCQ on the site / on request (.xls) Anonymous Free benchmark report in return

ISBSG

Mission: “To improve the management of IT resources by both business and government, through the provision and exploitation of public repositories of software engineering knowledge that are standardized, verified, recent and representative of current technologies”.

All ISBSG data is validated and rated in accordance with its quality guidelines current representative of the industry independent and trusted captured from a range of organization sizes and industries

ISBSG

www.isbsg.org

ISBSG wishes to help restore the confidence in IT, by giving transparency of what is realistic, resulting inImproved project success rates;Better customer satisfaction;Shorter time to market;More efficient software development;Better alignment with the business/customer;Easier contracting, resulting in better

relationships between customer and supplier.

ISBSG IT Confidence

3 Cases from my experience

Case 1: Telecom project reality check on the expert estimate (project manager)

Case 2: Assessing the competitive position of an organization (Senior management)

Case 3: Supplier Performance Measurement (procurement)

Case 1: Reality Check

Case 1A telecom company wished to develop a new Java system

for the maintenance of subscription types;

A team of experts studied the requirements documents and filled in the WBS-based estimation calculation (bottom-up estimate);

They decide that an estimate of 5.500 hours and a duration of 6 months should be feasible;

The project manager decided not to believe the experts ‘on their blue eyes’ only and wished to carry out a reality check.

Why a realistic estimate?Non-linear extra costs

-Planning errors

-team enlargement more expensive, not faster

-Extra management attention / overhead

-Stress: More defects, lower maintainability !!

Linear extra costs

Extra hours will be used

ISBSG Reality Check: EffortAn estimated FPA comes up with the expected size:

Min: 550 FP, likely 850 FP, Max 1300 FPImplicit likely expert PDR: 5.500/850 = 6,5 h/FP

Selecting the most relevant projects in the ISBSG D&E repository show the next results:

PDR (h/FP)Min. 3,2Percentile 10% 4,3Percentile 25% 6,2Median 8,9Percentile 75% 12,9Percentile 90% 19,8Max. 34,2N 89

PDR (h/FP)Min. 3,2Percentile 10% 4,3Percentile 25% 6,2Median 8,9Percentile 75% 12,9Percentile 90% 19,8Max. 34,2N 89

550 850 1300

3.410 5.270 8.0604.895 7.565 11.5707.095 10.965 16.770

Functional Size550 850 1300

3.410 5.270 8.0604.895 7.565 11.5707.095 10.965 16.770

Functional Size

5.500 hours seems optimistic

ISBSG Reality Check: DurationSame analysis is possibleAlso, formulas have been published in the Practical

Project Estimation bookFor instance:table C-2.2 Project Duration, estimated from software size onlyFunctionele omvang 550 FPC uit tabel 0,507E1 uit tabel 0,429Duration = C * Size^E1Duration = 7,6 elapsed months

550 850 1300Duration 7,6 9,2 11,0

ResultExpert estimate was assessed optimisticAdjusted Estimate:

Effort: 8.000 hoursDuration: 10 months

This turned out to be quite accurate!

The project manager now always carries out reality checks and is ‘spreading the word’.

Case 2: Competitiveness

Case 2Senior management of a software company wondered

how competitive they were when it comes to productivity.

Many bids for projects were lost and they wished to improve, especially their Microsoft .Net department.

Analysis of the bids by department showed the next figures:Nr. of bids 23

Average PDR in bid 16,3 h/FPAverage Size (FP) 230 FPAverage teamsize 6 fte

PDR (h/FP)Min. 3,2Percentile 10% 3,8Percentile 25% 5,9Median 7,6Percentile 75% 12,9Percentile 90% 18,9Max. 34,2N 35

ISBSG data analysis

IssuesAnalysis of the bid phase showed a number of issues:

Estimates were extremely pessimistic due to severe penalties in case of overruns;

In a number of stages, risk surcharges were added;They wished to work in fixed team of 6 fte, but

ISBSG data shows that the project size was usually too small for this teams size to be efficient;

Because of the knowledge that the department bids were not market average (or better), the bid process was redesigned, making the company more successful!

Case 3: Supplier selection and management

Case 3An organization decided to outsource all of

their IT work to one supplier;Therefore, the internal competition has been

gone and potentially the chosen supplier could charge whatever they wish;

The ISBSG data was used to assess the productivity and competitiveness of the suppliers;

After the bidding procedure, the chosen supplier had to commit to productivity targets and quality targets.

Bidding process5 ICT suppliers bid for the 5-year contract;They had to submit data of 6 completed projects,

sized in IFPUG function points;The company metrics desk (MD) assessed the

productivity and competitiveness of the organizations;The MD constructed a model to do the assessment;The model was part of the overall bid process and an

important part of the selection of the one winner;The data submitted was also the basis for the baseline

performance that he had to live up to.

Assessment modelReality value; Compliancy value to data requirements;Productivity / Quality value.

Project Delivery Rate (PDR) = spent project effort related to function point (h/FP);

Productivity Index (PI) = metric from QSM, derived from size, duration and effort;

Quality: delivered defects per FP;

Benchmarks:• PI against the QSM Business trend line• PDR against ISBSG Repository• Adjusted = normalised to Construction+Test activities

Productivity/quality assessment

Supplier PI scoreRank

PI scorePoints

PI score

Supplier A 3,9 2 8

Supplier B 5,0 1 10

Supplier C 3,4 3 6

Supplier D 3,0 5 2

Supplier E 3,2 4 4

SupplierPDR score

Rank PDR score

Points PDR score

Supplier A -3,2 1 10

Supplier B -2,1 2 8

Supplier C 16,6 4 4

Supplier D 6,2 3 6

Supplier E 18,3 5 2

SupplierQuality Score

Rank Quality score

Points Quality score

Supplier A 3,1 1 10

Supplier B 13,9 2 8

Supplier C 52,6 3 6

Supplier D 1000,0 5 2

Supplier E 94,6 4 4

SupplierPoints

PI scorePoints

PDR scorePoints

Quality scoreProductivity/ Quality value

Supplier A 8 10 10 9,0

Supplier B 10 8 8 9,0

Supplier C 6 4 6 5,4

Supplier D 2 6 2 3,2

Supplier E 4 2 4 3,4

weight 50% 30% 20%

ResultsSupplier

Compliancy value

Reality value

Productivity/Quality value

Total Points Rank

Supplier A 0 6 9,0 7,2 2

Supplier B 0 8 9,0 7,8 1

Supplier C 4 10 5,4 6,6 3

Supplier D 0 10 3,2 4,9 4

Supplier E 0 8 3,4 4,4 5

weight 10% 30% 60%

Metrics Based Software Supplier Selection - Kuijpers & van Heeringen (IWSM2012)

ISBSG benchmark

Supplier targets and management

7

8

9

10

11

12

13

PDR

(h/F

P)

Target PDR PDR (h/FP)

500

600

700

800

900

1000

1100

1200

1300

EUR/

FP

Target PCR PCR (EUR/FP)

0

5

10

15

20

25

30

35

40

45

jan-13 feb-13 mrt-13 apr-13 mei-13 jun-13 jul-13 aug-13 sep-13 okt-13

Target Defects/1000 FP Defects/1000 FP

The supplier is measured continuously and still has to make his target for the first time!

The organization is happy that the trends show improvement and they feel in control.

Other useful purposes of ISBSG dataAnalyze the difference in productivity or

quality between two (or more) types of projects:Traditional vs. AgileOutsourced vs. In-houseGovernment vs. Non-governmentOne site, multi siteReuse vs. no reuse

ISBSG Special Analysis reports

Special reports, examplesImpact of Software Size on ProductivityGovernment and Non-Government Software Project

PerformanceISBSG Software Industry Performance reportISBSG/COSMIC The Performance of Business

Application, Real-Time and Component Software Projects

Estimates – How accurate are they?Planning Projects – Role PercentagesTeam size impact on productivity Manage your M&S environment – what to expect?Many more

2013 Country Analysis Report

USA

Thank you!

NEtherlands Software Metrics users Association

Historical landmarksHistorical landmarks

Modern landmarksModern landmarks

European Cultural Capital 2001European Cultural Capital 2001

The city of modern architectureThe city of modern architecture

Largest soccerstadium in the NetherlandsLargest soccerstadium in the Netherlands

Welcome toWelcome to

RotterdamRotterdam

Busiest port in EuropeBusiest port in Europe