The value of benchmarking

IT projects

Harold van Heeringen

Software Cost Engineer, Sogeti Nederland B.V. ISBSG president

NESMA board

The Russian Managers Association

Location: Moscow

Date: October 2013

3

Overview

Benchmarking

Software Project Industry

Functional Size Measurement

Software Metrics

Historical Data: ISBSG

Project Benchmark Example

Organization Benchmark

Other uses for Benchmarking

Conclusions and final remarks

Scope: Software Development

4

Benchmarking (wikipedia)

Benchmarking is the process of comparing one's business processes

and performance metrics to industry bests or best practices from other

industries.

Benchmarking is used to measure performance using a specific indicator (cost

per unit of measure, productivity per unit of measure, cycle time of x per unit

of measure or defects per unit of measure) resulting in a metric of

performance that is then compared to others

This then allows organizations to develop plans on how to make

improvements or adapt specific best practices, usually with the aim of

increasing some aspect of performance. Benchmarking may be a one-off

event, but is often treated as a continuous process in which organizations

continually seek to improve their practices.

5

Where are we?

“Even the most detailed navigation map of an area is

useless if you don’t know where you are”

?

?

?

6

Benchmarking

Senior Management of IT departments/organizations need

to make decisions need to make decisions based on

‘where they are’ and ‘where they want to go’.

Benchmarking is about determining ‘where you are’

compared to relevant peers, in order to make informed

decisions.

But how to measure and determine where you are?

7

Software project industry

Low ‘performance metrics’ maturity Few Performance Measurement Process implemented

Few Benchmarking processes implemented

Most organizations don’t know how good or how bad they

are in delivering or maintaining software.

These organizations are not able to assess their

competitive position, nor able to make informed strategic

decisions to improve their competitive position.

8

But…

Best in Class organizations deliver software up to 30 times

more productively than Worst in Class organizations High Productivity, High Quality

More functionality for the users against lower costs

Shorter Time to Market – competitive advantage!

Worst in Class organizations will find themselves in trouble

in an increasingly competitive market

Outperformed by competition

Internal IT departments get outsourced

Commercial software houses fail to win new contracts

Important to know where you are! Benchmark is essential!

9

Performance means balance

Delivery to time and budget Project Productivity

Project Speed Project Quality

Actual vs. Estimated Cost

Actual vs. Estimated Duration

Size / Effort

Size / Duration Post delivery defects / Size

10

Difficulty – low industry maturity

How to measure metrics like productivity, quality, time-to-

market in such a way that a meaningful comparison is

possible?

Comparing apples to apples

11

12

Functional Size Measurement

Function Point Analysis (NESMA, IFPUG or COSMIC) Measure the functional user requirements – size in function points;

ISO standards – objective, independent, verifiable, repeatable;

Strong relation between functional size and project effort needed;

What to do with the results? Project effort/duration/cost estimation

Benchmarking/performance measurement

Use in Request for Proposal management (answer price/FP questions)

What about historical data? Company data (preferably for estimation)

Industry data (necessary for external benchmarking)

13

Unit of Measure (UoM) Why are Function Points the best UoM to use in Benchmarking? Functionality is of value for the client/business. More functionality means more

value. More Lines of code (technical size) is not necessarily of value.

Function Points are measured independent from technical requirements

500 FP of functionality implemented in Java SOA architecture

= 500 FP of functionality implemented in Cobol mainframe

Function Points are measured independent from implementation method

500 FP delivered in an agile development project

= 500 FP delivered in a COST package implementation

14

Software metrics – some examples

Productivity

Productivity Rate: #Function points per staff month

PDR: #Effort hours per function Point

Quality

Defect Density: #Defects delivered per 1000 function points

Time to Market

Speed: #Function points delivered per calendar month

15

Performance Measurement

Measure the size of completed projects

Project size in Function Points

Product size in Function Points

Collect and analyze the data

Effort hours, duration, defects

Normalize the data when necessary

Store the data in the corporate database

Benchmark the project, internally and externally

Report metrics and trends

Different reports for different stakeholders

Depending on goals of the stakeholder

16

Historical data: ISBSG repositories

International Software Benchmarking Standards Group

Independent and not-for-profit

Grows and exploits two repositories of software data: New development projects and enhancements (> 6000 projects)

Maintenance and support (> 1000 applications)

Everybody can submit project data DCQ on the site

Anonymous

Free benchmark report in return

17

ISBSG

Mission: “To improve the management of IT resources by both

business and government, through the provision and exploitation of

public repositories of software engineering knowledge that are

standardized, verified, recent and representative of current

technologies”.

All ISBSG data is validated and rated in accordance with its quality guidelines

current

representative of the industry

independent and trusted

captured from a range of organization sizes and industries

Industry leaders around the world contribute to the ISBSG’s

development, offering the highest metrics expertise worldwide

18

www.isbsg.org

19

Example – project benchmark

Project X was completed and the following data was collected:

Primary programming language: Java

Effort hours spent: 5730

Duration: 11 months

Defects found after delivery: 23

The functional size of the project was measured: 411 FP

Software metrics:

Project Delivery Rate: 5730/411 = 13,9 h/FP

Project Speed: 411/11 = 37,4 FP per calendar month

Defect Density: (23/411) *1000 = 56,0 defects/1000 FP

20

Example: Benchmark

ISBSG ‘New Developments & Enhancements’

Select the right ‘peer group’

Data Quality A or B

Count approach: IFPUG 4.x or NESMA

Primary Programming Language = ‘Java’

300 FP < Project Size < 500 FP

21

Results – project benchmark PDR

N 488

Minimum 0,1

Percentile 10 2,5

Percentile25 4,7

Median 9,8

Percentile 75 18,4

Percentile 90 28,9

Maximum 621,3

Average 15,2

Project Delivery Rate: 5730/411 = 13,9 h/FP

Project Speed: 411/11 = 37,4 FP per calendar month

Defect Density: (23/411) *1000 = 56,0 defects/1000 FP

Speed

N 428

Minimum 9,4

Percentile 10 23,1

Percentile 25 32,5

Median 53,8

Percentile 75 95,4

Percentile 90 130,2

Maximum 476,0

Average 70,9

Defect Density

N 154

Minimum 0,0

Percentile 10 0,0

Percentile 25 0,0

Median 3,7

Percentile 75 17,9

Percentile 90 40,1

Maximum 366,5

Average 18,6

This project was carried out less productive and slower than market

average, and the quality is worse than average.

22

Organization Benchmark

0,40,60,81,01,21,41,61,82,02,2

<2009 2009 2010 2011 2012

Pro

du

ctiv

ity

In

de

x

Organization Y - Productivity lndex

Organization Y PI Target (baseline +50%)

Industry Productivity Lower bound (baseline -40%)

0,4

0,6

0,8

1,0

1,2

1,4

1,6

1,8

2,0

2,2

<2009 2009 2010 2011 2012

Spee

d In

dex

Organization Y - Speed index

Organization Y Speed index Target (baseline +50%)

Industry Speed Lower bound (baseline -40%)

0,20,40,60,81,01,21,41,61,82,0

<2009 2009 2010 2011 2012

Qu

alit

y In

de

x

Organization Y - Quality lndex

organization Y - Quality index Target (baseline +50%)

Industry Quality level Lower bound (baseline -50%)

Analysis: - Until 2010, the organization was improving

- After 2010/2011, the trends go the wrong way

- Recommendation: find the cause and draw up

improvement plan

23

Other uses for Benchmarking

Vendor selection, based on productivity, speed or quality metrics

Definition of SLA agreements (or other KPI’s) based on market

average performance

Establish a baseline from which to measure future improvement

Explain to the client/business that a project was carried out in a

‘better-than-average’ way, while the client may perceive otherwise

24

Conclusions and final remarks

Benchmarking is essential in the strategic management of an

organization. It helps understanding the competitive position and it

help to identify the ‘problem areas’, or ‘improvement areas’.

There are proven tools, techniques, models and historical data

available to carry out benchmarks in a fairly low-cost way.

This presentation focused on software development, because this

is usually the main concern of the business/client. Benchmarking

of other aspects of IT is similar, but with other performance metrics.

Harold van Heeringen harold.van.heeringen@sogeti.nl

@haroldveendam

haroldveendam

haroldvanheeringen

Software Cost Engineer NESMA board member

ISBSG president

COSMIC International Advisory Council, NL representative