iwsm2014 defect density measurements using cosmic (thomas fehlmann)

1

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Defect Density Measurements Using COSMIC

Thomas M. Fehlmann, Zürich

Eberhard Kranich, Duisburg

Euro Project Office AG

E: [email protected]

H: www.e-p-o.com

mailto:[email protected]

http://www.e-p-o.com/

2

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Dr. Thomas Fehlmann

1981: Dr. Math. ETHZ

1991: Six Sigma for Software Black Belt

1999: Euro Project Office AG, Zürich

2001: Akao Price 2001 for original contributions to QFD

2003: SwissICT Expert for Software Metrics, ICTscope.ch

2004: Member of the Board QFD Institute Deutschland – QFD Architect

2007: CMMI for Software – Level 4 & 5

2011: Net Promoter® Certified Associate

2012: Member of the DASMA Board

2013: Vice-President ISBSG

1981: Dr. Math. ETHZ

1991: Six Sigma for Software Black Belt

1999: Euro Project Office AG, Zürich

2001: Akao Price 2001 for original contributions to QFD

2003: SwissICT Expert for Software Metrics, ICTscope.ch

2004: Member of the Board QFD Institute Deutschland – QFD Architect

2007: CMMI for Software – Level 4 & 5

2011: Net Promoter® Certified Associate

2012: Member of the DASMA Board

2013: Vice-President ISBSG

Dr. Thomas Fehlmann

3

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Mathematics and Computer Science

Emphasis on Mathematical Statistics

Mathematical Optimization

Theory of Polynomial Complexity of Algorithms

Working at T-Systems International GmbH in Bonn, Germany

Six Sigma Black Belt for Software Development

Software Quality Assurance Manager

Member of the DASMA Board

Eberhard KranichEberhard Kranich

Mathematics and Computer Science

Emphasis on Mathematical Statistics

Mathematical Optimization

Theory of Polynomial Complexity of Algorithms

Worked at T-Systems International GmbH in Bonn, Germany

Six Sigma Black Belt for Software Development

Software Quality Assurance Manager

Member of the DASMA Board

4

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

What is a Defect?

Defect = Behavior impacting expected or required functionality of software

How many bugs?

By counting the

size of defect repositories?

By number of entries???

5

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Software Testing as a Game

Tester sees selected sequences in

the UML sequence diagram

Tester can “walk” the data

movements when planning or

executing tests

Functionality becomes visible to the

agile team

Defects impacting functionality

become visible to testers

Functional

Process

Other

Application

Some

Device

8.// Move some data

9.// Move some data

10.// Move some data


Other

Device

6

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Functionality, Defect Size, and Defect Density

What happens if data movements

have defects?

Testers mark the data movement

where a defect has been detected

Same Metric:

ISO/IEC 19761 COSMIC

Functional

Process

Other

Application

Some

Device

8.// Move some data

Move some data



Other

Device

Functional Size Number of Data Movements needed to implement all FUR

Test Size Number of Data Movements executed in Tests

Test Story Collection of Test Cases aiming at certain FURs

Defect Count Number of Data Movements affected by some defect detected in a test story

7

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Defects Density Prediction?

Now he counts the defects!

And counts and adjusts test size

By ISO/IEC 19761 COSMIC

Functional

Process

Other

Application

Some

Device

8.// Move some data

9.// Move some data



Other

Device

How does he know

that he found

all the defects?

8

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Research Questions

What is Defect Density?

Defects per KDLOC?

What is Test Coverage?

Code lines executed by some test case?

9

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

SW Testing and SW Metrics

Counting practices for defect counting are undocumented

“Number of Defects Found” per Stages / with Tests / etc.

How do you count “Number of Defects”?

Is it simply the number of entries in a defect repository?

How can you avoid double reporting?

Or make sure two defects are reported twice and not in a single report?

A successor to the “Defect Measurement Manual” published by UKSMA in

October 2000 is under review: “Defect Measurement and Analysis Handbook”

By European cooperation

Important enhancement for ISBSG’s Data Collection!

10

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

SW Testing and SW Metrics

Counting practices for defect counting are undocumented

“Number of Defects Found” per Stages / with Tests / etc.

How do you count “Number of Defects”?

Is it simply the number of entries in a defect repository?

How can you avoid double reporting?

Or make sure two defects are reported twice and not in a single report?

A successor to the “Defect Measurement Manual” published by UKSMA in

October 2000 is under review: “Defect Measurement and Analysis Handbook”

By European cooperation

Important enhancement for ISBSG’s Data Collection!

11

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Learn & Remember

Software Development the Six Sigma Way

Use COSMICISO/IEC 19761

Benchmark!

Count Defects basedon Reference Model

12

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

As a … [Functional User] I want to … [get something done] Such that …[quality characteristic] So that … [value or benefit]

1) Q001 Propose Standard Destinations User of public transportation be able to store my preferred

destinations

they are valid for the Ticket

Shop

I no longer have to pay fees

when catched without tickets

2) Q002 Find Nearest Boarding Station User of public transportation locate nearest station with GPS that's being served right now I immediately can see whether

it's right

3) Q003 Process Payment Provider of transportation

services

give user access to their

preferred payment options

all payments are traceable in

Ticket Shop

they can manage spending

4) Q004 Issue Ticket User of public transportation get a valid ticket with settings from Ticket Shop I no longer have to pay fees

when catched without tickets

5) Q005 Show Ticket Ticket controller see the validated ticket I can check validity period and

travel range

I don't need to go into a dispute

with a client

Functional User Requirements

Example: The Ticket Apps

Customer’s Voice:

Give me a ticket subito!

Without much ado and questions

if I simply want to go home

13

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

The Ticket Apps

12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP

App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service Local Time

1.// Add New Destination

Traveler

2.// Check whether Destination Exists

3.// Collect Matching Destinations

4.// Show Matching Destinations

5.// Select Exact Destination

6.// Record Selected Destinations

7.// Ask App for a Ticket

8.// Propose Destinations

9.// Proposed Destinations

10.// Select Destination

Prepare

11.// GPS Coordinates

12.// Date & Time

13.// Ask GIS

14.// Nearest Boarding Station

15.// Search Connection

16.// Next Connection

17.// Request Ticket

18.// Send Pricing Proposal

19.// Proposed Price

20.// Confirm Price

21.// Confirm Transaction

22.// Issue Ticket

23.// Show Ticket

24.// Record Selected Destination

14

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

The Ticket Apps – Objects of Interest

Description Type

Wants to buy a ticket to ride Device User

Store a list of destinations that are valid for "getting home" Functional Process

List of standard stations eligible as travel destination Persistent Data

Finds nearest origin station Other Application

Buy the ticket needed for that route Functional Process

External ticket shop able to identify the App’s user Other Application

Finds connections between stations at a given time & day Other Application

Provides GPS coordinates where phone is located Device User

Synchronized time service located in smartphone Device User

App User

Home Destinations

Ticket Purchase

Ticket Shop

Phone's GPS

Timetable Service

Saved Stations

GIS Application

Local Time

15

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

The Ticket Apps – Prepare Destinations

App User Ticket Shop


Prepare





Home Destinations Saved Stations

6.// Record Selected Destination

Preparation:

Store standard destination stations on Smartphone

After checking with the Ticket Shop whether they exist or which stop to select

Involves Ticket Shop and local data ‘Standard Stations”

16

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

The Ticket App in the IFPUG Model

Boundary IFP=51

EI

EO

EQ

ILF

EIF

EQ1 / 2

Select DestinationEI

1 / 3

Enter New Destination

EI2 / 2

Request Ticket

EI2 / 1

Confirm Purchase

EIF2 / 12

Ticket Shop

EIF1 / 3

GIS Application

EIF1 / 5

Timetable Service

EIF1 / 1

Local Time

EO2 / 2

Boarding StationEO3 / 3

Mobile TicketEO

1 / 2

Show Matching Destinations

ILF1 / 2

Home Destinations

Less complicated than

UML Sequence Diagrams

Suits business better

Better boundary identification

between different layers

17

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Prepare Destination in the IFPUG Model

It is not obvious what happens

Two elementary process seem

unrelated

Nevertheless, they update an ILF

[Uh, they forgot the elementary

process for deleting obsolete

home destinations!]

Boundary IFP=19

EI

EO

EQ

ILF

EIF

ILF1 / 2

Home Destinations

EO

1 / 2

Show Matching Destinations

EI1 / 3

Enter New Destination

EIF2 / 12

Ticket Shop

18

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Buy Ticket in the IFPUG Model

Elementary processes stay

The ILF ‘Home Destinations’

becomes an EIF

Adding the two counts yields

19 + 42 > 51

IFPUG is useless for functional

sizing when developing Apps or

counting defects

IFPUG 4.3 is great for identifying

system boundaries splitting apps!Boundary IFP=42

EI

EO

EQ

ILF

EIF

EIF2 / 12

Ticket Shop

EIF1 / 3

GIS Application

EIF1 / 5

Timetable Service

EIF1 / 1

Local Time

EIF1 / 2

Home Destinations

EI2 / 2

Request Ticket

EI1 / 1

Confirm Purchase

EO2 / 2

Boarding StationEO2 / 3

Mobile Ticket

EQ1 / 2

Select Destination

19

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Requirements for a Defect Measurement Reference Model

It must be additive

The total size must be the sum of the components’ sizes.

It must be understandable

Support UML Sequence Diagrams

Works well with UML Use Cases

Also with Business Process Model & Notation (BPMN) 2.0

It must fit into agile delivery

Buglione-Trudel Matrix for managing agile projects

Story Cards for Sprints

Story Card for Prepare Destinations Test isReady

Draft isReady

ReviewDone

Final-ized

Appro-ved

Func-tional

R001-01: Prepare Destinations6 x reworked

Refactoring

Count:

5

0

0

Business Impact:

Functional Size:

Story Points:

App User Prepare Destinations Standard Stations Ticket Shop







The user prepares his ticket app by entering standard

destinations. The ticket shop is involved for resolvingstation names matches and avoid spelling errors

20

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Learn & Remember



Benchmark!


21

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

What is a Test?

A Software Test has

Several Test Stories

• Weighted by Customer’s Priority for the Test Story,

reflecting the value for the customer

Each Test Story has many Test Cases

• For various kind of test data

A Test Size attribute

• Number of data movements executed by test cases

A Test Coverage attribute

• Percentage of data movements covered with test cases

An Outcome

• Passed or Failed

– Passed: All responses according expectations

– Failed: at least one test case didn’t yield the expected response

22

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Sample Test Stories

Eight Test Stories for the Ticket Apps

Each Test Story has several Test Cases

Each Test Case has defined Test Data

The Expected Response is known as per Test Case

Test Story Case 1 Test Data Expected Response Case 2 Test Data Expected Response

CT-A Prepare CT-A.1 Find Nearest Station CT-A.1.1 Enter GPS exactly Returns correct station CT-A.1.2 Enter GPS nearby Returns nearest station

CT-A.2 Served Stations only CT-A.2.1 Select time of service Returns next available connection CT-A.2.2 Select time out of service Return next best served station

CT-A.3 Enter New Destination CT-A.3.1 Enter valid station name Destination stored CT-A.3.2 Enter invalid station name Destination rejected

CT-B Ticketing CT-B.1 Select Destination CT-B.1.1 Station of origin known Asks for destination CT-B.1.2 No destination selected Stops without contacting Ticket Shop

CT-B.2 Get Ticket CT-B.2.1 Both stations known Asks to confirm price CT-B.2.2 Boarding station undefined Doesn't ask for destination

CT-B.3 Price Calculation CT-B.3.1 Both stations known Presents price of next actual connection CT-B.3.2 Both stations known Display intermediate change stations

CT-B.4 Issue Ticket CT-B.4.1 Issue successful, ticket shown All credentials visible CT-B.4.2 Ticket Shop blocked Explanation given

CT-B.5 Payment Tests CT-B.5.1 Credit limit exceeded Returns exceed notice CT-B.5.2 Payment service out of order Returns warning

23

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

What is a Test Case?

A Test Cases has

Entry Data (“Test Data”)

• Explaining the environment for the test case

• Typically valid, invalid, borderline data

• Normal and disturbed communication services

A known Expected Response

• The response of the system is known in advance

A known sequence of data movements executed

• Defining Test Coverage

• Each Test Case has a Size

24

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

The sequence of data movements can be visualized in the sequence diagram

per test case

By double-

clicking the

respective

data movements

Find Route GIS Application Phone's GPS


9.// Date & Time

10.// Ask GIS


Local Time

What is a Test Case?

Test Story No. 1

User Stories

CT-A.1 Find Nearest Station Q001: Propose Standard Destinations Q002: Find Nearest Boarding Station Q003: Process Payment Q004: Issue Ticket Q005: Show Ticket Expected Response

CT-A.1.1 Enter GPS exactly E012 E012,X009 Returns correct station

CT-A.1.2 Enter GPS nearby E012,X009 Returns nearest station

CT-A.1.3 Enter malformatted GPS E006 Returns no station

CT-A.1.4 Enter far away GPS position E006,E007,X004,E008 E012,X009 Returns nearest station plus warning

CT-A.1.5 GPS not working E005,E006 E006 Returns warning

Test Story Contribution (CFP): 2 6 0 1 6 Test Size

Test Case Measurements

for Test Story CT-A.1

25

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Defect Reporting

When a defect has been identified

in a test case, it can be recorded by

double-clicking the suspect

sequence of data movements

User Stories

CT-B.1 Select Destination Q001: Propose Standard Destinations

CT-B.1.1 Station of origin known R001,X001,E005

CT-B.1.2 No destination selected R001,X001

CT-B.1.3 Valid boarding & destination W001,E005,R001,X001

CT-B.1.4 List of destinations X003,R001,W001

Test Story Contribution (CFP): 12

Test Case Measurements

for Test Story CT-B.1

Test Story No. 4

Expected Response CFP Name Label Description Name Label

Asks for destination 6

Stops without contacting Ticket Shop 2

Returns next available connection 15

All readable and visible 3 #003 Multiple References If a station contains more than one transport

mode, e.g., bus and train, both are recorded as

separate destinations

W001 Record Selected Destinations

Test Size 26 1

Defects Observed Data Movements Affected

Defect Count

26

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP

App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service


Traveler






7.// Ask App for a Ticket

8.// Propose Destinations

9.// Proposed Destinations

10.// Select Destination

Prepare


12.// Date & Time

13.// Ask GIS


Extract Report

Select Data Movements forTest Case CT-B.1.4: List of destinations

when executed in view of Q001: Propose Standard Destinations

This Test Case identifies Fault #003: Multiple Referencesaffecting Data Movement 6.// Record Selected Destinations

Finish

Record Defects

27

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Count Defects

You can count

One Defect

Per

Data Movement

Identified by

One Test Story

And never more…©2012-2014 Rhafiel

28

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Compare

This means

The more test stories you have, the more defects you can uncover

Test size is equally important as functional size

Defect density depends from test size

Test

Benchmarking

Test Status Summary

Total CFP: 24

Defects Pending for Removal: 3 Test Size in CFP: 185

Defects Found in Total: 3 Test Intensity in CFP: 7.7

Defect Density: 13% Test Coverage: 79%

29

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Test Stories

Go

al T

est

Co

vera

ge

Fin

d N

eare

st S

tatio

n

Ser

ved

Sta

tions

onl

y

Ent

er N

ew D

estin

atio

n

Sel

ect

Des

tinat

ion

Get

Tic

ket

Pric

e C

alcu

latio

n

Issu

e T

icke

t

Pay

men

t T

ests

Ach

ieve

d C

ove

rag

e

CT

-A.1

CT

-A.2

CT

-A.3

CT

-B.1

CT

-B.2

CT

-B.3

CT

-B.4

CT

-B.5

Q001 Propose Standard Destinations 0.52 2 9 12 8 9 7 0.55

Q002 Find Nearest Boarding Station 0.45 6 9 7 10 3 5 4 0.45

Q003 Process Payment 0.62 3 4 4 4 10 7 16 0.59

Q004 Issue Ticket 0.24 1 3 1 3 6 2 3 0.23

Q005 Show Ticket 0.29 6 3 2 1 9 3 3 0.31

Ideal Profile for Test Stories 0.18 0.20 0.26 0.40 0.38 0.51 0.36 0.40 Convergence Gap

0.18 0.2 0.2 0.4 0.4 0.5 0.4 0.4 0.06

0.10 Convergence Range

0.20 Convergence Limit

Test StoriesDeployment Combinator

User Stories

Toggle Measured Controls

Measuring Test Coverage

Achieved

Response 𝒚𝐸

Expected Response 𝒚Number of

data movements

executed

30

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Learn & Remember



Benchmark!


31

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Conclusions

You need more than one model when measuring modern software

ISO/IEC 19761 COSMIC is most appropriate for measuring defects

Other methods might be suitable for cost estimations

However, in these days, what drives cost more than faulty software?

Stop the nonsense with counting defects in repositories

Include defect density goals in contracts

Do independent auditing as long as you don’t trust your measurement equipment

Start Measuring Defects. Now.

32

Customer

Orientation

Lean

Six Sigma

Agile

Processes

Project

Estimations

Transfer

Functions

Questions?

iwsm2014 defect density measurements using cosmic (thomas fehlmann)

Documents