1

Proving Our Worth

Quantifying The

Value Of Testing

Lee Copeland lee@sqe.com

2

• Recently, I visited

MegaCorp, my new

consulting client. I’d

been asked to review

their testing process.

• Here’s what I heard:

MegaCorp

3

My Interview With Rajan

―Testers are such a nuisance.

They waste time, create bugs,

get in the way of shipping our

products, cost too much, and

make our developers feel bad.‖ Project Manager

4

My Interview With Yunshu

―Project Managers don’t have a

clue. They don’t see the value in

our work. And why can’t they

make up their minds — one day

quality is the most important

thing, the next day shipping the

product is.‖

Test Manager

5

• I hear these verbal bombardments a lot.

• Perhaps you hear them too.

• Perhaps they are flying overhead in your

organization right now.

―The Bombs Bursting In Air‖

Idiots Turkeys Fools Losers

6

• In The Art of Software Testing,

Glenford Myers defined testing

as ―the process of executing a

program or system with the

intent of finding errors.‖

• Bill Hetzel expanded the

definition, ―Testing is any activity

aimed at evaluating an attribute

of a process or a system.‖

Definitions Of Testing

7

• More recently, Rick Craig and

Stefan Jaskiel wrote, ―Testing

is a concurrent lifecycle

process of engineering, using,

and maintaining testware in

order to measure and improve

the quality of the software

being tested.‖

Definitions Of Testing

8

• Note the similarities:

– The process of finding

– The process of evaluating

– The process of measuring

– The process of improving

• For forty years we as testers have been

focusing on the internal process of testing,

while generally disregarding its real

purpose.

Definitions Of Testing

9

• The real purpose of testing is:

• James Bach wrote, ―The ultimate

reason testers exist is to provide

information that others on the

project use to create things of

value.‖

The Real Purpose Of Testing

TO CREATE VALUABLE INFORMATION

10

Perhaps What He’s Trying To Say

―Testers are such a nuisance.

They waste time, create bugs,

get in the way of shipping our

products, make our developers

feel bad, and cost too much.‖ Project Manager

―I’m not getting enough

valuable information to justify

the cost of testing.‖

11

Perhaps What She’s Trying To Say

―Project Managers don’t have a

clue. They don’t see the value in

our work. And why can’t they

make up their minds — one day

quality is the most important

thing, the next day shipping the

product is.‖

Test Manager

―Project Managers don’t value

the information I’m providing to

them.‖

12

• The value of testing is the value of

the information we create for others.

• We do not determine its value; THEY DO.

• In a utopian world, the value of information

would be ―measured by observing

differences in the decision maker’s

performance when provided with different

types of information.‖

The Value Of Testing

– Kamran Parsaye

13

• Most testing organizations do not quantify

that value and thus cannot communicate it to

executive management.

• Rather than having solid economic data, all

we have are ―campfire stories,‖ anecdotes

that may be interesting around the campfire

but that are ineffective in the boardroom.

The Value Of Testing

14

• Developers

• Development management

• Testers

• Test management

• Project management

• Quality assurance

• Executive management

Possible Clients For Information

15

• Rather than inundating clients with piles of

information, let’s first begin by understanding

their needs.

• The Goal-Question-Metric model originated

by Dr. Victor Basili is an excellent way to

start.

The GQM Model

16

• GOAL

– Ask, what are your quality related goals?

• QUESTION

– What question(s) would you have to be able to

answer to know if you were meeting those goals?

• METRIC

– What would you have to measure to answer those

questions?

The GQM Model

17

• Give your clients a Magic Wand.

• Or try the ―Magical Information

Machine‖

– It can answer any of your quality

questions regarding the systems you’re

developing.

– What questions would you like it to

answer?

If Your Clients Are Stuck

18

• Aside from specific types of information,

what would be general attributes of all (or

almost all) valuable information?

– Accurate

– Timely / Pre-emptive

– Complete

The Value Of Information

Lookout on

the Titanic

―There are no

icebergs in the

vicinity,

Captain.‖

(As the ship is

sinking)

―There are

icebergs in the

vicinity,

Captain.‖

―There are no

icebergs on

the port side,

Captain.‖

19

• … general attributes? (continued)

– Relevant

– Unique

– Actionable

The Value Of Information

Lookout on

the Titanic

―Look Captain,

the Northern

Lights. Aren’t

they

beautiful?‖

―It’s cold out

here tonight,

Captain.‖

―Captain!

Icebergs on

the starboard

side, coming

at us fast.‖

20

• … general attributes? (continued)

– Client-oriented — expressed in terms meaningful

and useful to our clients.

The Value Of Information

• # Test cases written /

executed / passed /

failed / blocked

• # Sev 1 / 2 / 3 defects

• DDP/DRE / Other TLAs

• Branch coverage

• Loss of sales, revenue,

market share

• Claims for compensation

• Cost of restorative actions

• Cost of repair and

redistribution

21

The Developer

• Goal:

My goal is to get my code out on time without

making any real blunders in it. I don’t want the

customer finding them. Then I look foolish in front

of the whole team, especially my manager.

22

• Questions / Metrics:

– How many defects have I created?

– Are my defects scattered randomly throughout the

code or are they clumped in a few modules?

– Am I making random mistakes or am I creating the

same type of defects again and again?

The Developer

23

• Value:

Quantifying the value of this information is difficult

for me. Having information about the defects I

created could help make fewer mistakes in the

future. If it cost x to find and fix each one of my

defects, and if I made y fewer mistakes, then the

information has saved xy dollars so that would be

its value. But I don’t know what x and y are and I

don’t know how to find out.

The Developer

24

The Test Manager

• Goal:

My goal is to ensure that my testers find most of

the defects and that very few, if any, escape into

production.

25

• Questions / Metrics:

– What percent of defects are being found during

system testing, before system testing, and after

system testing, that is, in production?

– What test techniques are effective at finding

defects?

– How did the defects that escaped get by us?

– Where are the holes in our testing net?

The Test Manager

26

• Value:

If the percentage of defects we found during testing

was very high compared to those that escaped into

production, that makes my test team look good. But, I

know how this will play out. My management won’t

see any real value in this information; in fact, they will

probably use it against me. They will quickly forget

how many defects we’re finding and focus just on the

one’s we’ve missed — ―That was a very critical

defect, how did you miss that one?‖ ―It’s apparent

your team can only find the unimportant bugs. Our

users find the real ones.‖ Perhaps we’d all be better

off not reporting this information.

The Test Manager

27

The Project Manager

• Goal:

My goals are a high quality product that gives our

customers a good experience while solving their

problems, and doesn’t create major post-release

support costs.

28

• Questions / Metrics:

– How many defects have we found so far?

– What would their impact be on our organization if

they had escaped into production?

– How many more serious defects remain

undiscovered?

– What would their impact be?

The Project Manager

29

• Value:

If we could add up the cost impact of each of the

defects had they escaped to the user and caused

problems in the field, then subtract the cost of

finding them, that would be the value of the

information that testing created for the

organization (assuming others acted on that

information to improve the product’s quality).

The Project Manager

30

• These three scenarios show us that

quantifying the value of testing is difficult

work.

• Perhaps that’s why we concentrate so much

on the testing process — it’s much easier.

• But, until we do this difficult work,

we should expect the bombardment

to continue — and we as testers

will be the target.

Quantifying The Value Of Testing

31

Thanks

Shameless

commercial

messages

lee@sqe.com

32

Bibliography

Bach, James. ―Value Without Numbers.‖ STQE Magazine, July/August 2002 (Vol. 4

Issue 4).

Basili, Victor and D.M. Weiss. ―A Methodology for Collecting Valid Software

Engineering Data.‖ TR-1235, University of Maryland, Technical Report,

December 1982, 22 pages.

Craig, Rick D. and Stefan P. Jaskiel. Systematic Software Testing. Artech House,

2002. ISBN 1-58053-508-9.

Hendrickson, Elisabeth. ―I’ve Looked at Bugs from Both Sides Now.‖ STARWest,

2004.

Hetzel, William. The Complete Guide to Software Testing. QED Information

Sciences, 1983. ISBN 0-89435-110-9.

Myers, Glenford. The Art of Software Testing. John Wiley & Son, 1979. ISBN 0-471-

04328-1.

Parsaye, Kamran and David Petrie. ―Measuring the Value of Mined Information.‖ DM

Review Magazine, March 1998.

33

Acknowledgements

• Thanks to Google.com for helping

find information and images used

in this presentation.

• All were used in compliance with the Fair Use

section of the U.S. Copyright Act as found in the

U.S. Code Title 17, Chapter 1, Section 107:

―Notwithstanding the provisions of sections 106 and 106A, the fair

use of a copyrighted work, including such use by reproduction in

copies or phonorecords or by any other means specified by that

section, for purposes such as criticism, comment, news reporting,

teaching (including multiple copies for classroom use),

scholarship, or research, is not an infringement of copyright.‖