Lecture 2: testingBook: Chapter 9

What is testing?

Testing is not showing that there are no errors in the program.

Testing cannot show that the program performs its intended goal correctly.

So, what is testing?Testing is the process of executing the

program in order to find errors.A successful test is one that finds an error.

Some drawbacks of testing

There are never sufficiently many test cases.

Testing does not find all the errors. Testing is hard and takes a lot of time. Testing is still a largely informal task.

Black-Box (data-driven, input-output) testing

The testing is not based on the structure of the program (which is unknown).

In order to ensure correctness, every possible input needs to be tested - this is impossible!

The goal: to maximize the number of errors found.

White-Box testing

Is based on the internal structure of the program.

There are several alternative criterions for checking “enough” paths in the program.

Even checking all paths (highly impractical) does not guarantee finding all errors (e.g., missing paths!)

Some testing principles

A programmer should not test his/her own program. One should test not only that the program does what

it is supposed to do, but that it does not do what it is not supposed to.

The goal of testing is to find errors, not to show that the program is errorless.

No amount of testing can guarantee error-free program.

Parts of programs where a lot of errors have already been found are a good place to look for more errors.

The goal is not to humiliate the programmer!

Inspections and Walkthroughs

Manual testing methods. Done by a team of people. Performed at a meeting

(brainstorming). Takes 90-120 minutes. Can find 30%-70% of errors.

Code Inspection

Team of 3-5 people. One is the moderator.

He distributes materials and records the errors.

The programmer explains the program line by line.

Questions are raised. The program is analyzed

w.r.t. a checklist of errors.

An example: Sieve of Erathosthenes

2 3 5

6 - is dropped because it is divisible by 2.7 - is added to the next additional slot.8 - dropped (divisible by 2).9 - dropped (divisible by 3)....

The Sieve program

int P[101];int V=1; int N=1; int I;main() { while (N<101) { for (I=1;I<=N;I++) if (P[I]=0) { P[I]=V++ ; N++ ; }

else if (V%P[I]==0) I=N+1; else I++}}

Is this program correct?

Checklist for inspections

Data declarationAll variables declared?Default values

understood?Arrays and strings

initialized?Variables with similar

names?Correct initialization?

Control flowEach loop terminates?DO/END statements

match?

Input/outputOPEN statements correct?Format specification

correct?End-of-file case handled?

Walkthrough

Team of 3-5 people. Moderator, as

before. Secretary, records

errors. Tester, play the role

of a computer on some test suits on paper and board.

Selection of test cases (for white-box testing)

The main problem is to select a good coveragecriterion. Some options are:

Cover all paths of the program. Execute every statement at least once. Each decision has a true or false value at least

once. Each condition is taking each truth value at least

once. Check all possible combinations of conditions in

each decision.

Cover all the paths of the program

Infeasible.

Consider the flow diagram on the left.

It corresponds to a loop.

The loop body has 5 paths.

If the loops executes 20

times there are 5^20 different paths!

May also be unbounded!

How to cover the executions?

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Choose values for A,B,X. Value of X may change, depending on

A,B. What do we want to cover? Paths?

Statements? Conditions?

Edge coverageExecute every statement at least once

By choosingA=2,B=0,X=3each statement will

be chosen.The case that the

tests fail is not checked!

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Decision coverageEach decision has a true and false outcome at least once.

Can be achieved using A=3,B=0,X=3 A=2,B=1,X=1

Problem: Does not test individual conditions. E.g., when X>1 is erroneous in second decision.

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Condition coverageEach condition has a true and false value at least once.

For example: A=1,B=0,X=3 A=2,B=1,X=0

lets each condition be true and false once.

Problem:covers only the path where the first test fails and the second succeeds.

IF (A>1)(A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Multiple Condition CoverageTest all combinations of all conditions in each test.

A>1,B=0 A>1,B≠0 A<=1,B=0 A<=1,B≠0 A=2,X>1 A=2,X<=1 A≠2,X>1 A≠2,X<=1

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

A smaller number of cases: A=2,B=0,X=4 A=2,B=1,X=1 A=1,B=0,X=2 A=1,B=1,X=1Note the X=4 in the

firstcase: it is due to the

factthat X changes beforebeing used!

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

How to find values for coverage?

A>1 & B=0

A=2 | X>1

X=X+1

X=X/Ano

no

yes

yes

true

true

A=2 | X>1

(A=2 | X>1) & ¬(A>1 & B=0)•Put true at end of path.

•Propagate path backwards.

•On assignment, relativize expression.

•On “yes” edge of decision, add decision as conjunction.

•On “no” edge, add negation of decision as conjunction.

•May need to give more specific choice of covering condition, which ones are true or false.

How to find values for coverage?

A>1 & B=0

A=2 | X>1

X=X+1

X=X/Ano

no

yes

yes

true

true

A≠2 & X>1

(A≠2 & X/A>1) & (A>1 & B=0)

A≠2 & X/A>1Need to find a

satisfying assignment:

A=3, X=6, B=0

Test case design for black box testing

Equivalence partition Boundary value analysis Cause-effect graphs

Test cases based on data-flow analysis

Partition the program into pieces of code with a single entry/exit point.

For each piece find which variables are set/used/tested.

Various covering criteria: from each set to each

use/test From each set to

some use/test.

Equivalence partition

Goals: Find a small number of test cases. Cover as much possibilities as you can.

Try to group together inputs for which the program would likely to behave the same.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Example: A legal variable

Begins with A-Z Contains [A-Z0-9] Has 1-6 characters.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Starting char

Chars

Length

Starts A-Z Starts other

[A-Z0-9] Has others

1-6 chars 0 chars, >6 chars

1 2

3 4

56 7

Equivalence partition (cont.)

Add a new test case until all valid equivalence classes have been covered. A test case can cover multiple such classes.

Add a new test case until all invalid equivalence class have been covered. Each test case can cover only one such class.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

AB36P (1,3,5) 1XY12 (2) A17#%X (4)

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Starting char

Chars

Length

Starts A-Z Starts other

[A-Z0-9] Has others

1-6 chars 0 chars, >6 chars

1 2

3 4

56 7

(6) VERYLONG (7)

Boundary value analysis

In every element class, select values that are closed to the boundary. If input is within range -1.0 -- +1.0,

select values -1.001, -1.0, -0.999, 0.999, 1.0, 1.001.

If needs to read N data elements, check with N-1, N, N+1. Also, check with N=0.