Chapter 14Testing Tactics

• What is a “Good” test?

14.1 Software Testing Fundamentals

A good test has a high probability of finding an error

A good test is not redundant

A good test should be “best of breed”

A good test should be neither too simple nor too complex

• Test Case DesignOBJECTIVE: to uncover errors

CRITERIA: in a complete manner

CONSTRAINT: with a minimum of effort and time

1/21

14.1 Software Testing Fundamentals

• Testability

Operability - the better it works the more efficiently it can be tested

Observability - what you see is what you test

Controllability - the better software can be controlled the more testing can be automated and optimized

Decomposability - by controlling the scope of testing, the more quickly problems can be isolated and retested intelligently

Simplicity - the less there is to test, the more quickly we can test

Stability - the fewer the changes, the fewer the disruptions to testing

Understandability - the more information known, the smarter the testing

2/21

14.2 Black-box an White-box Testing

• Why not try exhaustive testing?

loop = 20 XThere are 5201014 possible paths! If we execute one test per millisecond, it would take 3,170 years to test this program!!

Note: Testing cannot show the absence of errors and defects.

3/21

14.3 White-box Testing

• Definition: White-box or Glass-box Testing Knowing the internal workings of a product, tests are performed to

check the workings of all possible logic paths -- testing in the small.• Questions Can you guarantee that all independent paths within a module

will be executed at least once?

Can you exercise all logical decisions on their true and false branches?

Will all loops execute at their boundaries and within their operational bounds?

Can you exercise internal data structures to ensure their validity?

4/21

14.4 Basis Path Testing

• Goal: execute every statement at least once

TOTAL=TOTAL+A

K++

Input A

Output K, L, TOTAL

Stop

T

Begin

K=0L=0

TOTAL=0

Input A

Do While TOTAL<1000 and

A0F

A>0T

L++

F

a （ begin ）b （ in ）

c

d

e

f

g

h

i

j （ out）k （ stop）

a

b

c

d

e

f

g

h

i

j

k

flow graph

Predicate node

d’

R1R2

R3

R4

Compound conditions

Cyclomatic Complexity =

# of regionsE N + 2P + 1

Defines the number of independent paths in the basis

set of a program and provides us with an upper bound for the number of tests

that must be conducted.

5/21

14.4 Basis Path Testing

• Prepare test cases that will force execution of each path in the basis set – only to execute every statement at least once is NOT enough

Example:

Note: If AND is written as OR, the error will not be detected by the above test case.

in

A > 1AND B=0

T

A=2OR X > 1

T

X = X / A

X = X + 1

out

F

F

Test case : A=3, B=0, X=4.

CC = ?5

6/21

14.5 Control Structure Testing

• Condition Testing: test every condition at least once

Examples: Test ( E1 > E2 ) AND ( E3 = E4 )

B1 AND B2{ ( t, t ), ( f, t ), ( t, f ) }

B1 AND ( E3 = E4 ){ ( t, = ), ( f, = ), ( t, < ), ( t, > ) }

( E1 > E2 ) AND ( E3 = E4 )

{ ( >, = ), ( =, = ), ( <, = ), ( >, < ), ( >, > ) }

7/21

14.5 Control Structure Testing

• Data Flow Testing: selects test paths according to the locations of variable definitions and uses in the program

DEF(S) = { X| S contains a definition of X };

USE(S) = { X| S contains a use of X }.

DU chain of X = [X, S, S’] where XDEF(S) and XUSE(S’), and the definition of X in S is live at S’ ---- the flow of X.

Every DU chain is covered once.

8/21

14.5 Control Structure Testing

• Loop Testing

Simple loop

Concatenated Loops Unstructured

Loops

Nested Loops

012

2<m<nn-1n

n+1

0,1

9/21

14.6 Black-box Testing

• Definition: Black-box or Behavioral TestingKnowing the specified function a product is to perform and demonstrating correct operation based solely on its specification without regard for its internal logic – testing in the large.

• Questions How is functional validity tested?

How is system behavior and performance tested?

What classes of input will make good test cases?

Is the system particularly sensitive to certain input values?

How are the boundaries of a data class isolated?

What data rates and data volume can the system tolerate?

What effect will specific combinations of data have on system operation?

10/21

14.6 Black-box Testing

• Techniques

Graph-Based Testing

Equivalence Partitioning

Boundary Value Analysis

Orthogonal Array Testing

11/21

14.6 Black-box Testing

Equivalence Partitioning An equivalence class represents a set of valid or invalid states for input conditions, so that there is no particular reason to choose one element over another as a class representative.

Guidelines

• If an input condition specifies a range, one valid and two invalid equivalence classes are defined.

• If an input condition requires a specific value, one valid and two invalid equivalence classes are defined.

• If an input condition specifies a member of a set, one valid and one invalid equivalence classes are defined.

• If an input condition is Boolean, one valid and one invalid equivalence classes are defined.

12/21

14.6 Black-box Testing

Boundary Value Analysis (BVA)

"Bugs lurk in corners and congregate at boundaries ..."

Boris Beizer

Guidelines

• If input condition specifies a range bounded by values a and b, test cases should include a and b, values just above and just below a and b.

• If an input condition specifies and number of values, test cases should be exercise the minimum and maximum numbers, as well as values just above and just below the minimum and maximum values.

• Apply guidelines 1 and 2 to output conditions, test cases should be designed to produce the minimum and maxim output reports.

• If internal program data structures have boundaries (e.g., size limitations), be certain to test the boundaries.

13/21

14.7 Object-Oriented Testing Methods

• Test Case Design1. Each test case should be uniquely identified and should be explicitly associated with the class to be tested,2. The purpose of the test should be stated,3. A list of testing steps should be developed for each test and should contain [BER94]:

a. a list of specified states for the object that is to be testedb. a list of messages and operations that will be exercised as

a consequence of the testc. a list of exceptions that may occur as the object is testedd. a list of external conditions e. supplementary information that will aid in understanding or

implementing the test.

Unlike conventional test case design, which is driven by an I-P-O view of software or the algorithmic detail of individual modules, OO testing focuses on designing appropriate sequences of operations to exercise the states of a class.

14/21

14.7 Object-Oriented Testing Methods

• Fault-Based Testing– The tester looks for plausible faults (i.e., aspects of the

implementation of the system that may result in defects). To determine whether these faults exist, test cases are designed to exercise the design or code.

• Class Testing and the Class Hierarchy– Inheritance does not obviate the need for thorough testing of

all derived classes. In fact, it can actually complicate the testing process.

• Scenario-Based Test Design – Scenario-based testing concentrates on what the user does,

not what the product does. This means capturing the tasks (via use-cases) that the user has to perform, then applying them and their variants as tests.

15/21

14.8 Class Level Testing Methods

• Random testing– identify operations applicable to a class– define constraints on their use– identify a minimum test sequence

• an operation sequence that defines the minimum life history of the class (object)

– generate a variety of random (but valid) test sequences• exercise other (more complex) class instance life histories

Requires large number of data permutations and

combinations and can be inefficient.

16/21

14.8 Class Level Testing Methods

• Partition Testing– reduces the number of test cases required to test a class in

much the same way as equivalence partitioning for conventional software

– state-based partitioning• categorize and test operations based on their ability to

change the state of a class– attribute-based partitioning

• categorize and test operations based on the attributes that they use

– category-based partitioning• categorize and test operations based on the generic function

each performs

17/21

14.9 Inter-class Test Case Design

• Multiple Class Testing (random)

– For each client class, use the list of class operators to generate a series of random test sequences. The operators will send messages to other server classes.

– For each message that is generated, determine the collaborator class and the corresponding operator in the server object.

– For each operator in the server object (that has been invoked by messages sent from the client object), determine the messages that it transmits.

– For each of the messages, determine the next level of operators that are invoked and incorporate these into the test sequence.

18/21

14.9 Inter-class Test Case Design

• Behavior Testing

– The tests to be designed should achieve all state coverage [KIR94].

emptyacctopen setup Accnt

set upacct

depos it(init ial)

work ingacct

withdrawal(f inal)

deadacct c lose

nonwork ingacct

depos it

withdrawbalance

creditaccntInfo

Figure 14.12 State diagram f or Account class (adapted f rom [ KIR94])

19/21

14.10 Specialized Testing

• Graphical User Interfaces

• Client/Server Architectures

• Documentation and Help Facilities

• Real-Time Systems

14.11 Testing Patterns

• Define the patterns and REUSE

20/21

Final Words About Testing

Testing never ends, it just gets transferred from you [ the software engineer ] to your customer. Every time your customer uses the program, a test is being conducted.

21/21