software engineering ch14

3/18/20093/18/2009 S.Sreenivasa RaoS.Sreenivasa Rao 1

Chapter 14Chapter 14Software Testing TechniquesSoftware Testing Techniques


Testability:Testability: how easily a software can be how easily a software can be testedtested

OperabilityOperability the better it works, the more efficiently it can be tested. the better it works, the more efficiently it can be tested. -- It It operates cleanlyoperates cleanly

ObservabilityObservabilityWhat you see is what you test. The results of each test What you see is what you test. The results of each test case are readily observed (clear and visible variables and statecase are readily observed (clear and visible variables and states, incorrect s, incorrect output, output, ))

ControllabilityControllabilitythe degree to which testing can be automated and the degree to which testing can be automated and optimizedoptimized

DecomposabilityDecomposabilitytesting can be targeted to independent modulestesting can be targeted to independent modules SimplicitySimplicityreduce complex architecture and logic to simplify testsreduce complex architecture and logic to simplify tests StabilityStabilityfew changes are requested during testingfew changes are requested during testing UnderstandabilityUnderstandabilityof the designof the design


What is a What is a GoodGood Test?Test?

A good test has a high probability of finding an A good test has a high probability of finding an errorerror

A good test is not redundant.A good test is not redundant. A good test should be A good test should be best of breedbest of breed (best (best

between those with similar intent) between those with similar intent) A good test should be neither too simple nor A good test should be neither too simple nor

too complextoo complex


Test Case DesignTest Case Design"Bugs lurk in corners "Bugs lurk in corners and congregate at and congregate at boundaries ..."boundaries ..."

Boris Boris BeizerBeizer

OBJECTIVEOBJECTIVE

CRITERIACRITERIA

CONSTRAINTCONSTRAINT

to uncover errorsto uncover errors

in a complete mannerin a complete manner

with a minimum of effort and timewith a minimum of effort and time


Exhaustive TestingExhaustive Testing

loop < 20 Xloop < 20 X

There are 10 possible paths! If we execute oneThere are 10 possible paths! If we execute onetest per millisecond, it would take 3,170 years totest per millisecond, it would take 3,170 years totest this program!!test this program!!

1414


Selective TestingSelective Testing

loop < 20 Xloop < 20 X

Selected pathSelected path


Software TestingSoftware Testing

Methods

Strategies

white-boxmethods

black-boxmethods


WhiteWhite--Box TestingBox Testing

... our goal is to ensure that all ... our goal is to ensure that all statements and conditions have statements and conditions have been executed at least once ...been executed at least once ...


Why Cover?Why Cover?logic errors and incorrect assumptions logic errors and incorrect assumptions are inversely proportional to a path's are inversely proportional to a path's execution probabilityexecution probability

we often we often believe believe that a path is not that a path is not likely to be executed; in fact, reality is likely to be executed; in fact, reality is often counter intuitiveoften counter intuitive

typographical errors are random; it's typographical errors are random; it's likely that untested paths will contain likely that untested paths will contain some some

3/18/20093/18/2009 S.Sreenivasa RaoS.Sreenivasa Rao

Basic Path TestingBasic Path Testing

Define a basic set of execution pathsDefine a basic set of execution paths Guarantee that every statement is Guarantee that every statement is

executed at least onceexecuted at least once Using a logical complexity measure of a Using a logical complexity measure of a

procedural design and use it as a guide for procedural design and use it as a guide for defining a basic set of execution path defining a basic set of execution path

10


Flow Graph NotationFlow Graph Notation

A representation for control flow of A representation for control flow of programprogram

Each structured construct Each structured construct --> a flow graph > a flow graph symbolsymbol

11


Independent program pathIndependent program path

Any path through the program that introduces at Any path through the program that introduces at least one new set of processing statements or a least one new set of processing statements or a new condition.new condition.

An independent path must move along at least An independent path must move along at least one edge that has not been traversed before the one edge that has not been traversed before the path is defined.path is defined.

The basic set is not unique.The basic set is not unique. How do we know how many paths to look for?How do we know how many paths to look for?

CyclomaticCyclomatic complexitycomplexity12


Basis Path Testing: Basis Path Testing: CyclomaticCyclomaticComplexityComplexityFirst, we compute the cyclomaticcomplexity:

number of simple decisions + 1 (simple decisions: predicate node)

or

number of enclosed areas + 1or

E-N+2 (E number of edges, N number of nodes)

In this case, V(G) = 4


CyclomaticCyclomatic ComplexityComplexityA number of industry studies have indicated A number of industry studies have indicated that the higher V(G), the higher the probability that the higher V(G), the higher the probability or errors.or errors.

V(G)V(G)

modulesmodules

modules in this range are modules in this range are more error pronemore error prone


Basis Path TestingBasis Path TestingNext, we derive the Next, we derive the independent paths:independent paths:

Since V(G) = 4,Since V(G) = 4,there are four pathsthere are four paths

Path 1: 1,2,3,6,7,8Path 1: 1,2,3,6,7,8Path 2: 1,2,3,5,7,8Path 2: 1,2,3,5,7,8Path 3: 1,2,4,7,8Path 3: 1,2,4,7,8Path 4: 1,2,4,7,2,4,...7,8Path 4: 1,2,4,7,2,4,...7,8

Finally, we derive testFinally, we derive testcases to exercise these cases to exercise these paths.paths.

11

22

3344

55 66

77

88


Basis Path Testing NotesBasis Path Testing Notes

you don't need a flow chart, you don't need a flow chart, but the picture will help when but the picture will help when you trace program pathsyou trace program paths

count each simple logical test, count each simple logical test, compound tests count as 2 or compound tests count as 2 or moremore

basis path testing should be basis path testing should be applied to critical modulesapplied to critical modules


Deriving test casesDeriving test cases

Using the design or code, draw a flow Using the design or code, draw a flow diagramdiagram

Determine the Determine the cyclomaticcyclomatic complexity of complexity of the flow graphthe flow graph

Determine a basis set of independent Determine a basis set of independent pathspaths

Prepare test cases to force the execution Prepare test cases to force the execution of each pathof each path 17


Graph MatricesGraph Matrices A tabular representation of a flow graphA tabular representation of a flow graph A graph matrix is a square matrix whose size A graph matrix is a square matrix whose size

(i.e., number of rows and columns) is equal to (i.e., number of rows and columns) is equal to the number of nodes on a flow graphthe number of nodes on a flow graph Each row and column corresponds to an identified Each row and column corresponds to an identified

node, and node, and matrix entries correspond to connections (an edge) matrix entries correspond to connections (an edge)

between nodes. between nodes. By adding a By adding a link weightlink weight to each matrix entry, to each matrix entry,

the graph matrix can become a powerful tool the graph matrix can become a powerful tool for evaluating program control structure during for evaluating program control structure during testingtesting


Link weights can beLink weights can be The probability of execution of that edgeThe probability of execution of that edge The processing time for execution that edgeThe processing time for execution that edge The memory requiredThe memory required The resource requiredThe resource required

Help to design test casesHelp to design test cases19


Other Control Structure TestingOther Control Structure Testing

Condition testingCondition testing a test case design method a test case design method that exercises the logical conditions contained that exercises the logical conditions contained in a program modulein a program module

Data flow testingData flow testing selects test paths of a selects test paths of a program according to the locations of program according to the locations of definitions (D) and uses (U) of variables in the definitions (D) and uses (U) of variables in the program (every DU chain be covered at least program (every DU chain be covered at least once)once)

Loop testing Loop testing focus on validity of loop focus on validity of loop constructsconstructs


Loop TestingLoop Testingfour different classes of loopfour different classes of loop

Nested Nested LoopsLoops

ConcatenatedConcatenatedLoops Loops Unstructured Unstructured

LoopsLoops

Simple Simple looploop


Loop Testing: Simple Loop Testing: Simple LoopsLoopsMinimum conditionsMinimum conditionsSimple LoopsSimple Loops

1. skip the loop entirely1. skip the loop entirely2. only one pass through the loop2. only one pass through the loop3. two passes through the loop3. two passes through the loop4. m passes through the loop m < n4. m passes through the loop m < n5. (n5. (n--1), n, and (n+1) passes through 1), n, and (n+1) passes through the loopthe loop

where n is the maximum number where n is the maximum number of allowable passesof allowable passes


Loop Testing: Nested LoopsLoop Testing: Nested Loops

Start at the innermost loop. Set all outer loops to their Start at the innermost loop. Set all outer loops to their minimum iteration parameter values.minimum iteration parameter values.

Test the min+1, typical, maxTest the min+1, typical, max--1 and max for the 1 and max for the innermost loop, while holding the outer loops at their innermost loop, while holding the outer loops at their minimum values.minimum values.

Move out one loop and set it up as in step 2, holding all Move out one loop and set it up as in step 2, holding all other loops at typical values. Continue this step until other loops at typical values. Continue this step until the outermost loop has been tested.the outermost loop has been tested.

NestedNested LoopsLoops


Loop Testing: Concatenated Loop Testing: Concatenated LoopsLoops

If the loops are independent of one another If the loops are independent of one another then treat each as a simple loopthen treat each as a simple loopelse* treat as nested loopselse* treat as nested loops

endifendif* *

for example, the final loop counter value of loop 1 is for example, the final loop counter value of loop 1 is used to initialize loop 2.used to initialize loop 2.

Concatenated LoopsConcatenated Loops


BlackBlack--Box TestingBox Testing

requirementsrequirements

eventseventsinputinput

outputoutput


Incorrect or missing functionsIncorrect or missing functions Interface errorsInterface errors Errors in Data structures or external data Errors in Data structures or external data

base accessbase access Behavior or performance errorsBehavior or performance errors initialization and termination errorsinitialization and termination errors

26


BlackBlack--Box TestingBox Testing

How is functional validity tested?How is functional validity tested? How is system behavior and performance tested?How is system behavior and performance tested? What classes of input will make good test cases?What classes of input will make good test cases? Is the system particularly sensitive to certain input Is the system particularly sensitive to certain input

values?values? How are the boundaries of a data class isolated?How are the boundaries of a data class isolated? What data rates and data volume can the system What data rates and data volume can the system

tolerate?tolerate? What effect will specific combinations of data have on What effect will specific combinations of data have on

system operation?system operation?


GraphGraph--Based MethodsBased Methods

newfile

menu select generates

(generation time < 1.0 sec)document

window

documenttex

t

is represented as

contains

Attributes:

background color: whitetext color: default color

or preferences

(b)

object#1

Directed link

(link weight)object

#2

object#3

Undirected link

Parallel links

Node weight(value

)

(a)

allows editingof

To understand the To understand the objects that are objects that are modeled in modeled in software and the software and the relationships that relationships that connect these connect these objectsobjects

In this context, we In this context, we consider the term consider the term objectsobjects in the broadest in the broadest possible context. It possible context. It encompasses data encompasses data objects, traditional objects, traditional components (modules), components (modules), and objectand object--oriented oriented elements of computer elements of computer software.software.


Graph based testingGraph based testing

Transaction flow modelingTransaction flow modeling Finite state modelingFinite state modeling Data flow modelingData flow modeling Timing modeling (sequence of execution Timing modeling (sequence of execution

and its time)and its time)

29


Equivalence PartitioningEquivalence Partitioning

Divides the input domain of a program Divides the input domain of a program into classes of data from which test cases into classes of data from which test cases can be derived.can be derived.

Define test cases that uncovers classes of Define test cases that uncovers classes of errorserrors

30


If an Input condition isIf an Input condition is A rangeA range

One valid and two invalid equivalence classesOne valid and two invalid equivalence classes A specific valueA specific value

One valid and two invalid equivalence classesOne valid and two invalid equivalence classes A member of a setA member of a set

One valid and one invalid equivalence classesOne valid and one invalid equivalence classes BooleanBoolean

One valid and one invalid equivalence classesOne valid and one invalid equivalence classes 31


Boundary value checkingBoundary value checking

A greater number of errors occurs at the A greater number of errors occurs at the boundaries of the input domainboundaries of the input domain

32


Comparison Testing (NComparison Testing (N--versioning)versioning)

Used only in situations in which the reliability of Used only in situations in which the reliability of software is absolutely critical (e.g., humansoftware is absolutely critical (e.g., human--rated systems)rated systems) Separate software engineering teams develop Separate software engineering teams develop

independent versions of an application using the independent versions of an application using the same specificationsame specification

Each version can be tested with the same test data Each version can be tested with the same test data to ensure that all provide identical output to ensure that all provide identical output

Then all versions are executed in parallel with realThen all versions are executed in parallel with real--time comparison of results to ensure consistencytime comparison of results to ensure consistency


Orthogonal Array TestingOrthogonal Array Testing Used when the number of input parameters is Used when the number of input parameters is

small and the values that each of the small and the values that each of the parameters may take are clearly boundedparameters may take are clearly bounded

Dispersed uniformly throughout the test domain Dispersed uniformly throughout the test domain (balancing property)(balancing property)

One input item at a time L9 orthogonal array

XY

Z

XY

Z


OOTOOTTest Case DesignTest Case DesignBerardBerard [BER93] proposes the following approach:[BER93] proposes the following approach:

11.. Each test case should be uniquely identified and should be expliEach test case should be uniquely identified and should be explicitly citly associated with the class to be tested,associated with the class to be tested,

2.2. The purpose of the test should be stated,The purpose of the test should be stated,

3.3. A list of testing steps should be developed for each test and shA list of testing steps should be developed for each test and should ould contain [BER94]:contain [BER94]:

a.a. a list of specified states for the object that is to be testeda list of specified states for the object that is to be tested

b.b. a list of messages and operations that will be exercised as a list of messages and operations that will be exercised as a consequence of the testa consequence of the test

c.c. a list of exceptions that may occur as the object is testeda list of exceptions that may occur as the object is tested

d.d. a list of external conditions (i.e., changes in the environment a list of external conditions (i.e., changes in the environment external to the software that must exist in order to properly coexternal to the software that must exist in order to properly conduct nduct the test)the test)

e.e. supplementary information that will aid in understanding or supplementary information that will aid in understanding or implementing the test.implementing the test.


Testing MethodsTesting Methods FaultFault--based testing (same for conventional)based testing (same for conventional)

The tester looks for plausible faults The tester looks for plausible faults aspects of the implementation of the system that may result aspects of the implementation of the system that may result

in defects. in defects. To determine whether these faults exist, test cases are designedTo determine whether these faults exist, test cases are designed

to exercise the design or code. to exercise the design or code.

Class Testing and the Class HierarchyClass Testing and the Class Hierarchy Inheritance does not obviate the need for thorough testing of alInheritance does not obviate the need for thorough testing of all l

derived classes. derived classes. In fact, it can actually complicate the testing process.In fact, it can actually complicate the testing process.

ScenarioScenario--Based Test DesignBased Test Design ScenarioScenario--based testing concentrates on what the user does, not based testing concentrates on what the user does, not

what the product does. what the product does. This means capturing the tasks (via useThis means capturing the tasks (via use--cases) that the user has cases) that the user has

to perform, then applying them and their variants as tests.to perform, then applying them and their variants as tests.


OOT Methods: Random Testing class basedOOT Methods: Random Testing class based

Random testing: Random testing: check class instance life historiescheck class instance life histories identify operations applicable to a classidentify operations applicable to a class define constraints on their usedefine constraints on their use identify a minimum test sequenceidentify a minimum test sequence

an operation sequence that defines the minimum life history of tan operation sequence that defines the minimum life history of the class (object)he class (object)

generate a variety of random (but valid) test generate a variety of random (but valid) test sequencessequences

exercise other (more complex) class instance life historiesexercise other (more complex) class instance life histories

Like for an instance of Account class: open, setup, depositLike for an instance of Account class: open, setup, deposit, , withdraw, close And open, setup, deposit, [deposit, withdraw, withdraw, close And open, setup, deposit, [deposit, withdraw, balance, balance, creditlimitcreditlimit], withdraw, close], withdraw, close


OOT Methods: Partition OOT Methods: Partition TestingclassTestingclass basedbased

Partition TestingPartition Testing reduces the number of test cases required to test a class in reduces the number of test cases required to test a class in

much the same way as equivalence partitioning for much the same way as equivalence partitioning for conventional softwareconventional software

statestate--based partitioningbased partitioning categorize and test operations based on their ability to change categorize and test operations based on their ability to change the state of a classthe state of a class

attributeattribute--based partitioningbased partitioning categorize and test operations based on the attributes that theycategorize and test operations based on the attributes that they useuse For example: partition the operations according to their use/modFor example: partition the operations according to their use/modify/donify/dont use or t use or

modify an attributemodify an attribute

categorycategory--based partitioningbased partitioning categorize and test operations based on the generic function eaccategorize and test operations based on the generic function each performsh performs For example: operations can be categorized as For example: operations can be categorized as

initialization/computation/queries/terminationinitialization/computation/queries/termination


OOT Methods: InterOOT Methods: Inter--Class TestingClass Testing(During integration)(During integration)

RandomRandom Like in class testingLike in class testing

PartitioningPartitioning In (intraIn (intra--)class testing: state)class testing: state--based, attributebased, attribute--based, based,

categorycategory--basedbasedHere:Here: Partition according to messages which serve different Partition according to messages which serve different

classesclasses StateState--basedbased

BehavioralBehavioral StateState--basedbased


OOT Methods: InterOOT Methods: Inter--Class Testing Class Testing (During integration)(During integration)

RandomRandom For each client class, use the list of class operators to For each client class, use the list of class operators to

generate a series of random test sequences. The operators generate a series of random test sequences. The operators will send messages to other server classes.will send messages to other server classes.

For each message that is generated, determine the For each message that is generated, determine the collaborator class and the corresponding operator in the collaborator class and the corresponding operator in the server object.server object.

For each operator in the server object (that has been For each operator in the server object (that has been invoked by messages sent from the client object), determine invoked by messages sent from the client object), determine the messages that it transmits.the messages that it transmits.

For each of the messages, determine the next level of For each of the messages, determine the next level of operators that are invoked and incorporate these into the operators that are invoked and incorporate these into the test sequencetest sequence


OOT Methods: Behavior TestingOOT Methods: Behavior Testingemptyacctopen setup Accnt

set upacct

deposit(initial)

workingacct

withdrawal(final)

deadacct close

nonworkingacct

deposit

withdrawbalance

creditaccntInfo

Figure 14.3 St at e diagram f or Account class (adapt ed f rom [ KIR94] )

The tests to be The tests to be designed should designed should achieve all state achieve all state coverage [KIR94]. coverage [KIR94]. That is, the That is, the operation operation sequences should sequences should cause the cause the Account class to Account class to make transition make transition through all through all allowable statesallowable states

Chapter 14Software Testing TechniquesTestability: how easily a software can be testedWhat is a Good Test?Test Case DesignExhaustive TestingSelective TestingSoftware TestingWhite-Box TestingWhy Cover?Basic Path TestingFlow Graph NotationIndependent program pathBasis Path Testing: Cyclomatic ComplexityCyclomatic ComplexityBasis Path TestingBasis Path Testing NotesDeriving test casesGraph MatricesOther Control Structure TestingLoop Testingfour different classes of loopLoop Testing: Simple LoopsLoop Testing: Nested LoopsLoop Testing: Concatenated LoopsBlack-Box TestingBlack-Box TestingGraph-Based MethodsGraph based testingEquivalence PartitioningBoundary value checkingComparison Testing (N-versioning)Orthogonal Array TestingOOTTest Case DesignTesting MethodsOOT Methods: Random Testing class basedOOT Methods: Partition Testingclass basedOOT Methods: Inter-Class Testing(During integration)OOT Methods: Inter-Class Testing (During integration)OOT Methods: Behavior Testing

software engineering ch14

Documents