02 structural testing

8/3/2019 02 Structural Testing

1/23

Testing Tactics

Tests based on spec

Test covers as muchspecifiedbehavioras possible

Tests based on code

Test covers as muchimplementedbehavioras possible

Why Structural?

If a part of the program is never executed, adefect may loom in that partA part can be a statement, function, transition. condition

Attractive because automated

From Pressman, Software Engineeringa practitioners approach, Chapter 14and Pezze + Young, Software Testingand Analysis, Chapters 12-13

1

In contrast to functional tests(discussedthe last time), structuraltests are basedon the code structure

2

Structural tests are automated andcan be much more fine-grained than

functional tests.

3


2/23

Why Structural?

Complements functional testsRun functional tests first, then measure what is missing

Can cover low-level details missed in high-level specification

A Challenge

class Roots {

// Solve ax2 + bx + c = 0

public roots(double a, double b, double c)

{ }

// Result: values for xdouble root_one, root_two;

}

Which values for a, b, cshould we test?assuming a, b, c, were 32-bit integers, wed have (232)3 1028 legal inputswith 1.000.000.000.000 tests/s, we would still require 2.5 billion years

The Code// Solve ax2 + bx + c = 0


{

double q = b * b - 4 * a * c;

if (q > 0 && a 0) {

// code for handling two roots

}

else if (q == 0) {// code for handling one root

}

else {

// code for handling no roots

}

}

Test this case

and this

and this!

Typically, both techniques are used.

4

Recall this example from last lecture.

5

If we know the code (white box) andthus the structure, we can design testcases accordingly

6


3/23

Test this case

and this

and this!

The Test Cases// Solve ax2 + bx + c = 0


{

double q = b * b - 4 * a * c;

if (q > 0 && a 0) {


}else if (q == 0) {

// code for handling one root

}

else {


}

}

(a, b, c) = (3, 4, 1)

(a, b, c) = (0, 0, 1)

(a, b, c) = (3, 2, 1)

A Defect// Solve ax2 + bx + c = 0


{

double q = b * b - 4 * a * c;

if (q > 0 && a 0) {


}

else if (q == 0) {

x = (-b) / (2 * a);

}

else {


}

}

code must handle a = 0

(a, b, c) = (0, 0, 1)

Expressing Structure// Solve ax2 + bx + c = 0


{

double q = b * b - 4 * a * c;

if (q > 0 && a 0) {


}

else if (q == 0) {x = (-b) / (2 * a);

}

else {


}

}

Finding appropriate input values is achallenge in itself which may requireexternal theory but in this case, theexternal theory is just maths.

7

The test case that executes the q = 0branch reveals a defect the case a = 0

8

What is relevant in her is the programstructure the failure occurs only if aspecific condition is true and a specificbranch is taken.

9


4/23

Control Flow Graphpublic roots(double a, double b, double c)

double q = b * b - 4 * a * c;

q > 0 && a != 0

// code for two roots

q == 0

// code for one root

// code for no roots

return

A control flow graph expressespaths of program execution

Nodes are basic blocks sequences of statements withone entry and one exit point

Edges represent control flowthe possibility that theprogram execution proceedsfrom the end of one basicblock to the beginning ofanother

Structural Testingpublic roots(double a, double b, double c)

double q = b * b - 4 * a * c;

q > 0 && a != 0

// code for two roots

q == 0

// code for one root

// code for no roots

return

The CFG can serve as anadequacy criterion for testcases

The more partsare covered(executed), the higher thechance of a test to uncover adefect

parts can be: nodes, edges,paths, conditions

Control Flow Patternswhile (COND)

BODY

if (COND)

THEN-BLOCK ELSE-BLOCK

while (COND)

BODY

do

COND

BODY

for

INIT

INCR

while (COND)

BODY;

if (COND)

THEN-BLOCK;

elseELSE-BLOCK;

do {

BODY

}while (COND);

for (INIT; COND; INCR)

BODY;

To express structure, we turn theprogram into a control flow graph, wherestatements are represented as nodes,and edges show the possible controlflow between statements.

10

To talk about structure, we turn theprogram into a control flow graph, wherestatements are represented as nodes,and edges show the possible controlflow between statements.

11

Every part of the program induces itsown patterns in the CFG.

12


5/23


6/23


7/23


8/23


9/23


10/23

Demo

TestCriteria

TestCriteria

subsumes

See the package cgi_decode.zip onthe course page for instructions on howto do this yourself.

28

Statement testing is a simple criterionfor assessing the adequacy of a testsuite but there are many more suchcriteria.

29

As an example, consider branch testing,which is a criterion that subsumesstatement testing. In other words, if thebranch testing criterion is satisfied by apair program, test suite, so is the

statement testing criterion for the samepair.

30


11/23


12/23

!" #$%&!'()*&!+!(,#-.(./#$%&!'.)*&!+!.(#-.(./0,*!-1!+!2/

#$%&!#/#!+!'()*&/03!4#!++!5657!"!!

'.)*&!+!5!5/8!

9$0:(!4'()*&7!"

;&


13/23

Condition Testing

Key idea: also cover individual conditions incompound boolean expressione.g., both parts ofdigit_high == 1 || digit_low == -1

Condition Testing

Adequacy criterion: each basic conditionmust be evaluated at least once

Coverage:# truth values taken by all basic conditions 2 * # basic conditions

Example:100% basic condition coveragebut only 87% branch coverage

TestCriteria

subsumes

37

38

The basic condition criterion is notcomparable with branch or statementcoverage criteria neither implies(subsumes) the other.

39


14/23

TestCriteria

subsumes

TestCriteria

subsumes

Assume (((a b) c) d) e)

We need 13 teststo cover all possiblecombinations

In general case, weget a combinatorialexplosion

Compound Conditions

||

|| ||

Test Case a b c d e

(1) True True True

(2) Fa ls e Tr ue Tr ue True

(3) True Fa ls e Tr ue Tr ue

(4) Fa ls e Tr ue Fa ls e Tr ue Tr ue

(5) False False True True

(6) True True False(7) Fa ls e Tr ue Tr ue False

(8) True Fal se True Fal se

(9) Fa ls e Tr ue Fa ls e Tr ue Fa ls e

(10) False False True False

(11) True False False

(12) Fa ls e Tr ue Fa ls e F al se

(13) False False False

The idea here is to cover both branchand condition testing by covering allconditions and all decisions. That is,every sub-condition must be true andfalse, but the entire condition just aswell.

40

Another idea might be simply to test allpossible combinations. This is calledcompound condition testing.

41

42


15/23

TestCriteria

PracticalCriteria

TheoreticalCriteria

subsumes

TestCriteria

PracticalCriteria

TheoreticalCriteria

subsumes

Key idea: Test importantcombinations ofconditions, avoiding exponential blowup

A combination is important if each basiccondition is shown to independently affectthe outcome of each decision

MCDC TestingModified Condition Decision Coverage

The combinatorial explosion is thereason why compound condition testingis a theoretical, rather than a practicalcriterion.

43

A possible compromise is MCDC orModified Condition/Decision Coveragetesting.

44

45


16/23

For each basic condition C, we needtwo test cases T1 and T2

Values of all evaluatedconditions except C

are the same

Compound condition as a whole evaluatesto True for T1 and false for T2

A good balance of thoroughness and testsize (and therefore widely used)


Assume (((a b) c) d) e)

We need six tests to cover MCDCcombinations


a b c d e Decision(1) True True True True

(2) False True True True True

(3) True False True True True

(6) True True False False

(11) True False False False

(13) False False False False

Path Testingbeyond individual branches

Key idea: exploresequences of

branches incontrol flow

Many more pathsthan branchescalls for compromises

46

Underlined values independently affectthe outcome of the decision. Note thatthe same test case can cover the valuesof several basic conditions. Forexample, test case (1) covers valueTrue for the basic conditions a, c and e.Note also that this is not the onlypossible set of test cases to satisfy thecriterion; a different selection of booleancombinations could be equally effective.

47

48


17/23

TestCriteria

PracticalCriteria

TheoreticalCriteria

subsumes

TestCriteria

PracticalCriteria

TheoreticalCriteria

subsumes

TestCriteria

PracticalCrit

eria

TheoreticalCriteria

subsumes

49

For one thing, there is general pathtesting, i.e. covering all paths in theprogram. Since loops are unbounded,this is generally not feasible andtherefore just a theoretical criterion. Itsadvantage, though, is that it subsumesalmost all criteria.

50

Boundary interior testing groupstogether paths that differ only in thesubpath they follow when repeating thebody of a loop. In other words, wefollow each path in the CFG up to thefirst repeated node.

51


18/23


19/23

TestCriteria

PracticalCriteria

TheoreticalCriteria

subsumes

Loop Boundary Adequacy

A test suite satisfies the loop boundary adequacycriterion if for every loop L:

There is a test case which iterates L zero times

There is a test case which iterates L once

There is a test case which iterates L more than once

Typically combined with other adequacy criteriasuch as MCDC

!" #$%&!'()*&!+!(,#-.(./#$%&!'.)*&!+!.(#-.(./0,*!-1!+!2/

#$%&!#/#!+!'()*&/03!4#!++!5657!"!!

'.)*&!+!5!5/8!

9$0:(!4'()*&7!"

;&


20/23

TestCriteria

PracticalCriteria

TheoreticalCriteria

subsumes

TestCriteria

PracticalCriteria

TheoreticalCriteria

subsumes

LCSAJ AdequacyTesting all paths up to a fixed length

LCSAJ = Linear Code Sequence And Jump

A LCSAJ is a sequential subpath in the CFGstarting and ending in a branch

LCSAJ length corresponds to

1 statement coverage2 branch coverage

ncoverage ofn

consecutive LCSAJs

path coverage

Another alternative is loop boundarytesting which forces constraints on howloops are to be executed.

58

LCSAJ is a generalization over branchand statement coverage.

59

Considering the exponential blowup insequences of conditional statements(even when not in loops), we mightchoose to consider only sub-sequencesof a given length. This is what LCSAJgives us --- essentially considering fullpath coverage of (short) sequences ofdecisions.

60


21/23

TestCriteria

PracticalCriteria

TheoreticalCriteria

subsumes

Weyukers Hypothesis

Satisfying Criteria

Sometimes criteria may not be satisfiable:

Statements may not be executed because ofdefensive programmingor code reuse

Conditions may not be satisfiable because ofinterdependent conditions

Paths may not be executable because ofinterdependent decisions

And this is the summary of structuraltesting techniques.

61

Established by a number of studiesdone by E. Weyuker at AT&T. Anyexplicit relationship between coverageand error detection would mean that wehave a fixed distribution of errors overall statements and paths, which isclearly not the case.

62

63


22/23

Satisfying Criteria

Reaching specific code can be very hard!

Even the best-designed, best-maintainedsystems may contain unreachable code

A large amount of unreachable code/paths/conditions is a serious maintainability problem

Solutions: allow coverage less than 100%,or require justification for exceptions

More Testing Criteria

Object-oriented testinge,g,Every transition in the objects FSM must be covered orEvery method pair in the objects FSM must be covered

Interclass testinge.g, Every interaction between two objects must be covered

Data flow testinge.g.,Every definition-use pair of a variable must be covered

!" #$%&!'()*&!+!(,#-.(./#$%&!'.)*&!+!.(#-.(./0,*!-1!+!2/

#$%&!#/#!+!'()*&/03!4#!++!5657!"!!

'.)*&!+!5!5/8!

9$0:(!4'()*&7!"

;&


23/23

Summary

67

02 structural testing

Documents