1

Lecture 14: Testing Techniques and Strategies

SACHIN

Today’s TopicsChapters 17 & 18 in SEPA 5/eTesting Principles & TestabilityTest CharacteristicsBlack-Box vs. White-Box TestingFlow Graphs & Basis Path TestingTesting & Integration Strategies

2SACHIN

Software TestingOpportunities for human error

Specifications, design, codingCommunication

“Testing is the ultimate review”Can take 30-40% of total effortFor critical apps, can be 3 to 5 times all

other efforts combined!

3SACHIN

Testing ObjectivesExecute a program with the intent of

finding errorsGood tests have a high probability of

discovering errorsSuccessful tests uncover errors‘No errors found’: not a good test!Verifying functionality is a secondary goal

4SACHIN

Testing PrinciplesTests traceable to requirementsTests planned before testing Pareto principle: majority of errors traced

to minority of componentsComponent testing first, then

integrated testingExhaustive testing is not possibleIndependent tests: more effective

5SACHIN

Software Testability

OperabilityObservabilityControllabilityDecomposability

6

Characteristics that lead totestable software:

• Simplicity• Stability• Understandability

SACHIN

Operability

System has few bugsNo bugs block execution of testsProduct evolves in functional stages

7

The better it works, the more efficientlyit can be tested

SACHIN

Observability

Distinct output for each inputStates & variables may be queriedPast states are loggedFactors affecting output are visibleIncorrect output easily identifiedInternal errors reportedSource code accessible

8

What you see is what you test

SACHIN

Controllability

All possible outputs can be generated by some input

All code executable by some inputStates, variables directly controlledInput/output consistent, structuredTests are specified, automated, and

reproduced

9

The better we can control the software,the more the testing can be automated

SACHIN

Decomposability

Independent modulesModules can be tested separately

10

By controlling the scope of testing, wecan more quickly isolate problems andperform smarter retesting

SACHIN

Simplicity

Minimum feature setMinimal architectureCode simplicity

11

The less there is to test, the morequickly we can test it

SACHIN

Stability

Changes made to system:are infrequentare controlleddon’t invalidate existing tests

Software recovers from failure

12

The fewer the changes, the fewer thedisruptions to testing

SACHIN

Understandability

Design is well-understoodDependencies are well understoodDesign changes are communicatedDocumentation is:

accessiblewell-organizedspecific, detailed and accurate

13

The fewer the changes, the fewer thedisruptions to testing

SACHIN

Test CharacteristicsGood test has a high probability of finding

an errorGood test is not redundantA good test should be “best of breed”A good test is neither too simple nor too

complex

14SACHIN

Test Case Design‘Black Box’ Testing

Consider only inputs and outputs‘White Box’ or ‘Glass Box’ Testing

Also consider internal logic paths, program states, intermediate data structures, etc.

15SACHIN

White-Box TestingGuarantee that all independent paths have

been testedExercise all conditions for ‘true’ and ‘false’Execute all loops for boundary conditionsExercise internal data structures

16SACHIN

Why White-Box Testing?More errors in ‘special case’ code which is

infrequently executedControl flow can’t be predicted accurately

in black-box testingTypo errors can happen anywhere!

17SACHIN

Basis Path TestingWhite-box method [McCabe ‘76]Analyze procedural designDefine basis set of execution pathsTest cases for basis set execute every

program statement at least once

18SACHIN

Basis Path Testing [2]

19

Flow Graph: Representation of Structured Programming Constructs

[From SEPA 5/e]

SACHIN

20

Cyclomatic Complexity

V(G)=E-N+2 = 4

Independent Paths1: 1,112: 1,2,3,4,5,10,1,113: 1,2,3,6,8,9,10,1,114: 1,2,3,6,7,9,10,1,11

V(G): upper bound on number of teststo ensure all code has been executed

[From SEPA 5/e]

SACHIN

Black Box TestingFocus on functional requirementsIncorrect / missing functionsInterface errorsErrors in external data accessPerformance errorsInitialization and termination errors

21SACHIN

Black Box Testing [2]How is functional validity tested?What classes of input will make good test

cases?Is the system sensitive to certain inputs?How are data boundaries isolated?

22SACHIN

Black Box Testing [3]What data rates and volume can the system

tolerate?What effect will specific combinations of data

have on system operation?

23SACHIN

Comparison TestingCompare software versions“Regression testing”: finding the outputs

that changed Improvements vs. degradationsNet effect depends on frequency and

impact of degradationsWhen error rate is low, a large corpus can

be used

24SACHIN

Generic Testing StrategiesTesting starts at module level and moves

“outward”Different testing techniques used at

different timesTesting by developer(s) and independent

testersTesting and debugging are separate

activities

25SACHIN

Verification and ValidationVerification

“Are we building the product right?”Validation

“Are we building the right product?”Achieved by life-cycle SQA activities,

assessed by testing“You can’t create quality by testing”

26SACHIN

27

Organization of Testing

[From SEPA 5/e]

SACHIN

28

Logarithmic Poissonexecution-time model

With sufficient fit,model predicts testingtime required to reachacceptable failure rate

[From SEPA 5/e]

SACHIN

29

[From SEPA 5/e]

SACHIN

30

PRO: Higher-level (logic) modules tested earlyCON: Lower-level (reusable) modules tested late

[From SEPA 5/e]

SACHIN

31

PRO: Lower-level (reusable) modules tested earlyCON: Higher-level (logic) modules tested late

[From SEPA 5/e]

SACHIN

Hybrid ApproachesSandwich Integration: combination of top-

down and bottom-upCritical Modules

address several requirementshigh level of controlcomplex or error pronedefinite performance requirements

Test Critical Modules ASAP!

32SACHIN

33SACHIN