1 note content copyright © 2004 ian sommerville. nu-specific content copyright © 2004 m. e. kabay....

1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved.

Software Testing

IS301 – Software EngineeringLecture #31 – 2004-11-12

M. E. Kabay, PhD, CISSP

Assoc. Prof. Information AssuranceDivision of Business & Management, Norwich University mailto:[email protected] V: 802.479.7937

mailto:[email protected]


Topics

Defect Testing Integration TestingObject-Oriented TestingTesting Workbenches


Defect Testing

Testing programs to establish presence of system defects


Testing Process

Component testing Testing of individual program componentsUsually responsibility of component

developer (except sometimes for critical systems)

Tests derived from developer’s experience Integration testing

Testing of groups of components integrated to create system or sub-system

responsibility of independent testing teamTests based on system specification


Testing Phases

Componenttesting

Integrationtesting

Software developer Independent testing team


Defect Testing

Goal of defect testing to discover defects in programs

Successful defect test test which causes program to behave in anomalous way

Tests show presence not absence of defects


Testing priorities

Only exhaustive testing can show program free from defects. However, exhaustive testing impossible

Tests should exercise system's capabilities rather than its components

Testing old capabilities more important than testing new capabilities

Testing typical situations more important than boundary value cases


Test Data and Test Cases

Test data Inputs which have been devised to test

systemTest cases

Inputs to test system plusPredicted outputs from these inputs

If system operates according to its specification


Defect Testing Process

Design testcases

Prepare testdata

Run programwith test data

Compare resultsto test cases

Testcases

Testdata

Testresults

Testreports


Black-Box Testing

Program considered as ‘black-box’Test cases based on system specification Test planning can begin early in software

process


Black-Box Testing

Ie

Input test data

OeOutput test results

System

Inputs causinganomalousbehaviour

Outputs which revealthe presence ofdefects


Equivalence Partitioning

Input data and output results often fall into different classes where all members of class related

Each of these classes = equivalence partition where program behaves in equivalent way for each class member

Test cases should be chosen from each partition


Equivalence Partitioning (1)

System

Outputs

Invalid inputs Valid inputs


Equivalence partitioning (2)

Partition system inputs and outputs into ‘equivalence sets’; e.g.,If input 5-digit integer

between 10,000 and 99,999, then equivalence partitions are

<10,000, 10,000-99,999 and >99,999

Choose test cases at boundaries of these sets:00000, 09999, 10000, 10001, 99998, 99999,

& 100000


Equivalence Partitioning (3)

Between 10000 and 99999Less than 10000 More than 99999

999910000 50000

10000099999

Input values

Between 4 and 10Less than 4 More than 10

34 7

1110

Number of input values


Search Routine Specification

procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;

Pre-condition-- array has at least one elementT’FIRST <= T’LAST

Post-condition-- element found and referenced by L( Found and T (L) = Key)

or -- element not in array( not Found and

not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))


Search Routine - Input Partitions

Inputs which conform to pre-conditions Inputs where pre-condition does not hold Inputs where key element is member of

array Inputs where key element not member

of array


Testing Guidelines (Sequences)

Test software with sequences which have only single value

Use sequences of different sizes in different tests

Derive tests so that first, middle and last elements of sequence accessed

Test with sequences of zero length


Search Routine - Input Partitions


Structural Testing

Sometime called white-box testingDerivation of test cases according to program

structure. Knowledge of program used to identify

additional test casesObjective to exercise

all program statements not all path combinations


White-Box Testing

Componentcode

Testoutputs

Test data

DerivesTests


Binary Search – Equiv. Partitions

Pre-conditions satisfied, key element in arrayPre-conditions satisfied, key element not in

arrayPre-conditions unsatisfied, key element in

arrayPre-conditions unsatisfied, key element not in

array Input array has single value Input array has even number of values Input array has odd number of values


Binary Search – Equiv. Partitions

Mid-point

Elements < Mid Elements > Mid

Equivalence class boundaries


Binary Search – Test Cases


Path Testing

Objective: ensure each path through program executed at least once

Starting point: flow graphShows nodes representing program

decisionsArcs representing flow of control

Statements with conditions therefore = nodes in flow graph


Program Flow Graphs

Each branch shown as separate pathLoops shown by arrows looping back to loop

condition nodeBasis for computing cyclomatic

complexityCyclomatic complexity = Number of edges -

Number of nodes + 2


Cyclomatic Complexity

Number of tests needed to test all control statements equals cyclomatic complexity

Cyclomatic complexity equals number of conditions in program

Useful if used with care. Does not imply adequacy of testing.

Although all paths executed, all combinations of paths not necessarily executed

Binary Search Flow Graph1

2

3

4

65

7

while bottom <= top

if (elemArray [mid] == key

(if (elemArray [mid]< key8

9

bottom > top


Independent Paths

1, 2, 3, 8, 9 1, 2, 3, 4, 6, 7, 2 1, 2, 3, 4, 5, 7, 2 1, 2, 3, 4, 6, 7, 2, 8, 9 Test cases should be

derived so that all of these paths executed

Dynamic program analyzer may be used to check that paths have been executed

1

2

3

4

65

7

while bottom <= top

if (elemArray [mid] == key

(if (elemArray [mid]< key8

9

bottom > top


Integration Testing

Tests complete systems or subsystems composed of integrated components

Integration testing should be black-box testing with tests derived from specification

Main difficulty is localizing errors Incremental integration testing reduces this

problem


Incremental Integration Testing

T3

T2

T1

T4

T5

A

B

C

D

T2

T1

T3

T4

A

B

C

T1

T2

T3

A

B

Test sequence1

Test sequence2

Test sequence3


Approaches to Integration Testing

Top-down testingStart with high-level system and integrate

from top-down replacing individual components by stubs where appropriate

Bottom-up testingIntegrate individual components in levels

until complete system created In practice, most integration involves

combination of these strategies


Top-Down Testing

Level 2Level 2Level 2Level 2

Level 1 Level 1Testing

sequence

Level 2stubs

Level 3stubs

. . .


Bottom-Up Testing

Level NLevel NLevel NLevel NLevel N

Level N–1 Level N–1Level N–1

Testingsequence

Testdrivers

Testdrivers


Testing Approaches

Architectural validationTop-down integration testing better at

discovering errors in system architectureSystem demonstration

Top-down integration testing allows limited demonstration at early stage in development

Test implementationOften easier with bottom-up integration testing

Test observation – what’s happening during test?Problems with both approachesExtra code may be required to observe tests

Instrumenting the code


Interface Testing

Takes place when modules or sub-systems integrated to create larger systems

Objective: Detect faults due toInterface errors or Invalid assumptions about interfaces

Particularly important for object-oriented development Objects defined by their interfaces


Interface Testing

Testcases

BA

C


Interface Types

Parameter interfacesData passed from one procedure to another

Shared-memory interfacesBlock of memory shared between / among

proceduresProcedural interfaces

Sub-system encapsulates set of procedures to be called by other sub-systems

Message-passing interfacesSub-systems request services from other

sub-systems


Interface Errors

Interface misuseCalling component makes error use of

interface; e.g. parameters in wrong order Interface misunderstanding

Calling component embeds incorrect assumptions about behavior of called component

Timing errorsCalled and calling component operate at

different speedsOut-of-date information accessed“Race conditions”


Interface Testing Guidelines

Parameters to called procedure at extreme ends of their ranges

Always test pointer parameters with null pointers

Design tests which cause component to failUse stress testing in message passing

systems (see next slide) In shared-memory systems, vary order in

which components activatedWhy?


Stress Testing

Exercises system beyond its maximum design loadStressing system often causes defects to

come to light Stressing system tests failure behavior

Should not fail catastrophicallyCheck for unacceptable loss of service or

dataDistributed systems

Non-linear performance degradation


Approaches to Cluster Testing

Use-case or scenario testingTesting based on user interactions with

systemTests system features as experienced by

usersThread testing

Tests systems response to events as processing threads through system

Object interaction testingTests sequences of object interactions Stop when object operation does not call

on services from another object


Scenario-Based Testing

Identify scenarios from use-cases Supplement with interaction diagrams Show objects involved in scenario

Consider scenario in weather station system where report generated (next slide)


Testing Workbenches

Testing expensiveWorkbenches

Range of toolsReduce

Time required and Total testing costs

Most testing workbenches are open systemsTesting needs are organization-specificDifficult to integrate with closed design

and analysis workbenches


Testing Workbench


Testing Workbench Adaptation

Scripts User interface simulators andPatterns for test data generators

Test outputsMay have to be prepared manually for

comparisonSpecial-purpose file comparators may be

developed


Required Homework

Read-recite-review Chapter 23 of Sommerville’s text

Survey-Question Chapter 24 for MondayQuiz on WEDNESDAY 17th Nov:

Chapters 17-21Required

For Fri 19 Nov 2004 for 35 points23.1 & 23.3 (@5)23.4 & 23.5 (@10)23.7 (@5)


Optional Homework

By Mon 29 Nov 2004, for up to 14 extra points, complete any or all of23.2, 23.6, 23.8 (@2)23.9 (@5)23.10 (@3)


DISCUSSION

1 note content copyright © 2004 ian sommerville. nu-specific content copyright © 2004 m. e. kabay....

Documents

note content copyright

nuspecific content copyright

ian sommerville

testing phases slide

testing priorities

defect testing process

blackbox testing slide

blackbox testing program