Lecture 6: Testing Techniques

Software Quality Assurance (INSE 6260/4-UU)

Winter 2016

2

INSE 6260/4-UU

Software Quality

Assurance

Software Quality

Factors and Models

Metrics

Quality Assurance

Inspection Testing

Techniques Reachability

Analysis

INSE 6260

3

• Several definitions:

1- “Testing is the process of establishing confidence that a program

or system does what it is supposed to.”

by Hetzel 1973

2- “Testing is the process of executing a program or system with the

intent of finding errors.”

by Myers 1979

3- “Testing is any activity aimed at evaluating an attribute or

capability of a program or system and determining that it meets its

required results.” by Hetzel 1983

What is Software Testing?

INSE 6260

4

- Test manager - manage and control a software test project - supervise test engineers - define and specify a test plan - Software test engineers and testers - define test cases, write test specifications, run tests - Independent test group - Development engineers - Only perform unit tests and integration tests - Quality assurance group and engineers - Perform system testing - Define software testing standards and quality control process

Who does Software Testing

INSE 6260

Testing

5

Software Apply input Observe output

Validate the observed output

Is the observed output the same as the expected output?

Oracle

INSE 6260

6

- Due to the testing time limit, it is impossible to achieve total confidence - We can never be sure the specifications are 100% correct - We can never be certain that a testing system (or tool) is correct - No testing tools can cop with every software program - Tester engineers never be sure that they completely understand a software product

Software Testing Limits

INSE 6260

7

Software Testing Process

Unit test

Integration test

Validation test

System test

System engineering

Requirements

Software Design

Code & Implementation

V&V Targets

INSE 6260

8

Unit Test (Component Level Test) What?

Unit testing: Individual components are tested independently to

ensure their quality

Why? The focus is to detect errors in design and implementation, including:

- data structure in a component

- program logic and program structure in a component

- component interface

- functions and operations of a component

Who? developers of the components

Operations and Functions with I/O

White-box interface

input

output

Internal logic, data, structure

output

input

interface

operation

Black-box

INSE 6260

9

Integration Testing What? Integration test: A group of dependent components are composed and tested together to ensure the quality of their integration unit Why? The objective is to detect errors in: - Design and construction of software architecture - Integrated functions or operations at sub-system level - Interfaces and interactions between them Who? Either developers and/or test engineers

Component #1

Operations and Functions with I/O

input

interface

operation

Component #2

Operations and Functions with I/O

output

interface

operation

INSE 6260

10

System Validation Testing

What? System val. test: The entire integrated software that composes the system is tested based on requirements to ensure that we have the right product Why ? The objective is to detect errors in: - System input/output - System functions and information data - System interfaces with external parts - User interfaces - System behavior and performance Who? test engineers: SQA team

System

(Operations & Functions

& Behavior)

User interface

User External interfaces

INSE 6260

11

Types of Testing

Aspect

Accessibility

Level

functional

robustness

performance

reliability

usability

unit

integration

system

acceptance

white box

grey box

black box

regression

INSE 6260

12

Levels of Testing

What users really need

Requirements

Design

Code

Acceptance testing

System testing

Integration testing

Unit testing

INSE 6260

13

System Testing

• Tests the overall system (the integrated hardware and

software) to determine whether the system meets its

requirements

• Focuses on the use and interaction of system functionalities

rather than details of implementations

• Should be carried out by a group independent of the code

developers

• Should be planned with the same rigor as other phases of

the software development

INSE 6260

14

Regression Testing

• Development versus maintenance

– Development costs: 1/3

– Maintenance costs: 2/3

• Testing in maintenance phase

– How can we test modified or newly inserted

programs? • Ignore old test suites and make new ones from the scratch or

• Reuse old test suites and reduce the number of new test suites as

many as possible

INSE 6260

Testing Techniques

Functional testing (Black box testing):

1. Equivalence class analysis,

2. Cause effect graphs

3. Syntactic testing

4. Random testing

5. Transaction analysis

6. Testing based on models

7. Testing state transitions

15

INSE 6260

1) Equivalence class

A common approach is to divide the input domain into Equivalence Classes such that the program can reasonably be expected to have the same behavior for any points within a class

when dealing with predominantly numerical input space, we can define the boundary of an equivalence class through the use of an algebraic equation

Based on specification

Let E = space of values,

find classes such that

ci = E , union of ci from i = 1 to n (state cover)

i, j, ci cj =

16

INSE 6260

Equivalence class and boundary values

• Case #1: domain of data is an interval

– Class(es) of valid values

– Class of invalid values (out of boundary, inf. and sup.)

– Class of non member (for numerical values, characters are non

members)

17

• Case #2: Domain of data is a number of values

– N classes for valid values

– One class: absence of values

– One class: too much values

INSE 6260

18

Equivalence class

• Case #3: data is a constraint

– One class for valid constraint

– One class for constraint violation

INSE 6260

19

2) Cause effect graph

Establish a relation between the effect (an output) and its causes

(inputs)

4 basic symbols express the interdependency between inputs

and outputs:

Identity

Negation

Logic And

V

Logic OR

INSE 6260

20

Cause effect graph, an example

File management example:

The character of the first column should be “A” or “B”. The second column

should be a number. In this case the file is considered up dated.

If the first character is erroneous, we should print then the message X12.

If in the second column we don’t have a number, we print the message X13.

A1

A2

NE

M2

M1

M3

A3

V

A1: The first char is “A”

A2: The first char is “B”

A3: The second is a number

M1: File updated

M2: Message X12

M3: Message X13

INSE 6260

21

Cause effect graph: decision table

Causes effect

A1 A2 A3 M1 M2 M3

1 0 0 0 0 1 1

2 0 0 1 0 1 0

3 0 1 0 0 0 1

4 0 1 1 1 0 0

5 1 0 0 0 0 1

6 1 0 1 1 0 0

Test

data

3 inputs, 8 possibilities

But A1 = true and A2= true is an impossible combination

1 = true, 0 = false

INSE 6260

22

Cause effect graph: constraints

X

A

B

A

B

C

X V

Disjunctive case

R1: for X at 1 then

don’t take the case A= B=1

R2: for X at 0, enumerates all the

situations where A=B=0

Conjunctive case

R3: for X at 1, enumerate all the situations

Where A=B=C=1

R4: For X at 0

R4.1 include only the case A=B=C=0

R4.2 for all other cases( 001, 010, 100,

011,, 101, 110), include only one sample

INSE 6260

23

Cause effect graph: methodology

Test technique based on cause effect graph consists in the following steps:

1- decompose the unit to be tested, if it has many functionalities

2- identify the distinct input conditions “causes”

(input combinations or classes of input conditions)

3- identify the distinct output conditions “effects”

(output conditions, or system transformations)

4- draw the graph of relations between causes and effects

5- complete the graph by adding the constraints between causes and effects

6- convert the graph to a decision table

- assign true in sequence to all effects

- create a line for each combination of causes that produces an effect

- for each combination determine the state of all effects

7- Produce a test per line

INSE 6260

24

3) Data validation & Syntax testing Generators & Recognizers

The goal is to determine data tests based on formal description

The formal description is a BNF or an FSM

BNF= Backus- Naur Form ( grammar)

FSM= Finite State Machine

“The routine is a string recognizer”

“The tester is string generator”

What to capture?

2 possibilities:

- the recognizer does not recognize a good string

- it accepts a bad string

INSE 6260

25

Data validation & Syntax testing: example

Recognizer for

copy aa to bb

rename aa to bb

copy a to b

First step: define the grammar

<command>::= < command type> < file name> to < file name>

<command type>::= copy | rename

<file name> ::= <letter> | <letter> <letter>

<letter>::= a | b

INSE 6260

26

Data validation & Syntax testing

1

<command>

2

< command type>

3

<file name>

4

to 5

<file name>

6

copy or

7

rename

14 15

a b or

or 10

<letter>

11

<letter> <letter>

20 21

a b or

22 23

a b or

or 8

<letter>

9

<letter> <letter>

16 17

a b or

18 19

a b or

12 13

a b or

Second step: build a corresponding derivation tree

INSE 6260

27

Data validation & Syntax testing

Third step: Produce data test based on the derivation tree

Cover valid commands, then invalid commands.

For the example,

Valid commands

Cover the non terminal nodes (1,2,3,5,8,9,10,11) to check if the 2 commands are recognized:

copy a to b (covers the non terminals nodes 1,2,3,5,8,10 and terminal nodes 6,12,15)

copy aa to bb (covers the non terminals nodes 1,2,3,5,9, 11

and terminal nodes 6,16, 18,21,23)

Cover at least one time all terminal nodes (those not covered yet are: 7,13,14,17,19,20,22.

We have to add then the following tests:

rename b to a (that covers 7,13,14)

copy bb to aa (17, 19, 20, 22)

INSE 6260

28

Data validation & Syntax testing

Invalid commands

It may exist an infinite number of invalid commands. There is need for sampling

We distinguish 3 levels in the derivation tree

Level 1: nodes 2,3,4,5

Level 2: nodes 8,9,10,11

Level 3: all other nodes

Level 1:

a to b ( node 2 is omitted)

copy to b (node 3 is omitted)

copy a b (node 4 is omitted)

copy a to (node 5 is omitted)

rename a to (node 5 is omitted)

rename to b (node 3 is omitted)

etc

Produce various faults and simulate the addition or omission of nodes at each level

INSE 6260

29

Level 2:

copy a a to b

copy a to b b

rename a a to b

etc.

Level 3:

copy c to b

copy a to c

copy aaa to b

copy a to bbb

copy a to +

rename aaa to b

cop a to b

copy a t b

renome a to b

etc.

INSE 6260

30

4) Random testing

All test selection techniques seen before are deterministic.

Random testing is based on a probabilistic approach.

It uses a uniform sampling on definition domains of input variables.

It uses Random Number Generator (RNG)

Spec: x is an integer between 1 and 10

If x > 8 then action1

else action2

The probability of executing the then is 2/10,

The else is 8/10 for input selection based on random testing.

INSE 6260

31

5) Transaction flows

“A transaction is a unit of work seen from a system user’s

point of view. A transaction consists of a set of operations, some

of which are performed by a system, persons, or devices that

are outside the system. A transaction typically consists of a set

of operations that begins with an input and ends with one or more

outputs…”

Boris Beizer

INSE 6260

32

Transaction flow: an example

Insert

card

Invalid

NIP

message

Invalid card

message

Withdraw

Balance

Output

Input

The test of transaction’s flow is a functional testing

Transaction flow

INSE 6260

33

Transaction Flow Graph (TFG)

Choice 1

Choice 2

Parent

child

parent Parent

Child

Child

Path 1

Path 2

Next

Predator

Prey

Predator

Junction

Parent

Parent

Child

Absorb Combine

Decision

“Biosis” “Mitosis”

INSE 6260

34

The modeling with TFG

Divergences

Decision: similar to a decision in control graph.

Boisis: the transaction creates a new transaction called “child”, only the parent transaction

keeps its previous identity.

Mitosis: parent transaction creates two new transactions (each of them is a child).

Convergence

Junction: Similar to a join in the control graph. One of the two paths will be followed by a transaction.

Absorption: the predator transaction will absorb the prey. The predator transaction keeps its previous

identity.

Combine: Two transactions are assembled to create a new transaction.

INSE 6260

35

The testing technique of TF

• Extract all transaction flows documented in the specification.

• Test every link, every decision, every process…

• Additional paths should be selected to cover loops, boundary..

• Additional paths should be selected to cover longest transactions,

complexes and transactions that present an identified risk.

INSE 6260

36

Readings: Path testing and transaction flaws

from B. Beizer “ Software testing techniques”

INSE 6260

37

Model-based Testing

Formal

Specification

FSM extractor

FSM

Specification

Automatic

Test Generator Abstract

Test suite

USER Completed Test Case

TSL (Test Scripting

Language)

Validation

Translation

Executable Test Case

(C Code for example)

Validated Test Case

USER

TEST ENGINE

Test

Results Implementation

Under Test (IUT)

INSE 6260

38

Fault models

A fault model is a hypothetical model of what types of faults may occur in an implementation – most fault models are “structural”, i.e. the model is a refinement of

the specification formalism (or of an implementation model)

• e.g., mutations of the specification or of a correct implementation

A fault model is useful for the following problems:

– Test suite development

– For existing test suite: coverage evaluation

– Diagnostics

INSE 6260

39

Testing Sequential Software

• White-box vs. black-box testing

• Coverage criteria

– Control flow criteria

– Data flow criteria

– Parameter variation

INSE 6260

40

Testing based on Finite State Models

• Conformance relations: based on I/O sequences

• Testing based on FSM specifications

– Fault model

– Test derivation methods

• Transition Tour

• State identification methods

• Testing based on partially specified behavior

• Testing against non-deterministic specifications

– Testing non-deterministic FSMs with input queuing

• Coverage analysis

INSE 6260

41

FSM

S 1 S 2

S 4 S 3

t1: 1/1

t2: 2/2 t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

S1 is an initial state

Is a transition:

It has a starting state

S1, and an ending state

S2.

Its label is t1.

The input is 1 and the

output is 1.

It separates the input

from the output

t1: 1/1

INSE 6260

42

: Ds --> S

: Ds --> Y

Mealy Machine

state

set

initial

state

M = < S, S1, X, Y, Ds, , >

input

set

output

set

spec.

domain

transfer

function

output

function

Ds S x X

partially defined (specified), deterministic, initialized

S = {S1, S2, S3, S4}

X = {1, 2}

Y = {1, 2}

Ds = S x X - {<S3, 1>}

S 1 S 2

S 4 S 3

t1: 1/1

t2: 2/2 t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/?

?

An FSM Example

INSE 6260

43

1) Output fault

2) Transfer fault

3) Transfer fault with additional states

4) Additional or missing transitions

5) additional or missing states

Fault Model for Finite State Machine

(FSM)

INSE 6260

44

Specification

S 1 S 2

S 4 S 3

t1: 1/1

t2: 2/2 t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

S 1 S 2

S 4 S 3

t1: 1/2

t2: 2/2 t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

Output Fault on transition t1

Implementation under test

IUT

INSE 6260

45

S 1 S 2

S 4 S 3

t1: 1/1

t2: 2/2 t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

S 1 S 2

S 4 S 3

t1: 1/1

t2: 2/2 t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

Transfer fault on t2

The ending state is now S3

Specification IUT

INSE 6260

46

S 1 S 2

S 4 S 3

t1: 1/1

t2: 2/2 t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/?

?

S 1 S 2

S 4 S 3

t1: 1/1

t2: 2/2 t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/2

Transfer fault on t5 with

Additional state

Specification IUT

INSE 6260

47

Example of implementation with additional state

S1

S2

S0

b/f

a/e

a/f

c/f

b/fc/ec/e

b/e

a/f

I0

b/f

a/e

a/f

c/e

b/e

I1

I2

b/fc/e

a/f

c/f

c/f

a/f

I 1

I 2

I o

b/f

a/e

a/f

b/fc/ec/e

b/e

I 3

a/e

b/f

c/e

Specification Impl. 1

Impl. 2

INSE 6260

48

Example of a test suite

S 1 S 2

S 4 S 3

t1: 1/1

t2: 2/2 t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

TS = { r.1.1.2.1, r.2.2.1.2.2}

A test suite is a set of input

sequences starting from the

initial state of the machine

r.1.1.2.1

r.2.2.1.2.2

Test Case MS

1.1.2.2

2.2.1.2.2

M’I MI

1.1.2.2

2.2.1.2.2

1.1.2.2

2.2.2.2.2

Conforming Non-conforming

Pass TS Fail to pass TS

INSE 6260

49

An undetected faulty implementation

TS = { r.1.1.2.1, r.2.2.1.2.2}

Expected output sequences: 1.1.2.2, 2.2.1.2.2

I1

I2 I3 I4 I5

I6 I7 I8 I9 I10

I11 1/1

1/1 2/2 1/2

2/2

2/2 1/1 2/2 2/2

1/1

Consider input sequence: r.1.1.2.1.1

Expected output sequence: 1.1.2.2.2

Observed output sequence: 1.1.2.2.1

INSE 6260

50

Possible changes made by a developer

• Type 1: change the tail state of a transition

• Type 2: change the output of a transition

• Type 3: add a transition; and

• Type 4: add an extra state.

S 1 S 2

S 4 S 3

t1: 1/1

t2: 2/2 t4: 2/2

t3:1/1

t6: 2/2

t7: 1/2

t8: 2/2

t5: 1/?

?

No limitation on the number

of such changes allows for

an infinite set of possible

implementations !!!