software testing and reliability southern methodist university cse 7314

Software Testing

and Reliability

Southern Methodist University

CSE 7314

“A working program remains an elusive thing of beauty”

Robert Dunn

Syllabus

• Instructor; Rob Oshana

• Office hours: By appointment

• Phone; (281) 274-3211

• Fax; (214) 768-3085

• E-mail; oshana@airmail.net• Web site;

www.engr.smu.edu/cse/roshana/cse7314

Syllabus

• Required Text Book: Systematic Software Testing, by Rick Craig and Stefan Jaskiel, Artech House, ISBN 1-58053-508-9

• Practical Guide to Testing Object-Oriented Software by David A. Sykes, John D. McGregor Addison-Wesley Pub Co; ISBN: 0201325640

Syllabus

• Student Evaluation; The course grade will be computed as follows:–Midterm Exam 30%– Final Exam 30%– Homework 15%– Project 25%

Course Outcomes

• Upon successful completion of this course the student will be able to:

• 1. Determine the test techniques applicable to a given program

• 2. Construct a test suite using the techniques discussed in class

Course Outcomes

• 3. Determine various test and quality metrics of a program

• 4. Create and manage an effective software testing team

Course is / is not

• Is a roadmap approach for test professionals

• Is not an implementation course

• Is not a software testing tools course

Outline

Trip Topics Readings

1 Overview of the testing process

Risk analysis

Master test planning

Detailed test planning

Craig, Chapters 1-4

2 Analysis and design

Test implementation

Test execution

Craig, Chapters 5-7

3 Test organization

The software tester

The test manager

Improving the process

Craig, chapters 8-11

Outline

Trip Topics Readings

4 Statistical testing techniques

Testing OO systems

Notes

Sykes, Chapters 1-6

5 Testing OO systems Sykes, Chapters 7-10

6 Testing RT systems

Testing safety critical systems

Testing web based systems

Notes

Outline

• Selected readings will be sent to the students on a periodic basis

• Homework; assignments/schedule will be posted on the web site shortly

• Project will be discussed next trip

Testing style

Competence Test question cues

Knowledge List, describe

Comprehension Summarize, discuss, describe

Evaluation Explain, compare

Analysis Analyze, explain, compare

“Errors are more common, more pervasive, and more troublesome in software than with other technologies”

David Parnas

Homework 1a

• Please send me a couple paragraphs describing your background and experience. Describe to me what you want to get out of the course

Homework 1a

• Please read the paper entitled “Improving Software Testability”

• www.stlabs.com/newsletters/testnet/docs/testability.htm

CSE 7314

Software Testing and Reliability

What is testing?

• How does testing software compare with testing students?

What is testing?

• “Software testing is the process of comparing the invisible to the ambiguous as to avoid the unthinkable.” James Bach, Borland corp.

What is testing?

• Software testing is the process of predicting the behavior of a product and comparing that prediction to the actual results." R. Vanderwall

Purpose of testing

• Build confidence in the product

• Judge the quality of the product

• Find bugs

Finding bugs can be difficult

x

x

x

x

x

x

x

x

x

x

x

Mine field

A path through themine field (use case) A path through the

mine field (use case)

Why is testing important?

• Therac25: Cost 6 lives

• Ariane 5 Rocket: Cost $500M

• Denver Airport: Cost $360M

• Mars missions, orbital explorer & polar lander: Cost $300M

Why is testing so hard?

Reasons for customer reported bugs

• User executed untested code• Order in which statements were

executed in actual use different from that during testing

• User applied a combination of untested input values

• User’s operating environment was never tested

Interfaces to your software

• Human interfaces

• Software interfaces (APIs)

• File system interfaces

• Communication interfaces–Physical devices (device drivers)

–controllers

Selecting test scenarios

• Execution path criteria (control)–Statement coverage–Branching coverage

• Data flow – Initialize each data structure–Use each data structure

• Operational profile • Statistical sampling….

What is a bug?

• Error: mistake made in translation or interpretation ( many taxonomies exist to describe errors)

• Fault: manifestation of the error in implementation (very nebulous)

• Failure: observable deviation in behavior of the system

Example

• Requirement: “print the speed, defined as distance divided by time”

• Code: s = d/t; print s

Example

• Error; I forgot to account for t = 0

• Fault: omission of code to catch t=0

• Failure: exception is thrown

Severity taxonomy

• Mild - trivial

• Annoying - minor

• Serious - major

• Catastrophic - Critical

• Infectious - run for the hills

What is your taxonomy ?

IEEE 1044-1993

Life cycle

Requirements

Design

Code

Testing

error

error

error

Errors can be introduced ateach of these stages

Resolve

Isolate

Classify

error

error

error

error

Testing and repair process can bejust as error prone as the developmentProcess (more so ??)

Ok, so lets just design our systems with “testability” in

mind…..

Testability

• How easily a computer program can be tested (Bach)

• We can relate this to “design for testability” techniques applied in hardware systems

JTAG

A standard Integrated Circuit

CoreIC

Logic

Test access portcontroller

Test mode Select (TMS)

Test clock (TCK)

Test data out (TDO)

Test data in (TDI)

BoundaryScan cells

BoundaryScan path

I/O pads

Data in

Data out

TDI TDOcell

Operability

• “The better it works, the more efficiently it can be tested”–System has few bugs (bugs add

analysis and reporting overhead)–No bugs block execution of tests–Product evolves in functional

stages (simultaneous development and testing)

Observability

• “What you see is what you get”– Distinct output is generated for each

input– System states and variables are visible

and queriable during execution– Past system states are ….. (transaction

logs)– All factors affecting output are visible

Observability

– Incorrect output is easily identified

– Internal errors are automatically detected through self-testing mechanisms

– Internal errors are automatically reported

–Source code is accessible

Visibility Spectrum

DSPvisibility

GPPvisibility

Factoryvisibility

End customervisibility

Controllability

• “The better we can control the software, the more the testing can be automated and optimized”–All possible outputs can be

generated through some combination of input–All code is executable through some

combination of input

Controllability

–SW and HW states and variables can be controlled directly by the test engineer

– Input and output formats are consistent and structured

Decomposability

• “By controlling the scope of testing, we can more quickly isolate problems and perform smarter testing”–The software system is built from

independent modules–Software modules can be tested

independently

Simplicity

• “The less there is to test, the more quickly we can test it”–Functional simplicity (feature set is

minimum necessary to meet requirements)–Structural simplicity (architecture

is modularized)–Code simplicity (coding standards)

Stability

• “The fewer the changes, the fewer the disruptions to testing”–Changes to the software are

infrequent, controlled, and do not invalidate existing tests

–Software recovers well from failures

Understandability

• “The more information we have, the smarter we will test”– Design is well understood– Dependencies between external,

internal, and shared components are well understood

– Technical documentation is accessible, well organized, specific and detailed, and accurate

“Bugs lurk in corners and congregate at boundaries”

Boris Beizer

Types of errors

• What is a Testing error?–Claiming behavior is erroneous

when it is in fact correct

– ‘fixing’ this type of error actually breaks the product

Errors in classification

• What is a Classification error ?–Classifying the error into the

wrong category

• Why is this bad ?–This puts you on the wrong path

for a solution

Example Bug Report

• “Screen locks up for 10 seconds after ‘submit’ button is pressed”

• Classification 1: Usability Error • Solution may be to catch user events

and present an hour-glass icon• Classification 2: Performance error• solution may be a modification to a sort

algorithm (or visa-versa)

Isolation error

• Incorrectly isolating the erroneous modules• Example: consider a client server

architecture. An improperly formed client request results in an improperly formed server response

• The isolation determined (incorrectly) that the server was at fault and was changed

• Resulted in regression failure for other clients

Resolve errors

• Modifications to remediate the failure are themselves erroneous

• Example: Fixing one fault may introduce another

What is the ideal test case?

• Run one test whose output is "Modify line n of module i."

• Run one test whose output is "Input Vector v produces the wrong output"

• Run one test whose output is "The program has a bug" (Useless, we know this)

More realistic test case

• One input vector and expected output vector– A collection of these make of a Test

Suite

• Typical (naïve) Test Case– Type or select a few inputs and observe

output– Inputs not selected systematically– Outputs not predicted in advance

Test case definition

• A test case consists of;– an input vector– a set of environmental conditions– an expected output.

• A test suite is a set of test cases chosen to meet some criteria (e.g. Regression)

• A test set is any set of test cases

Requirements as theory model

• Suppose we consider a specification to be a theory describing a program

• How do we test theories?• By examining the theory and using it

to make predictions• First Principle of testing. The

expected results of a test should be known before the test is run

Requirements as theory

• "Accumulating evidence to support a theory is not the appropriate way to test it. What you should do is try to falsify it, to challenge it with your best efforts at proving it false."–Karl Popper

Requirements as theory

• Implications for us doing testing• Testing should not be used to

build confidence, to easy• Testing should attempt to find

deviations from the theory, that is, bugs

• Any other purpose sets up the wrong goal

Requirements as theory

• "Program testing can be used to show the presence of bugs, but never show their absence!" O.-J. Dahl, E. W. Dijkstra, and C.A.R. Hoare, Structured Programming, New York: Academic, 1972.

• "Absence of proof (of bugs) is not proof of absence."; Logic 101

A few words about computability

• From the theory of computability, we know:

• It is undecidable whether a given program will halt on a given input. (Halting problem)

• It is undecidable whether two programs will always output the same answer for a given input. (Equivalence)

Implications for testing

• There is no general solution for the automated oracle problem–no automatic testing strategy can be

devised that will work in all cases

• There is no general way to find the input that causes a specific line of code to be executed

• Coverage is undecidable

All of the following are undecidable

• Will a given statement ever be exercised by any input?

• What input will exercise a given statement?• Will a given input exercise some specified

statement?• Will a given path ever be exercised by any

input?• What input will exercise a given path?• Will a given input exercise some specified

path?

Computability

• Note that even though it is in general undecidable, there is a large class of programs for which these issues can be decided– a large testing tools industry has

emerged because of this

• When examining a tool, make sure that the class of programs for which it works is well understood

Reference book

A few words on combinatorics

• Based on the Cartesian product of sets, we can count the number of possible inputs that a program has, i.e. | I |

Example

• Assume a program has a single input, Customer ID (CID)

• May be any value in the domain {00000-99999}

• What is | I | ?

• | I | = 100000

Example

• Now assume we add a second input, the Order ID (OID)

• This may be any value in the domain {00000-99999} as well

• Now what is | I |?

• 100000*100000 = 10,000,000,000

Example

• Finally, add a credit card number to the input

• This is a 12 digit number• | I | has now reached 10**22

• If we can execute 1 million tests per second, it will take 1016 seconds, or about 300 million years!!

Example

• Since we cannot know what data may exercise a given statement/path in general, we may attempt to resort to exhaustive testing• This attempt is doomed to fail

due to the combinatorial explosion

Functional and structural approaches to testing

Engineering the testing process

• Any engineered product (and most other things) can be tested in one of two ways–Knowing the specified function that

a product has been designed to perform, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function

Engineering the testing process

–Knowing the internal workings of a product, tests can be conducted to ensure that “all gears mesh” (internal operations performed according to specifications)

Structural testing

• Uses knowledge of the internal workings

• Also known as Clear box/glass box• Code based• Can be useful for finding interesting

inputs• Misses an entire class of faults,

missing code

Behavioral

• Uses knowledge of the specific function that is to be performed

• Based solely on the specification without regard for the internals

• Also known as Black box• More user oriented• Misses an entire class of faults, extra

code (surprises) except by accident

Passing criteria

• How do we know when

• 1. a single test has passed

• 2. when we are done testing

Passing criteria

• A single test passes when its output is correct–This requires a specific definition

of correct and ties into the automated oracle problem

When are we done?

• Conway Criteria:• No syntactic errors (it compiles)• No compile errors or immediate

execution failures• There exists Some set of data for

which the program gives the correct output• A typical set of data produces the

correct output

When are we done?

• Difficult sets of data produce the correct output.

• All possible data sets in the problem specification produce the correct output

• All possible data sets and likely erroneous input succeeds.

• All inputs produce the correct output

Nature of software defects

• Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed

Nature of software defects

• We often believe that a logical path is not likely to be executed when, in fact, it may be executed on a regular basis

• Typographical errors are random

More of a case for WHITE box testing……

Summary

• Zeroth Principle of Testing; The purpose of testing is to find bugs

• Corollary to the zeroth principle; "The program is wrong"

Summary

• First Principle of Testing; The results of a test must be known before the test is run

• Second Principle of Testing: Testing is difficult

Summary

• Exhaustive testing is doomed by the combinatorial explosion

• Any other technique is undecidable

• Third Principle of Testing: No single technique will suffice for any non-trivial testing effort

Homework 1b

• Discuss a software failure from your experience or knowledge and attempt to explain the role of testing (or lack thereof) in that failure

Another reference

• Testing techniques newsletter

• www.testworks.com/News/TTN-Online

More on coverage

Logical coverage

• What does statement coverage imply ?

• Each statement must be executed at least once

Example

• 1 IF (( X > 1) AND ( Y == 1))

• 2 Z = Z / X

• 3 END

• 4 IF ((X == 2) OR ( Z >1))

• 5 Z = Z + 1

• 6 END

What is required for statement coverage ?

Example

• To achieve statement coverage, one can choose x = 2, y = 1

• Many possible errors are missed:

• 1 IF ((X > 1) OR (Y ==1))• 1 IF ((X >=1) AND (Y ==1))• 1 IF ((X > 1) AND (Y >=1))

Blindness

• Statement coverage is blind to several classes of error when choosing values to achieve coverage

Assignment blindness

• Due to assignment of a particular value to a variable, the error does not propagate

Equality blindness

• Due to an Equality check of a particular variable, the error does not propagate

Self blindness

• Conditional itself covers up the error

Assignment Blindness example

• 1 IF (<conditional>)• 2 X = 1• 3 END• 4 IF (X + Y > 1) //Should have

been (Y > 0)• 5 ....• 6 END

Assignment Blindness example

• In this example, values are chosen to make the conditional true, the statement X=1 is executed and the error in line 4 is not seen

Assignment Blindness example

• If every path forces the same assignment, then the 'error' doesn't really matter, (does it exist??) –For instance, if the conditional in

statement 1 always evaluated to true

Equality Blindness Example

• 1 IF (A == 2)

• 2 .....

• 3 IF (B > 1) // Should have been (A + B > 3)

Equality Blindness Example

• In this example, the value of 2 is chosen for A to force execution of the body

• The error in statement 3 is missed

Self Blindness

• 1 IF (X < 1) // Should have been (2X < 1)

• In this example, the value of 0 is chosen for X

Observation

• Statement coverage is the weakest form of coverage

• Many classes of errors can escape this testing

• Typical projects have less than 80% statement coverage!!

Branch coverage

• Each branch of a decision must be executed at least once

• This is stronger that statement coverage, but it is still weak

• Note: A decision is a logical combination of one or more conditions

Branch coverage example

• 1 IF (( X > 1) AND ( Y == 1))

• 2 Z = Z / X

• 3 END

• 4 IF ((X == 2) OR ( Z >1))

• 5 Z = Z + 1

• 6 END

Branch coverage example

• To achieve branch coverage, one option is to choose–x = 2, y = 1

–x = 0, y = 2

Branch coverage example

• Many possible errors are still missed:

• 1 IF ((X > 1) OR ( Y ==1)

• 1 IF ((X >=1) AND (Y ==1))

• 1 IF ((X > 1) AND (Y <=1))

Blindness in branch coverage

• Branch coverage is blind to several classes of error when choosing values to achieve coverage–Compound decisions (more than

one condition) are weakly tested–Boundaries of conditions (within a

decision) are not explicitly tested

Condition coverage

• Each condition within a decision must assume all possible values

• This is 'potentially' stronger than branch coverage, but not always

• It may, in fact, be weaker

Example of condition coverage

• 1 IF (( X > 1) AND ( Y == 1))

• 2 Z = Z / X

• 3 END

• 4 IF ((X == 2) OR ( Z >1))

• 5 Z = Z + 1

• 6 END

Example of condition coverage

• To achieve condition coverage for statement 1, one must choose values such that:

• X > 1 ~(Y == 1)

• ~(X > 1) Y == 1

Example of condition coverage

• Both of these vectors miss the execution of statement 2

• A better choice may be:

• ~(X > 1) ~(Y == 1)

• X > 1 Y == 1

Multi-condition coverage

• Every combination of possible condition values within a decision must be chosen

• This is strictly stronger than branch or condition coverage–still has weaknesses!

Multi-condition coverage example

• 1 IF (( X > 1) AND ( Y == 1))

• 2 Z = Z / X

• 3 END

• 4 IF ((X == 2) OR ( Z >1))

• 5 Z = Z + 1

• 6 END

Multi-condition coverage example

• To achieve multi-condition coverage for statement 1, one must choose values such that:

• 1. X > 1 Y == 1• 2. X > 1 ~(Y == 1)• 3. ~(X > 1) Y == 1• 4. ~(X > 1) ~(Y == 1)

Multi-condition coverage example

• To achieve multi-condition coverage for statement 2, one must choose values such that:

• 5. X == 2 Z > 1• 6. X == 2 ~(Z > 1)• 7. ~(X == 2) Z > 1• 8. ~(X == 2) ~(Z > 1)

Coverage

X Y Z Covers

1 0 1 1,5

2 1 4 4,8

1 1 3 2,6

2 0 1 3,7

Multi-condition coverage example

• There are 9 paths through this code and this test set only covers 4 of those

Multi-condition coverage example

• Consider this error

• Line 2 is incorrectly written as Z = Z –X

• Only the second vector will execute this line but for the values chosen, Z-X = Z/X

Multi-condition coverage example

• Also consider that no vector will execute both line 2 and line 5

• In a more complex example, it is easy to imagine variables being set in line 2 that are then incorrectly used in line 5

Multi-condition coverage example

• For instance:

• 2. Z = Z/X; A = 0

• ....

• 5. Z = Z + 1; B = 1/A;

Path coverage

• Every possible path must be chosen

• Finding vectors to accomplish this is, in general, undecidable, and usually difficult

Path coverage

• This still misses boundary conditions

• If statement 1 was mis-typed as X >= 1, it is conceivable that the chosen test set does not contain a vector with X=1, missing this error

Backup

Homework • Suppose we have a routine r1 that sends its output to

routine r2, as in a unix pipe. Discuss the implications of R1 != D2, where R1 is the range of routine r1 and D2 is the domain of routine r2

• Suppose a program accepts as input, a 16 bit positive integer, i, and a single 8 bit character, c. The program has several faults. the first fault is the absence of 0 checking in a denominator. The parameter i is used as this denominator. There is another fault in that if the parameter c has a value between a and z, i.e. it is a lower case character, values for the parameter i less than 100 will fail. What is the probability that an input vector chosen at random will discover a fault?