Introduction ToSoftware Testing

Brian Nielsen bnielsen@cs.auc.dkCenter of Embedded Software Systems

Aalborg University, Denmark

C SS10101110110101011011010101110111

CISS

Ariane 5 crash• An overrun exception stopped both replicated

position calculation computers (SRI’s) at time H0+36.7. Fail-stop was a wrong design decision

• Overrun caused by higher velocity in Ariane 5 compared to Ariane 4

• Overrun occured in a module which was irrelevant after take-off

• Simulations were not run with the new velocity (would have caught the failure)

CISS

Introducing, Detecting andRepairing Errors Liggesmeyer 98

CISS

What is testing?

CISS

TestingTesting:• to check the quality (functionality, reliability,

performance, …) of an (software) object

-by performing experiments-in a controlled way

• In avg. 10-20 errors per 1000 LOC• 30-50 % of development time and cost in embedded software

CISS

Testing Objectives• To identify as many errors as possible• To bring the software to an acceptable level of

quality• To determine risk of release• Perform testing efficiently, effectively and within

budget and time constraints• Compile a record of software errors for future

error prevention and correction (learning organization).

CISS

A Self-Assessment Test [Myers]

• “A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.”

• Write a set of test cases to test this program

CISS

Test cases for:

A Self-Assessment Test [Myers]

1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?6. negative side ?7. one side is sum of others ?8. 3 permutations of previous ?

9. one side larger than sum ofothers ?

10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?13. wrong number of values ?14. for each test case: is

expected output specified ?15. check behaviour after

output was produced ?

What is a Test?

Software under Test

Test Data Output

Test Cases

Correct result?

Oracle

CISS

Testing vs. Model-Checking• examines physical

system• as finished as possible• in an environment close

to its real usage• no proof• very shallow test

sample• ad-hoc, error prone,

laborious ?!!

• examines an abstract model

• proof of correctness• cheap and early design

verification• only selected, user

specified properties, or generic

• state space explosion• requires formal models

Complementary Techniques!! Apply Both!!

CISS

Types of Testing

unit

integration

system

performancerobustness

functional behaviour

white box black box

Level

Accessibility

Aspect

usability

reliability

CISS

Quality-Characteristics (ISO-9126)• Functionality

• Suitability, accuracy, security, compliance, interoperability

• Reliability• maturity, fault tolerance, recoverability

• Usability• understandability, learnability, operability

• Efficiency• time behaviour, resource utilization

• Maintainability• Analysability, changeability, stability, testability

• Portability• Adaptability, intallability, conformance, replaceability

⇒ functional testing

⇒ reliability testing

⇒ usability testing

⇒ performance testing

⇒maintainability testing ??

⇒ portability testing ?

CISS

Whitebox Exampleint invoice (int x, int y) {int d1, d2, s;if (x<=30) d2=100;else d2=90;s=5*x + 10 *y;if (s<=200) d1=100;else if (s<=1000) d1 = 95;

else d1 = 80;return (s*d1*d2/10000);

}

ifd2=100 d2=90

ifd1=100

if

d1=95

d1=80

return

x>30

s>200

s>1000

Test Data Expected Output

X=5 Y=5 75

X=31 Y=10 229.5

X=30 Y=100 977.5

Test Cases

CISS

Black box testing

SUT

requirements

input

events

output

y

x

domain testing

CISS

Unit test

test driver

Unit under test

test stub

A

FE

B C D

G

Component Hierarchy

test stub

•C-function•Abstract Data Type•Object•Process / thread

•Function call•Method invocation•Message passing•Shared variables

Stub: a code fragment that simulates a missing unit

•Simulate•Pre-programmed return values•Simplified impl.

Driver: a code fragment that invokes the U-ut Simulate

•Supplies input data•Invokes Unit•Collects and compares results•~xUnit testing frame work

CISS

Unit-testing• Goal: Ensure that module functions correctly

according to module spec• Module interfaces are tested for proper

information flow.• Local data are examined to ensure that integrity is

maintained.• Boundary conditions are tested.• Basis path testing should be used.• All return values• All error handling paths should be tested.

• Drivers and/or stubs need to be developed to test incomplete software.

• Mostly white-box techniques.

CISS

Integration Testing• Goal: Systematic construction of program

structure and testing for interfacing errors• Loss of data/information across interfaces• Side-effects (shared data-structures)• Accumulation of errors (imprecision)

• Incremental• Consider required cost and thoroughness• Mostly blackbox, supplemented with white-

box tests to ensure coverage

CISS

TestA

TestA,B,C,DE,F,G

TestB

TestC

TestD

TestE

TestF

A

FE

B C D

G

Big-bang integration

Integration Order

1. Test all components in isolation2. Integrate all in one step

Component Hierarchy

- Stubs and drivers needed!- Difficult to localize cause of errors- Difficult to distinguish interface faults

TestG

CISS

TestE

TestA,B,C,DE,F,G

TestF

TestG

TestB,E,F

TestC

TestD,G

A

FE

B C D

G

Buttom-Up integration

Integration Order

1. Test lower levels first (utility routines)2. Implement driver (easy) to invoke module3. Use tested modules as stubs

Component Hierarchy

+No stubs needed!-Mundane low-level utilities tested first, but most

important top level modules tested late • Critical errors found late• Main design problems are at higher-

levels• Timing determined by high-level

modules

CISS

TestA,B,C,DE,F,G

TestA

TestA,B,C,D

A

FE

B C D

G

Top-Down integration

Integration Order

1. Test higher levels first (controlling modules)2. Implement stubs to deliver responses3. Use tested modules as drivers

Component Hierarchy

+No drivers needed!-Many stubs may be needeed-Stubs may be problematic to write

Must provide values as expected by testerMay require frequent alteration

CISS

TestE

TestA,B,C,DE,F,G

TestF

TestG

TestB,E,F

TestD,G

A

FE

B C D

G

Sandwich integration

Integration Order

1. Combination of B-up and T-Down2. Divide system into upper,

middle (target), and lower levels3. B-up in low-levels4. T-Down in upper levels

Component Hierarchy

+ combines B-up and T-Dn+ Integration possible early-Individual components not tested thoroughly

before integration, especially middle level

TestA

CISS

Testability = how easy can a program be tested

• operability - it operates cleanly• observability

• the results are easy to see• distinct output is generated for each input• incorrect output is easily identified

• controllability• processing can be controlled• tests can be automated & reproduced

• decomposability - software modules can be tested independently

• simplicity - no complex architecture and logic• stability - few changes are requested during testing• understandability - program is easy to understand

CISS

Verification & Validation• Verification :

Software correctly implements a specific function

“building the product right” (Boehm81)

• concerns development process

• intermediate products

• e.g. checking the testability of the specification

• Validation :

Tracable to user requirements

“building the right product” (Boehm81)

• whether final product meets requirements and needs

• e.g. by reviewing the specification

CISS

System Test (Pfleger)• Unit & integration: code implements design correctly• System testing: system does what customers expect

• Constructed software tested against established requirements during requirements analysis

• Validation criteria satisfied?• Are all user-visible functional, behavioral, performance

requirements satisfied?

Functiontest

Non-functionaltest

Acceptancetest

Installation/Production

test

Systemfunctional

requirements

RequirementsFor other

Quality aspects

•Performance•Stress / load•Reliability,•recoverability•Security •Installability,•…

Customer RequirementsSpecification

UserEnvironment

SystemIn

useIntegratedmodules

functioningsystem

accepted,validatedsystem

verifiedsystem

CISS

System test

• 2*CRTG (4 channels) 2 * 200 k€

CISS

Test Equipment

• Complete Type Approval Test System (3 M€)

CISS

Acceptance Test (Pfleger)• By customers,

• User’s understanding of requirements• Customer writes, conducts, and evaluates

acceptance tests• Functional Acceptance Test (FAT)• Production Acceptance Test (PAT)• Techniques

• Benchmark tests• Pilot tests ~ experimental, limited edition

– Alpha test: in-house, controlled environment, developer supervised

– Beta test: “live” test at customer sites, unsupervised• Parallel test: run concurrently with old system

CISS

Who Should Test?

• Developer• Understands the system• But, will test gently• And, is driven by

deadlines

• Independent tester• Must learn system• But, will attempt to break it• And, is driven by “quality”

CISS

Risk Based Test Strategy• Make best possible use of resources by

identifying and prioritizing quality aspects and subsystems• Higher risk ⇒ more testing• No risk ⇒ no testing

• Risk = chance of failure × damage

Use frequencyChance of error being present

ComplexityNew tools/techniquesInexperienced developers

Cost of repairLoss of market shareLegal claim

CISS

Testing• Dynamic testing is the process of executing a

program or system with the intent of finding error (Glenford Meyers’ definition)

• Static testing is any activity that aims at finding defects by inspecting, reviewing, walking through, and analyzing any static component of the software (code, documents, and models)

• Debugging is an ad hoc activity performed by individual developers to find and remove bugs from a program.

• Testing is a planned activity

CISS

Testing• Quality assurance activities by running code –

examines functionality in the form used by customers

• Software testing is a formal process carried out by a specialized testing team in which a software unit, several integrated software units or an entire software package are examined by running programs on a computer. All the associated tests are performed according to approved test procedures on approved test cases.

CISS

Lifecycles & testing

CISSimplementation

code

detailed design

requirements

specification

acceptancetest

systemtest

moduletest

unit-test

V - Model

integrationtest

architecturespec

acceptance test spec

system test spec

integration test spec

module test spec

unit test spec

CISS

• Testing as phase of development process vs.

• Testing as a process itself• with its own phases

• in parallel with development process

Life Cycle : Testing as a Process

plan

ning

& co

ntro

l

prep

arat

ion

test

gene

ratio

n

test

exec

utio

n

com

plet

ion

quality report

implementationspecificationrequirements

T

L

OI

Techniques

Life-cyclefor testing

OrganizationInfrastructureand tools

CISS

Challenges of Testing• New embedded systems

• more functionality• increasingly advanced• faster time-to-market• higher quality

• Testing • more to be tested• more complicated• in less time• more thorough

• skilled developers and testers• advanced testing tools and techniques• well organized• using solid development method

CISS

• No realistic reliability models for software

• Moving implementation deadlines...... but fixed delivery deadlines

systemdelivery

startproject

code delivery =start test execution

Challenges of Testing

CISS

Challenges of Testing• Infinity of testing:

• too many possible input combinations -- infinite breadth• too many possible input sequences -- infinite depth• too many invalid and unexpected inputs

• Exhaustive testing never possible:• when to stop testing ?• how to invent effective and efficient test cases with high

probability of detecting errors ?

• Optimization problem of testing yield and invested effort• usually stop when time is over ......

CISS

Challenges of Testing• What is an effective method to measure coverage ?

• Regression testing: • very important• very boring and expensive• must be automated

• Bad specification or no specification at all• what do you test ?• test oracle problem

• Many operating environments and contexts

CISS

Challenges of Testing• Lack of appropriate tools

• Diversified fields• Experts dispersed, but also doesn’t talk

across application domain.• Tools are specialized, sells in low volume• Tools are expensive, • Tools are immature• No money available for test tools

CISS

Software Quality

CISS

Software Qualities

MaintainerGood Documentation

Readable CodeGood Design

Customer

Low CostPortability Increased

productivity

ReliabilityCorrectnessEfficiency

User

FunctionalityEase of useEase of learning

CISS

Costs of Poor Quality• Increased time to find and fix problems• Increased cost to distribute

modifications• Increased customer support• Product liability• Failure in the market place

CISS

Software Quality Assurance• Quality Control

• Process definition and standards • Formal technical review• Test planning and review

• Quality Assurance• Analysis and reporting• Measurement

CISS

Assuring that a software system meets a user’s needs

Verification vs. Validation:

• Verification: • “are we building the product right?”• The software should conform to its specification

• Validation:• “are we building the right product?”• The software should do what the user really

requires

CISS

static analysisstatic analysis

reviewingreviewinginspectioninspection

walk-throughwalk-throughdebuggingdebugging

certificationcertification

CMMCMM

quality controlquality controlrequirement managementrequirement managementsoftware processsoftware process

There is More than Testing

CISS

Summary

CISS

Some Testing Principles• Testing starts during the requirements phase• The programmer shall not be the tester• A test case specifies the test inputs

and the expected outputs• Test cases shall also cover invalid and unexpected inputs• Test cases shall test that the program does what it should do

and that it does not do what it should not do• Test cases shall be recorded for reuse• A test is successful when it detects an error

( but the project manager thinks differently ! )• No risk, no test

CISS

Lessons• Testing starts when project starts• Have a development approach

• How to develop (waterfall, cyclic, etc) : roles and activities• Clear expectations between testers and developers• Clarity about phases and tasks• Easier planning

• Test execution takes 3 steps• Test, fix, re-test

• Let the programmers find bugs• 80 % of bugs can be found by programmers (but they don’t)

• Every morning discuss the issues found• Testers create test report at end of the day• Next morning discuss which issues have to be solved• Programmer and tester fix issue together

CISS

Non-Structured testing• Start too late• Don’t organize and manage your test process• No test plan (V&V plan)• Unclear responsibilities• Conflicting interests• No fixed or stable specifications• Don’t learn

• from a defect• from a test• from statistics

• Use test tools in a chaotic environment• No configuration management and change control• No standard test procedures• Don’t educate• Do not provide career options

CISS

END