how does software fail?

Chapter 8 7-1Lecturer: Dr. AJ Bieszczad

How does software fail?

• Wrong requirement: not what the customer wants

• Missing requirement

• Requirement impossible to implement

• Faulty design

• Faulty code

• Improperly implemented design


Terminology

• Fault identification: what fault caused the failure?

• Fault correction: change the system

• Fault removal: take out the fault


Types of faults

• Algorithmic fault• Syntax fault• Computation and precision

fault• Documentation fault• Stress or overload fault• Capacity or boundary fault• Timing or coordination

fault

• Throughput or performance fault

• Recovery fault• Hardware and system

software fault• Documentation fault• Standards and procedures

fault


Table 8.1. IBM orthogonal defect classification.

Fault type MeaningFunction Fault that affects capability, end-user interfaces, product interfaces,

interface with hardware architecture, or global data structureInterface Fault in interacting with other components or drivers via calls, macros,

control blocks or parameter listsChecking Fault in program logic that fails to validate data and values properly before

they are usedAssignment Fault in data structure or code block initialization.Timing/serialization Fault that involves timing of shared and real-time resourcesBuild/package/merge Fault that occurs because of problems in repositories, management changes,

or version controlDocumentation Fault that affects publications and maintenance notesAlgorithm Fault involving efficiency or correctness of algorithm or data structure but

not design


HP fault classification


Faults distribution (HP)

Data handling

6%

Other code

11%

Documentation

19%

Computation

18%

Logic

32%

Process

5%Hardware

4%

Requirements

5%


Testing steps

Unittest

Unittest

Unittest

Integrationtest

Functiontest

Performancetest

Acceptancetest

Installationtest

Designspecifications

Systemfunctional

requirements

Othersoftware

requirements

Customerrequirementsspecifications

Userenvironment

SYSTEMIN USE!


Views of Test Objects• Closed box or black box

• the object seen from outside• tests developed without the knowledge of the internal structure• focus on inputs and outputs

• Open box or clear box• the object seen from inside• tests developed so the internal structure is tested• focus algorithms, data structures, etc.

• Mixed model• Selection factors:

• the number of possible logical paths• the nature of input data• the amount of computation involved• the complexity of algorithms


Exmining code

• Code walkthrough– the developer presents the code and the documentation to a review team

– the team comments on their correctness

• Code inspection– the review team checks the code and documentation against a prepared list of

concerns

– more formal than a walkthrough

• Inspections criticize the code, and not the developer


Table 8.2. Typical inspection preparation and meeting times.

Development artifact Preparation time Meeting timeRequirements document 25 pages per hour 12 pages per hourFunctional specification 45 pages per hour 15 pages per hourLogic specification 50 pages per hour 20 pages per hourSource code 150 lines of code per hour 75 lines of code per hourUser documents 35 pages per hour 20 pages per hour

Table 8.3. Faults found during discovery activities.

Discovery activity Faults found per thousand lines of codeRequirements review 2.5Design review 5.0Code inspection 10.0Integration test 3.0Acceptance test 2.0


Proving code correct• Formal proof techniques

• + formal understanding of the program

• + mathematical correctness

• - difficult to build a model

• - usually takes more time to prove correctness than to write the code itself

• Symbolic execution• + reduces the number of cases for the formal proof

• - difficult to build a model

• - may take more time to prove correctness than to write the code itself

• Automated theorem-proving• + extremely convenient

• - tools exists only for toy problems at research centers

• - ideal prover that that can infer program correctness from arbitrary code can never be built (the problem is equivalent to the halting problem of Turing machines, which is unsolvable)


Testing program components

• Test point or test case– A particular choice of input data to be used in

testing a program

• Test– A finite collection of test cases


Test thoroughness• Statement testing• Branch testing• Path testing• Definition-use testing• All-uses testing• All-predicate-uses/some-computational-uses testing• All-computational-uses/some-predicate-uses testing


Example: Array arrangement

Statement testing:•1-2-3-4-5-6-7

Branch testing:•1-2-3-4-5-6-7•1-2-4-5-6-1

Path testing:•1-2-3-4-5-6-7•1-2-3-4-5-6-1•1-2-4-5-6-7•1-2-4-5-6-1


Relative strength of test strategiesAll paths

All definition-use paths

All uses

All computational/some predicate uses

All definitions

All computational uses

All predicate/some computational uses

Statement

All predicate uses

Branch


Comparing techniquesTable 8.5. Fault discovery percentages by fault origin.

Discovery technique Requirements Design Coding DocumentationPrototyping 40 35 35 15Requirements review 40 15 0 5Design review 15 55 0 15Code inspection 20 40 65 25Unit testing 1 5 20 0

Table 8.6. Effectiveness of fault discovery techniques. (Jones 1991)

Requirementsfaults

Design faults Code faults Documentationfaults

Reviews Fair Excellent Excellent GoodPrototypes Good Fair Fair Not applicableTesting Poor Poor Good FairCorrectnessproofs

Poor Poor Fair Fair


Integration testing

• Bottom-up

• Top-down

• Big-bang

• Sandwich testing

• Modified top-down

• Modified sandwich


Component hierarchy for the examples


Bottom-up testing


Top-down testing


Modified top-down testing


Big-bang integration


Sandwich testing


Modifying sandwich testing


Bottom-up Top-down Modifiedtop-down

Big-bang Sandwich Modifiedsandwich

Integration Early Early Early Late Early EarlyTime tobasicworkingprogram

Late Early Early Late Early Early

Componentdriversneeded

Yes No Yes Yes Yes Yes

Stubsneeded

No Yes Yes Yes Yes Yes

Workparallelismat beginning

Medium Low Medium High Medium High

Ability totestparticularpaths

Easy Hard Easy Easy Medium Easy

Ability toplan andcontrolsequence

Easy Hard Hard Easy Hard Hard

Comparison of integration strategies.


Sync-and-stabilize approach (Microsoft)


Testing OO vs. testing other systems


Where testing OO is harder?


Test planning

• Establish test objectives

• Design test cases

• Write test cases

• Test test cases

• Execute tests

• Evaluate test results


Test plan• Purpose: to organize testing activities• Describes the way in which we will show our

customers that the software works correctly• Developed in parallel with the system

development• Contents:

– test objectives

– methods for test execution for each stage of testing

– tools

– exit criteria

– system integration plan

– source of test data

– detailed list of testcases


Automated testing tools

• Code analysis– Static analysis

• code analyzer

• structure checker

• data analyzer

• sequence checker

– Dynamic analysis• program monitor

• Test execution– Capture and replay – Stubs and drivers– Automated testing

environments

• Test case generators


Sample output from static code analysis


Probability of finding faults during development


When to stop testing• Coverage criteria• Fault seeding

• Estimate of remaining faultsS – seeded faults, n – discovered faults, N – actual faults

N = Sn/s

• Two test team testingx - faults found by group 1, y – by group 2, q – dicovered by both

E1=x/n E2=y/n q <=x, q<=y

also: E1=q/y (group 1 found q faults out of group’s 2 y)

so: y=q/E1 and n=y/E2

therefore: n=q/(E1E2)

detected seeded faults = detected non-seeded faults total seeded faults total non-seeded faults


When to stop testing• Coverage criteria

• Fault seeding

• Confidence in the software

detected seeded faults = detected non-seeded faults total seeded faults total non-seeded faults

C = 1 if n > NS/(S-N+1) if n < N

S – seeded faults, n – discovered faults, N – actual faults (e.g., N=0, S=10, n=10 91%

N=0, S=49, n=49 98%)


Classification tree to identify fault-prone components

how does software fail?

Documents