lecture1_extensive prologue for software quality

8/7/2019 Lecture1_Extensive Prologue for Software Quality

1/48

Introduction to Software Testing

An Extensive Prologue forSoftware Quality


2/48

Syed Raheel Abbas- Delivered at Mohammad Ali Jinnah University

Software testing motivation.

What is a software testing?

Verification and Validation?

Error , Defect and Bug

Quality Control and Quality Assurance

Why software fail?

What is cost of software failures?

What makes a good software tester?

22

Lecture Outline


3/48


The Lion King Animated Storybook

Disneys first multimedia CD-ROM game forkids. Released at Christmas season.

26th December.. Customer SupportsNightmare.

The CD was testing only for specific PCplatform.

It failed on many popular PC operating system.

33

Software Testing Motivation


4/48


Intel Pentium Floating-Point Bug

(4195835/3145727)* 3145727 - 4195835

If answer is ZERO, your computer is fine.

Anything else, you have old Intel Pentium CPUwith floating-point division bug.

A software bug burned into a computer chip and

reproduced over and over in the manufacturingprocess.

44



5/48


NASA Mars Polar Lander, 1999

3rd December 1999, Mars Polar Landerdisappeared during its landing attempt.

Failure Review Board concludes the likely failurereason was the unexpected setting of a singledata bit.

Most alarming was why the problem was notcaught by internal tests.

55



6/48


Malaysia Airlines Jetliner, August 2005

Flight between Perth, Australia and KualaLampur, Malaysia zoomed 3,000 feet upwards.

A defective software program had providedincorrect data about the aircrafts speed andacceleration, confusing flight computers.

66



7/48 Syed Raheel Abbas- Delivered at Mohammad Ali Jinnah University

The process of finding evidence of defects insoftware systems.

Establishing confidence that a program does

what it is supposed to do.

77

What is Software Testing?

1. Software testing is not debugging.

2. Software testing is not quality assurance


8/48 Syed Raheel Abbas- Delivered at Mohammad Ali Jinnah University

Correctness of software with respect torequirements or intent;

Performance of software under various

conditions;Robustness of software, its ability to handleerroneous input and unanticipatedconditions;

Installation and other facets of a softwarerelease.

88

What is Software Testing?


9/48

Software Testing

To find and prevent defect

To have confidence that the systemperforms adequately

To provide information about quality

To provide the insight of product riskand quality

To ensure a program corresponds to itsspecification

Defect Detection Mechanism


10/48

Software Test Process

Test Strategy

Test Plan

Test Design

Setup Test EnvironmentTest Execution

Defect Tracking

Testing Tools

Evaluate Test Exit Criteria

Reporting

Improve Test Process


11/48

Some terminologies

Error

Defect, Fault, Bug

Incident, Issue

Failure

Risk

Related to

human=Mistake

Related to product

Bug= slang / dialect

Related to event

Accident

Risk = likelihood of failure

x impact


12/48


Simply any thing find by tester its defect,when the developer accepts the defect itsbug. Any thing found faulty i.e. fault is anerror.

Fault--> Error--> Defect--> Bug

12

12

Summary


13/48


For any integer n, square (n) = n*n.

int square (int x)

{

return x*2;

}

13

13

Trivial Example

Square (3) = 6

Failure

Fault


14/48


For any integer n, square (n) = n*n.

int square (int x)

{

return x*2;

}

14

14

Trivial Example

Square (2) = 4

Correct Result


15/48


MyersTesting is the process of executing a program withintention of finding errors

Paul Jorgensen

Testing is obviously concerned with errors, faults,failures and incidents. A test is the act of exercisingsoftware with test cases with an objective of Findingfailure and Demonstrate correct execution

ISOTechnical operation that consist of the determination ofone or more characteristics of a given product, processor service according to a specified procedure

15

15

Testing...Definition Revisited


16/48

Verification and Validation

Validation does the software do what was wanted? Are we building the right system?

Validation is the process of evaluating software artifacts during the software

development process in an attempt to determine if the system works as

required by the customers.

Verification does the software meet its specification?

Are we building the System right??

This can only be objective if the specifications are sufficiently precise

Everything must be verified including the verification process itself


17/48


Verification Are we building the productright?

Verification is any checking processconducted on software artifacts in an

attempt to determine if they work asspecified by the designers of the system.

Includes reviews, inspections, walkthroughs,

unit testing and integration testing.

Verification


18/48


Validation Are we building the right product?

Validation is the process of evaluating softwareartifacts during the software developmentprocess in an attempt to determine if thesystem works as required by the customers.Any evaluation activity that involves thecustomer can be used for validation purposes.

Includes program reviews, system testing,customer acceptance testing.

Validation


19/48


Verification Main purpose is to detect defects

in the artifacts of the system underdevelopment.

Validation Main purpose is to show that the

system under development meetsuser needs, requirements, andexpectations.

Validation and Verification

process continue in a cycle till

Verification vs. Validation


20/48

20

20

What is the objectives ofSoftware Testing?


21/48

Quality Assurance vs.

Quality ControlQuality Assurance

An overall

management plan to

guarantee the

integrity of data

(The system)

Quality Control

A series of

analyticalmeasurements used

to assess the

quality of theanalytical data

(The tools)


22/48


To identify and reveal as many errors aspossible in the tested software

To bring the tested software, after correctionof the identified errors and retesting to an

acceptable level of quality

To perform the required tests efficiency andeffectively, within budgetary and scheduling

limitation

22

22

Direct Objectives


23/48


To compile a record of software errors for usein error prevention (by corrective andpreventive actions)

23

23

Indirect Objective


24/48

Goals of Testing

Goal: show a program meets its

specification But: testing can never be complete for non-

trivial programs

What is a successful test? One in which no errors were found?

One in which one or more errors were

found?


25/48

Goals of Testing - 2

Testing should be: repeatable

if you find an error, you want to repeat the test to

show others

if you correct an error, you want to repeat the test to

check that you fixed it systematic

random testing is not enough

select test sets that are representative of real uses

select test sets that cover the range of behaviors ofthe program

documented

keep track of what tests were performed, and what

the results were


26/48

Goals of Testing - 3.

Therefore you need a way to document

test cases showing: the input

the expected output

the actual result

These test plans/scripts are critical to

project success!


27/48


27

27

Why do Failure Occur?


28/48


Inaccurate understanding of end userrequirements.

Inability to deal with changing requirements.

Late discovery of serious project flaws.

For example, modules that do not fittogether.

Untrustworthy build & release process.

Implementation teams chaos.

28

28

Root Causes of Failures

F il C t


29/48


29

29

Failure Costs

Disneys Lion King CD


30/48


Good understanding of the developmentprocess and its products.

Ability to anticipate likely faults and errors.

30

30

Software Testers


31/48


31

31


32/48


Correctness accuracy, completeness of requiredoutput

uptodateness, availability of theinformation

Reliability maximum failure rate

Efficiency resources needed to perform software function

Integrity software system security, access rights

Usability ability to learn, perform required task

32

32

Software Quality Factors


33/48


Maintainability

effort to identify and fix software failures (modularity,

documentation, etc)

Flexibility

degree of adaptability (to new customers, tasks, etc)

Testability

support for testing (e.g. log files, automatic diagnostics,

etc)

33

33

Software Quality Factors.


34/48


Portability

adaptation to other environments (hardware,

software) Reusability

use of software components for other projects

Interoperability

ability to interface with othercomponents/systems

34

34

Software Quality Factors.


35/48


1. It is impossible to test a program completely.

2. Software testing is a risk-based exercise.

3. Testing cannot show the absence of bugs.

4.

The more bugs you find, the more bugs there are.5. Not all bugs found will be fixed.

6. It is difficult to say when a bug is indeed a bug.

7. Specifications are never final.

8.

Software testers are not the most popular members of aproject.

9. Software testing is a disciplined and technical profession.

35

35

Software Testing proverbs


36/48


How many test cases do you need to exhaustively test:

Powerpoint

A calculator

MS Word

Any interesting software!

The only way to be absolutely sure software works is torun it against all possible inputs and observe all of itsoutputs

Oh, and the specification must be correct and complete.

36

36

proverb 1It is impossible to test a program

completely

proverb 1 (contd)


37/48


The number of possible inputs is very large.

The number of possible outputs is verylarge.

The number of paths through the software isvery large.

The software specification open tointerpretation.

37

37

proverb 1 (contd)It is impossible to test a programcompletely

b


38/48


If you do not test the software for all inputs (a wisechoice) you take a risk.

Hopefully you will skip a lot of inputs that workcorrectly.

What if you skip inputs that cause a fault?Risk: financial loss, security, loss of money, loss of life!

That is a lot of pressure for a tester!

This course is all about techniques and practices to

help reduce the risk without breaking the bank.

38

38

proverb 2Software testing is a risk-based exercise

b 2 ( td)


39/48


If you try to test toomuch, thedevelopment costbecomes prohibitive.

If you test too little,the probability ofsoftware failureincreases and as we

discussed softwarefailures can cost usbig time!

39

39

proverb 2 (contd)Software testing is a risk-based exercise

Cost of

TestingNumber of

Missed Bugs

Over

Testing

Amount of Testing

Under

Testing

TestingEquilibrium

Q

u

an

t

i

t

y


40/48


Program testing can be used toshow the presence of bugs, but neverto show their absence!

- Edsger Dijkstra

40

40

proverb 3Testing cannot show the absence of bugs


41/48


Bugs appear in groups, where you see one you will likely find more Why?

Programmers can have bad days

Programmers tend to make the same mistakes

Some bugs are just the tip of the iceberg.

Boris Beizer coined the term pesticide paradox to describe thephenomenon that the more you test software the more immune it

becomes to your test cases.Remedy: continually write new and different tests to exercise different parts ofthe software.

41

41

proverb 4The more bugs you find, the more bugs thereare

proverb 5


42/48


Why wouldnt you fix a bug you knew about?

Theres not enough time

Some deadlines cannot be extended (e.g., Y2K)

Its not really a bug

Specifications can be wrong

Its too risky to fix

Im not touching Shahryarzzzz code!

Its just not worth it

Bugs in fringe features may have to wait

Why not charge the customer for bug fixes in the next release(sound familiar?) :-)

42

42

proverb 5Not all bugs found will be fixed

proverb 6


43/48


If there is a problem in the software but no oneever discovers it is it a bug?

Parody of if a tree falls in the forest does it

really make a noise?What is your opinion? Does a bug have to beobservable in order for it to me a bug?

Bugs that are undiscovered are called latent bugs.

43

43

proverb 6It is difficult to say when a bug is indeed a bug

proverb 7


44/48


Building a product based on a moving target specificationis fairly unique to software development.

Competition is fierce

Very rapid release cycles

Software is easy to change

Not true in other engineering domains

E.g., the Brooklyn Bridge could not be adjusted to allow train traffic tocross it once its construction started.

44

44

proverb 7Specifications are never final


45/48


Goal of a software tester:

Find bugs

Find bugs early

Make sure bugs get fixed

Tips to avoid becoming unpopular:

Find bugs early

Temper your enthusiasm act in a professional manner

Dont report just the bad news

45

45

proverb 8Software testers are not the most popularmembers of a project

proverb 9


46/48


When software was simpler and more manageable softwaretesters were often untrained and testing was not donemethodically.

Contrary to the urban legend, if you hire a million monkeys and have themtest for a million years you will not find all of the bugs in your software.

It is too costly to build buggy software. As a result testing hasmatured as a discipline.

Sophisticated techniquesTool support

Rewarding careers

46

46

proverb 9Software testing is a disciplined and technicalprofession

K P i t


47/48


Software testing is the process of discoveringevidence of defects and failures in softwaresystems.

Test early, test often, test enough.

Testers should have good understanding ofthe development process, product.

47

47

Key Points


48/48

the 9 proverbs of software testing

what is software verification

what is software validation

the relationship between software quality andreliability

the relationship between Quality Control and qualityassurance or Software Testing and Software Quality

Priority and Severity of Defect.

48

You now know

lecture1_extensive prologue for software quality

Documents