Lecture 9 Testing

Lecture 9 Testing

Topics Topics Testing

Readings:Readings:

Spring, 2008

CSCE 492 Software Engineering

– 2 – CSCE 492 Spring 2008

OverviewOverviewLast TimeLast Time

Achieving Quality Attributes (Nonfunctional) requirements

Today’s Lecture Today’s Lecture Testing = Achieving Functional requirements

References: References: Chapter 8 - Testing

Next Time:Next Time:

Requirements meetings with individual groupsRequirements meetings with individual groups

Start at 10:15Start at 10:15

Sample test - Sample test -

– 3 – CSCE 492 Spring 2008

TestingTesting

Why Test?Why Test?

The earlier an error is found the cheaper it is to fix.The earlier an error is found the cheaper it is to fix.

Errors/bugs terminologyErrors/bugs terminology

A fault is a condition that causes the software to fail.A fault is a condition that causes the software to fail.

A failure is an inability of a piece of software to A failure is an inability of a piece of software to perform according to specifications perform according to specifications

– 4 – CSCE 492 Spring 2008

Testing ApproachesTesting Approaches

Development time techniquesDevelopment time techniques

Automated tools: compilers, lint, etcAutomated tools: compilers, lint, etc

Offline techniquesOffline techniques

WalkthroughsWalkthroughs

InspectionsInspections

Online TechniquesOnline Techniques

Black box testing (not looking at the code)Black box testing (not looking at the code)

White box testingWhite box testing

– 5 – CSCE 492 Spring 2008

Testing LevelsTesting Levels

Unit level testingUnit level testing

Integration testingIntegration testing

System testingSystem testing

Test cases/test suitesTest cases/test suites

Regression testsRegression tests

– 6 – CSCE 492 Spring 2008

Simple Test for a Simple FunctionSimple Test for a Simple Function

Test cases for the function ConvertToFahrenheitTest cases for the function ConvertToFahrenheit

Formula Formula

Fahrenheit = Celsius * scale + 32 ; // scale = 1.8Fahrenheit = Celsius * scale + 32 ; // scale = 1.8

Test Cases for f = convertToFahrenheit(input);Test Cases for f = convertToFahrenheit(input);

1.1. convertToFahrenheit(0); // result should be 32convertToFahrenheit(0); // result should be 32

2.2. convertToFahrenheit(100); // result should be 212convertToFahrenheit(100); // result should be 212

3.3. convertToFahrenheit(-10); // result should be ???convertToFahrenheit(-10); // result should be ???

– 7 – CSCE 492 Spring 2008

Principles of Object-Oriented TestingPrinciples of Object-Oriented TestingObject-oriented systems are built out of two or Object-oriented systems are built out of two or

more interrelated objectsmore interrelated objects

Determining the correctness of O-O systems Determining the correctness of O-O systems requires testing the methods that change or requires testing the methods that change or communicate the state of an objectcommunicate the state of an object

Testing methods in an object-oriented system is Testing methods in an object-oriented system is similar to testing subprograms in process-similar to testing subprograms in process-oriented systemsoriented systems

– 8 – CSCE 492 Spring 2008

Testing TerminologyTesting Terminology

Error - refers to any discrepancy between an actual, Error - refers to any discrepancy between an actual, measured value and a theoretical, predicted value. measured value and a theoretical, predicted value. Error also refers to some human action that results in Error also refers to some human action that results in some sort of failure or fault in the softwaresome sort of failure or fault in the software

Fault - is a condition that causes the software to Fault - is a condition that causes the software to malfunction or failmalfunction or fail

Failure - is the inability of a piece of software to Failure - is the inability of a piece of software to perform according to its specifications. Failures are perform according to its specifications. Failures are caused by faults, but not all faults cause failures. A caused by faults, but not all faults cause failures. A piece of software has failed if its actual behaviour piece of software has failed if its actual behaviour differs in any way from its expected behaviourdiffers in any way from its expected behaviour

– 9 – CSCE 492 Spring 2008

Code InspectionsCode Inspections

Formal procedure, where a team of programmers read through code, explaining what it does.

Inspectors play “devils advocate”, trying to find bugs.

Time consuming process!

Can be divisive/lead to interpersonal problems.

Often used only for safety/time critical systems.

– 10 – CSCE 492 Spring 2008

WalkthroughsWalkthroughs

Similar to inspections, except that inspectors “mentally execute” the code using simple test data.

Expensive in terms of human resources.

Impossible for many systems.

Usually used as discussion aid.

– 11 – CSCE 492 Spring 2008

Test PlanTest Plan

A test plan specifies how we will demonstrate that A test plan specifies how we will demonstrate that the software is free of faults and behaves the software is free of faults and behaves according to the requirements specificationaccording to the requirements specification

A test plan breaks the testing process into specific A test plan breaks the testing process into specific tests, addressing specific data items and valuestests, addressing specific data items and values

Each test has a Each test has a test specificationtest specification that documents that documents the purpose of the testthe purpose of the test

– 12 – CSCE 492 Spring 2008

Test PlanTest Plan

If a test is to be accomplished by a series of smaller If a test is to be accomplished by a series of smaller tests, the test specification describes the tests, the test specification describes the relationship between the smaller and the larger relationship between the smaller and the larger teststests

The test specification must describe the conditions The test specification must describe the conditions that indicate when the test is complete and a that indicate when the test is complete and a means for evaluating the resultsmeans for evaluating the results

– 13 – CSCE 492 Spring 2008

Example Test Plan Deliverable 8.1 p267Example Test Plan Deliverable 8.1 p267

Test #15 Specification: addPatron() while checking out Test #15 Specification: addPatron() while checking out resourceresource

1.1. Requirement #3Requirement #3

2.2. Purpose: Create a new patron object when a new Purpose: Create a new patron object when a new Patron is attempting to check out a resourcesPatron is attempting to check out a resources

3.3. Test Description:Test Description:a. Enter check out screenb. Press new patron buttonc. … (next slide)

4.4. Test MessagesTest Messages

5.5. Evaluation – print patron list to ensure uniqueness Evaluation – print patron list to ensure uniqueness and that data was entered correctly and that data was entered correctly

– 14 – CSCE 492 Spring 2008

Example Test Description of Test PlanExample Test Description of Test Plan

3.3. Test Description:Test Description:a. Enter check out screen

b. Press new patron button

c. Enter Jill Smith Patron Name field

d. Enter New Boston Rd. Address field

e. Enter …

f. Choose Student from status choice box

g. A new Patron ID Number is generated if the name?? is new.

– 15 – CSCE 492 Spring 2008

Test OracleTest Oracle

A test oracle is the set of predicted results for a set of A test oracle is the set of predicted results for a set of tests, and is used to determine the success of testingtests, and is used to determine the success of testing

Test oracles are extremely difficult to create and are Test oracles are extremely difficult to create and are ideally created from the requirements specificationideally created from the requirements specification

– 16 – CSCE 492 Spring 2008

Test CasesTest Cases

A test case is a set of inputs to the system A test case is a set of inputs to the system

Successfully testing a system hinges on selecting Successfully testing a system hinges on selecting representative test casesrepresentative test cases

Poorly chosen test cases may fail to illuminate the Poorly chosen test cases may fail to illuminate the faults in a systemfaults in a system

In most systems In most systems exhaustive testingexhaustive testing is impossible, is impossible, so a so a white boxwhite box or or black boxblack box testing strategy is testing strategy is typically selectedtypically selected

– 17 – CSCE 492 Spring 2008

Black Box TestingBlack Box Testing

The tester knows nothing about the internal The tester knows nothing about the internal structure of the codestructure of the code

Test cases are formulated based on expected Test cases are formulated based on expected output of methodsoutput of methods

Tester generates test cases to represent all Tester generates test cases to represent all possible situations in order to ensure that the possible situations in order to ensure that the observed and expected behaviour is the sameobserved and expected behaviour is the same

– 18 – CSCE 492 Spring 2008

Black Box TestingBlack Box Testing

In black box testing, we ignore the internals of the system, and focus on relationship between inputs and outputs.

Exhaustive testing would mean examining output of system for every conceivable input.

Clearly not practical for any real system!

Instead, we use equivalence partitioning and boundary analysis to identify characteristic inputs.

– 19 – CSCE 492 Spring 2008

Equivalence PartitioningEquivalence Partitioning

Suppose system asks for “a number between 100 and 999 inclusive”.

This gives three equivalence classes of input: – less that 100 – 100 to 999 – greater than 999

We thus test the system against characteristic values from each equivalence class.

Example: 50 (invalid), 500 (valid), 1500(invalid).

– 20 – CSCE 492 Spring 2008

Boundary ValuesBoundary Values

Arises from the fact that most program fail at input boundaries.

Suppose system asks for “a number between 100 and 999 inclusive”.

The boundaries are 100 and 999.

We therefore test for values: 99 100 101 the lower boundary 998 999 1000 the upper boundary

– 21 – CSCE 492 Spring 2008

White Box TestingWhite Box Testing

The tester uses knowledge of the programming constructs to The tester uses knowledge of the programming constructs to determine the test cases to usedetermine the test cases to use

If one or more loops exist in a method, the tester would wish If one or more loops exist in a method, the tester would wish to test the execution of this loop for 0, 1, max, and max + 1, to test the execution of this loop for 0, 1, max, and max + 1, where max represents a possible maximum number of where max represents a possible maximum number of iterationsiterations

Similarly, conditions would be tested for true and falseSimilarly, conditions would be tested for true and false

– 22 – CSCE 492 Spring 2008

White Box TestingWhite Box Testing In white box testing, we use knowledge of the internal structure of

systems to guide development of tests.

The ideal: examine every possible run of a system.

Not possible in practice!

Instead: aim to test every statement at least once!

EXAMPLE.

if (x > 5) {

System.out.println(‘‘hello’’);

} else {

System.out.println(‘‘bye’’);

}

There are two possible paths through this code, corresponding to x > 5 and x 5.

– 23 – CSCE 492 Spring 2008

Unit TestingUnit Testing

The units comprising a system are individually testedThe units comprising a system are individually tested

The code is examined for faults in algorithms, data and syntaxThe code is examined for faults in algorithms, data and syntax

A set of test cases is formulated and input and the results are A set of test cases is formulated and input and the results are evaluatedevaluated

The module being tested should be reviewed in context of the The module being tested should be reviewed in context of the requirements specificationrequirements specification

– 24 – CSCE 492 Spring 2008

Integration TestingIntegration Testing

The goal is to ensure that groups of components The goal is to ensure that groups of components work together as specified in the requirements work together as specified in the requirements documentdocument

Four kinds of integration tests existFour kinds of integration tests exist Structure tests Functional tests Stress tests Performance tests

– 25 – CSCE 492 Spring 2008

System TestingSystem Testing

The goal is to ensure that the system actually does The goal is to ensure that the system actually does what the customer expects it to dowhat the customer expects it to do

Testing is carried out by customers mimicking real Testing is carried out by customers mimicking real world activitiesworld activities

Customers should also intentionally enter Customers should also intentionally enter erroneous values to determine the system erroneous values to determine the system behaviour in those instancesbehaviour in those instances

– 26 – CSCE 492 Spring 2008

Testing StepsTesting Steps

Determine what the test is supposed to measureDetermine what the test is supposed to measure

Decide how to carry out the testsDecide how to carry out the tests

Develop the test casesDevelop the test cases

Determine the expected results of each test (test Determine the expected results of each test (test oracle)oracle)

Execute the testsExecute the tests

Compare results to the test oracleCompare results to the test oracle

– 27 – CSCE 492 Spring 2008

Analysis of Test ResultsAnalysis of Test Results

The test analysis report documents testing and The test analysis report documents testing and provides information that allows a failure to be provides information that allows a failure to be duplicated, found, and fixedduplicated, found, and fixed

The test analysis report mentions the sections of The test analysis report mentions the sections of the requirements specification, the the requirements specification, the implementation plan, the test plan, and connects implementation plan, the test plan, and connects these to each testthese to each test

– 28 – CSCE 492 Spring 2008

Special Issues for Testing Object-Oriented SystemsSpecial Issues for Testing Object-Oriented Systems

Because object interaction is essential to O-O systems, Because object interaction is essential to O-O systems, integration testing must be more extensiveintegration testing must be more extensive

Inheritance makes testing more difficult by requiring Inheritance makes testing more difficult by requiring more contexts (all sub classes) for testing an more contexts (all sub classes) for testing an inherited moduleinherited module

– 29 – CSCE 492 Spring 2008

Configuration ManagementConfiguration Management

Software systems often have multiple versions or releasesSoftware systems often have multiple versions or releases

Configuration management is the process of controlling Configuration management is the process of controlling development that produces multiple software systemsdevelopment that produces multiple software systems

An evolutionary development approach often results in An evolutionary development approach often results in multiple versions of the systemmultiple versions of the system

Regression testing is the process of retesting elements of the Regression testing is the process of retesting elements of the system that were tested in a previous version or release system that were tested in a previous version or release

– 30 – CSCE 492 Spring 2008

Alpha/beta testingAlpha/beta testing

In-house testing is usually called alpha testing.

For software products, there is usually an additional stage of testing, called beta testing.

Involves distributing tested code to “beta test sites” (usually prospective customers) for evaluation and use.

Typically involves a formal procedure for reporting bugs.

Delivering buggy beta test code is embarrassing!