devon m. simmonds computer science department, csc550 1 1 csc 550 slides adapted from various...

Devon M. Simmonds Computer Science Department, CSC550 1

1

• CSC 550

Slides adapted from various sources including Software Engineering: A Practitioner’s Approach, 6/e by R.S. Pressman and Associates Inc.

Introduction to Software Testing


2

What is software testing?

“Testing is the process of exercising a program with the

intent of finding Errors.”

“Testing is the process of exercising a program with the

intent of proving its correctness.”


3

What is software testing?

“Observing the execution of a software system to validate

whether it behaves as intended and identify potential malfunctions.”

Antonia Bertolino


4

errors

requirements conformance

performance

an indicationof quality

What does testing show?


5

Why is testing

important?

Because the cost o

f failures

is significant!


6


7


Expectation vs. Reality

• Do we expect software to work correctly?

• A carefully made program– 5 faults/KLOC

• Windows XP has 45M LOC– How many faults?– 5 * 45M/1000 = 225,000!

• Why not remove all the faults?

8


9

Why is testing so difficult?

• How many paths are there through this loop?• How long would it take to execute all paths @ 1 test per

millisecond (10-3)?

1

32 54loop < =20 X

a

b

c d

e


10

A Testing Scenario

1

32 54loop < =20 X

a

b

c d

e

• There are about 1014 possible paths• How long would it take to execute all paths?


11

A Testing Scenario

1

32 54loop < =20 X

ab

c d

e

• How long would it take to execute 1014 possible paths @ 1 paths per 10-3 seconds?– 1 year =31,536,000 seconds ~= 32,000,000,000

milliseconds

– Total time = 1014 / 32 * 109 ~= 105 /32 years– About 3,170 years @ one test/millisecond– About 3 years @ one test/microsecond– About 11 days @ one test/nanosecond


12


• Because exhaustive testing is impossible– Exhaustive testing means testing a program with all

combinations of input values and preconditions for each element of the software under test.

132 54

loop <= 20 X

ab

c d

e


13


Loop header

Conditional

statementstatement

Looping : for (i=0; i< n; i++) etc.

• How long would it take to execute all paths @

1 test per millisecond (10-3)?


14

Terms! Terms! Terms!

Anomaly

Exception

BUG

Error Defect

Failure

Fault


15

Testing Terminology

• Fault/defect– An incorrect step, process, condition, or data definition in a computer program. – Software problem found after release (Pressman)

• Error– An incorrect internal state resulting from a fault.– A discrepancy between a computed, observed, or measured value or condition

and the true, specified, or theoretically correct value or condition (ISO). – Software problem found before release (Pressman)

• Failure– The inability of a system or component to perform its required functions within

specified performance requirements (IEEE). • Failures are caused by faults, but not all faults cause failures.


Testing Terminology: Fault vs. Error vs. Failure

public static int numZaro(int[] x) {//Effects: if x == null throw NullPointerException// else return the number of occurrences of “0” (zero) in x. int count = 0; for (int i = 1; i < x.length; i++) {

if (x[i] == 0) {count++;

} } return count;}

• Fault = search starts at index 1 instead of 0.• Error: numZero([2, 7, 0]) error but no failure• Failure: numZero([0, 7, 2]) error and failure

• State at if given [2, 7, 0] = (x=[2,7,0], count=0, i=1, PC=if). i Should be

zero• S0, if required sequence of states is s0, s1, s2 and

actual sequence is s0, s2, s3, then s2 is in error in second sequence.


17

Testing Terminology

• Anomaly– Anything observed in the documentation or operation of software that

deviates from expectations based on previously verified software products or reference documents (IEEE).

• Bug– Things the software does that it is not supposed to do– Something the software doesn't do that it is supposed to.

• Exception• An event that causes suspension of normal program execution; types

include addressing exception, data exception, operation exception, overflow exception, protection exception, underflow exception (IEEE).


18

Testing Terminology: Testing vs. Debugging

• Testing– Evaluating software to determine conformance to some

objective.

• Debugging – Finding a fault, given a failure.


19

Testing Terminology: The RIP Fault/Failure Model

Three conditions are necessary for failures to be observable.

• Reachability: The location of the fault must be reached.

• Infection: After executing the location, the state of the program must be incorrect.

• Propagation: The infected state must propagate to cause incorrect program output.


20

Testing Terminology

• Validation – Does software meet its goals? User-centric!

• Verification – Are we building the product right? E.g., does the

code reflect the design? Developer-centric!• Name for the artifact being tested

– Implementation under test (IUT)– Method under test (MUT), object under test (OUT)– Class/component under test (CUT), etc.


21

Testing Terminology: Test Plan

• A test plan specifies how we will demonstrate that the software is free of faults and behaves according to the requirements specification

• A test plan breaks the testing process into specific tests, addressing specific data items and values

• Each test has a test specification that documents the purpose of the test

• If a test is to be accomplished by a series of smaller tests, the test specification describes the relationship between the smaller and the larger tests

• The test specification must describe the conditions that indicate when the test is complete and a means for evaluating the results


22

Testing Terminology: Test Oracle

• A test oracle is the set of predicted results for a set of tests, and is used to determine the success of testing

• Test oracles are extremely difficult to create and are ideally created from the requirements specification


23

Testing Terminology: Test Case

• A test case is a specification of– The pretest state of the IUT

• Prefix values: values needed to put software into state required before test execution.

– a set of inputs to the system– Postfix values: values that need to be sent to the

software after test execution.– The expected results


24

Testing Terminology

• Test Point– A specific value for test case input and state variables.

• Domain– Set of values that input or state variables of IUT may take.

• Heuristics for test point selection– Equivalence classes – one values in set represents all values in set.– Partition testing techniques

• Boundary value analysis• Special values testing

• Test suite– A collection of test cases.


25

Testing Terminology• Fault model

– A description about where faults are likely to occur in a program.

• Test strategy– An algorithm or heuristic for creating test cases from a

representation, an implementation or a test model

• Test model– A description of relationships between elements of a

representation or implementation.

• Test design– The process of creating a test suite using a test strategy.

• Test effectiveness: relative ability of testing strategy to find bugs.

• Test efficiency: relative cost of finding bugs.


26

Testing Terminology: Testing Strategies

• Responsibility-based test design– Behavioural, functional, black-box, etc. testing

• Implementation-based test design– Relies on source code,, e.g., white-box

• Fault-based testing


27

Testing Strategy

• We begin by ‘testing-in-the-small’ and move toward ‘testing-in-the-large’

• For conventional software– The module (component) is our initial focus– Integration of modules follows

• For OO software– “testing in the small” OO class

• Has attributes and operations and implies communication and collaboration


28

Testing Strategyunit test integration

test

validationtest

systemtest

Black Box Testing White Box Testing


29

Unit Testing

moduleto betested

results

test cases

softwareengineer

• The units comprising a system are individually tested The code is examined for faults in algorithms, data and syntax

A set of test cases is formulated and input and the results are evaluated

The module being tested should be reviewed in context of the requirements specification

interface local data structures

boundary conditionsindependent pathserror handling paths


30

Unit Test Environment

Module

stub stub

driver

RESULTS

interface

local data structures

boundary conditions

independent paths

error handling paths

test cases


31

Integration Testing

• The goal is to ensure that groups of components work together as specified in the requirements document

• Four kinds of integration tests exist– Structure tests– Functional tests– Stress tests– Performance tests


32

Integration Testing Strategies

Options:• the “big bang” approach• an incremental construction strategy


33

Top Down Integration

top module is tested with stubs

stubs are replaced one at a time, "depth first"

as new modules are integrated, some subset of tests is re-run

A

B

C

D E

F G


34

Bottom-Up Integration

drivers are replaced one at a time, "depth first"

worker modules are grouped into builds and integrated

A

B

C

D E

F G

cluster


35

Sandwich Testing

Top modules aretested with stubs

Worker modules are grouped into builds and integrated

A

B

C

D E

F G

cluster


36

System Testing

• The goal is to ensure that all elements that interact to solve the problem do so correctly.– Software, databases, hardware, people,

procedures.– Overall functionality and performance must be

achieved.


37

Validation Testing

• The goal is to ensure that the system actually does what the customer expects it to do – fulfils requirements

• Testing is carried out by customers mimicking real world activities

• Customers should also intentionally enter erroneous values to determine the system behavior in those instances


38

Black Box Testing

• The tester knows nothing about the internal structure of the code

• Test cases are formulated based on expected output of methods

• Tester generates test cases to represent all possible situations in order to ensure that the observed and expected behavior is the same


39

White Box Testing

• The tester uses knowledge of the programming constructs to determine the test cases to use

• If one or more loops exist in a method, the tester would wish to test the execution of this loop for 0, 1, max, and max + 1, where max represents a possible maximum number of iterations

• Similarly, conditions would be tested for true and false


40

High Order Testing• Validation testing

– Focus is on software requirements• System testing

– Focus is on system integration• Alpha/Beta testing

– Focus is on customer usage• Recovery testing

– forces the software to fail in a variety of ways and verifies that recovery is properly performed

• Security testing– verifies that protection mechanisms built into a system will, in

fact, protect it from improper penetration• Stress testing

– executes a system in a manner that demands resources in abnormal quantity, frequency, or volume

• Performance Testing– test the run-time performance of software within the context of

an integrated system


A Classification of Testing: The V Model

Requirements Analysis

Architectural Design

Subsystem Design

Detailed Design

Implementation

Acceptance Test

System Test

Integration Test

Module Test

Unit Test


A Classification of Testing

• Graphs• Logical expressions• Input domain characterizations• Syntactic descriptions

42


43

Test-Case Exercise

• UML class model for Figure Hierarchy


44

The Polygon Class

abstract class Polygon {abstract void draw(int r, int g, int b); /* color closed area*/abstract void erase(); /* set to background rgb */abstract float area(); /* return area */abstract float perimeter(); /* return sum of sides */abstract Point center(); /* return centroid pixel */

}


45

The Triangle Class

class Triangle extends Polygon {public Triangle(LineSegment a, LineSegment b, LineSegment c){…}public void setA(LineSegment a) {…}public void setB(LineSegment b) {…}public void setC(LineSegment c) {…}

public LineSegment getA() {…}public LineSegment getB() {…}public LineSegment getC() {…}

public boolean is_isoceles(){…}public boolean is_scalene(){…}public boolean is_equilateral(){…}

public void draw(int r, int g, int b){…}public void erase(){…}public float area(){…}public float perimeter(){…}public Point center(){…}

}


46

The LineSegment Class

class LineSegment extends Figure {public LineSegment(Point x1, Point y1, Point x2, Point y2){…}public void setx1(Point x1) {…}public void sety1(Point y1) {…}public void setx2(Point x2) {…}public void sety2(Point y2) {…}

public Point getx1() {…}public Point gety1() {…}public Point getx2() {…}public Point gety2() {…}

}


47

Test-Case Exercise

• Specify as many test cases as you can for the Triangle class defined above.

• The class is used to determine if a triangle is: isosceles, equilateral or scalene. – A triangle is isosceles if two of its sides are equal.– A triangle is equilateral if all its sides are equal.– A triangle is scalene if its sides are of different sizes.


48

Test-Case Exercise

• A valid triangle must meet two conditions:• No side may have a length of zero.• Each side must be smaller than the sum of all sides

divided by two.• i.e. Given s = (a + b + c)/2, then a < s, b < s, c < s must

hold. – Stated differently, a < s && b < s && c < s a < b + c, b

< a + c and c < a + b.


49

Test-Case Exercise

• Specify as many test cases as you can for the Triangle class defined above.

Test Case DescriptionTest Case Input

a b c Expected Output

1

Valid isosceles triangle 3 3 4 Isosceles22829303132333435


• Solution To Test Case Exercise


Introduction to Software Testing (Ch 1), www.introsoftwaretesting.com

© Ammann & Offutt 54

Caveat: Impact of New Tools and Techniques

They’re teaching a new way of plowing over at the Grange tonight - you going?

Naw - I already don’t plow as good as I know how...

“Knowing is not enough, we must apply. Willing is not enough, we must do.” Goethe



Testing in the 21st Century• We are going through a time of change• Software Defines Behavior

– network routers– financial networks– telephone switching networks– other infrastructure

• Embedded Control Applications– airplanes, air traffic control– spaceships– watches– ovens– remote controllers

• Safety critical, real-time software

– PDAs– memory seats – DVD players– garage door openers– cell phones

Testing ideas have matured enough to be used in practice


Types of Test Activities• Testing can be broken up into four general

types of activities1. Test Design2. Test Automation3. Test Execution4. Test Evaluation

• Each type of activity requires different skills, background knowledge, education and training

• No reasonable software development organization uses the same people for requirements, design, implementation, integration and configuration control


Why do test organizations still use the same people for all four test activities??

This is clearly a waste of resources

1.a) Criteria-based1.b) Human-based


1. Test Design – (a) Criteria-Based

• This is the most technical job in software testing• Requires knowledge of :

– Discrete math– Programming– Testing

• Requires much of a traditional CS degree• This is intellectually stimulating, rewarding, and challenging• Test design is analogous to software architecture on the

development side• Using people who are not qualified to design tests is a sure way

to get ineffective tests


Design test values to satisfy coverage criteria or other engineering goal


1. Test Design – (b) Human-Based

• This is much harder than it may seem to developers• Criteria-based approaches can be blind to special

situations• Requires knowledge of :

– Domain, testing, and user interfaces

• Requires almost no traditional CS– A background in the domain of the software is essential– An empirical background is very helpful (biology, psychology,

…)– A logic background is very helpful (law, philosophy, math, …)

• This is intellectually stimulating, rewarding, and challenging– But not to typical CS majors – they want to solve problems

and build things


Design test values based on domain knowledge of the program and human

knowledge of testing


2. Test Automation

• This is slightly less technical• Requires knowledge of programming

– Fairly straightforward programming – small pieces and simple algorithms

• Requires very little theory• May be boring for test designers• Programming is out of reach for many domain experts• Who is responsible for determining and embedding

the expected outputs ?– Test designers may not always know the expected outputs– Test evaluators need to get involved early to help with this


Embed test values into executable scripts


3. Test Execution

• This is easy – and trivial if the tests are well automated

• Requires basic computer skills– Interns– Employees with no technical background

• Asking qualified test designers to execute tests is a sure way to convince them to look for a development job

• If, for example, GUI tests are not well automated, this requires a lot of manual labor

• Test executors have to be very careful and meticulous with bookkeeping


Run tests on the software and record the results


4. Test Evaluation

• This is much harder than it may seem• Requires knowledge of :

– Domain– Testing– User interfaces and psychology

• Usually requires almost no traditional CS– A background in the domain of the software is essential– An empirical background is very helpful (biology, psychology,

…)– A logic background is very helpful (law, philosophy, math, …)

• This is intellectually stimulating, rewarding, and challenging– But not to typical CS majors – they want to solve problems

and build things 61

Evaluate results of testing, report to developers


62

White-box testing: control-flow graph

• A control flow graph of a program is a graph that shows the paths that may be traversed when the program is executed.


63

Control-flow Graph

int n = 0; do{ if(n++ % 2 == 1)

dBase[i]++; else

if(dBase[i] > 100) dBase[i] -= n;else dBase[i] += n;

}while(n < 20);

1

2

34

5 6

7

8

1

2

4

3

5

6

7


64

Cyclomatic Complexity – V(G)

• If G is the control flow graph of program P and G has e edges and n nodes, then the cyclomatic complexity of P, V(G) is given by:– V(G) = e – n + 2– Or V(G) = the number of decision nodes + 1


65


• cyclomatic complexity– V(G) = e – n + 2

• = 11 – 7 = 4– Or V(G) = the number of

decision nodes + 1 • = 3 + 1 = 4

1

2

34

5 6

7

8


66

Cyclomatic ComplexityA number of industry studies have indicated

that the higher V(G), the higher the probability or errors.

V(G)

modules

modules in this range are more error prone


67


• The cyclomatic complexity is a measure of the number of linearly independent paths in the graph.– An independent path must move along at

least one edge not previously traversed• What are the independent paths for this

graph?– 1-2-3-5-7-8– 1-2-3-6-7-8– 1-2-4-7-8– 1-2-4-7-2…8

1

2

34

5 6

7

8


68

White-box testing: Code coverage

• White box testing methods emphasize code coverage– Statement coverage

• Every statement is executed at least once– Branch coverage

• Every branch is exercised at least once• i.e. every path from a node is traversed at least once

– Once true and once false evaluation– Multiple condition coverage

• All combinations of simple conditions are exercised– Basis-path model

• Exercise V(G) independent paths– Data flow coverage

• Evaluation of state of instance variables.


69

Statement Coverage

1

2

34

5 6

7

8

• Write enough test cases to execute each statement al least once, – 1-2-3-5-7-8– 1-2-3-6-7-8– 1-2-4-7-8

• May miss bugs for complex predicates and loops


70

Branch Coverage

1

2

34

5 6

7

8

• Determine number of branches• Compute path for each branch

– 1-2-3-5-7-2– 1-2-3-6-7-8– 1-2-4-7-8

• Write test cases to exercise each path


71

Multiple Condition Coverage

1

2

34

5 6

7

8

• Determine true/false combinations for simple conditions

• Write test cases to exercise each combination


72

Basis-path Testing

1

2

34

5 6

7

8

• Compute the cyclomatic complexity• Derive independent paths

– 1-2-3-5-7-8– 1-2-3-6-7-8– 1-2-4-7-8– 1-2-4-7-2…7-8

• Write test cases to exercise each path


73

Example 1

• V(G) = 2 + 1 = 3• Statement coverage requires 2 test cases• Branch coverage requires 2 test cases• Basis-path coverage requires 3 test cases• There are four distinct entry-exit paths

– 1-2-4-6-7– 1-2-4-5-7– 1-3-4-5-7– 1-3-4-6-7– Which should be chosen?

1

2

4

3

5 6

7


74

Example 2

• What is V(G)?• How many test cases are required for

statement coverage?• How many test cases are required for

branch coverage?• How many test cases are required for

basis-path coverage?• How many unique entry-exit paths exist?

A

D

E F

K L


75

Introduction to Mutation Testing &

Program Perturbation


76

Mutation Testing

• Faults are introduced into a program by creating many versions of the program– Each version is called a mutant.– Each mutant contains a single fault.

• Test cases are applied to the original program and to the mutant program.

• The goal is to cause the mutant program to fail, thus demonstrating the effectiveness of the test case.


77

Example of a Program Mutation

1 int max(int x, int y)2 {3 int mx = x;4 if (x > y)5 mx = x;6 else7 mx = y;8 return mx;9 }

1 int max(int x, int y)2 {3 int mx = x;

4 if (x < y)5 mx = x;6 else7 mx = y;8 return mx;9 }


78

Mutation Testing Process

Figure 2. Mutation Testing Flowchart, Source: Mutation 2000, pg 3

• Process (cont.)


79

Terminology

• Mutation testing– The creation of a set of mutant programs of the

program being tested.• Mutation

– a single syntactic change that is made to a program statement.

– Each mutant differs from the original program by one mutation.


80

Terminology

• Mutant– A faulty program; the original with one fault

introduced• Mutant Killed

– Test suite detects the mutant• Mutation Score

– #of killed mutants/ # of non-equivalent mutants• Mutation Operator

– The function that changes the source code to create a mutant


81

Mutation Operators

• Replace each operand with every other syntactically correct operand

• Modify expressions by replacing operators and inserting new operators

• Delete entire statements


82

Categories of Mutation Operators

• Operand Replacement Operators:– Replace a single operand with another operand

or constant. E.g.,• if(x > y)• if (5 > y) Replacing x by constant 5.• if (x > 5) Replacing y by constant 5.• if (y > x) Replacing x and y with each other.

– E.g., if all operators are {+,-,*,**,/} then the following expression a = b * (c - d) will generate 8 mutants:

• 4 by replacing * • 4 by replacing -.


83


• Expression Modification Operators:– Replace an operator or insert new operators. E.g.,

• if (x == y) • if (x >= y) Replacing == by >=.• if (x == ++y) Inserting ++.


84

• Statement Modification Operators:– E.g.,

• Delete the else part of the if-else statement.

• Delete the entire if-else statement.

• Replace line 3 by a return statement.


1 int max(int x, int y)2 {3 int mx = x;4 if (x > y)5 mx = x;6 else7 mx = y;8 return mx;9 }

return x;


85

Mutation System Example

• The Mothra mutation system for FORTRAN77 supports 22 mutation operators. E.g.,– absolute value insertion– constant for array reference replacement– GOTO label replacement– RETURN statement replacement– statement deletion– unary operator insertion– logical connector replacement– comparable array name replacement


86

Why Does Mutation Testing Work?

• The operators are limited to simple single syntactic changes on the basis of the competent programmer hypothesis.


87

The Competent Programmer Hypothesis

• Programmers are generally very competent and do not create “random” programs.

• For a given problem, a programmer, if mistaken, will create a program that is very close to a correct program.

• An incorrect program can be created from a correct program by making some minor change to the correct program.


88

Mutation Testing Algorithm

• Generate program test cases.• Run each test case against the original

program.– If the output is incorrect, the program must be

modified and re-tested. – If the output is correct go to the next step ...

• Construct mutants – E.g. using a tool like Mothra.


89

Mutation Testing Algorithm (Cont’d)

• Execute each test case against each alive mutant. – If the output of the mutant differs from the output of

the original program, the mutant is considered incorrect and is killed.

• Two kinds of mutants survive:– Functionally equivalent to the original program:

Cannot be killed.– Killable: Test cases are insufficient to kill the mutant.

New test cases must be created.


90

Mutation Score

• The mutation score for a set of test cases is the percentage of non-equivalent mutants killed by the test data.

• Mutation Score = 100 * D / (N - E) – D = Dead mutants– N = Number of mutants– E = Number of equivalent mutants

• A set of test cases is mutation adequate if its mutation score is 100%.


91

Program Perturbation

• Program Perturbation is a technique to test a program’s robustness.

• It is based on unexpectedly changing the values of program data during run-time.


92

The Perturbation Function

• The perturbation function is a mathematical function that:– takes a data state as its input;– changes the data state according to some

specified criteria;– produces a modified data state as output.


93

Program Perturbation Example

• Assume the following perturbation function:

1. int perturbation (int x)2. {3. int newX;4. newX = x + 20;5. return newX;6. }


94

Example of a Fault Injection

1. main()2. {3. int x;4. x = ReadInt();5. if (x > 0)6. printf(“X positive”);7. else8. printf(“X negative”);9. }

1. main()2. {3. int x;4. x = ReadInt();4.1 x = perturbation(x);5. if (x > 0)6. printf(“X positive”);7. else8. printf(“X negative”);9. }


Model-Driven Test Design


software artifact

model / structure

test requirements

refined requirements /

test specs

input values

test cases

test scripts

test results

pass / fail

IMPLEMENTATIONABSTRACTION

LEVEL

DESIGNABSTRACTION

LEVEL


Model-Driven Test Design – Steps


software artifact

model / structure

test requirements


test specs

input values

test cases

test scripts

test results

pass / fail


LEVEL

DESIGNABSTRACTION

LEVEL

analysis

criterion refine

generate

prefixpostfix

expected

automateexecuteevaluate


Model-Driven Test Design – Activities


software artifact

model / structure

test requirements


test specs

input values

test cases

test scripts

test results

pass / fail


LEVEL

DESIGNABSTRACTION

LEVEL

Test Design

Test

Automation

Test Executio

n

Test Evaluatio

n

Raising our abstraction level makestest design MUCH easier


Junit

Overv

iew

From: http://www.google.com/url?sa=t&source=web&cd=1&sqi=2&ved=0CBcQFjAA&url=http%3A%2F%2Fwww2.informatik.hu-berlin.de%2F~hs%2FLehre%2F2004-WS_SWQS%2F20041029_Ex_JUnit.ppt&rct=j&q=ppt%20junit%20testing&ei=Y7yUTf7EEe2C0QHO7vn0Cw&usg=AFQjCNHiSKk_Md3YoqEUBez54lS0MJ1Evw&cad=rja and other sources.

http://www.google.com/url?sa=t&source=web&cd=1&sqi=2&ved=0CBcQFjAA&url=http%3A%2F%2Fwww2.informatik.hu-berlin.de%2F~hs%2FLehre%2F2004-WS_SWQS%2F20041029_Ex_JUnit.ppt&rct=j&q=ppt%20junit%20testing&ei=Y7yUTf7EEe2C0QHO7vn0Cw&usg=AFQjCNHiSKk_Md3YoqEUBez54lS0MJ1Evw&cad=rja




JUnit 3


Satish Mishra Unit testing with JUnit 100

JUnit• JUnit is a framework for writing unit tests

– A unit test is a test of a single class• A test case is a single test of a single method• A test suite is a collection of test cases

• Unit testing is particularly important when software requirements change frequently– Code often has to be refactored to incorporate the

changes– Unit testing helps ensure that the refactored code

continues to work

From: http://www.google.com/url?sa=t&source=web&cd=1&sqi=2&ved=0CBcQFjAA&url=http%3A%2F%2Fwww2.informatik.hu-berlin.de%2F~hs%2FLehre%2F2004-WS_SWQS%2F20041029_Ex_JUnit.ppt&rct=j&q=ppt%20junit%20testing&ei=Y7yUTf7EEe2C0QHO7vn0Cw&usg=AFQjCNHiSKk_Md3YoqEUBez54lS0MJ1Evw&cad=rja






JUnit..

• JUnit helps the programmer:– Define and execute tests and test suites– Formalize requirements and clarify architecture– Write and debug code– Integrate code and always be ready to release a

working version



What JUnit does

• JUnit runs a suite of tests and reports results• For each test in the test suite:

– JUnit calls setUp()• This method should create any objects you may need

for testing



What JUnit does…

– JUnit calls one test method• The test method may comprise multiple test cases; that is, it

may make multiple calls to the method you are testing• In fact, since it’s your code, the test method can do anything

you want• The setUp() method ensures you entered the test method

with a virgin set of objects; what you do with them is up to you

– JUnit calls tearDown()

• This method should remove any objects you created



Creating a test class in JUnit

• Define a subclass of TestCase • Override the setUp() method to initialize object(s) under test. • Override the tearDown() method to release object(s) under test. • Define one or more public testXXX() methods that exercise the object(s)

under test and assert expected results. • Define a static suite() factory method that creates a TestSuite containing

all the testXXX() methods of the TestCase. • Optionally define a main() method that runs the TestCase in batch mode.



Fixtures

• A fixture is just a some code you want run before every test• You get a fixture by overriding the method

– protected void setUp() { …}• The general rule for running a test is:

– protected void runTest() {setUp(); <run the test> tearDown();

}– so we can override setUp and/or tearDown, and

that code will be run prior to or after every test case



Implementing setUp() method

• Override setUp() to initialize the variables, and objects

• Since setUp() is your code, you can modify it any way you like (such as creating new objects in it)

• Reduces the duplication of code

http://junit.sourceforge.net/javadoc/junit/framework/TestCase.html



Implementing the tearDown() method

• In most cases, the tearDown() method doesn’t need to do anything– The next time you run setUp(), your objects

will be replaced, and the old objects will be available for garbage collection

– Like the finally clause in a try-catch-finally statement, tearDown() is where you would release system resources (such as streams)



The structure of a test method

• A test method doesn’t return a result• If the tests run correctly, a test method

does nothing• If a test fails, it throws an

AssertionFailedError• The JUnit framework catches the error and

deals with it; you don’t have to do anything



Test suites

• In practice, you want to run a group of related tests (e.g. all the tests for a class)

• To do so, group your test methods in a class which extends TestCase

• Running suites we will see in examples



assertX methods

• static void assertTrue(boolean test)• static void assertFalse(boolean test)• assertEquals(expected, actual)• assertSame(Object expected, Object actual)• assertNotSame(Object expected, Object actual)• assertNull(Object object)



assertX methods

• assertNotNull(Object object) • fail()• All the above may take an optional String message as the first

argument, for example,static void assertTrue(String message, boolean test)



Organize The Tests

• Create test cases in the same package as the code under test• For each Java package in your application, define a TestSuite

class that contains all the tests for validating the code in the package

• Define similar TestSuite classes that create higher-level and lower-level test suites in the other packages (and sub-packages) of the application

• Make sure your build process includes the compilation of all tests



JUnit framework

Testing client

run(TestResult)setUp()runTest()tearDown()

TestCase

fName

setUp()runTest()tearDown()test1()test2()

ConcreteTestCase

action()

TestedClass

setUp()runTest()tearDown()

TestResult

runTest()test1()ortest2()

Test

run(TestResult)addTest(Test)

TestSuite

fTests

forall test in fTests test.run(TestResult)



Example: Counter class

• For the sake of example, we will create and test a trivial “counter” class– The constructor will create a counter and set it to zero– The increment method will add one to the counter and

return the new value– The decrement method will subtract one from the counter

and return the new value



The Counter class itself

public class Counter {int count = 0;public int increment() { return ++count;}public int decrement() { return --count;}

public int getCount() { return count; }}



JUnit tests for Counter

public class CounterTest extends junit.framework.TestCase { Counter counter1;

public CounterTest() { } // default constructor

protected void setUp() { // creates a (simple) test fixture counter1 = new Counter(); }

protected void tearDown() { } // no resources to release



JUnit tests for Counter…

public void testIncrement() { assertTrue(counter1.increment() == 1); assertTrue(counter1.increment() == 2); }

public void testDecrement() { assertTrue(counter1.decrement() == -1); }} // End from last slide


JUnit tests for Counter

public class CounterTest extends junit.framework.TestCase { Counter counter1; public CounterTest() { } // default constructor

protected void setUp() { // creates a (simple) test fixture counter1 = new Counter(); }

public void testIncrement() { assertTrue(counter1.increment() == 1); assertTrue(counter1.increment() == 2); }

public void testDecrement() { assertTrue(counter1.decrement() == -1); }}

Note that each test begins with a brand new counter

This means you don’t have to worry about the order in which the tests are run



Why JUnit• Allow you to write code faster while increasing quality• Elegantly simple • Check their own results and provide immediate feedback • Tests is inexpensive • Increase the stability of software • Developer tests • Written in Java • Free • Gives proper uniderstanding of unit testing


120

A Case Studyclass Money {

private int fAmount; private String fCurrency;

public Money(int amount, String currency) { fAmount= amount; fCurrency= currency; }

public int amount() { return fAmount; } public String currency() { return fCurrency; }

public Money add(Money m) { return new Money(amount()+m.amount(), currency()); }

}


121

How to Write A TestCasepublic class MoneyTest extends TestCase { //… public void testSimpleAdd() {

Money m12CHF= new Money(12, "CHF"); // (1)

Money m14CHF= new Money(14, "CHF"); Money expected= new Money(26, "CHF"); Money result= m12CHF.add(m14CHF); // (2) Assert.assertTrue(expected.equals(result)); // (3) } } (1) Creates the objects we will interact with during the test. This testing

context is commonly referred to as a test's fixture. All we need for the testSimpleAdd test are some Money objects.

(2) Exercises the objects in the fixture. (3) Verifies the result


122

Assert• assertEquals(expected, actual) • assertEquals(message, expected, actual) • assertEquals(expected, actual, delta)• assertEquals(message, expected, actual, delta) • assertFalse(condition) • assertFalse(message, condition) • AssertNotNull(object) • AssertNotNull(message, object) • AssertNotSame(expected, actual) • AssertNotSame(message, expected, actual) • assertTrue(condition) • assertTrue(message, condition)


123

Structure • setUp() Storing the fixture's objects in instance variables of your TestCase

subclass and initialize them by overriding the setUp method

• tearDown() Releasing the fixture’s

• run() Defining how to run an individual test case. Defining how to run a test suite.

• testCase()


124

Structure of Writing A Test public class MoneyTest extends TestCase { private Money f12CHF; private Money f14CHF; protected void setUp() { f12CHF= new Money(12, "CHF"); f14CHF= new Money(14, "CHF"); } public void testSimpleAdd() { Money expected= new Money(26, "CHF"); Money result= f12CHF.add(f14CHF); Assert.assertTrue(expected.equals(result)); }

TestCase test= new MoneyTest("simple add") { public void runTest() { testSimpleAdd(); } }}


JUnit 4 with Annotations


126

Start to Use it1. Download the latest version of JUnit from http://download.sourceforge.net/junit/2. Installation• unzip the junit.zip file • add junit.jar to the CLASSPATH. For example: set classpath=%classpath%;INSTALL_DIR\junit3\

junit.jar 3. Testing Test the installation by using either the batch or the graphical TestRunner tool to run the tests that

come with this release. All the tests should pass OK.

– for the batch TestRunner type: java junit.textui.TestRunner junit.samples.AllTests

– for the graphical TestRunner type: java junit.awtui.TestRunner junit.samples.AllTests

– for the Swing based graphical TestRunner type: java junit.swingui.TestRunner junit.samples.AllTests

Notice: The tests are not contained in the junit.jar but in the installation directory directly. Therefore make sure that the installation directory is on the class path

Important: Don't install the junit.jar into the extension directory of your JDK installation.If you do so the test class on the files system will not be found. JUnit plug-in for Eclipse

http://download.sourceforge.net/junit/


Running JUnit

• java junit.textui.TestRunner JavaClassfileName

• java junit.swingui.TestRunner JavaClassfileName

• java junit.awtui.TestRunner JavaClassfileName

• Use eclipse


128

Eclipse plug-in

http://dev.eclipse.org/viewcvs/index.cgi/jdt-ui-home/plugins/org.eclipse.jdt.junit/org.eclipse.pde.junit_3.0.0.zip





JUnit in Eclipse

• To create a test class, select File New Other... Java, JUnit, TestCase and enter the name of the class you will test

Fill this in

This will be filled in automatically


Running JUnit

First, select a Test classSecond, use this pulldown menu

Third, Run As JUnit Test


ResultsYour results are here


JUnit 4


Comparing JUnit 3 to JUnit 4

• All the old assertXXX methods are the same• Most things are about equally easy

– JUnit 4 makes it easier to test that exceptions are thrown when they should be

– JUnit 4 can still run JUnit 3 tests• JUnit 4 provides protection against infinite

loops• JUnit 4 has some additional features


Migrating from JUnit 3

• JUnit 4 requires Java 5 or newer• Don’t extend junit.framework.TestCase; just use an

ordinary class• Import org.junit.* and org.junit.Assert.*

– Use a static import for org.junit.Assert.*– Static imports replace inheritance from

junit.framework.TestCase

• Use annotations instead of special method names:– Instead of a setUp method, put @Before before some method– Instead of a tearDown method, put @After before some method– Instead of beginning test method names with ‘test’, put @Test

before each test method


Writing a JUnit test class, I

– Start by importing the JUnit 4 classes you need• import org.junit.*;

import static org.junit.Assert.*;

– Declare your class in the usual way• public class MyProgramTest {

– Declare any variables you are going to use frequently, typically including an instance of the class being tested

• MyProgram program;int [] array;int solution;


Writing a JUnit test class, II– If you wish, you can declare one method to be executed just once,

when the class is first loaded– This is for expensive setup, such as connecting to a database

• @BeforeClasspublic static void setUpClass() throws Exception { // one-time initialization code}

– If you wish, you can declare one method to be executed just once, to do cleanup after all the tests have been completed

• @AfterClasspublic static void tearDownClass() throws Exception { // one-time cleanup code}


Writing a JUnit test class, III

– You can define one or more methods to be executed before each test; typically such methods initialize values, so that each test starts with a fresh set

• @Beforepublic void setUp() { program = new MyProgram(); array = new int[] { 1, 2, 3, 4, 5 };}

– You can define one or more methods to be executed after each test; typically such methods release resources, such as files

• @Afterpublic void tearDown() {}


@Before and @After methods

• You can have as many @Before and @After methods as you want– Be warned: You don’t know in what order they will

execute• You can inherit @Before and @After

methods from a superclass; execution is as follows:– Execute the @Before methods in the superclass– Execute the @Before methods in this class– Execute a @Test method in this class– Execute the @After methods in this class– Execute the @After methods in the superclass


Writing a JUnit test class, IV

– A test method is annotated with @Test, takes no parameters, and returns no result

– All the usual assertXXX methods can be used• @Test

public void sum() { assertEquals(15, program.sum(array)); assertTrue(program.min(array) > 0);}


Special features of @Test– You can limit how long a method is allowed to take– This is good protection against infinite loops– The time limit is specified in milliseconds– The test fails if the method takes too long

• @Test (timeout=10) public void greatBig() { assertTrue(program.ackerman(5, 5) > 10e12);}

– Some method calls should throw an exception– You can specify that a particular exception is expected– The test will pass if the expected exception is thrown, and fail otherwise

• @Test (expected=IllegalArgumentException.class)public void factorial() { program.factorial(-5);}


Parameterized tests– Using @RunWith(value=Parameterized.class) and a @Parameters method, you

can run the same tests with multiple datasets• @RunWith(value=Parameterized.class)

public class FactorialTest { private long expected; private int value;

@Parameters public static Collection data() { return Arrays.asList( new Object[ ][ ] { { 1, 0 }, { 1, 1 }, { 2, 2 }, { 120, 5 } }); }

public FactorialTest(long expected, int value) { // constructor this.expected = expected; this.value = value; }

@Test public void factorial() { assertEquals(expected, new Calculator().factorial(value)); }}

• Source: http://today.java.net/pub/a/today/2006/12/07/junit-reloaded.html


Ignoring a test

– The @Ignore annotation says to not run a test

• @Ignore("I don’t want Dave to know this doesn’t work")@Testpublic void add() { assertEquals(4, program.sum(2, 2));}– You shouldn’t use @Ignore without a very good reason!


Test suites

– As before, you can define a suite of tests• @RunWith(value=Suite.class)

@SuiteClasses(value={MyProgramTest.class, AnotherTest.class})public class AllTests { … }


Other stuff

• Failed tests now throw an AssertionError, rather than JUnit 3’s AssertionFailedError

• There is now a version of assertEquals for arrays of objects:assertEquals(Object[] expected, Object[] actual)– Unfortunately, there is still no assertEquals for arrays of primitives

• JUnit 3 had an assertEquals(p, p) method for each kind of primitive p, but JUnit 4 only has an assertEquals(object, object) and depends on autoboxing


A gotcha

• The following method:– long sum(long x, long y) { return x + y; }

• with the following test:– @Test

public void sum() { assertEquals(4, s.sum(2, 2));}

• gives:– expected: <4> but was: <4>

• This is due to your friend, autoboxing– assertEquals no longer exists for primitives, only for objects– Hence, the 4 is autoboxed to an Integer, while sum returns a long– The error message means: expected int 4, but got long 4– To make this work, change the 4 to a 4L


JUnit 4 in Eclipse and NetBeans

• As usual, the easiest way to create a test class is just to let your IDE do it for you

• Here is the recommended test-driven approach– Create a class containing all the “stub” methods you initially think you

will need– Have the IDE create the test class, with all the test methods– Repeat:

• Write a test• Make sure the test fails• Write the method being tested• Make sure the test now succeeds

• Note: When you create the test class, NetBeans in particular puts a lot of garbage lines into each test method; you can just delete these and put in your own code


Summary?

JUnit Exercise


More Information

• http://www.junit.org– Download of JUnit– Lots of information on using JUnit

• http://sourceforge.net/projects/cppunit– C++ port of Junit

• http://www.thecoadletter.com– Information on Test-Driven Development

http://www.junit.org/

http://sourceforge.net/projects/cppunit

http://www.thecoadletter.com/


Q u e s t i o n s ?

The End

CSC550, Devon M. Simmonds, Computer Science Department, University of North Carolina Wilmington


150


• There are about 1014 possible paths• How long would it take to execute all paths?

– About 3,170 years @ one test/millisecond

1

32 54loop < =20 X

a

b

c d

e


154

Example 2• What is V(G)? - 7• How many test cases are required for

statement coverage? - 4• How many test cases are required for

branch coverage? - 4• How many test cases are required for

basis-path coverage? - 7• How many unique entry-exit paths exist?

– 16

A

D

E F

K L – ABDEGKMQS– ABDEGKNQS– ABDEHKMQ

S– ABDEHKNQS

– ACDEGKMQS

– ACDEGKNQS– ACDEHKMQ

S– ACDEHKNQS

–Similar for right side

devon m. simmonds computer science department, csc550 1 1 csc 550 slides adapted from various...

Documents

software testing devon

testnanosecond devon

xabcde devon

antonia bertolino devon

testing scenario13254loop

impossibleexhaustive

software engineering

software system