seminar on software testing

software evolution & architecture lab

University of ZurichDepartment of Informatics

Software Testing

Beat FluriZurich SoftSummer 2009Summer School on Complex Software Systems

As software quality control

Dr. Beat Fluri © 2009

Me, myself, and I

2

Beat FluriSenior research associate (UZH)

ETH ZurichStudies of Computerscience

UZHDoctorate in area of SoftwareEvolution Analysis

Lecturer @ UZHSoftware Testing

fluri.computerscience.chMaterial of this seminar


Agenda

3

Introduction into Software Testing

Continuous Integration

Unit Testing

Integration Testing

System and GUI Testing

Regression Testing

Tools you may like in you CI environment


Resources

4

John F. Smart’s blog

http://easytesting.org

http://java-source.net/open-source/testing-tools



Introduction


What is software testing?

6

Design and implementation of a special kind of software system; one that exercises another software system with the intent to finding bugs.

Rober V. Binder, TOOSMPT


Test design

7

Identify, model, and analyze the responsibilities of the system under test

On different level of abstraction: methods, classes, packages, subsystems, system

Design test cases based on this external perspectiveFunctional testing

Add test cases based on code analysis, suspicions, and heuristicsStructural testing

Develop expected results for each test case or choose an approach to evaluate the pass or no pass status of each test case


Test execution

Implementation under test is minimally operationalBottom-up to reduce test driver and test double overhead

Test suite execution: result is evaluated as pass or no pass

Coverage tool to instrument the implementation under testDefine goal: 95% statements, 80% branches, 70% conditions, etc.

If necessary, develop additional tests to exercise uncovered code

Stop: coverage goal is met and all tests pass

8


Why is software testing so hard?

9

for (int i = 0; i < n; i++) {if (a.get(i) == b.get(i)) {

x[i] += 100;} else {

x[i] /= 2;}

}

Input/state spacepublic Line(int x0, int y0, int x1, int y1)

Input per int: 232 different values: 232 x 232 x 232 x 232 = 2128

1000 lines per second: 1028 years

Execution sequences

2n + 1 paths with n iterations


Why is software testing so hard?Coincidental correctness

10

public class Account {protected long lastTxDate;public long daysSinceLastTx() {

long today = Calendar.getInstance().getTimeInMillis();return 1 + (today - lastTxDate) / (24 * 3600 * 1000);

} }

public class DepositAccount extends Account {public long daysSinceLastTx() {

long today = Calendar.getInstance().getTimeInMillis();return (today - lastTxDate) / (24 * 3600 * 1000);

} }


Why is software testing so hard?

Undecidability problem (halting problem)There isn’t any program that can decide whether the execution of another program will halt or not

For simple program it is decidable but not in general

11


Why software testing?

Find bugs earlyAirbag does not work: find out before or after releasing the car?

Part of quality assuranceWould you trust the airbag software if it wasn’t tested?

Supports changing a software systemVerify component integrity after a change

12


Why is testing neglected?

Expensive (software testing is hard)Test case design and selection

Building test environment

Integration environment

Few methodologies for test automation

Not part of development processHard to include for specific projects (customer won’t pay extra effort)

Not part of the software engineering culture

Missing testing environment

13


Short glossary

Test caseSet of inputs, execution conditions, and expected result (pass and no pass criterion)

Test suiteSet of test cases, usually related to a testing goal

Test driverClass or utility program that applies test cases

Double (stub, mock)Partial, temporary implementation of a component

Test harnessSubstitution for parts of the deployment environment, i.e., a system of test drivers and other tools to support test execution

14


Short glossary

OracleA mechanism to evaluate the actual result of a test case as pass or no pass

Equivalence classSet of input values: if one value is processed correctly/incorrectly, all other values will also be processed correctly/incorrectly

CoveragePercentage of elements required by a test strategy that have been exercised by a given test suite

15


Short glossary

Fault modelIdentifies relationships and components that are most likely to have faults

Test modelTestable representation of the relationships among elements of an implementation based on a fault model

Test strategyAlgorithm or heuristic to create test cases from an implementation or test model

Test designProduces a test suite using a test strategy: identification of interesting test inputs, placing these test inputs into a sequence, and defining the expected results

16


Short glossary

Fault-directed testingTesting that seeks to reveal implementation faults

Conformance-directed testingTesting that seeks to establish conformance to requirements or specifications

17


V-Model

18

User needs

System Spec

SubsystemDesign/Spec

UnitDesign/Spec

Unit

Subsystem

System

Delivery

Specification Implementation

Unit Test

Integration Test

Acceptance Test

System Test


V-Model

19

User needs

System Spec


UnitDesign/Spec

Unit

Subsystem

System

Delivery


Verification

Validation

Continuous Integration


Continuous Integration - What

21

... is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

Martin Fowler, Article

Source: http://martinfowler.com/articles/continuousIntegration.html


Continuous Integration - How

22

unit tests

integration tests

quality assurance

deployment

commit

changes

commit

changes

commit

changes

triggers

build

multi-stage

builds


Continuous Integration - How

23

Version control repository: CVS, Subversion, Git, Mercurial, etc.

Build tools: Ant, Maven, Make, Gant, Grails, Rake, etc.

Continuous integration environment (server): CruiseControl, Continuum, Hudson

Our experienceSubversion (http://subversion.tigris.org/)

Apache Maven (http://maven.apache.org/)

Hudson (https://hudson.dev.java.net/)


Continuous Integration - Why

Reduce risksDefects are detected when they are introduced

Measure the health of a software

Environment always the same and build starts clean => no assumptions

Reduce repetitive manual processes (safe time and costs)Process runs the same every time

Ordered process: compile, unit tests, integration tests, qa, etc.

Generate deployable software at any time and at any placeGoing back in the build history to deploy older, maybe stable, builds

24

Source: Paul M. Duval. Continuous Integration. Pearson Education, Inc., 2007


Continuous Integration - Why

Enable better project visibilityDecisions on quality improvements and tests are shown immediately

Ability to notice trends in various quality metrics (# bugs; # checkstyle, findbugs, pmd violations; code coverage of test cases)

Establish greater confidence in the software product from the development team

Making progress is visible and encourages developers

Confidence increases if the increase of overall product quality is visible

Same code basis for every developer: reduces communication overhead

Go to build #x, do you see...

25

Source: Paul M. Duval. Continuous Integration. Pearson Education, Inc., 2007


uDoo - Todo Application

26

udoo.controllerudoo.model


uDoo - Model

27

TodoList

ITodoListModel

<<interface>>

Todo

Project

1..*

IObservable

<<interface>>

IEventProvider

<<interface>>

Dispatcher*

IObserver

<<interface>>

*

Event*EventID


uDoo - Controller

28

execute()

undo()

ITodoCommand<<interface>>

AddTodoToProject DescriptionChange MoveTodo RemoveTodo

ITodoListModel

<<interface>>

Todo

Project2

ProjectTitleChange

AddTodo DueDateChange PriorityChange TodoTitleChangeRemoveProjectAddProject

*

CompletionStatusChange

ITodoListController<<interface>>

TodoListController *

DemouDoo with Maven and Hudson

Unit Testing


Unit Testing

31

User needs

System Spec


UnitDesign/Spec

Unit

Subsystem

System

Delivery


Unit Test

Test units in isolationConcentrate on possible bugs within smallest possible units


Unit Testing

32

public void doSomething(int number)Method scopeConstraints on input

private List<Todo> fTodos;Class scopeEncapsulationState dependant behavior

1

2

3

Method & class scopeInheritancePolymorphism

Method Scope TestConstraints on Input

1


Method Scope - General Approach

Decompose the specification into independently testable features

Identify representative values (domain analysis)Identify constraints for all input variables (boundaries)

Select test values for each variable in each boundary

Select test values for variables not given in the boundary

Generate test case specificationsDetermine expected results for these inputs

Generate test cases and instantiate tests

34


Domain Analysis - On, Off, In Points

On pointA value that lies on a boundary

Off pointA value that lies not on the boundary

In pointA value that lies not on the boundary and satisfies all boundary conditions

35


Open boundary: x > 10On point: 10

Off point: 11

On point makes the boundary condition false

Off point makes the boundary condition true

Closed boundary: x <= 100On point: 100

Off point: 101

On point makes the boundary condition true

Off point makes the boundary condition false

36



Listempty: List.size() == 0

loaded: List.size() > 0 && List.size() < List.MAX_CAPACITY

full: List.size() == List.MAX_CAPACITY

Abstract state on pointSmallest possible change in any state variable would produce a state change

empty: empty list

loaded: list with one element list with (List.MAX_CAPACITY - 1) elements

full: full list

37



Abstract state off pointempty: list with one element

loaded: empty list list with List.MAX_CAPACITY elements

full: list with (List.MAX_CAPACITY - 1) elements

Abstract state in pointempty: empty list

loaded: list with (0 < x < List.MAX_CAPACITY) elements list with (0 < x < List.MAX_CAPACITY) elements

full: full list

38



One-by-One (1x1) Selection Criteria

One on point and one off point for each relational conditionx > 10: on point = 10; off point = 11

One on point and two off points for each strict equality conditionx == 10: on point = 10; off point1 = 9; off point2 = 11

One on point and one off point for each non-scalar typex == “Testing”: on point = “Testing”; off point = “Testing!”

39


One-by-One (1x1) Selection Criteria

One on point and at least one off point for each abstract state invariant

!list.isEmpty()

on point: list with one element

off point: empty list

Don’t repeat identical tests for adjacent subdomainsAdjacent subdomains may share boundaries

State 1: x >= 10, on point = 10, off point = 9

State 2: x < 10, on point = 10, off point = 11

Test points: 9, 10, 11

40


Domain Test Matrix

41

Boundary Test CasesTest Cases

Variable Condition Type 1 2 3 4 5 6

> 10On 10

> 10Off 11

x<= 100

On 100<= 100

Off 101

Typical In 15 90

!isEmpty()On false

aList!isEmpty()

Off true

Typical In true true true true

Expected resultExpected resultExpected result Reject Accept Accept Reject Reject Accept


Category-Partition Test Pattern

Design method scope test suites based on input/output analysis

Method to test:TodoList.addTodo(Todo todo, Project project)

ProcedureDecompose the specification into independently testable features

1. Identify the functions of the method under test

2. Identify the input and output parameters of each function

Identify representative values

3. Identify categories for each input parameter

4. Partition each category into choices

5. Identify constraints on choices

Generate test case specifications

6. Generate test cases by enumerating all choice combinations

7. Develop expected results for each test case42


1. Identify the functions of the method under testAdds the given Todo to the given Project and returns true

Does nothing if the given Project does not exist and returns false

2. Identify the input/output parameters of each testable functionInput: Given Project

Input: Given Todo

Input: Content of the todo-list

Output: true or false

Output: Given Todo is in given Project or not

43



3. Identify categories for each input parameterParameter Category

Todo value

Project value

Todo-list special case

4. Partition each category into choicesParameter Category Choices

Todo value null not null

Project value null not null

Todo-list special case contains Project does not contain Project

44



5. Identify constraint on choicesIf todo-list throws an exception for (todo != null) and (project == null) it will also throw one for (todo == null) and (project == null)

If either (todo == null) or (project == null) the state of the todo-list does not care (DC)

6. Generate test cases by enumerating all choice combinations2 x 2 x 2 choices: 8 test cases

Remove cases identified by Step 5: 4 test cases

45




46

Parameter/ChoicesParameter/ChoicesParameter/Choices Expected Result

TC Todo Project Todo-list Returned Exception Todo-list

1 DC null DC NullPointerException !todo

2 DC not null DC NullPointerException !todo

3 not null not null project true todo

4 not null not null !project false !todo

Legendproject todo-list contains project!project todo-list does not contain projecttodo project contains todo!todo project does not contain todo

Class Scope TestEncapsulation

2


Encapsulation

Don’t test private methods

Pollute public APIPrivate -> Package

Delegators for private methods

Inner test classDeployment mess

Reflection

48

Class Scope TestState dependent behavior

2


State Dependent Behavior

Non-modal classesDoes not impose any constraints on the sequence of message accepted, i.e., does not have a state

Quasi-modal classesImposes sequential constraints on message acceptance that change with the content of the object (Collections)

Modal classesPlaces both message and domain constraints on the acceptable sequence of a message (Account)

50


Invariant Boundaries Test Pattern

Select test-efficient test value combinations for classes, interfaces, and components composed of complex and primitive data types

ProcedureDecompose the specification into independently testable features

1. Define the class invariants

Identify representative values

2. Develop on and off points for each condition using 1x1 selection criteria

Generate test case specifications

3. Complete test suite by developing in points for variables not referenced in a condition

4. Represent all test cases in a domain test matrix

51


Invariant Boundaries Test Pattern

Elevator Example

Elevator can only run if the following invariants are met0 < weight <= MAX_WEIGHT

LOWEST_FLOOR <= selectedFloor <= HIGHEST_FLOOR

selectedFloor != currentFloor

door.isClosed

!inEmergency

52


Elevator - Domain Test Matrix

53

Boundary Test CasesTest CasesTest Cases

Variable Condition Type 1 2 3 4 5 6 7

> 0On 0

> 0Off 1

weight<= MAX_WEIGHT

On max<= MAX_WEIGHT

Off max+1Typical In 23 50 66

>= LOWEST_FLOOROn low

>= LOWEST_FLOOROff low-1

<= HEIGHEST_FLOOROn high

selectedFloor<= HEIGHEST_FLOOR

OffselectedFloor

On

!= currentFloor Off

In 1 2 0 -1 2 6 1Typical In 0 3 5 6

isClosed()On

doorisClosed()

Off

Typical In true true true true true true true

falseOn

inEmergencyfalse

Off

Typical In false false false false false false falseExpected result Reject Accept Accept Reject Accept Reject Accept


Elevator - Domain Test Matrix

54

Boundary Test CasesTest Cases

Variable Condition Type 8 9 10 11 12 13 14 15

> 0On

> 0Off

weight<= MAX_WEIGHT

On<= MAX_WEIGHT

Off

Typical In 23 55 56 54 10 11 18 14

>= LOWEST_FLOOROn

>= LOWEST_FLOOROff

<= HEIGHEST_FLOOROn

selectedFloor<= HEIGHEST_FLOOR

Off high+1selectedFloor

On 0 -1!= currentFloor Off 0

In 3 -1 4 0 3Typical In -1 0 0 1 5 4 2

isClosed()On true

doorisClosed()

Off falseTypical In true true true true true true

falseOn false

inEmergencyfalse

Off trueTypical In false false false false false false

Expected result Reject Accept Accept Reject Accept Reject Accept Reject


Modal Class Test Pattern

55

public class Account { private int balance; private Date lastActivity; public void open() {} public int balance() {} public void credit(int amount) {} public void debit(int amount) {} public void freeze() {} public void unfreeze() {} public void close() {}}



Constraints on message sequencesAccount: debit is not accepted if the balance is <= 0

Account: freeze is not accepted if account is closed or already frozen

Modal class test1. Method scope test

2. Alpha-omega cycle

3. Generate state model and transition tree

4. Elaborate transition tree with a full expansion of conditional transition variants

5. Tabulate events and actions along each path to form test cases

6. Develop test data for each path by using Invariant Boundaries

7. Develop sneak path test cases

56



57

State model of Account

Overdrawn

Open Frozen

Inactive

Closed

freezeunfreeze

balance

balance

[balance == 0] close

balance

balance

credit

debit

credit

debit [balance < 0]

credit [balance >= 0]

[today - lastActivity > 5y]

[balance == 0] close

!

"



58

Transition tree of Account

Open

Open

Opendebit

Opencredit

Overdrawndebit [balance < 0]

Overdrawnbalance

Overdrawncredit

Frozenfreeze Frozen

balance

Opencredit [balance >= 0]

Openunfreeze

Inactive[today - lastActivity > 5y]

Closed[balance == 0] close

Inactivebalance

Closedclose

balance



59

Conditional Transitions

Open

Open

Opencredit

Opendebit [balance > 0]

Overdrawndebit [balance < 0]

Overdrawnbalance

Overdrawncredit [balance < 0]

Frozenfreeze Frozen

balance

Opencredit [balance == 0]

Openunfreeze

Inactive[today - lastActivity > 5y]


Inactivebalance


balance

Opendebit [balance == 0]

Opencredit [balance > 0]

Open[balance != 0] close/error

Inactive[balance != 0] close/error



60

Sneak PathBug that allows an illegal message to be accepted, resulting in an illegal transition

Test sneak pathSend illegal messages

Example: Account in state overdrawnSend message close

Expected responses to sneak pathsMessage should be rejected

State of the object should be unchanged



61

Test case specificationFor each path in the final transition tree specify one test case

Test Case/Event PathTest Case/Event PathTest Case/Event Path Expected

TC Level 1 Level 2 Level 3 Terminal State

1 New Open

2 New balance Open

3 New debit [balance < 0] balance Overdrawn

Test case specificationFor each sneak path specify one test case

Test Case/Event PathTest Case/Event Path Expected ResultExpected Result

TC Test State Test Event State Exception

1 Inactive freeze Inactive IllegalEventException

2 Overdrawn close Overdrawn IllegalEventException

3 Frozen debit Frozen IllegalEventException

Method & Class Scope TestInheritance and polymorphism

3


Inheritance-Related Bugs

Incorrect InitializationSuperclass initialization is omitted or incorrect

Inadvertent BindingsPrivates are invisible

Non-private are invisible if subclass has member with the same name

Missing OverrideA subclass-specific implementation of a superclass method is omitted

Java: use abstract classes

63



Naked AccessA superclass instance variable is visible in a subclass and subclass methods update variables directly

Square Peg in a Round HoleA subclass is incorrectly located in a hierarchy

For example: Square subclass of Rectangular

Naughty ChildrenA subclass either does not accept all messages that the superclass accepts or leaves the object in a state that is illegal in the superclass

64



Worm HolesA subclass computes values that are not consistent with the superclass invariant or superclass state invariants

Gnarly HierarchiesA subclass creates new holes, boundaries, or intersections that are inconsistent with the superclass domain

For example, strengthen precondition (in Eiffel, e.g., not possible)

Weird HierarchiesInheritance is abused as a kind of code-sharing macro to support hacks without regard to the resulting semantics

Fat InterfaceA subclass inherits methods that are inappropriate or irrelevant

65


Polymorphic Message Test Pattern

Develop tests for a client of a polymorphic server that exercise all client bindings to the server

Class hierarchy that overrides methods of the superclassClient may fail to meet all preconditions for all possible bindings

The implementation of a server class is changed or extended

Approach (different than in Binder)Determine all possible dynamic types

Call method with each possible dynamic type

66


Polymorphic Message Test Pattern

67

@Testpublic void getChargeTest() { Movie m = new Movie(new ChildrensPrice()); Assert.assertThat(m.getCharge(1), is(2)); m = new Movie(new NewReleasePrice()); Assert.assertThat(m.getCharge(1), is(4)); m = new Movie(new RegularPrice()); Assert.assertThat(m.getCharge(1), is(3));}

getCharge(days: int)getFrequentRenterPoints(days: int)

NewReleasePrice


title: StringMovie


Price

getCharge(days: int)

RegularPrice


ChildrensPrice

1

@Testpublic void getFrequentRenterPointsTest() { // similar to getChargeTest() for ChildrensPrice and NewReleasePrice} Adapted from: Martin Fowler. Refactoring. Addison-Wesley Professional, 1999.


Polymorphic Server Test Pattern

Design a test suite at class scope that verifies LSP compliance for polymorphic server hierarchy

LSP: Liskov substitution principleIf for each object o1 of type S there is an object o2 of type T such that for all programs P defined in terms of T, the behavior of P is unchanged when o1 is substituted for o2, then S is a subtype of T.

Barbara Liskov, Data Abstraction and Hierarchy, Keynote OOPSLA 1987

Hierarchy under test is nonmodalNo constraints on the sequence of messages accepted

Goal: exercise overridden methods in a polymorphic hierarchy

68


Polymorphic Server Test Pattern

Test each method in its defining class and all subclasses that override it

Reuse of the test suite for an overridden superclass methodIt could, but check whether additional test cases are necessary to test functionality and to satisfy coverage criteria

Overriding a superclass method used by a superclass methodAdditionally test subclass inherited methods within the context of the subclass

Check for conformance with LSPMake a client stub (simulate a client in test suite)

Clients of a polymorphic server can substitute any subtype without causing a failure

69


Polymorphic Server Test

70

class ASTNode { final void accept(ASTVisitor v) { v.preVisit(this); accept0(v); v.postVisit(this); } void accept0(ASTVisitor v) {}}class IfStatement extends ASTNode { void accept0(ASTVisitor v) { boolean children = v.visit(this); if (children) { acceptChild(v, getExpression()); acceptChild(v, getThenStatement()); acceptChild(v, getElseStatement()); } v.endVisit(this); }}class WhileStatement extends ASTNode { void accept0(ASTVisitor v) { // ... }}

class ASTNodeTest { @Test void accept0Test() {/*..*/} @Test void acceptTest() {/*..*/}}class IfStatementTest { @Test void accept0Test() {/*..*/} @Test void acceptTest() { ASTNode a = new IfStatement(); a.accept(new VisitorStub()); // ...

}}class WhileStatementTest { @Test void accept0Test() {/*..*/} @Test void acceptTest() { ASTNode a = new WhileStatement(); a.accept(new VisitorStub()); // ... }}

Structural TestingFind structural problems in source code


Structural Testing

Test cases that satisfy coverage criteria

Simple, but effective, coverage criteriaStatement coverage

Branch coverage

Condition coverage

72

X

A

B C

D

E

F

G

H

I

J

K


Statement coverage

All statements in a method have been executed at least once

Why is statement coverage is not enoughStatement coverage can be achieved without exercising all branches

Single loop iteration can cover all statements but may not reveal loop control bugs

Statement coverage can be achieved without exercising all true-false combinations of conditions

73


Branch Coverage

Every path from a node is executed at least once

Why is branch coverage not enoughBranch coverage can be achieved withoutexercising all true-false combinations of conditions

Branch coverage does not ensure that all entry-exit paths are executed

74

int foo(int x) { if (a == b) ++x; if (x == y) --x; return x;}


Condition Coverage

Condition coverageEach basic condition to be evaluated as true and false at lease once

It is not required to test all branches

(true || false) and (false || true)

Not possible: Short-circuit evaluation

Branch/condition coverageCondition coverage + each branch taken at least once

Still not all true-false combinations of conditions exercised

(true || false) and (false || true) and (false || false)

75

if (x > 0 || x <= 10) {}


Who does the coverage calculation?

Yes, you can! But you shouldn’t...

Use Coberturaintegrates nicely into Maven and Hudson

76

Test in IsolationMock’em


What about Testing in Isolation?

78

Always possible; but dependencies must be simulatedStubs or mocks

StubsProvide answers to calls made during tests. Re-implemented interface of the original class

MocksStubs + expectations which form a specification of the calls they are expected to receive


When to Use Mocks

79

Classical way: Use real objects if possibleMocks if using real objects is awkward or if real objects cannot be used

Focus on result of behavior

Cost/benefit balance

Mockist way: Always use a mock for any objectProblem: strong coupling to the implementation

Mocking model objects is overkill

Focus on how a unit/component is implemented to write expectations

Source: http://martinfowler.com/articles/mocksArentStubs.html

http://martinfowler.com/articles/mocksArentStubs.html

http://martinfowler.com/articles/mocksArentStubs.html


Tools for Mocking

EasyMockhttp://www.easymock.org

based on reflection

mocking of static, final, or private methods; constructors not possible

JMockhttp://www.jmock.org

based on reflection

mocking of static, final, or private methods; constructors not possible

80


Tools for Mocking

Amockhttp://amock.blogspot.com/

based on aspect oriented programming

mocking of static, final, or private methods; constructors possible

AspectJ must be used

JMockit (0.95)https://jmockit.dev.java.net/

based on instrumentation in Java 1.5(VM arg: -javaagent:<path>/jmockit.jar=junit4)

mocking of static, final, or private methods and constructors possible

no other tool necessary

81


uDoo

82

TodoList

ITodoListModel

<<interface>>

Todo

Project

1..*

IObservable

<<interface>>

Dispatcher*

Goal: test TodoList in isolation

ProblemsProject and Todo objects are necessary

Dispatcher object is necessary to test notification


JMockit in uDoo

Classical way: Use real objects if possible

Project and Todo objects are normally used (container objects)

Since Dispatcher can be replaced; test against IObserableUse expectations to test whether or not IObservable notifies observers

Use Hamcrest-Matchers to test whether or not the observers are notified with the correct Event

Use mock for Project to test corrupted Project (i.e.,does not store Todos)

83

DemoUnit Testing in uDoo

Integration Testing


V-Model

86

User needs

System Spec


UnitDesign/Spec

Unit

Subsystem

System

Delivery


Unit Test

Integration Test


Integration Testing

Integration testing is a search for components faults that cause intercomponent failures

Breaking up component isolation step-by-step

Integration strategyWhich components are the focus of the integration test?

In what sequence should component interfaces be exercised?

Integration testing in object-oriented development begins earlyWithin a class (integrating methods)

Within a class hierarchy (integrating superclasses)

Between a client and its server

Within a package (integrating classes)

... 87


Dependency Analysis

Integration strategy follows sequences in which components have to be tested

Dependency analysis among components is used to define the sequences

Dependency analysis: approachTake component diagram, e.g., UML class diagram

Make topology (i.e., directed graph) of the dependencies

88


Dependency Analysis

89

CC C

P

A S

O M

CP

*

*

* *


Dependency Analysis

90

S

M CP

O A

CC

CP

depends on


Big Bang Integration

Demonstrate stability by attempting to exercise an entire system with a few test runs

Integrate all components at onceMonolithic system

Components tightly coupled

No doubles are necessary

ProblemsFault locations are difficult to find

Big bang test suites are usually to small to reveal all interface faults

Try to avoid big bang integration!

91


Big Bang Integration

92

S

M CP

O A

CC

CP

Driver


Bottom-up Integration

Demonstrate stability by adding components to the system under test in uses-dependency order, beginning with components having the fewest dependencies

ApproachTest leaf components in dependency graph with drivers for each leaf. Use doubles when necessary

Repeat: test components on the next level; until the entire system is tested

93



94

S

M CP

O A

CC

CP

Driver



95

S

M CP

O A

CC

CP

Driver

Driver



96

AdvantagesEarly integration of leaf components

Parallel development on components in subtrees of dependency graph

Reduces the number of doubles (only for cycles)

DisadvantagesMany drivers have to be developed

Interaction testing is limited to collaborations implemented in the component (root of subtree)

Postpones checking critical control interfaces and collaborations until the end (root of dependency graph)


Top-down Integration

Demonstrate stability by adding components to the system under test in control hierarchy order, beginning with the top level control objects

ApproachTest component(s) at the highest level first. Use doubles for lower level components

Continue in a breath-first order at each level in the dependency graph until all components are integrated

97



98

S

M CP

O A

CC

CP

Driver



99

S

M CP

O A

CC

CP

Driver

Driver



100

AdvantagesEarly demonstration of end-to-end functionality

Fewer drivers necessary than with bottom-up integration

Test cases can be reused to drive lower-level tests

DisadvantagesDevelopment and maintenance of doubles

Test cases in high-level components may not be sufficient for lower-level components


Collaboration Integration

Demonstrate stability by adding sets of components to the system under test that are required to support a particular collaboration

ApproachMap collaborations onto the dependency graph

Choose a sequence in which to apply testing on collaborations

Test collaboration in the chosen sequence

101



102

O

S

M CP

A

CC

CP



103

O

S

M CP

A

CC

CP

Driver



104

S

M CP

O A

CC

CP

Driver



105

O

S

M CP

A

CC

CP

Driver



106

AdvantagesFew drivers and doubles necessary

Focus on end-to-end functionality; useful for system scope testing

Reuse of collaboration integration test suite for system test

DisadvantagesInter-collaboration may not be tested thoroughly

Participants in a collaboration are not exercised separately (big bang)

Probably many doubles for testing initial collaborations


Backbone Integration

Combination of top-down integration, bottom-up integration, and big bang integration to verify interoperability among tightly coupled subsystems

Used, for instance, for embedded system applicationsBackbone provides services that are essential for running tests and the application

Creating doubles for the entire backbone would be impractical

ApproachTest each component in backbone in isolation and do a bottom-up integration or big bang integration of the backbone

Do a top-down integration on the application control components

107



108

Backbone

Top-d

ow

n Inte

gra

tion

Bottom

-up Inte

gra

tion



109

AdvantagesMitigates the disadvantages of top-down and bottom-up integration

DisadvantagesDoubles and drivers are necessary

Integration Testing and Continuous Integration


Multi-Stage Builds

Special test classes for integration testingjohn ferguson smart

Say Maven to run integration tests only for goal “integration-test”mvn integration-test

Make a separate job in Hudson for integration testingRapid feedback on unit testing level is still satisfied

Integration may and are allowed to run longer than unit tests

111


Integration Testing for uDoo

Integrating controller with model

Integrating UI with controller and model comes later

Bottom-upITodoCommands with TodoList

TodoListController with ITodoCommands and TodoList

112

execute()

undo()

ITodoCommand<<interface>>

AddTodoToProject DescriptionChange MoveTodo RemoveTodo

ITodoListModel

<<interface>>

Todo

Project2

ProjectTitleChange

AddTodo DueDateChange PriorityChange TodoTitleChangeRemoveProjectAddProject

*

CompletionStatusChange

ITodoListController<<interface>>

TodoListController *

Top-downTodoListController with TodoList (including ITodoCommands, i.e., no mocks for commands)

Decision: top-downThe two steps with bottom-up are more or less the same

DemoIntegration Testing for uDoo

GUI and System Testing


V-Model

115

User needs

System Spec


UnitDesign/Spec

Unit

Subsystem

System

Delivery


Unit Test

Integration Test

System Test


System and GUI Testing

116

Testing at application level

Why testing at system level scope?Individual verification of components cannot guarantee a correctly functioning system.

Demonstrate that the system implements all required capabilities.

Answer the question: “Can we ship it yet?”

Objectives for testing at system level scopeRequirement documents (formal)

Natural language documents (informal)

Prototypes

Use cases


Two-step approach

117

UI Integration testVerify if UI interaction works as intended with the subsystem

User story or use case testing with UIValidate if system works as intended when simulating user stories or use cases with UI interaction


GUI Integration Testing

Manual GUI testingTesting script that is executed by a user

Hard to repeat (e.g., in a CI environment)

Semi-automated GUI testingCapture and replay

Have to be adapted on changing UI

Automated GUI testing with a frameworkFEST-swing: http://fest.easytesting.org

Abbot: http://abbot.sourceforge.net/doc/overview.shtml

118

http://abbot.sourceforge.net/doc/overview.shtml

http://abbot.sourceforge.net/doc/overview.shtml


FEST-Swing

FEST-Swing featuresSimulation of user interaction with a GUI (e.g., mouse and keyboard input)

Reliable GUI component lookup (by type, by name, or custom search criteria)

Support for all Swing components included in the JDK

Compact and powerful API for creation and maintenance of functional GUI tests

Supports Applet testing

Ability to embed screenshots of failed GUI tests in HTML test reports

Can be used with either TestNG or JUnit

Supports testing violations of Swing's threading rules

119

Source: FEST-swing: http://fest.easytesting.org/swing/

http://docs.codehaus.org/display/FEST/Performing+GUI+Component+Lookup

http://docs.codehaus.org/display/FEST/Performing+GUI+Component+Lookup

http://docs.codehaus.org/display/FEST/Testing+Applets

http://docs.codehaus.org/display/FEST/Testing+Applets

http://docs.codehaus.org/display/FEST/Screenshots+of+Test+Failures

http://docs.codehaus.org/display/FEST/Screenshots+of+Test+Failures

http://testng.org/

http://testng.org/

http://www.junit.org/

http://www.junit.org/

http://docs.codehaus.org/display/FEST/Testing+that+access+to+Swing+components+is+done+in+the+EDT

http://docs.codehaus.org/display/FEST/Testing+that+access+to+Swing+components+is+done+in+the+EDT

http://java.sun.com/javase/6/docs/api/javax/swing/package-summary.html#threading

http://java.sun.com/javase/6/docs/api/javax/swing/package-summary.html#threading

http://fest.easytesting.org/swing/

http://fest.easytesting.org/swing/


uDoo and FEST-Swing

uDoo UI integration with controller component

For each functionality in the TodoControllerTrigger it via the UI

Verify that the model is as expected (trigger via UI does not have side-effects in the model)

Verify that the model is consistently displayed in the UI (trigger via UI does not have side-effects in the UI)

Verify that the entire model is still accessible via UI

120


User Story Testing

Develop a system test suite by simulating a user story

Difference to integration testingUser stories cover a chain of actions

Integration testing starts over for each test case

ApproachWrite down a story that covers all features of the application

Specify a test case that runs the story as a system test including UI interaction

Verify consistency after each feature invocation

Use coverage tool to verify that each method was called at least once

121


uDoo User Story Testing

Specify a user story that uses each feature of uDoo

After executing each feature verify consistencyCreate a new todo and change its title: verify

Move a todo and undo the action: verify

etc.

Implement the user story as a FEST-Swing test case

122

DemoFEST-swing in uDooUI Integration TestingUser Story Testing

Regression Testing


Regression Testing

A new or modified component can fail when used with unchanged components

Failures by generating side-effects or feature interactions

When a new or modified component causes a system failure, the system is said to regress

A regression test case is a test case that the baseline version of a component has passed and which is expected to pass when run on a delta version

125


Regression Testing

Regression test suite is a set of existing test cases

Does not contain tests for new or changed functionality

Plays an important role in revealing bugsThe software failure in the Ariane 5 rocket controller was in large part due to an assumption that previously tested code would work when it was reused, obviating the need for regression testing.

126


Regression Testing - When and How

When a new subclass has been developedRerun the superclass test cases on the subclass, beside new test cases for the new subclass

When a superclass is changedRerun the superclass test cases on the superclass and on its subclasses. Rerun the subclass test cases

When a server class is changedRerun test cases for the clients of the changed class plus testing the changed class

When a bug fix has been completedRerun test cases on any part of the software that depend on the changed code to reveal unintended side effects of the bug fix

127


Regression Faults

Two components B and D pass individually their test suite, but fail when used together

D has a side effect on BD changes an entity used by B

D is a client of BD sends a msg that violates B’s invariant or a precondition

D is a server of BB sends a msg to D. D’s postconditions have changed or are buggy. The return value may cause a failure of B

128


Regression Faults

D is a new subclass of a polymorphic hierarchy that is not LSP-compliant

B is a client of D and expects the new subtype to be compatible

D is incompatible with BAriane 5 disaster resulted from an incorrect floating-point conversion. The requirements the algorithm , and the code did not change

An undesirable feature interaction occurs between B and D

129


Reducing a Test Suite

Full regression test can be too time consuming (setup or run)

Reduced regression test: Select a set of test cases for a specific test scenario

In other words, which test cases can be skipped?

Safe reduction (likelihood of finding regression faults is not reduced)

130


Safe Reduction

InclusivenessPercentage of baseline test cases that may show regression faults; a safe regression test suite is 100 percent inclusive

PrecisionPercentage of baseline test cases in a reduced test suite that cannot reveal regression faults and that are not selected for the reduced test suite

EfficiencyCost of identifying a reduced regression test suite

GeneralityThe range of application for the selection strategy

131


Regression Test Patterns

Patterns describe which baseline test cases should be selected

Each pattern has a “safe reduction” evaluation with the predicates inclusiveness, precision, efficiency, and generality

PatternsRetest All

Retest Risky Use Cases

Retest Changed Code

132


Rest All

Rerun the entire baseline test suite on a delta version

Just run all existing tests

Safe reduction evaluationInclusiveness: All baseline tests are selected; => is safe

Precision: No tests that could be skipped are skipped; => the least precise

Efficiency: Lowest cost of analysis and setup. Highest run cost

Generality: Can always be applied. Easy to set up, no analysis or special development is necessary

133



Use risk-based heuristics to select a partial regression test suite. Run this subset on a delta version

Suspicious use cases: Select use cases that depend on components that

are individually unstable or unproven

have not been shown to work together before

implement complex business rules

have a complex implementation

were subject to high churn during development (many changes spread all over the system)

134



Critical use cases: Select tests for use cases that are necessary for safe, effective operation

Safe reduction evaluationInclusiveness: Unsafe; test cases are not selected by analysis of dependencies

Precision: Some tests that could be skipped may be repeated

Efficiency: Setup cost are low: no code analysis is necessary; run cost may be high

Generality: Can always be applied. Easy to set up, no analysis or special development is necessary

135


Retest Changed Code

Use code change analysis to select a partial regression test suite. Run this subset on a delta version

Procedure1. Use Cobertura to check which test case cover which source code parts

2. Use CVS/SVN diff information to see which source code parts changed

3. Rerun test cases that cover deleted or changed source code parts

136


Retest Changed Code

Safe reduction evaluationInclusiveness: All baseline test cases that can produce a different result are selected; => is safe

Precision: Most precise of the white box regression test strategies

Efficiency: If test case selection can be automated, the setup cost is low; in worst case (all baseline test case have to be selected) the total cost of this pattern can be higher than Retest All pattern

Generality: Class scope; => analysis knowledge and special development is necessary

137


Regression Testing and CI

Recommendation: make a separate Maven module and Hudson job

Retest allCopy all existing tests to the regression-testing module

Retest risky use casesCreate a test suite with the risky use cases and configure maven

Retest changed codeWrite a script...

138

DemouDoo Regression Tests

CI Tools


Hudson Plugins

Besides automated build and test Hudson offers a wealth of plugins

Source code management, build triggers, build notifiers, build tools, build wrappers, build reports, Maven, etc.

http://wiki.hudson-ci.org/display/HUDSON/Plugins

Build report plugins for QACheckstyle

Cobertura (already shown)

FindBugs

PMD

141


Checkstyle

“Checkstyle is a development tool to help programmers write Java code that adheres to a coding standard. It automates the process of checking Java code to spare humans of this boring (but important) task. This makes it ideal for projects that want to enforce a coding standard.”

http://checkstyle.sourceforge.net/

Maven pluginhttp://maven.apache.org/plugins/maven-checkstyle-plugin/

Configure via XML or wizard in Eclipsehttp://eclipse-cs.sourceforge.net/

142

http://eclipse-cs.sourceforge.net

http://eclipse-cs.sourceforge.net


FindBugs

“[FindBugs] a program which uses static analysis to look for bugs in Java code.”

http://findbugs.sourceforge.net/

Over 350 bug patternshttp://findbugs.sourceforge.net/bugDescriptions.html

Maven pluginhttp://mojo.codehaus.org/findbugs-maven-plugin/2.0.1/

143

http://findbugs.sourceforge.net/bugDescriptions.html

http://findbugs.sourceforge.net/bugDescriptions.html


PMD

PMD scans Java source code and looks for potential problems like:Possible bugs - empty try/catch/finally/switch statements

Dead code - unused local variables, parameters and private methods

Suboptimal code - wasteful String/StringBuffer usage

Overcomplicated expressions - unnecessary if statements, for loops that could be while loops

Duplicate code - copied/pasted code means copied/pasted bugs

http://pmd.sourceforge.net/

Over 280 ruleshttp://pmd.sourceforge.net/rules/index.html

Maven pluginhttp://maven.apache.org/plugins/maven-pmd-plugin/

144

http://pmd.sourceforge.net/rules/index.html

http://pmd.sourceforge.net/rules/index.html

DemouDoo-QA

seminar on software testing

Education

testing testing

test automationnot

systemunder test

test executionimplementation

test designidentify

expensive software testing

system of test drivers

systemdesign test cases