End-To-End TestingWhat is different?

SAST Stockholm,14 October 2010

Rob Baarda

2

Agenda

1. Introduction

2. Scope and definition

3. End-to-end testing?!

4. Research and conclusions

5. What is needed and different?

6. Conclusion

3

Importance of chains is growing

• Chains grow across organizations like government, specialized organizations

• E-mail, SOA and the internet make data exchange simple

4

But: Chains are vulnerable

• Several disasters in the Dutch press 700,000 income

tax statements lost

• Damage in one system inflicts other chain partners railway tickets

• Not all systems/processes are developed for use in chainre-use, identity theft

• Increasing customer demands reduce tolerance for failures

5

Example of a chain: webshop

Order System

StockSystem

InvoiceSystem

FinancialSystem

ExternalBank system

Client mgtsystem

Online Banking

PurchasingSystem

StockSystem

Webshop Portal

WarehouseMailSystem

Placeorder

Receiveorder

Createinvoice

Receiveinvoice

Payinvoice

Receivepayment

Send item

Receiveitem

6

Definition of the end-to-end test

• The dynamic test intended to demonstrate that the consecutive series of systems supports the (business) process according to specifications

A

B

End-to-End

A B

Interfacetest

7

Do we need E2E?

Each part is unit tested and approved

according to specification

8

Is there a need to test the car?

Each part is tested and approved, but the car isn’t!

Test depends on the risk!

9

Research study E2E Testing

60+ programs & projects analyzed & interviewed

• End-to-end test strategy and plan− Test strategy determined : < 10%

− Within budget and planning : < 30%

− Average budget overrun : > 300%

• Chain risk analyses− Chain risks : < 10%

− No ownership of the full chain

− Part of a chain without owner

• Not always an E2E test performed: balance risk feasibility/cost

10

How to get in control?

• To better estimate and plan: − make a process model, estimate and measure

• Develop a risk analysis for chains

• Look at the feasibility of E2E testing

• Get management involvement

11

End-to-end test process

Inventorying

processes

and

systems

Analysing

chain risks and

determining

feasibility

Determining

-Scope

-Sub-chains

-Strategy

-Infrastructure

Creating

the IETP

Planning

and

Estimating

Determining

limitations and

measures

Determining

-Strategy

-Planning

-Estimate

Creating the

SETP

Detailing

dataflows

Completing the

end-to-end test

Executing

the end-to-end

test

Specifying test

infrastructure

build and

setup

Realising the

end-to-end test

infrastructure

and intake

Specifying

logical

end-to-end test

cases

Filling basic

files and

creating

physical end-

to-end test

cases

Creating the

end-to-end test

scenario

Adaptive

Preparation

Creating the

infrastructure

design

document

E2E Test

Management

E2E

Infrastructure

Coordination

E2E Test

Execution

E2E Test

Direction

GO

NO GO

GO

NO GO

GO

NO GO

GO

NO GO

12

Risk in a chain

Damage

Ch of Fail

Ch of Fail

Ch of Fail

Ch of FailCh of Fail

Ch of Fail

Ch of Fail

Ch of Fail

Ch of Fail

Ch of Fail

Ch of Fail

Ch of Fail

Ch of Fail

• Ch of Fail = Chance of failure

13

EPF (elec. patient file) a chain of systems

Region.

Hospital

National

switch

Etc.

Identity

Mgt

Patient

Physio-

therapist

General

Pract.

Pharmacy

Academ.

Hospital

Identity

Mgt

Med staff

14

You

survived

Chance of failure all over, damage at the end

Hospital

National

Switch

General

Pract.Pharmacy

Identity

mgt

Ambu-

lance

chance of

failure

chance of

failure

chance of

failure

chance of

failure

chance of

failure

chance of

failure

chance of

failure

Holiday

Loss of

conscience

Ambulance

gets info

Pharmacist

adjusts

GP

informed

You

survived

Damage

= you

15

Chance of failure by dependencies

C* DA B F

K

E

H

GSecondary

chain

Primary chain

NL M PO

Tertiary chain

C* = adapted for a chain reason

16

Feasibility of E2E test

• How difficult/feasible is it to test End-to-End?

• Areas:1. Knowledge & Accessibility

2. Organization & Management

3. Project environment

4. Test environment

17

Strategy = Scope & Sub-chains

• Possible strategy outcome = scope− Full E2E test

− No E2E test

− Testing some sub-chains

• Sub-chains

Systems or manual processes

A B C D E F G H I J K L M N O PO O O O O O X X O O O O O O O O

Sub chain test 1

Sub chain test 3

Sub chain test 2

Available for sub end-to-end testX: not availableO: available

18

Strategy & Infrastructure

• Strategy− Intensity of test for each sub-chain (dependent on risk)

− Happy flow and/or failure flow

− Performance, back-up etc.?

• Infrastructure− Develop a specific E2E test environment and/or

− Use test environments of the underlying systems and sometimes production

19

Planning & Estimation

• First make planning than estimate!

• Planning − Depending on milestones and availability of SW & test

environments

• Estimation− 20-50 test cases

− In 8-system chain about 4-10 hours a test case

− Possibly a lot more time is needed for finding information and organizing the test infrastructure (feasibility)

20

First results of the E2E TMap method

− Elapse time E2ET : - 50%

− Elapse time UAT : - 40%

− Incidents : -90%

Insurance, launching customer

•E2ET no more on critical path

•Clear view on risks

•Project quality improved

21

End-to-end testing

with TMap NEXT®

www.tmap.net

For checklists and templates

(partly under construction)

Want to know more?