Outsourcing Test activities

How to prove cost reductionsHarold van Heeringen, Metrics Consultant

Sogeti

Wednesday May 28, Milan, Italy

Outline

• Problem definition• Factors that influence Test

Performance• Model CASE• Results & Conclusions

Problem definition

• More and more outsourcing test activities

• Contracts - x% of cost reductions for the client in 1,2,3 years

• So far: Impossible to prove!>Compare bills ?>What about volume increase/decrease ?>What about defects found… and defects not

found?

Test activities

Functional design

Technical design

Coding + Unit test

Systems test

User Acceptance Test

Client

Client

Sogeti

Client

Factors that influence Test Performance

• The most important factors:>Number of defects found (in relation to

number of defects present)Many defects found – did we test well?Few defects found – did we test badly?

>Number of hours spentWe have spent 1.000 hours testing this project. Is that

OK?

>Time interval availableBuild phase ended late, but end date is fixed. What

can we do in the time interval available?

The relation between these factors

• Extra short duration...>How many extra

hours do we need?

• Extra long duration...>How many extra

defects will we find?>How many defects

are not found?

Impossib

le

Unpractical

Eff

ort

(h

our)

DurationD

efe

cts

rem

ain

ing

When do we test well?

• More defects found than expected?• Less hours spent than expected?

• But how do we know the number of defects and hours expected??

• We need an estimation tool!

Defects found < expected

Defects found = expectated

Defects found > expected

hours spent < expected

? + ++hours spent = expected

- ± +hours spent > expected

-- - ?

QSM SLIM Estimate

• Project Estimation> Hours, duration, fte, costs, etc

> But also: defects !!!

Benchmark test performance

• Benchmark supplier performance against client performance – but how??>Every project is different >Duration is critical variable>Test hours and defects found are highly

dependent on the Build performance

• How can we know the client’s performance?

• Hours spent• Defects found• Defects not found

Model proposed: 3 stages

>1: Baseline analysis – assess the client situation

• PI (productivity duration)• Cost per test hour• Defect Tuning factor

>2: Operational execution – Estimate, measure and benchmark performance

• Simulate project with actual duration, but with client baseline parameters

• Measure actuals• Calculate Cost reductions

>3: Evaluation – Evaluate whether the cost reduction targets have been met

Model step 1: Baseline analysis

• Select 4 – 5 representative projects>Size the projects in COSMIC/FPA/slocs>Administrate results in SLIM Datamanager

• Results:>Detailed analysis per project>SLIM Estimate WBS>Average/median Productivity index>Defect tuning factor>Costs per test hour

CASE: Baseline analysis

• Contract: outsourcing systems testing • Target: 8% Cost reductions 1st year

• Baselinedata:

• 1 test hour = 80 euro> activities?> Fixed costs?> Waiting hours?

Project Size in

CFP

Hours S-test

Defects S-test

Defects A-test /1st M production

A 310 554 12 8

B 512 937 27 5

C 212 508 22 12

D 224 732 52 18

E 487 878 35 8

Model step 2: Operational execution

• Choice: Measure everything, or measure predefined set of projects1 – Measure size 2 – Estimate project with client parameter

set and actual duration3 – Compare client estimation with Sogeti

actual data4 – Calculate Test Performance5 – Calculate Cost Reduction

CASE: Operational execution

Step 1 - 5 projects selected to measureStep 1 - Size measurement in COSMICStep 2 - Estimation in SLIM, using client baseline parameters and actual durationProject Q

Scenario Defects S-test FS

Defects A-test + 1M Prod. FAP

Defects Total FT

FS/FT

Client baseline

30 31 61 0,49

Scenario Test hours TU

Hour rate R

Costs C (TU*R)

Client baseline

2.045 80 163.600

CASE: Operational execution

Step 3 – Compare Sogeti actuals against baseline scenarioProject Q

Scenario Test hours TU

Hour rate R

Costs C (TU*R)

Client baseline

2.045 80 163.600

Sogeti actuals

1.800 82 147.600

Scenario Defects S-test FS

Defects A-test + 1M Prod. FAP

Defects Total FT

FS/FT

Client baseline

30 31 61 0,49

Sogeti actuals

35 30 65 0,53

CASE: Operational execution

Step 4 – Assess Test Performance projects> Outsourcing Effectiveness

Did we test better than the client would have done?

> Outsourcing EfficiencyDid we test more efficiently than the client would have done?

Outsourcing Efficiency OF

Baseline C – Actual C Baseline C

Outsourcing Effectiveness OV

(Actual FS/FT) – (Baseline FS/FT) Baseline (FS/FT)

*100%

*100%

CASE: Operational execution

• Project Q>Outsourcing Effectiveness:

OV = (0.53-0.49)/0.49 ≈ 8,2 %

>Outsourcing Efficiency: OF = (163.600–147.600) / 147.600 ≈ 9,7%

Outsourcing effectiveness

Outsourcing efficiency

- 100 %

100 % 0 %

100 %

- 100 %

0 %

No Cost Reduction

Cost Reduction

CASE: Operational execution

• Step 5 – Calculate Cost Reduction>Maybe client rates OV higher than OF or

vice versa>Contract – agree on weight factors

OV = Outsourcing EffectivenessOF = Outsourcing Efficiencyw1 = weight factor OVw2 = weight factor OF

Cost Reduction CR

(w1*OV) + (w2*OF) w1+w2

Model step 3: Evaluation

• After contract period: Analyze results

Outsourcing effectiveness

Outsourcing efficiency

- 100 %

100 % 0 %

100 %

- 100 %

0 %

No Cost Reduction

Cost Reduction

8%

8%

CASE: Evaluation

• Evaluation client X, 1st year

• Total CR: € 129.143 / € 1.383.570 ≈ 9,3%• Substantial cost reductions, even more than

was promised!!

Project OV OF C Baseline CR % CR €

Q 9,7% 8,2% € 163.600 8,95% € 14.642

R 12,3% 6,1% € 312.150 9,2% € 28.718

S 8,2% 3,0% € 122.650 5,6% € 6.868

T 6,4% 13,3% € 718.120 9,85% € 70.735

U 15,6% 8,8% € 67.050 12,2% € 8.180

Total € 1.383.570 € 129.143

First results

• Clients reactions are very positive• Model should be evaluated in

practice and should be developed further

• Please help us further develop the model:>Metrieken.sogeti.nl>sec@sogeti.nl

Conclusions

• The model proposed is objective and can be used in all organizations and all projects

• The client can now see the real cost reductions they achieve by outsourcing

• The supplier can also use the model as a commercial instrument and is able to support claims of cost reduction based on experience

metrieken.sogeti.nl