swiss testing day 2013 - how to avoid the testing swiss cheese syndrome
TRANSCRIPT
- 11/02/2008 2Marc Rambert - Kalistick
Our journey
New testing challenges
Testing Swiss Cheese Syndrome
Medication: Test/Code Coverage
How to cure this syndrome
Conclusion & Q & A
- 11/02/2008 3Marc Rambert - Kalistick
New Testing Challenges?
- Time 2 market: from 1 release per year to 1 per day/month/quarter
- Adapt testing to Speed & Quality
- Being Agile
- From one QA team to a whole team doing testing
- Adapt to time & cost constraints
- While managing risks
- 11/02/2008 4Marc Rambert - Kalistick
From 1 testing team to a whole team doing testing
Mike Cohn: Test PyramidFounding member of the Scrum alliance
- 11/02/2008 5Marc Rambert - Kalistick
Application
Version X
Manuel Automated
Unit
Integration
Functional
Regression
User AcceptanceExploratory
The “Testing Swiss Cheese Syndrome” !
Testing ‘holes’ / Gaps
modificationsmodifications
Bug catched !
High Risks!
“Testing is an investment and the investment we make at one layer should be influenced by how well testing has been done at the other layers” Mike Cohn
- 11/02/2008 6Marc Rambert - Kalistick
Where are the testing gaps?
Application – release X
Done by dev Tested
Risks ? Ok Effectiveness ?
8% to 16% of all bugs are created by fixes
Requirements
Capers Jones, “Software Quality in 2011: a survey of the state of the art”
“Dev are from Mars & Test are from Venus”
- 11/02/2008 7Marc Rambert - Kalistick
Application V x.1.11.1.2
Lot of questions.Few information
Modifications?
Impacts? Tests relevance?
‘Build’Tests coverage?
Risks? Risks Coverage?
Test Focus?
How much testing?
Go Live?
- 11/02/2008 8Marc Rambert - Kalistick
Changes
Application
Medication:Capture & Aggregate Test Coverage
Unit Tests
Automated Functional Tests
Integration Tests
Holes = Risks
=> Which manual tests are relevant?
- 11/02/2008 9Marc Rambert - Kalistick
What’s Test/Code coverageor more effective: Test Footprint
Test Team
Manual & automated
tests
Test 1
Application build/Version X
Test 2
Test 3
Test 1 - footprint
Test 2 - footprint
Test 3 - footprint Detect Code Changes
Testing Holes
Tests/Code Coverage
- 11/02/2008 10Marc Rambert - Kalistick
Improve your Testing Strategy
Automated tests executed
Code modifications
Functional tests executed
Functional tests not yet executed
Overall application code base
1
1
2
3Functional tests not yet executed
1
1
- 11/02/2008 12Marc Rambert - Kalistick
How to cure this syndrome
Functional tests
Unit tests
Application
AggregatedTest
Coverage
Softwarechanges
Integration tests
Testing Gaps
Identify relevant tests to fill the gaps
- 11/02/2008 13Marc Rambert - Kalistick
Impact on Testing Effectiveness
– Team 1• After executing planned campaigns• Selected 10 tests with the highest impact (Test Scoring) 3 regressions were found.
Best ratio ever: Bugs/Tests executed
– Team 2• Adopted Test Coverage & Test Scoring to improve its testing activities KPI: number of bugs found within 3 months after deployment - 50%
- 11/02/2008 14Marc Rambert - Kalistick
Conclusion + Q&A
– Different kind of testers with different tools
– Adopt an aggregated view of all testing activities spread along the application life-cycle
– Nurture collaboration between Dev & Test
Code Coverage is not only for developers
Test Footprint is more than coverage
More questions: @MarcRambert