put risk based testing in place right now!
TRANSCRIPT
23/11/2016 1
Eric RIOU du COSQUERMinsk, November 24th 2015
Put Risk Based Testing in place, right now!
23/11/2016 2
You took the risk to attend my presentation
Busines Analyst / Product Owner
Project Manager
Test Manager
Functional or Technical Tester
Software Quality and Testing consultant
Sales person
About you
23/11/2016 3
Eric RIOU du COSQUER, [email protected]
• Business Analysis www.iqbba.org• Member of the executive committee
• Requirements Engineering www.reqb.org• Member of the executive committee
• International Software Testing www.istqb.org • General Secretary from 2011 to 2015, France
Representative afterwards
• French Software Testing Qualification Board www.cftl.fr • Manager since 2013
• Test organizations assessment www.tmmi.org • Lead Assessor since 2015
About me
23/11/2016 4
The goal is to explain how to implement a Risk Based Testing approach based on PRISMA® (Product RIsk MAnagement)
Introduction
Risk Management Basics
RBT approach
What next?
Summary
Agenda
23/11/2016 5
Testing, Risk, and Risk Based Testing
Introduction
23/11/2016 6
Main activities (after ISTQB)
What is testing ?
Planning
Cont
rol
Closure
Acceptance
System
Integration
Component 1
Analysis andDesign
Implementation and Execution Evaluation &
Reporting
Planification
Closure
Cont
rol
23/11/2016 7
Definitions (ISTQB)
Risk• A factor that could result in future negative
consequences; usually expressed as impact and likelihood
Product Risk• A risk directly related to the test object
Project Risk• A risk related to management and control of the (test)
project, e.g. lack of staffing, strict deadlines, changing requirements…
What is a risk ?
23/11/2016 8
Definition
Risk Based Testing• An approach to testing to reduce
the level of product risks and inform stakeholders of their status (…). It involves the identification of product risks and the use of risk levels to guide the process
What is « RBT » ?(Risk Based Testing)
23/11/2016 9
A general risk management approach applied to product risks
Risk Management Basics
23/11/2016 10
A process with 4 main activities
Risk Management
Risk assessment
Identification
Analysis
Risk control
Mitigation
Monitoring
What does the general risk management approach consist in ?
23/11/2016 11
The result is a list of risks
• Advice: 30 risks max !
1/4 Risk Identification
Risks TypeRisk 1 Fonctionnal
Risk 2 Security
Risk 3 Fonctionnal
Risk 4: Reliability
…
23/11/2016 12
Define Likelihood and Impact for each risk, and then a risk level
• Risk Level = Probability * Impact
2/4 Risk Analysis
Risks Type Likelihood Impact LevelRisk 1 Fonctionnal
Risk 2 Security
Risk 3 Fonctionnal
Risk 4: Reliability
… … … … …
23/11/2016 13
The risk level calculation may be supported by a table
2/4 Risk Analysis
23/11/2016 14
Implement actions to reduce the risks
• Four mains options1. Mitigate the risk through preventive measures to reduce likelihood
and/or impact2. Make contingency plans to reduce impact if the risk becomes an
actuality3. Transfer the risk to some other party to handle4. Ignore and accept the risk, which means doing nothing but wait and
see whether the problem occurs or not.
• Mitigation with testing• Associate test cases to the risks
3/4 Risk Mitigation
23/11/2016 15
Periodically review the risk status , identify new risks and communicate
4/4 Risk monitoring
Risks Type Proba. Impact Action Status LevelRisk 1 Fonctionnal
Risk 2 Security
Risk 3 Fonctionnal
Risk 4:
Reliability
… … … … …
New Risk
23/11/2016 16
A practical approach, step by step
RBT approachbased on PRISMA®(Product RISk Management)
23/11/2016 17
The decision to implement an RBTapproach must be made
#1RBT Selection
23/11/2016 18
Possible insights
Exhaustive testing is impossible
The allocated test design and execution time and budget is always reduced
The specifications and requirements may not cover the overall set of expected caracteristics
The quality and success of a product depend on the final users and customers view
How to (be) convice(d) to implement an RBT approach ?
23/11/2016 19
The right people to be involved mustbe identified
#2Stakeholders identification
23/11/2016 20
The Test Manager must select different kind of stakeholders
Who should be involved in the RBT process ?
On the vendor side
On the customer
side
• End user (client of the customer)• Other organizations (regulatory entities,
…)
• Customer representatives (called “Business”)
• Project sponsors• End users (from the customer company)• Installation and Operations personnel• Testers and Quality Assurance staff
• Project managers• Business and System Analysts• Developers and architects• DBA• GUI designers• Technical writers• Testers and Quality Assurance staff
23/11/2016 21
PRISMA provides a checklist for stakeholders identification
Who should be involved in the RBT process ?
- Project manager - Business experts- Designers - Testers- Client / sponsor - End users - Usability experts - Operations- Maintenance team - Security - Safety services - Inspectors- Support / helpdesk - Manufacturing- Marketing - Legal- Professional bodies - Special interest groups- Technology experts - Marketing- Customers - System development- Quality assurance - Regulatory bodies
23/11/2016 22
A first list of risks must be created
#3Risk identification
23/11/2016 23
Different techniques can be combined
How to involve the selected stakeholders in the risk identification ?
• Requirements based
• Interviews
• Workshops and Brainstorming sessions
Risks TypeRisk 1 Fonctionnal
Risk 2 Security
Risk 3 Fonctionnal
Risk 4: Reliability
…
Same result as above
23/11/2016 24
The initial set of product risks mustbe improved
#4#4 Risk triage or extended identification
23/11/2016 25
Review the list and check against requirements
• Remove the less relevant risk from the list
• What to do with• A risk but no requirement• A requirement but no risk
How to keep the most relevant risks in the list ?
Product Risk Requirement
ID Product Risk Risk Type Requirement01 Customer cannot start the
transaction at another bankFunctionality Customer shall be able to
perform a transaction at another bank
02 Customer not issued with receipt at the end of the transaction
Functionality Customer shall receive a receipt at the end of the transaction
03 The system is unavailable to the customer for longer than two hours
Reliability System shall be available to customers 24/7
……
Example of a set of product risks for an after Pinkster]
23/11/2016 26
The impact of each risk needs to be rated
#5Impact Rating
23/11/2016 27
PRISMA® suggested factors
1. Critical areas (damage, cost and consequences of failure)2. Visible areas (external visibility of a failure)3. Most used areas4. Business importance5. Cost of rework
Which factors shall we consider to rate the impact ?
Impact
Factor Criticity Visibility …
Weight 2 1 …
Risk 1 5 3 …
Risk 2 3 5 …
Risk 3 3 2 …
… … … …
23/11/2016 28
The likelihood of each risk needs to be rated
#6Likelihood Rating
23/11/2016 29
PRISMA® suggested factors1. Complexity2. Size3. Number of changes4. New technology and methods5. Inexperience6. New development vs. re-use7. Interfacing8. …
Which factors shall we consider to rate the likelihood ?
Impact LIkelihood
Factor Criticity Visibility … Complexity Size …
Weight 2 1 … 1 2 …
Risk 1 5 3 … 3 5 …
Risk 2 3 5 … 4 1 …
Risk 3 3 2 … 2 4 …
… … … … … … …
23/11/2016 30
Once impact and likelihood are scored, the risks are included in a Matrix
#7Risk Matrix creation
23/11/2016 31
Impact and Likelihood are scored for each risk
• Each risk may be rated by different profiles• Impact: business skills• Likelihood: technical skills
How to visualize the risk distribution ?
Impact Probabilité
Factor Criticity Visibility VALUE Complexity Size VALUE
Weight 2 1 na 1 2 na
Risk 1 5 3 13 3 5 13
Risk 2 3 5 11 4 1 6
… … … … … … …
23/11/2016 32
Each risk will be positioned in a matrix
What is the Product Risk Matrix ?
IIV
II IIII
IIIII
Like
lihoo
d of
Def
ects
(T
echn
ical
Ris
ks)
Impact of Defects(Business Risks)
33
15
15
R1
R2
R3
R4R5
23/11/2016 33
IIV
Consider the following advice1. Avoid the central circle2. Try not to have all the risks in the same areas3. Add a fifth area for safety-critical applications
How to ensure a right distribution of the risks ?
IIV
II IIII
IIIII
Like
lihoo
d
Impact33
15
15
R1
R2
R4R5
R5 R7
R6
23/11/2016 34
The test approach will be basedon the risk distribution
#8Test approach and Test techniques selection
23/11/2016 35
Impact and Likelihood help you focus on the right level(s)
How to allocate the test effort on the different levels ?
IIV
II IIII
IIIII
Like
lihoo
d
Impact33
15
15
component and Integration leveltest (focus on technical risk)
systemand acceptance level test (focus on business risk)
23/11/2016 36
This question should be adressed for each test level
How to select the right techniques and define the associated coverage goals ?
IIV
II IIII
IIIII
Like
lihoo
d
Impact33
15
15
Example for the component level
Decision coverage
(90%)
Code inspection
Instruction coverage
(90%)
Instruction coverage
(70%)
23/11/2016 37
This question should be adressed for each test level
How to select the right techniques and define the associated coverage goals ?
IIV
II IIII
IIIII
Prob
abili
té
Impact33
15
15
Use Case(incl alternative
paths)
Decision table
Use Case(main path)
Equivalence partitioning
Use Case(incl alternative
paths)
Equivalence partitioningUse Case
(main path
Exploratory testing
Example for the acceptance level
23/11/2016 38
The traceability from risks to test casesis implemented
# 9Test Design… and Execution
23/11/2016 39
Use the traceability
How to reach the final Risk Based Test Execution step ?
Product Risk Requirement Test CasesTest
Execution Results
Defects
23/11/2016 40
The risk likelihood and impactmust be reviewed based onthe test execution results
#10Risk Based reporting and Defect correction
23/11/2016 41
Update it !
What to do with the Product Risk Matrix
Product Risk Requirement Test CasesTest
Execution Results
DefectsDefects Likelihood is increased
Passed test cases Likelihood is decreased
New risks ?
23/11/2016 42
Increase your knowledge in RBT and implement it right now!
What next ?
23/11/2016 43
And at any time!
RBT is everywhere in the test process
23/11/2016 44
The Best seller about RBT
• ISBN 9789490986070
Sources
23/11/2016 45
With your own Excel file or the PRISMA® tool
The method can be tooled
23/11/2016 46
ISTQB Advanced Level Test Analyst Syllabus
Additional Sources
• http://www.istqb.org/downloads/send/10-advanced-level-syllabus-2012/53-advanced-level-syllabus-2012-test-analyst.html
23/11/2016 47
TMMi• http://
www.tmmi.org/wp-content/uploads/2016/09/TMMi.Framework.pdf
Additional Sources
