10 signs your testing is not enough
TRANSCRIPT
10 signs your testing is not enough
How do we know?
How do we know if we tested enough?
• Frequently asked question
• Fiascos
• Assuring the minimum acceptable level of quality is a must
• Need to analyze if you should be investing more resources, time, or money in your software testing
• “Not enough” vs “Not Good Enough”
Background
Background
Background
Another horror story
Knight Capital
• Only one defect in the trading algorithm for Knight Capital caused a loss of $ 440 million in about 45 minutes.
• That loss is 3 times the company’s annual earnings.
• The shock that followed caused Knight Capital’s stock to lose 75% of its value in 1 business day.
Infinium shut their algorithm down after just three seconds of trading, but it had already lost more than $1 million.
Lack of understanding
Management misunderstanding
• We don’t have bugs!
• Testing is expensive!• It is an expense• What is the alternative?• Preventive by nature
• We don’t need testers!• We do DevOps!• We do Agile!• Testing and Dev – 2 different skill sets
Positive trend
Budget for software testing grows
How good is your testing?
1: Glitches
Excessive amount of Production bugs:
• Unknown• How long have they been there? 1, 2 or more Test cycles?• Why not discovered earlier?• Not only investing more in testing but also considering changing
Testing Strategy
• Known – Why?
• If found by Customers - Worst from business point of view • Bad PR • May have disastrous effects• Most expensive to fix
2: Bugs in UAT
Too much bugs discovered in Acceptance Testing:
• UAT supposed to build confidence in product
• Last stage before releasing • goal is to find deviations from intended design • Goal is NOT to find bugs
3: Bug counts grow
Are you aware of such trends?
• If bug counts grow you should analyze asap to find reasons
• If you don’t take measures your project/product might go in a Death Spiral
But do you have the Metrics to see trends?
• Basic Metrics are: • easy to apply • Great for analyzing trends
Quality Metrics
Quality Metrics
Project – Level Metrics
1. Requirements and Requirement Coverage – How to measure the coverage?
2. Defect Open and Close Rates
Quality Metrics
Quality Metrics
Department – Level Metrics
1. Mean Time To Detect (MTTD) and Mean Time To Repair – MTTD measures how long it takes QA professionals to find a problem, while the MTTR demonstrates the amount of time needed to effectively address it.
Quality Metrics
Quality Metrics
Company – Level Metrics
1. Number of Issues reported by Customers
4: Not investing enough
Investment in Testing is below industry average
• Always good to know what competition is doing
• You are investing less than similar companies
5: No clear criteria
No definition of what “enough” testing is:
• Coverage• Execution• Acceptable levels of defects
• Major• Minor
• Plan for fixing known defects• etc
6: Listen to Testers’ NO
You receive negative advice from Testers to go ahead:
• Always listen to employees
• Value of good tester
• Analyze the legitimated issues raised
7: Weak Prevention
Get the Team involved in:
• Requirements assessments
• Definition of coding standards/ good practices
• Reviews/Inspections – Static Testing
Prevention
8: No Dev involvement
You don’t have Unit testing in place?
• Software quality is a team responsibility
• Unit testing is natural to developers
• Find bugs earlier
Picture so far
Choices and effects
9: Time problems
9: Time problems
Testing periods are regularly reduced
9: Time problems
Deadlines problems regularly?
• How often does it happen that testing is squeezed
• Lack of time for testing is a symptom of deeper project, process and organizational issues.
• Make sure you make enough time for Testing
10: Testers keep quitting
Dissatisfaction of Test Team:
• Ultimate sign!
• Happy employees – better performance
• Can be financial
• Can be also professional• Not enough time for testing• Not enough resources• Frustration levels grow up
Testing is enough!
Make it even better!
1. Use testing as a high – level risk management
2. Test reusability
3. Automate
Challenges
in the testing…
Questions?