From Quality Control to Quality Assurance…and Beyond
Alan PageMicrosoft
IntroductionMy testing career started in 1993
Testing always lastTesting often skippedAnyone can test“Testing in” quality
My role in testing has changed significantly in the past 13 yearsTesting as a whole has matured Disclaimer – I know what I know and don’t claim to know more (and are not representative of microsoft)
Software testing 13 years agoAnyone can do it
Technical knowledge not important
Test is a stepping stone to development (often still true today)
Testing occurs late in the product cycle
Software testing todayMany teams using automation to produce consistent and frequent test results
Often heavy emphasis on automation
Testers are generally focused on:Test authoring: writing test cases and automated testsTest execution: running automated or manual test casesFunctional and non-functional testingOther bug finding activities
Some analysisData analysis is often extensive, but is primarily based on defect data
Software testing tomorrowToo many bugs getting to customers
Bugs need to be found sooner
Where are we going?
Better bug prediction
More testable software
Defect prevention
Inspections
…
How do we get there?
More testable software
Testability is the degree to which components and systems are designed and implemented to make it easier for tests to achieve complete and repeatable code path coverage and simulate all usage situations in a cost efficient manner. In other words, testability determines how inexpensive it is to test.
or
What is the cost of testing?
Testability
Testability is a design time attribute…but testers often drive testability into the design
Ask questions and take action:How are we going to efficiently test this?Add testability sections to design and functional specificationsHold testability reviews
SOCK for testability
Testability: SOCKSimple components are less expensive to test.
Can the hardware be simulated?
Can we test a multi-machine scenario on a single machine?
Observable behavior and results are essential in order to determine pass or fail
Can I determine if the internal structures were updated correctly?
Can I tell which path the code took?
Control: can I test every nook and cranny?
Can I change timeout values or other thresholds to simulate failures?
Can I simulate every failure that a customer may hit?
Knowledge is needed to understand expected results and compare with actual results
What should happen in error cases?
What circumstances cause this error?
SOCK for testability
Defect analysis
Known defects need to be analyzed
Deep analysis can tell a lot, but it’s expensiveToo lightweight of an analysis may not tell you enough
Ask whyAnalyze to the point of action: Ask “why” until you have a sufficient root cause
Why did the program crash? The filename was longer than expected.
Why did the program not expect a longer filename?
The programmer was unfamiliar with file name limits
Why was the programmer unfamiliar with the limits?
They were new to windows programming and hadn’t been trained…or…
Why was this missed during code review?Code review rules were often skipped or rushed.
Ask Why?
Defect prevention
Root cause analysis leads to defect prevention
What could have been done to keep this defect from ever happening?
Examples:Change compiler settings or use code analysis tools or scriptsBuild code often (continuous integration)Automatically run unit tests or verification tests at check-in
Software Inspections
Not typically considered a “Test task” – but testers are well suited to drive this effort
Extremely effective method of removing defects, but…
Cost (training and time) tends to scare teams away
What happens when you inspect?
Example of changes from inspection:
Inspection MetricsExisting
codeRelease
ARelease
BRelease
CInspection Effort 0 hrs 406 hrs 276 hrs 691 hrsTotal LOC 400K 7777 10904 49343Inspection hrs / KLOC 0 52 25 14Total Effort n/a 1062 hrs 1101 hrs 2169 hrsPercent in inspection 0% 38% 25% 32%Total hrs / KLOC n/a 137 101 44Defects / KLOC (after check-in) 8.2 1.8 1 0.7
Measurements were performed on existing code
base, and again on subsequent releases
How to make inspections less scaryTeam members do not know how to inspect
Train the team; appoint a moderator
Inspections perceived to slow down the project
Schedule inspections; measure and track progress
Team members fearful of inspectionsNon-confrontational meetings; remove threats; use as a learning experience for author
What am I trying to say here?We need to stop waiting for bugs!
It’s great to find bugs before the customers do, but we need to find them earlier
We need more emphasis on early detection and ultimately, prevention
How can this be achieved?Be an equal member of the engineering team
Software quality is a tough problemTechnical, creative people are needed to solve this problem
We need clearly defined career paths for both managers and non-managers in order to keep these types of people in test
Provide learning opportunities when needed
Key pointsSoftware testing is a maturing professionNeed to move beyond quality control (writing test cases and running tests)Test needs to do more to drive quality improvements throughout the engineering processTechnical skills are important in order to drive quality initiatives
Resources
Software InspectionsTom Gilb and Dorothy Graham, Addison-Wesley
The Practical Guide to Defect Prevention Harry Emil et al. Microsoft Press
Random thoughts on testinghttp://blogs.msdn.com/alanpa/
Questions?