© workroom productions 2002 slide 1 further adventures in session-based testing james lyndsay...
TRANSCRIPT
Slide 1 © Workroom Productions 2002
www.workroom-productions.com
Further Adventures in Session-based Testing
James LyndsayWorkroom Productions
Slide 2 © Workroom Productions 2002
www.workroom-productions.com
?What?
Slide 3 © Workroom Productions 2002
www.workroom-productions.com
Session-Based Testing?
Manages and controls unscripted testing
• Limited Duration• Directed Exploration• Recorded Activities
Supports ‘Agile’ ProcessAllows quick metricsHas wider uses
Slide 4 © Workroom Productions 2002
www.workroom-productions.com
Adventures?
Two case studies
• Permanent team - • 20 coders, 4 testers• Inexperienced and under-motivated• Internet-based product, ASP model
• Part-time team• 2-6 coders, 0-3 testers bought in from pool of 6-8• No permanent test team• Windows client / server product around known
applicationShared business context
• Rapid change• Commercial constraints; budget, time-to-
market
Slide 5 © Workroom Productions 2002
www.workroom-productions.com
Further?
Paper - Adventures in Session-based Testing
• Contains detailed methods• Contains problems and (some)
solutions• Now a year old
STARWest 2002 -Further Adventures . . .
• Methods• Tools• Training, Coaching
Slide 6 © Workroom Productions 2002
www.workroom-productions.com
Strategy
Stay within business constraintsDeal with change rather than enforce stasisRapidly transfer knowledge to allow
independenceFour goals
• Control the scope• Control the work• Measure risk and coverage• Promote ongoing improvement
Slide 7 © Workroom Productions 2002
www.workroom-productions.com1
Methods
Slide 8 © Workroom Productions 2002
www.workroom-productions.com
Methods: Overview
Control and recording
• Control Scope• Control Work• Recording a Session
Assessment and Measurement
• Assessing Risk• Assessing Coverage
Infrastructure
• Real-time Metrics• Social techniques
Slide 9 © Workroom Productions 2002
www.workroom-productions.com
Example: Test Points (Excel-based tool)
Test / Test Group ID Description/instructions FC L RiskEstimated
cost Who Date planned Status Actual cost
SM-I-W20F20E136 Smoke test for release E1.3.6 H L M 0.8 James 7-Aug planned
STR-01 Stress test: Multiple users in one doc - automated H L M 1.0 James 7-Aug plannedSTR-02 Stress test: Multiple users in one doc H H H 4.0 Many 7-Aug plannedSTR-03 Stress test: One user in multiple docs - less important H L M 1.0 James 7-Aug plannedSTR-04 Stress test: Many users observing and working in many documentsH H H 4.0 Many 7-Aug planned
AT-01 Audit trail (both) presentation, usability, interaction with explorer, word, othersL H M 2.0 James 7-Aug plannedAT-02 Audit trail recording events within a doc H L M 0.5 James 7-Aug planned
NC-01 Narrowcast - reproduce and vary new bug H L M 1.0 James 7-Aug plannedNC-02 Further narrowcast tests (may be covered by stress test) H L M 1.0 James 7-Aug left out this time
UE-01 Test customized template (once delivered) H L M 1.0 James 7-Aug plannedUE-02 Test 'Get my doc into LD' use case H L M 1.0 James 7-Aug plannedUE-03 Test 'Get my doc out of LD' case H L M 1.0 James 7-Aug plannedUE-04 Test 'Insert a BLOB' use case H L M 1.0 James 7-Aug plannedUE-05 Test 'Bullets and numbering' use case H L M 1.0 James 7-Aug plannedUE-06 Test 'Setting permissions' use case H L M 1.0 James 7-Aug planned
WR-01 Single-user wrapper tests - automated for coverage H L M 1.0 James 7-Aug plannedWR-02 Tabs Indents H H H 1.0 James 7-Aug plannedWR-03 Bullets and numbers - also see enh #32 L H M 1.0 James 7-Aug plannedWR-04 User Settings - change/restore L H M 2.0 James 7-Aug plannedWR-05 Usability, interaction of error messages and conditions L H M 1.0 James 7-Aug plannedWR-06 Test effect of error conditions on wrapper L H M 2.0 James 7-Aug planned
BLB-01 Inserting Word BLOB - automate for coverage H L M 1.0 James 7-Aug plannedBLB-02 Editing § around BLOBs H L M 1.0 James 7-Aug plannedBLB-03 Editing BLOB contents H H H 1.0 James 7-Aug plannedBLB-04 Editing BLOB - copy/paste - automate for coverage H H H 1.0 James 7-Aug planned
Slide 10 © Workroom Productions 2002
www.workroom-productions.com
Methods: Control
Scope - made of Test Points
• Test Point - piece of work• Risk, Estimated cost• Four groups
• New tests• Regression tests• Retests for fixes• Investigatory tests
Work done - made of Test Sessions
• Test Session - piece of time • Actual cost, Coverage
Slide 11 © Workroom Productions 2002
www.workroom-productions.com
Example: A Test Session
Test Description
Test ID Risk Date + time done
Your name IDs, Machines, Files and other resources used
Estimated time
Actual time
How much more testing does this need?
% complete?
Plan
Notes
Slide 12 © Workroom Productions 2002
www.workroom-productions.com
Methods: Control
Recording a Test Session
• Plan: Approach, Scope, Expectations• Record: Conditions, Actions,
Observations• Bug details• Actual Cost• How done are we? Coverage
Allows
• Review, coaching• Forgetting• Retrospectives
Slide 13 © Workroom Productions 2002
www.workroom-productions.com
Methods: Assessment
Risk
Coverage
• Ask the Experts - the testers!• How done are you? How much is left?• Collect many subjective few objective
assessments • Overall figures can go down as well as up• Ongoing learning - estimates improved
Likelihood of failureRisk High Low
Cost of failure High 3 2Low 2 1
Slide 14 © Workroom Productions 2002
www.workroom-productions.com
Example: Metrics (Excel-based tool)We are 95% complete38% of our tests have been successfulWe think we have 7 hours left
No. failed tests 55No. still not fixed #NAME?No. fixed not tested 1
% planned tests complete 95%Time taken so far 125.15Est. time for these 100.7
total leftPre-test estimate of time 105.7 -19.45Post-test new guess 132 7
All tests Planned tests Completed Tests Successful CompletionEst. Actual %Done %Successful
Number Hours Number Hours Number Hours spent Number Number Number Total 139 182.5 94 92.6 89 106.1 34 95% 38%
H High risk 26 20.1 23 17.1 20 18.9 6 87% 30%M Med risk 68 80.4 48 46 46 54.5 18 96% 39%L Low risk 45 82 23 29.5 23 32.8 10 100% 43%
All retests Planned retests Fix tested Successfully fixed CompletionActual %Done %Successful
Number Hours Number Hours Number Hours spent Number Number Number Total 64 13.3 63 13.1 63 19.05 49 100% 78%
The spreadsheet looks consistent
Slide 15 © Workroom Productions 2002
www.workroom-productions.com
Methods: Support
Real-time metrics
• Immediate feedback• State of testing, not of system• Fast, frequent, informed re-planning
Soft skills/Social Techniques
• Experimental Approach• Ongoing learning• Communication• Empowerment
Slide 16 © Workroom Productions 2002
www.workroom-productions.com2Tools
Slide 17 © Workroom Productions 2002
www.workroom-productions.com
Tools
Freely available - under GPLExcel spreadsheet
• Single-user, single location• Download from:
Web: PHP/MySQL
• Multiple user• Download from:
Slide 18 © Workroom Productions 2002
www.workroom-productions.com3
Training and coaching
Slide 19 © Workroom Productions 2002
www.workroom-productions.com
Ongoing learning and development
Training - from unskilled to skilled
• Skilled involvement - pair expert / novice
• Show - practice - challenge• Careful choice of sessions
Coaching - from good to great!
• Peer involvement, neutral assessment• Challenge - innovate• Discussion, metrics, personal
investigation
Slide 20 © Workroom Productions 2002
www.workroom-productions.com
Supporting the learning process
Session-based testing
• Introduces discipline and framework• Helps overlap training and testing• Helps match challenge to individual
Experts, managers - get involved!Sometimes compromise:
• Good bugs found quickly, or• Better testers long-term
Feedback is integral
• Before testing - techniques, planning, charter
• After testing - session log, bug log• Stats and metrics - not just for management
Slide 21 © Workroom Productions 2002
www.workroom-productions.com
Encouraging the team
Involve the test team
• Allocate responsibility / ownership for tests
• Allow direct influence on clear, open metrics
• Relate metrics to business decisionsShare successes, failuresReward discoveryDiscourage blind faith, boredomStep away - look at the wider picture
Slide 22 © Workroom Productions 2002
www.workroom-productions.com
Changes: Case Study 1
Active learning process is encouraged
• Feedback has improved estimating skills• Session review has improved test techniques• Test Point ownership has improved planning
and scoping skills• Weekly 2-hour self-training session as a team• Greater interest in testing, greater motivation
Useful, visible, up-to-date metrics
• Increase openness, reduce surprise, blame• Joint ownership of decisions, successes,
failures
Slide 23 © Workroom Productions 2002
www.workroom-productions.com!
Conclusion
Slide 24 © Workroom Productions 2002
www.workroom-productions.com
Conclusion
Communication
• Sessions gave us the tools to communicate• Better communication improved trust and
effectivenessOngoing improvement
• Sessions allow directed and disciplined approach• Feedback improved techniques and estimation• Empowerment increased morale and confidence
Openness
• All figures available, all of the time, always current• Encouraged trust and interest• Allowed neutral feedback
Slide 25 © Workroom Productions 2002
www.workroom-productions.com
Conclusion
Test sessions provide:Control of unscripted testing
Framework for ongoing improvement
Overall lessons
• Simple measures are the best• Favour effective communication and
involvement over knee-jerk documentation
• Unobtrusive, immediate metrics allow real-time control and feedback
Slide 26 © Workroom Productions 2002
www.workroom-productions.com
More
Further details:
• Paper, updates, references etc.• www.workroom-productions.com
• Contact: • James Lyndsay