(exploratory) testing with scripted support
DESCRIPTION
Some consideration and reflections on applying good software test thinking in signalling protocol environments using "agile" and incremental development models.TRANSCRIPT
(Exploratory) TestingWith Scripted Support
Simon Morley October 2014
Experience & Considerations
Saturday 25 October 14
My Situation Context• Complex Environment
• Meaning overhead & set-up costs
• Meaning preparation, tooling or ordering
• Many signaling protocols
• Meaning script support for signal injection and check
• Agile & Incremental SW Development
Saturday 25 October 14
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Saturday 25 October 14
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
Testing before “testing”• Testing in the “Opportunity Analysis”,
“Planning” & “Design” “phases” might be:
• Feature Investigation - from a testing perspective
• Testable?
• Creates issues with existing framework, tools and maintenance?
• Feature Walkthrough
Saturday 25 October 14
Possible Testing Approach• Investigate the new feature
development
• Develop framework and tools, develop scripts, comms, test scripts & framework & feature, comms, debug scripts, comms, debug environment, comms, test feature, comms, report completion, archiving.
Saturday 25 October 14
Feature Investigation
• Feature Walkthrough
• Test environment impacts & walkthrough
• Test Analysis summary
Saturday 25 October 14
Test Analysis Summary• Describe* or reference the current
feature, test environment and framework situation
• Describe* impacts to test environment and framework
• Describe* approach
• Include coverage goals for product, feature and testing
Saturday 25 October 14
Comms?• Communication
• Can be: Verbal & Written
• Can be: Formal & Informal
• Always: Two-way!
• Reporting - stand-ups, progress boards, meetings
• Reflect & decide about next steps
Saturday 25 October 14
Reflect & Decide?• What has the testing so far revealed?
• Product coverage goals met?
• Testing coverage goals met?
• Perceived risk acceptable?
• Stakeholder view on these?
Saturday 25 October 14
Reflect & Decide #2
• Next steps?
• Additional test steps?
• Enough already?
• Change direction or approach?
Saturday 25 October 14
Reflect & Decide #3• Pause & Reflect
• Control Q: “What don’t we know?”
• Silent Evidence
• Note: This pause & reflection is important - see thinking about heuristic discovery.
Saturday 25 October 14
Necessary support?• Can the tools & framework support
changes?
• Is the tool & framework architecture modular?
• Remember to test the tool architecture!!!
• Do the tools & framework allow quick prototyping?
• If not -> fix it!
Saturday 25 October 14
Describe*• Describe: Meaning the level sufficient in
the team or project.
• If coordination is needed between teams this should be a format that is commonly understood (picture/symbols, dashboard or text document).
• You agree what’s appropriate!
Saturday 25 October 14
Exploratory Testing Traps• Sometimes people think of ET means
less:-
• Documentation
• Structure
• Traceability
• And more
• Random banging on a keyboard
Saturday 25 October 14
Exploratory Testing Traps• Sometimes people think of ET means
less:-
• Documentation
• Structure
• Traceability
• And more
• Random banging on a keyboard
Wrong !!!
Saturday 25 October 14
ET Traps #2• Actually ET is about more
responsibility in skilled testing
• The skilled tester is responsible for communicating his/her approach, thinking, results and any mandated archiving
Saturday 25 October 14
ET Traps #2• Actually ET is about more
responsibility in skilled testing
• The skilled tester is responsible for communicating his/her approach, thinking, results and any mandated archiving I.E. Plenty of structure,
traceability & documentation
where appropriate!
Saturday 25 October 14
ET Traps #3• No support for skilled testers and
testing.
• Support for skilled testing is a pre-requisite (hand-in-hand) to realizing emphasized tester responsibility.
Saturday 25 October 14
ET Traps #3• No support for skilled testers and
testing.
• Support for skilled testing is a pre-requisite (hand-in-hand) to realizing emphasized tester responsibility.
Fix these if you want good & valuable testing!
Saturday 25 October 14
Discussion• Exploratory testing can be performed
with scripting support
• Good testing (in complex environments) requires elements of:
• Domain knowledge - built through early investigation
• Test environment knowledge & understanding
Saturday 25 October 14
Discussion #2
• Plus
• Rapid learning
• System and problem analysis
• Decision making
• Framing, Consensus & Commumnication
Saturday 25 October 14
References• Experience Report: Introducing Exploratory
Testing
• Deliberated Exploration - A Walk in the Woods
• Experiences with Semi-Scripted Exploratory Testing
• Framing: Some Decision Analysis and Frames in Testing
• Silent Evidence in Testing
• On Thinking about Heuristic Discovery
Saturday 25 October 14
Thanks!• Simon Morley
• Twitter: @YorkyAbroad
• Blog: testers-headache.blogspot.com
• Questions?
• Get in touch!
Saturday 25 October 14
Feature Walkthrough
Saturday 25 October 14