exploratory testing in practice, short story how approach influenced on strategy

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Exploratory Tests in practice

Short story about how test approach influenced on test strategy

Zbyszek Moćkun, Łukasz Morawski


What is about?

Exploratory Testing


Exploratory tests definition

• Approach or technique?

• What about ISTQB?

• Test techniques (boundary values, decision table, ...)

• Scripting against exploratory

„A technique is a recipe for solving a problem, whereas a style (or approach) is a way of thinking around a theme that stretches far beyond solving a particular problem”

Henrik Emilsson (Blog Thoughts From The Test Eye)


Exploratory approach techniques

• „The Intelectual” tour

• Landmark tour

• Money tour

• Supermodel tour

• Feature

• Complexity

• Claims

• Configuration

• User

• Testability

• Scenario

• Variability

• Interoperability

• Data

• Structure

FCC CUTS VIDS tour by Michael Kelly

Touring by James Whittaker


Exploratory as extension for scripting

„Having formal scripts can provide a structure to frame exploration, and exploratory methods can add an element of variation to scripts that can amplify their effectiveness.”

James Whittaker


The „freedom” scale

Jon Bach, Telling your exploratory story

Where are you on this scale?


Cognifide journey

Exploratory Tests


Why?

• Requirements− Changeability− Clarification of the requirements in the development of production or

testing

• Tester – Dev cooperation− Close Dev – Tester cooperation− Start as fast as possible− Avoid miniwaterfall model− Quick feedback− CI environment − QA services− Retesting− Raising improvements (usability as example)


When?


Improvements - part I

• Repeatable− Notes− Checklists

• Visualization (Integration with dashboard)− Requirement status− Testing activities

• Review process− No time for debrief− Cross testing (mostly for juniors)


Auditable?

Pharmaceutical company performed the audit− Text notes are unclear and contains mistakes/not all information− Lack of test scripts − No traceability

Test scenarios against AC Issues against AC

− No visual status of Acceptance Criteria− No JIRA integration− No regression execution history− No automation coverage

Audit went well, but showed room to improve


Wolf and sheep

John Rieger – wolf and sheep


Improvements - visualization


AC state


Checklist: web domain


Heuristics

• Rule of thumb

• Test Heuristics Cheat Sheet by Quality Tree Software


Testing Dojo


Requirement review


Exploratory tests in steps

• Task delivered to QA (or small part of the task)• Learning phase (including documention)• Test planning (tour session)• Execute planned sessions (note, bugs, questions, AC

update)• Task rejected (back to developer)• Fixing• Retest planning (sessions, or no need)

− Retest bugs− Regression testing if needed (sessions)

• Writing test scenario based for regression purpose• Accept story


Session Based Test Management

• Time management

• Test planning (FCC CID VIDS tour)

• Charters

• Test status

• Debrief (PROOF concept)

• Notes

• Tools− Research existing tools− Session Tester (our own tool)− Bonfire


SBTM – break the rule

• Sessions are not interrupted?− Tester - developer pairing− Continuous Integration environment− Raising bugs take several seconds

Bonfire plugin as example

• Notes− Level of details

• One session per task• Session debrief

− Lonely knight− Time pressure− Cross testing− Project audit


SBTM – Bonfire (JIRA Capture)

• Session Management

• JIRA integration

• Plugin for most common browsers


Metrics

• Exploratory metrics againist scripting approach

• Internal against External Quality− Rejects− Bugs

• Requirement coverage− Scripts against sessions

• Sessions metrics


Summary

Jon Bach, Telling your exploratory story


Q&A

exploratory testing in practice, short story how approach influenced on strategy

Technology

cognifide limited

commercial confidence

test approach

exploratory testing

test eye

exploratory story

exploratory methods

traceability test scenarios