exploratory testing · exploratory testing: a definition testing is context-dependent testing as a...
Post on 23-Jun-2020
9 Views
Preview:
TRANSCRIPT
Exploratory Testing
Software Testing: INF3121 / INF4121
27.04.2017 / Yulai de Meer Fjeld
Learning Outcomes
Understanding the system to be tested
Building quality, not only checking for it
Scripted testing vs. exploratory testing
Let’s play: The Witness
Exploratory testing: A Definition
Testing is context-dependent
Testing as a sapient activity
Heuristics and questioning strategies
What is a computer program?
Computer Program: A Definition
“A set of instructions for a computer”
“A collection of instructions to perform a
specific task when executed by a
computer”
“A list of instructions that tell a computer
what to do”
Is the definition complete?
What is a house?
House: A Definition
“A collection of construction materials
assembled according to house-design
patterns”
House: Other Definitions
“A building for human habitation”
“A building that functions as a home”
“A building in which people meet for a
particular activity”
“A residence for a community”
Computer Program: A New Definition What are computer programs?
“A set of instructions for a computer”
Computer Program: A New Definition What are computer programs?
“A set of instructions for a computer”
Missing factors
Computer Program: A New Definition What are computer programs?
“A set of instructions for a computer”
Missing factors
Intent → Why?
Stakeholders → Who?
Computer Program: A New Definition What are computer programs?
“A set of instructions for a computer”
Missing factors
Intent → Why?
Stakeholders → Who?
Cem Kaner’s alternative definition
The purpose of a program is to provide value to people
“A communication among several individuals and computers, who are
distributed across space and time, that contains instructions that can be
executed by a computer”
Quality To provide value, we expect a certain degree of quality
A computer program is far more than its code
A software product is far more than the instructions for the device
Quality To provide value, we expect a certain degree of quality
A computer program is far more than its code
A software product is far more than the instructions for the device
Quality is far more than the absence of errors
Have you ever bought a program simply because it was bug-free?
Quality To provide value, we expect a certain degree of quality
A computer program is far more than its code
A software product is far more than the instructions for the device
Quality is far more than the absence of errors
Have you ever bought a program simply because it was bug-free?
Most likely, you have not
Instead, we use programs because we believe they provide some benefit
Quality To provide value, we expect a certain degree of quality
A computer program is far more than its code
A software product is far more than the instructions for the device
Quality is far more than the absence of errors
Have you ever bought a program simply because it was bug-free?
Most likely, you have not
Instead, we use programs because we believe they provide some benefit
Quality is value experienced
Improvement | Solution
How to strive for quality?
Quality Assurance
What is software testing?
Software Testing: A Definition Generally defined as:
“A process of executing a program or application with the intent of finding
software bugs”
Software Testing: A Definition Generally defined as:
ISTQB defines software testing as:
“A process of executing a program or application with the intent of finding
software bugs”
“The process consisting of all lifecycle activities, both static and dynamic,
concerned with planning, preparation and evaluation of software products
and related work products to determine that they satisfy specified
requirements, to demonstrate that they are fit for purpose and to detect
defects”
Testing: An Investigation Testing is far more than writing code
Testing is investigative work
Testing: An Investigation Testing is far more than writing code
Testing is investigative work
Testers examine
Code and documents
Systems
People
And the relationships between them
Testing: An Investigation Testing is far more than writing code
Testing is investigative work
Testers examine
Code and documents
Systems
People
And the relationships between them
We need to understand the objective
Why are we testing in the first place?
Testing: A Search for Information Testing is always a search for information
What information do we search for?
Testing: A Search for Information Testing is always a search for information
Find important bugs in order to get them fixed
Assess the quality of the product
Assess conformance to specifications
Find safe scenarios for use of the product
Testing: A Search for Information Testing is always a search for information
Find important bugs in order to get them fixed
Assess the quality of the product
Assess conformance to specifications
Find safe scenarios for use of the product
How do we search for information?
Testing: A Search for Information Testing is always a search for information
Find important bugs in order to get them fixed
Assess the quality of the product
Assess conformance to specifications
Find safe scenarios for use of the product
How do we search for information?
We examine the work products
We compare the results to expected outcomes
We explore!
Scripted testing
Scripted Testing A test script specifies
Test data and operations
Expected results
Comparisons to be made by humans / machines
Scripted Testing A test script specifies
Test data and operations
Expected results
Comparisons to be made by humans / machines
Scripts can control
Manual testing by humans
Automated test execution or comparison by a machine
Scripted Testing: Example Testing login functionality on a simple site
Test data
?
Scripted Testing: Example Testing login functionality on a simple site
Test data
Username and Password
Test procedure
?
Scripted Testing: Example Testing login functionality on a simple site
Test data
Username and Password
Test procedure
Go to site
Type in username and password
Click “Login”
Scripted Testing: Example Let us write a manual script for testing login functionality
Note that we have not included error handling (e.g. incorrect username / password)
Scripted Testing Benefits of scripted testing
What are the benefits of using scripts in testing?
Scripted Testing Benefits of scripted testing
Division of labour
Planning / Analysis / Implementation / Execution / Evaluation
Scripted Testing Benefits of scripted testing
Division of labour
Planning / Analysis / Implementation / Execution / Evaluation
Careful thinking about the design of each test
Created from requirements, code, etc.
Covers all attributes of the system that have been defined
Scripted Testing Benefits of scripted testing
Division of labour
Planning / Analysis / Implementation / Execution / Evaluation
Careful thinking about the design of each test
Created from requirements, code, etc.
Covers all attributes of the system that have been defined
Review by other stakeholders
Reuse- and repeatability
Other testers can replicate the test in no time
Scripted Testing Problems with test scripts
What are the problems with test scripts?
Scripted Testing Problems with test scripts
Programs fail in many ways, some of which may seem inexplicable
Scripted Testing Problems with test scripts
Programs fail in many ways, some of which may seem inexplicable
Scripts are “hit and miss”
Unintentional blindness
Pay attention to certain things, not to others
Scripted Testing Problems with test scripts
Programs fail in many ways, some of which may seem inexplicable
Scripts are “hit and miss”
Unintentional blindness
Pay attention to certain things, not to others
Computers focus only on what they are programmed to look at
Blind by design → Can leave many stones unturned
With a script, you miss the same things every single time
Scripted Testing Problems with test scripts
Programs fail in many ways, some of which may seem inexplicable
Scripts are “hit and miss”
Unintentional blindness
Pay attention to certain things, not to others
Computers focus only on what they are programmed to look at
Blind by design → Can leave many stones unturned
With a script, you miss the same things every single time
We have to explore!
The need for exploration
The Witness: An Example
“Explore an abandoned island”
A video game by Jonathan Blow
Released in 2016
Gameplay
Walk around an island
Solve puzzles
Let’s play!
The Witness: On Exploration What can we learn from The Witness?
The Witness: On Exploration
Failure
Without failing, there can be no learning
The need to embrace
The Witness: On Exploration
Failure
Without failing, there can be no learning
Curiosity
Exploring leads to discovery
The need to embrace
The Witness: On Exploration
Failure
Without failing, there can be no learning
Curiosity
Exploring leads to discovery
Logic
Acknowledge, and adhere to the rules
The need to embrace
The Witness: On Exploration
Failure
Without failing, there can be no learning
Curiosity
Exploring leads to discovery
Logic
Acknowledge, and adhere to the rules
Nonsense
Discard the rules; they are meant to be broken
The need to embrace
What is Exploratory Testing?
Exploratory Testing As defined by the ISTQB
“An informal test design technique where the tester actively controls
the design of the tests as those tests are performed, and uses
information gained during testing to design new and better tests”
Exploratory Testing As defined by the ISTQB
However, it was a term first coined by Cem Kaner
“An informal test design technique where the tester actively controls
the design of the tests as those tests are performed, and uses
information gained during testing to design new and better tests”
“A style of software testing that emphasises the personal freedom
and responsibility of the individual tester, to continually optimise the
value of his/her work, by treating test-related learning, test design,
test execution, and test result interpretation as mutually supportive
activities that run in parallel throughout the project”
Testers Explore For each software / application:
Doing Exploratory Testing Anything that can guide us in
What to test / How to test / Recognise a problem
Doing Exploratory Testing Anything that can guide us in
What to test / How to test / Recognise a problem
Creating / fashioning / constructing according to plan
Design is not scripting
Doing Exploratory Testing Anything that can guide us in
What to test / How to test / Recognise a problem
Creating / fashioning / constructing according to plan
Design is not scripting
Executing the test and collecting the results
Can be automated or manual
Doing Exploratory Testing Anything that can guide us in
What to test / How to test / Recognise a problem
Creating / fashioning / constructing according to plan
Design is not scripting
Executing the test and collecting the results
Can be automated or manual
What we learn from the system under test
Information about the system
Information about how we are testing the system
Testing is Context-Dependent Software systems are different
How are they different?
Testing is Context-Dependent Software systems are different
Intended for different users
Used in different ways
Pose different risks
Testing is Context-Dependent Software systems are different
Intended for different users
Used in different ways
Pose different risks
Require different focus in testing
We cannot test all systems the same way
We need information about the system under test
We let this information guide the test effort
Testing is Context-Dependent Testing depends on the following
Testing is Context-Dependent Testing depends on the following
Even for the same product → Testing differs from release to release
First release → Positive testing | Subsequent releases → Performance testing
Patch → Negative, regression testing
Testing is Context-Dependent Questions to understand the context and build quality
What do we need to understand?
Testing is Context-Dependent Questions to understand the context and build quality
Who are the customers and who are the users?
Is the customer the same as the user?
How can we describe the typical user?
Testing is Context-Dependent Questions to understand the context and build quality
Who are the customers and who are the users?
Is the customer the same as the user?
How can we describe the typical user?
What will the product be used for?
How important is feature X to the customers?
Can they live without this feature?
Testing is Context-Dependent Questions to understand the context and build quality
Who are the customers ad who are the users?
Is the customer the same as the user?
How can we describe the typical user?
What will the product be used for?
How important is feature X to the customers?
Can they live without this feature?
What is the time to market?
How much is invested in the development / testing of the product?
Testing is Context-Dependent Questions to understand the context and build quality
Who are the customers and who are the users?
Is the customer the same as the user?
How can we describe the typical user?
What will the product be used for?
How important is feature X to the customers?
Can they live without this feature?
What is the time to market?
How much is invested in the development / testing of the product?
What are the customers’ and users’ expectations?
Testing: A Sapient Activity Sapience
“The ability to act with judgement”
Homo sapiens → “The Wise Man”
Testing: A Sapient Activity Sapience
“The ability to act with judgement”
Testing always happens under conditions of uncertainty
Good testing stands up to scrutiny
Without scrutiny, software testing would be an easy task
One could test badly, and no one would ever notice / care
Testing: A Sapient Activity Sapience
“The ability to act with judgement”
Testing always happens under conditions of uncertainty
Good testing stands up to scrutiny
Without scrutiny, software testing would be an easy task
One could test badly, and no one would ever notice / care
The test effort will (at some point) be judged
Either by users, management, peers, etc.
This makes testing difficult!
Sapience in Testing Example requirement
“The system shall operate at an input voltage range of 100-250 VAC”
How would you test this?
Sapience in Testing Example requirement
“The system shall operate at an input voltage range of 100-250 VAC”
How would you test this?
Poor answer
“Test with an input voltage in the range of 100-250 VAC”
Sapience in Testing Example requirement
“The system shall operate at an input voltage range of 100-250 VAC”
How would you test this?
Poor answer
“Test with an input voltage in the range of 100-250 VAC”
An improved, but still poor answer
First: “Test with input voltage below 100 VAC”
Then: “Within 100-250 VAC”
Last: “Above 250 VAC”
Sapience in Testing Good testing should answer some fundamental questions
What are some good questions to ask?
Sapience in Testing Good testing should answer some fundamental questions
What happens when the system receives voltage outside its range?
Does it shut down? Blow up? Give erroneous results / error message?
If it is designed to pause and recover; what actually happens?
Sapience in Testing Good testing should answer some fundamental questions
What happens when the system receives voltage outside its range?
Does it shut down? Blow up? Give erroneous results / error message?
If it is designed to pause and recover; what actually happens?
Is there a special gear that can supply voltage at different levels?
If so; we could probably use it
What does it cost? / What are the risks?
Sapience in Testing Good testing should answer some fundamental questions
What happens when the system receives voltage outside its range?
Does it shut down? Blow up? Give erroneous results / error message?
If it is designed to pause and recover; what actually happens?
Is there a special gear that can supply voltage at different levels?
If so; we could probably use it
What does it cost? / What are the risks?
What happens if there is a power spike?
What happens if the power input suddenly drops and then recovers?
Sapience in Testing Good testing should answer some fundamental questions
How long can the product run above or below the rated levels?
Run the product for a long time at the upper and lower voltage ranges
Sapience in Testing Good testing should answer some fundamental questions
How long can the product run above or below the rated levels?
Run the product for a long time at the upper and lower voltage ranges
What happens when we turn the product on and off?
Does the product need time to reach some state before use?
If so; how long does that take?
Sapience in Testing Good testing should answer some fundamental questions
How long can the product run above or below the rated levels?
Run the product for a long time at the upper and lower voltage ranges
What happens when we turn the product on and off?
Does the product need time to reach some state before use?
If so; how long does that take?
Run the system with varying input voltages
Varying slowly and rapidly as we run other tests
Sapience in Testing Good testing should answer some fundamental questions
How long can the product run above or below the rated levels?
Run the product for a long time at the upper and lower voltage ranges
What happens when we turn the product on and off?
Does the product need time to reach some state before use?
If so; how long does that take?
Run the system with varying input voltages
Varying slowly and rapidly as we run other tests
What are the correlations between input voltage and output?
Can we improve this process?
Heuristics How can we structure our problem-solving skills?
Why is this important?
Heuristics How can we structure our problem-solving skills?
Testing happens under constraints → Limited resources
Exhaustive testing is impossible
Need to speed up the process of finding satisfactory solutions
Heuristics How can we structure our problem-solving skills?
Testing happens under constraints → Limited resources
Exhaustive testing is impossible
Need to speed up the process of finding satisfactory solutions
Heuristic → Heuriskein (greek) → Find
Experience-based technique
Problem-solving, learning, and discovery
Heuristics How can we structure our problem-solving skills?
Testing happens under constraints → Limited resources
Exhaustive testing is impossible
Need to speed up the process of finding satisfactory solutions
Heuristic → Heuriskein (greek) → Find
Experience-based technique
Problem-solving, learning, and discovery
Example heuristics
Rule of thumb | Educated guess | Intuitive judgement | Stereotyping
Heuristics: Different Types
Guideword Heuristics Simple strategy for making decisions and finding solutions
Does not provide the answers
Instead: Suggests key elements to consider
Keywords direct attention to the details that are most likely to matter
Guideword Heuristics Simple strategy for making decisions and finding solutions
Does not provide the answers
Instead: Suggests key elements to consider
Keywords direct attention to the details that are most likely to matter
Based on experience and probability
Ensure most crucial aspects are covered
Regardless of how little knowledge the tester has about a program
Guideword Heuristics Simple strategy for making decisions and finding solutions
Does not provide the answers
Instead: Suggests key elements to consider
Keywords direct attention to the details that are most likely to matter
Based on experience and probability
Ensure most crucial aspects are covered
Regardless of how little knowledge the tester has about a program
Words or labels to access your full spectrum of knowledge
Used to effectively analyse and approach a problem
Heuristics: SFDPO What is it?
What files does it have? Is it one program or many?
Heuristics: SFDPO What is it?
What files does it have? Is it one program or many?
What does it do?
What are its functions? What kind of user interface does it have?
Heuristics: SFDPO What is it?
What files does it have? Is it one program or many?
What does it do?
What are its functions? What kind of user interface does it have?
What does it process?
What does the input / output look like? What are the modes / states?
Heuristics: SFDPO What is it?
What files does it have? Is it one program or many?
What does it do?
What are its functions? What kind of user interface does it have?
What does it process?
What does the input / output look like? What are the modes / states?
What does it depend on?
What operating systems does it run on? Special configurations?
Heuristics: SFDPO What is it?
What files does it have? Is it one program or many?
What does it do?
What are its functions? What kind of user interface does it have?
What does it process?
What does the input / output look like? What are the modes / states?
What does it depend on?
What operating systems does it run on? Special configurations?
How will it be used?
Who will use it? Where and how will it be used?
Heuristics: Cheat Sheet
http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
Trigger Heuristics Ideas associated with an event or condition
Recognise when it is time to take further action
Shift focus to think in a different way
Trigger Heuristics Ideas associated with an event or condition
Recognise when it is time to take further action
Shift focus to think in a different way
Emotions and feelings are powerful sources for trigger heuristics
Impatience → An intolerable delay?
Frustration → A poorly conceived workflow?
Surprise → Inconsistency with expectations?
Confusion → Unclear interface? Poor testability?
Annoyance → A missing feature?
Boredom → An uninteresting feature / test?
Heuristics for The Witness Guideword heuristics
Dots → Cover all
Different coloured squares → Separate them
Trigger heuristics
Frustration → Use the environment, find the right perspective
Confusion → Check your assumptions
Tiredness → Time for a break?
Heuristics in Testing We use heuristics the same way when testing
Heuristics in Testing We use heuristics the same way when testing
Notice what you do when trying to approach a problem
What do you do?
What made you succeed?
What made you fail?
Heuristics in Testing We use heuristics the same way when testing
Notice what you do when trying to approach a problem
What do you do?
What made you succeed?
What made you fail?
Find an evocative label or phrase for that thought or process
Should be easy to remember
Heuristics in Testing We use heuristics the same way when testing
Notice what you do when trying to approach a problem
What do you do?
What made you succeed?
What made you fail?
Find an evocative label or phrase for that thought or process
Should be easy to remember
Try using the labels next time you approach a problem
See how effective they can be!
Questioning Strategies Each test case is viewed as a question we ask the product
“Can you do this?”
“What happens if I do that?”
Questioning Strategies Each test case is viewed as a question we ask the product
“Can you do this?”
“What happens if I do that?”
Shifting the focus from designing test cases
To designing a set of questions
Use heuristics to come up with appropriate questions
Questioning Strategies Each test case is viewed as a question we ask the product
“Can you do this?”
“What happens if I do that?”
Shifting the focus from designing test cases
To designing a set of questions
Use heuristics to come up with appropriate questions
Phoenix Checklist
Developed by the CIA → Guides the operatives when facing a problem
Context-free questions → Can be applied to exploratory software testing
Phoenix Checklist
The problem
Why is it necessary to solve the problem?
What is the unknown?
What is it you don’t yet understand?
What is not the problem?
What is the information you have?
Sufficient? Redundant? Contradictory?
Should you draw a diagram of the problem?
Can you separate various parts of the problem?
Have you seen this problem before?
Do you know of related problems?
Can these methods be used?
What are the most probable cases you can imagine?
The plan
Can you solve the whole problem? Part of it?
Can you derive something useful from the given info?
Have you used all the information?
Can you separate steps …
in the problem-solving process?
What should be done?
How should it be done?
What do you need to do at this time?
Who will be responsible for what?
What milestones can best mark your progress?
How will you know when you succeed?
How will you know when you are done?
Phoenix Checklist
The problem
Why is it necessary to solve the problem?
What is the unknown?
What is it you don’t yet understand?
What is not the problem?
What is the information you have?
Sufficient? Redundant? Contradictory?
Should you draw a diagram of the problem?
Can you separate various parts of the problem?
Have you seen this problem before?
Do you know of related problems?
Can these methods be used?
What are the most probable cases you can imagine?
The problem: Testing Wikipedia
The world’s most popular online encyclopaedia
Do not know all of the site’s functionality
The internal structure of the site
Testing the graphical user interface
Access to the site
Access to the database
Could be helpful
Yes → Content / User profiles / Tools
Yes
Testing other websites
Of course
Information search and editing
Phoenix Checklist
The Plan: Testing Wikipedia
Yes
Given info provides an overview of the functionality
Not as of yet
Yes
Plan, Analyse, Implement, Execute, Evaluate
Categorise functionality into areas of use
Create specific tests for each usage area
Examine Wikipedia’s functionality to identify patterns
Has yet to be decided
Completed testing of various usage areas
When my efforts pass someone else’s scrutiny
When all usage areas have been tested
The plan
Can you solve the whole problem? Part of it?
Can you derive something useful from the given info?
Have you used all the information?
Can you separate steps …
in the problem-solving process?
What should be done?
How should it be done?
What do you need to do at this time?
Who will be responsible for what?
What milestones can best mark your progress?
How will you know when you succeed?
How will you know when you are done?
Assessing Exploratory Testing Exploratory testing can prove hugely beneficial
However, it must be applied in the right context
When can it be useful?
Assessing Exploratory Testing Exploratory testing can prove hugely beneficial
However, it must be applied in the right context
Lacking specifications and documentation
When requirements are not available
When under extreme time pressure
Assessing Exploratory Testing Exploratory testing can prove hugely beneficial
However, it must be applied in the right context
Lacking specifications and documentation
When requirements are not available
When under extreme time pressure
Other considerations
What are the disadvantages of exploratory testing?
What should you pay extra attention to?
Assessing Exploratory Testing Exploratory testing can prove hugely beneficial
However, it must be applied in the right context
Lacking specifications and documentation
When requirements are not available
When under extreme time pressure
Other considerations
Quality of test effort depends on the tester’s skill
Limited by the domain knowledge of the tester
Should be used to complement other test techniques
Summary
Scripted testing Exploratory testing
Closing Remarks
The lecture slides were based on
Previous lecture slides by Raluca Florea (2016)
Black, R., van Veenendal, E., Graham, D. (2012). Foundations of Software
Testing: ISTQB Certification 3E. Cengage Learning.
IEEE 829: Standard for Software and System Documentation
top related