MODIFIED SESSION BASED TESTING ON GAIANaoki’s Method

Preface Step 1 : Open your mind. Step 2 : Breathe in. Step 3 : Breathe out. This does not replace automation of any

type This doc and supplementary pdfs are

available at http://people.mozilla.com/~nhirata/SBT/

So please stop reading this, and pay attention to the discussion

Sections: Section 1: boring terminology stuff Section 2: How we can apply this to Gaia Section 3: Demo

Section 1: What is QA? What’s exploratory testing? What’s session based testing? Why session based testing?

What is QA? To me, it’s about risk management and then some Unknown amount of bugs with unknown severity

to find and have fixed in a limited amount of time QA has to lower the risk of impact to the end user

before the time of ship. If it was a set number of bugs, we could probably

apply one or two methods. Since it’s unknown, there’s no real good one solution… we employ various techinques

Pick your battles well and try to plan ahead

What’s exploratory testing? “Exploratory testing is an approach to

software testing that is concisely described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1983,[1] now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”” - http://en.wikipedia.org/wiki/Exploratory_testing

Plain English What does the word explore mean to

you?

Now apply that to software.

Why Exploratory? Kaner: Exploratory testing better than scripted

testing- http

://searchsoftwarequality.techtarget.com/news/1323432/Kaner-Exploratory-testing-better-than-scripted-testing

My own thought : 1) If 1+x and x =1 and x never changed… why wouldn’t the actual answer be 2? What would cause you to run it over and over again?2) How much time do you spending testing reading and trying to understand not only the STRs but the results? Once you get to the point of familiarity, do you have tester fatigue?3) If you do the same task over and over… are you thinking? People should be paid to think, computers should be able to do repetitive tasks.

What is Session-Based Testing? “Session-based test management (SBTM) is a

technique for managing exploratory testing. Two of the major criticisms of exploratory testing are that it's hard to make progress visible, and it's hard to know what kind of coverage you might have after exploratory testing has been completed. Session-based test management is one answer to those problems, since it provides a metric for measuring progress (sessions) and takes coverage into account.” - http://searchsoftwarequality.techtarget.com/tip/Using-session-based-test-management-for-exploratory-testing

This was introduced by James and Jon Bach

WAT? Structure and metrics for unstructured

approach to testing Basically:

Set aside 1 hr of straight testing given mission goal

Area tested and your name Jot down notes and bugs at the end Give a summary/account of what you found in

an outline fashion Example : http://www.satisfice.com/sbtm/

sample_session.htm

Um, You said metrics? The session metrics are the primary means to express the status of the exploratory

test process. They contain the following elements: Number of sessions completed Number of problems found Function areas covered Percentage of session time spent setting up for testing Percentage of session time spent testing Percentage of session time spent investigating problems

http://www.satisfice.com/sbtm/index.shtml

Generate with a Pivot Chart: Bug density per each target area. Average bug yield per sessions Bug trend over a given period of time (Example: Total # of sessions and total # of bugs

week by week for the past 3 month).  This is another way to pulse check on how good/bad the product is doing.

Who ran the most sessions/who filed the most bugs http://blogs.msdn.com/b/testing123/archive/2009/03/11/session-based-testing.aspx

Difference between this and bug bash

http://blogs.msdn.com/b/testing123/archive/2009/03/11/session-based-testing.aspx

Supplemental PDFs and readings http://staqs.com/docs/sbtm/ James Lindsay’s 

“Adventures in Session-Based Testing” Jon Bach’s “Session Based Testing” Jon Bach’s “How to Manage and Measure

Exploratory Testing” Jon Bach’s “How to Measure Ad Hoc Testing”

(this was his earlier paper) http://www.mddionline.com/article/applying-

session-based-testing-medical-software

Tools:

http://staqs.com/docs/sbtm/#2 http://sessiontester.openqa.org/

http://openqa.org/ https://sites.google.com/site/

sessionbasedtester/

Sections 2: How Session Based Testing is applied Some modifications necessary Where it doesn’t apply Advantage

Change your mindset 1) This is the opportunity to make things better! 2) Change your mindset from this is working right

to… there are bugs and I will find them 3) Human Cognitive Behavior:

http://bigthink.com/ideas/20583 Pay attention not just to the task at hand but everything

around it. 4) You are working with the Devs not against them

ie make it easy for them to reproduce and figure out where the issues are.

The easier it is for them to figure things out the more inclined they are to fix things earlier.

How Session Based Testing is applied

Have an outline of the positive testings that can be done

Since nothing was mapped for Gaia and it is revolving, it’s easiest to use the requirements as a base

Take that outline and use each bullet point as a mission. Use the concept of states.

A function has a precondition a postcondition and a change. In the same way, this will leave the application in a state

Reset the device if you lose track of the state you are in.

What to test Remember your QA interview question? What would you test for or look for when

testing a mug? That’s right…. You’re applying the

interview question onto the application. Imagine that. An interview question that applies to the job. [/snark]

Generic Testing Outline Positive Test Cases MVC:

UI Dialogs Test for friendly error

messages Functionality Data

Special Case Conditions

Integration Testing Multiple/Cross Feature Network testing

Requirements Safety Installation Compatibility Usability Hardware

Negative Test Cases

Boundary Conditions

Initialization Deletion Screen size SD card space Memory

Race conditions Real World Stress test Performance

Testing Security

Accessibility Color blind Blind

L10n Date/time Number Double byte Hi-ascii Words fitting

ui Proper

translation

Risk Management You don’t have all the time in the world to cover this… what

do you do? Don’t Panic Test the requirements and explore from there. Find the base line by exploring as much as you can by testing

all the requirements Look at the code check-ins and get a gist of all the changes If you have coding experience, think… “how would I

build it and where would I most likely %$#%#$ up?” Then work on the “critical path” and defining it. Put yourselves in the shoes of the end user! “Optimize Quality for Business Outcomes” by Golze, Li, and

Prince (from Mercury Publishing)

Some modifications necessary I haven’t bothered taking notes on time

People often use metrics in the wrong fashion This isn’t a measure on QA time; this is a measure on quality of software

I take roughly 2 to 3 hrs. in a session because I write down bugs at the time of testing, and it requires reproducibility

I think in terms of states and do testing around the mission as well as changes

If you wonder why I painstakingly labeled all the open bugs… now you know why. It’s to figure out which areas are the buggiest.

Having the bugs as the source of truth, you can just look at the gaia issues page and see what area needs the most work.

Because of this it’s important to make sure that all the gaia bugs are up to date and still a source of an issue. Ie verify that these bugs are still bugs.

Where it doesn’t apply Regression testing Smoke Tests

Bottom Line: Automation is still needed to help cover these things! Sometimes regression bugs occur because of an oversight, or unexpected impact change. You still need to do overall testing somehow at the very least in a major cycle.

Why not verification testing? Change is where you find variation Test around the fix!

You might find more bugs!

Advantage Regression testing is basically the scripted testing

once it’s already run once. It takes less time to setup session based testing,

because there’s less reading, modifying, updating based on change. It’s great for a starting project or feature and you find bugs faster due to the less overhead.

Regression testing is better suited for automation, the downside is the overhead cost of creation and maintenance

Note: Unit test automation should still be done to cut down on the number of bugs/cost of fixing the bugs.

Section 3: Demo Look at the system requirements

Take some time… Test…

Look at a verification Take some time… Test around it…

Look at a group of code changes See what changed… Test around it…