rapid testing presentation

8/11/2019 Rapid Testing Presentation

1/46

MA

// 0

A Rapid Introduction to Rapid

Software Testing

Presented by:

Michael BoltonDevelopSense

0 , 00, , 0

0 00 . ..


2/46

Michael Bolton

Tester, consultant, and trainer Michael Bolton is the coauthor (with James Bach) of Rapid SoftwareTesting, a course that presents a methodology and mindset for testing software expertly in uncertainconditions and under extreme time pressure. Michael is a leader in the context-driven software testing

movement with twenty years of experience testing, developing, managing, and writing about software.Currently, he leads DevelopSense, a Toronto-based consultancy. Prior to DevelopSense, he was withQuarterdeck Corporation, where he managed the companys flagship products and directed project andtesting teams--both in-house and worldwide. Contact Michael at [email protected].


3/46

ARapidIntroduction

toRapidSoftwareTesting

JamesBach,Satisfice,Inc.

[email protected]

www.satisfice.com

+1(360)4401435

MichaelBolton,DevelopSense

[email protected]

www.developsense.com

+1(416)6565160

2

Acknowledgements Someofthismaterialwasdevelopedincollaborationwith

Dr.Cem Kaner,oftheFloridaInstituteofTechnology.Seewww.kaner.comandwww.testingeducation.org.

DougHoffman(www.softwarequalitymethods.com)hasalsocontributedtoandoccasionallyteachesfromthismaterial.

ManyoftheideasinthispresentationwerealsoinspiredbyoraugmentedbyothercolleaguesincludingJonathanBach,BretPettichord,BrianMarick,DaveGelperin,ElisabethHendrickson,JerryWeinberg,NoelNyman,andMaryAlton.

SomeofourexerciseswereintroducedtousbyPaysonHall,Jerry

Weinberg,Ross

Collard,

James

Lyndsay,

Dave

Smith,

Earl

Everett,

BrianMarick,Cem Kaner andJoeMcMahon.

Manyideaswereimprovedbystudentswhotookearlierversionsoftheclassgoingbackto1996.


4/46

3

AssumptionsAboutYou

Youtest

software,

or

any

other

complex

human

creation.

Youhaveatleastsomecontroloverthedesignofyour

testsandsometimetocreatenewtests.

Youareworriedthatyourtestprocessisspendingtoo

muchtimeandresourcesonthingsthatarent

important.

Youtestunderuncertaintyandtimepressure.

Yourmajor

goal

is

to

find

important

problems

quickly.

Youwanttogetverygoodat(software)testing.

A

Question

What makes

testing harder or

slower?


5/46

PremisesofRapidTesting

1. Softwareprojectsandproductsarerelationships

betweenpeople.

2. Eachprojectoccursunderconditionsofuncertainty

andtimepressure.

3. Despiteourbesthopesandintentions,some

degreeofinexperience,carelessness,and

incompetenceisnormal.

4. Atestisanactivity;itisperformance,notartifacts.

Premises

of

Rapid

Testing5. Testingspurposeistodiscoverthestatusofthe

productandanythreatstoitsvalue,sothatour

clientscanmakeinformeddecisionsaboutit.

6. Wecommittoperformingcredible,costeffective

testing,andwewillinformourclientsofanythingthat

threatensthatcommitment.

7. Wewillnotknowinglyornegligentlymisleadour

clientsand

colleagues

or

ourselves.

8. Testersacceptresponsibilityforthequalityoftheir

work,althoughtheycannotcontrolthequalityofthe

product.


6/46

7

RapidTesting

Rapidtestingisamindset

andaskillsetoftesting

focusedonhowtodotesting

morequickly,

lessexpensively,

withexcellentresults.

This

is

a

general

testingmethodology.Itadaptsto

anykindofprojectorproduct.

8

Rapidtestingmaynot

beexhaustive,butitis

thoroughenoughand

quickenough.Itsless

workthanponderous

testing.Itmightbeless

workthan

slapdash

testing.

Itfulfillsthemission

oftesting.

HowdoesRapidTestingcompare

withother

kinds

of

testing?

Managementlikestotalk

aboutexhaustivetesting,but

theydontwanttofunditand

theydontknowhowtodoit.

Youcanalwaystest

quickly...

Butitmightbepoor

testing.

Whentestingisturnedintoan

elaboratesetofrotetasks,

itbecomesponderouswithout

reallybeingthorough.

M

oreWork&Time

(Cost)

BetterThinking&BetterTesting

(Value)

Slapdash

Muchfaster,

cheaper,

andeasier

PonderousSlow,expensive,

andeasier

RapidFaster,lessexpensive,

stillchallenging

ExhaustiveSlow,veryexpensive,

anddifficult


7/46

9

ExcellentRapidTechnicalWork

BeginswithYou

When the ball comes to you

Doyouknowyouhavetheball?

Canyoureceivethepass?

Doyouknowwhatyour

roleandmissionis?

Doyouknowwhere

yourteammatesare?

Areyoureadytoact,rightnow?

Canyouletyourteammateshelpyou?

Doyouknowyouroptions?

Isyourequipmentready?

Canyoureadthe

situationonthefield?

Are

you

aware

of

the

criticalityofthesituation?

10

Rapidtestteamsareaboutdiversetalentscooperating

Wecallthistheellipticalteam,asopposedtotheteamof

perfectcircles.

Someimportantdimensionstovary:

Technicalskill

Domainexpertise

Temperament(e.g.introvertvs.extrovert)

Testingexperience

Projectexperience

Industry

experience Productknowledge

Educationalbackground

Writingskill

Diversitymakesexplorationfarmorepowerful

Yourteamispowerfulbecauseofyouruniquecontribution

butyoudonthavetobegreatateverything.


8/46

WhatItMeansToTestRapidly

Sincetesting

is

about

finding

apotentially

infinite

numberofproblemsinaninfinitespaceinafinite

amountoftime,wemust

understandourmissionandobstaclestofulfillingit

knowhowtorecognizeproblemsquickly

modeltheproductandthetestspacetoknowwheretolook

forproblems

preferinexpensive,lightweight,effectivetools

reducedependenceonexpensive,timeconsumingartifacts,

whilegetting

value

from

the

ones

weve

got

donothingthatwastestimeoreffort

tellacrediblestoryaboutallthat11

One

Big

Problem

in

Testing

Formality Bloat

Much of the time, your testing doesnt need to be very formal*

Evenwhenyourtestingdoesneedtobeformal,youllneedto

dosubstantialamountsofinformaltestinginorderfigureout

howtodoexcellentformaltesting.

Whosays? TheFDA. Seehttp://www.satisfice.com/blog/archives/602

Even in a highly regulated environment, you do formal testing

primarily for the auditors. You do informal testing to make sure

you dont lose money, blow things up, or kill people.

*Formaltestingmeanstestingthatmustbedonetoverifyaspecificfact,

orthatmustbedoneinaspecificway.


9/46

EXERCISE

Test the Famous Triangle

14

Whatistesting?Serving Your Client

Ifyoudonthaveanunderstandingandanagreement

onwhatisthemissionofyourtesting,thendoingit

rapidlywouldbepointless.


10/46

NotEnough

ProductandProjectInformation?

Where do we get

test information?

What

Is

A

Problem?

A problem is


11/46

HowDoWeRecognizeProblems?

An oracle is

a way to recognize

a problem.

Learn

About

HeuristicsHeuristicsarefallible,fastandfrugalmethodsofsolving

problems,makingdecisions,oraccomplishingtasks.

Theengineeringmethodis

theuseofheuristics

tocausethebestchange

in

a

poorly

understood

situation

withintheavailableresources.

BillyVaughanKoen

DiscussionoftheMethod


12/46

Heuristics: GeneratingSolutions

QuicklyandInexpensively

Heuristic(adjective):

servingtodiscoverorlearn

Heuristic(noun):

afalliblemethodforsolvingaproblem

ormakingadecision

Heuristicreasoningisnotregardedasfinalandstrict

butas

provisional

and

plausible

only,

whose

purpose

istodiscoverthesolutiontothepresentproblem.GeorgePolya,HowtoSolveIt

OraclesAnoracleisaheuristic principleormechanism

bywhichwerecognizeaproblem.

...itappearedatleastoncetomeetsome

requirementto

some

degree

...uh,whenIranit

...thatonetime

...onmymachine.

Itworks!

really means


13/46

FamiliarProblems

Ifaproductisconsistentwithproblemsweveseenbefore,

wesuspectthattheremightbeaproblem.

Explainability

Ifaproductisinconsistentwithourabilitytoexplainit

(orsomeoneelses),wesuspectthattheremightbeaproblem.


14/46

World

Ifaproductisinconsistentwiththewaytheworldworks,


History

Ifaproductisinconsistentwithpreviousversionsofitself,


Okay,

so how the#&@ do I prin t

now?


15/46

Image

Ifaproductisinconsistentwithanimagethat

thecompanywantstoproject,wesuspectaproblem.

Comparable

Products

WordPad Word

Whenaproductseemsinconsistentwithaproductthatis

insomewaycomparable,wesuspectthattheremightbeaproblem.


16/46

Claims

Whenaproductisinconsistentwithclaimsthatimportant

peoplemakeaboutit,wesuspectaproblem.

User

Expectations

Whenaproductisinconsistentwithexpectationsthata

reasonableusermighthave,wesuspectaproblem.


17/46

Purpose

When a product is inconsistent with its designers explicit

or impl icit purposes, we suspect a problem.

Product

When a product is inconsistent internallyas when it

contradicts itselfwe suspect a problem.


18/46

StatutesandStandards

Whenaproductisinconsistentwithlawsorwidely

acceptedstandards,wesuspectaproblem.

32

Consistency(thisagreeswiththat)an

important

theme

in

oracle

principles

Familiarity:Thesystemisnotconsistentwiththepatternofanyfamiliarproblem.

Explainability: Thesystemisconsistentwithourabilitytodescribeitclearly.

World: Thesystemisconsistentwiththingsthatwerecognizeintheworld.

History: Thepresentversionofthesystemisconsistentwithpastversionsofit.

Image:Thesystemisconsistentwithanimagethattheorganizationwantstoproject.

ComparableProducts:Thesystemisconsistentwithcomparablesystems.

Claims: Thesystemisconsistentwithwhatimportantpeoplesayitssupposedtobe.

UsersExpectations:Thesystemisconsistentwithwhatuserswant.

Product:Eachelementofthesystemisconsistentwithcomparableelementsinthe

samesystem.

Purpose: Thesystemisconsistentwithitspurposes,bothexplicitandimplicit.

StandardsandStatutes:Thesystemisconsistentwithapplicablelaws,orrelevant

implicitorexplicitstandards.

Consistencyheuristicsrelyonthequalityofyour

modelsoftheproductanditscontext.


19/46

Consistencyheuristicsrelyonthequalityof

yourmodelsoftheproductanditscontext.

AnoracledoesnttellyouthatthereISaproblem.

Anoracletellsyouthatyoumightbeseeingaproblem.

Relysolelyondocumented,anticipatedsourcesoforacles,

andyourtestingwilllikelybeslowerandweaker.

Trainyourmindtorecognizepatternsoforacles

andyourtestingwilllikelybefaster

andyourabilitytospotproblemswillbesharper.

AllOraclesAreHeuristic

Anoraclecanalertyoutoapossibleproblem,

butanoraclecannottellyouthatthereisnoproblem.

A person whose opinion matters.

An opinion held by a person who matters.

A disagreement among people who matter.

A reference document with useful information.

A known good example output.

A known bad example output.

A process or tool by which the output is checked.

A process or tool that helps a tester identify patterns.

A feeling like confusion or annoyance.

A desirable consistency between related things.

GeneralExamplesofOraclesthings that suggest problem or no problem

34

People

Mechanisms

Feelings

Principles


20/46

Tacit Explicit

Other

People

Tester

Your

Feelings

Mental Models

Shared Artifacts

(specs, tools, etc.)

Stakeholders

Feelings

Mental Models

Inference

Observable

Consistencies

Reference

onference

Experience

Oracles from the Inside Out

Oracle

Cost

and

Value Someoraclesaremoreauthoritative

butmoreresponsivetochange

Someoraclesaremoreconsistent butmaybenotuptodate

Someoraclesaremoreimmediate butlessreliable

Someoraclesaremoreprecise buttheprecisionmaybemisleading

Someoraclesaremoreaccurate

butless

precise

Someoraclesaremoreavailable butlessauthoritative

Someoraclesareeasiertointerpret butmorenarrowlyfocused


21/46

FeelingsAsHeuristicTriggersForOracles

Anemotional

reaction

is

atrigger

to

attention

andlearning

Withoutemotion,wedontreasonwell

SeeDamasio,TheFeelingofWhatHappens

Whenyoufindyourselfmildlyconcernedabout

something,someoneelsecouldbevery

concernedaboutit

Observeemotionstohelpovercomeyour

biasesand

to

evaluate

significance

An emotion is a signal; consider looking into it

38

All

Oracles

Are

Heuristic Weoftendonothaveoraclesthatestablishadefinitecorrectorincorrect

result,inadvance.Oraclesmayrevealthemselvestousonthefly,orlater.

Thatswhyweuseabductive inference.

Nosingleoraclecantelluswhetheraprogram(orafeature)isworking

correctlyatalltimesandinallcircumstances.

Thatswhyweuseavarietyoforacles.

Anyprogramthatlookslikeitsworking,toyou,mayinfactbefailingin

somewaythathappenstofoolallofyouroracles.Thatswhyweproceed

withhumilityandcriticalthinking.

Wenever

know

when

atest

is

finished.

Thatswhywetrytomaintainuncertaintywheneveryoneelseonthe

projectissure.

You(thetester)cantknowthedeeptruthaboutanyresult.

Thatswhywereportwhateverseemslikelytobeabug.


22/462

39

OraclesareNotPerfect

AndTestersareNotJudges

Youdontneedtoknowforsureifsomethingisabug;

itsnotyourjobtodecideifsomethingisabug;itsyour

jobtodecideifitsworthreporting.

YoudoneedtoformajustifiedbeliefthatitMIGHTbe

athreattoproductvalueintheopinionofsomeone

whomatters.

Andyoumustbeabletosaywhyyouthinkso;youmust

beabletocitegoodoraclesoryouwilllosecredibility.

MIPing VS.BlackFlagging

40

CopingWith

Difficult

Oracle

Problems

IgnoretheProblem Asksowhat?Maybethevalueoftheinformationdoesntjustifythecost.

SimplifytheProblem Askfortestability.Itusuallydoesnthappenbyaccident. Builtinoracle.Internalerrordetectionandhandling. Lowerthestandards. Youmaybeusinganunreasonablestandardofcorrectness.

ShifttheProblem Paralleltesting. Comparewithanotherinstanceofacomparablealgorithm. Liveoracle.Findanexpertwhocantelliftheoutputiscorrect. Reversethefunction. (e.g.2x2=4,then4/2=2)

DivideandConquertheProblem Spotcheck.Performadetailedinspectionononeinstanceoutofasetofoutputs. Blinktest.Compareorreviewoverwhelmingbatchesofdataforpatternsthatstand

out.

Easyinput. Useinputforwhichtheoutputiseasytoanalyze. Easyoutput. Someoutputmaybeobviouslywrong,regardlessofinput. Unittestfirst.Learnaboutthepiecesthatmakethewhole. Testincrementally.Learnabouttheproductbytestingoveraperiodoftime.


23/462

41

EasyInput

FixedMarkers.

Use

distinctive

fixed

input

patterns

that

are

easy

to

spotintheoutput.

StatisticalMarkers.Usepopulationsofdatathathavedistinguishablestatisticalproperties.

SelfReferentialData.Usedatathatembedsmetadataaboutitself.(e.g.counterstrings)

EasyInputRegions.Forspecificinputs,thecorrectoutputmaybeeasytocalculate.

OutrageousValues.Forsomeinputs,weexpecterrorhandling.

IdempotentInput.Tryacasewheretheoutputwillbethesameastheinput.

Match.Dothesamethingtwiceandlookforamatch.

ProgressiveMismatch.Doprogressivelydifferingthingsovertimeandaccountforeachdifference.(codebreakingtechnique)

42

OraclesAreLinkedToThreats

ToQuality

Criteria

Anyinconsistencymayrepresentdiminishedvalue.

Manytestapproachesfocusoncapability(functionality)

andunderemphasizetheothercriteria.

Capability Scalability

Reliability Compatibility

Usability Performance

Charisma Installability

Security Development


24/462

43

OraclesAreLinkedToThreats

ToQualityCriteria

Anyinconsistencymayrepresentdiminishedvalue.

Manytestapproachesfocusoncapability(functionality)

andunderemphasizetheothercriteria.

Supportability

Testability

Maintainability

Portability

Localization

FocusingonPreparationandSkill

CanReduce

Documentation

Bloat

3.0TestProcedures

3.1Generaltestingprotocol. Inthetestdescriptionsthatfollow,thewordverify"isusedtohighlightspecificitems

thatmustbechecked.Inadditiontothoseitemsatestershall,atalltimes,bealertfor

anyunexplainedorerroneousbehavioroftheproduct.Thetestershallbearinmind

that,regardlessofanyspecificrequirementsforanyspecifictest,thereisthe

overarchinggeneralrequirementthattheproductshallnotposeanunacceptablerisk

ofharmtothepatient,includinganunacceptableriskusingreasonablyforeseeable

misuse.

Testpersonnelrequirements: Thetestershallbethoroughlyfamiliarwiththe

generator

and

workstation

FRS,

as

well

as

with

the

working

principles

of

the

devices

themselves.Thetestershallalsoknowtheworkingprinciplesofthepowertestjigandassociatedsoftware,includinghowtoconfigureandcalibrateitandhowtorecognizeif

itisnotworkingcorrectly.Thetestershallhavesufficientskillindataanalysisand

measurementtheorytomakesenseofstatisticaltestresults.Thetestershallbe

sufficientlyfamiliarwithtestdesigntocomplementthisprotocolwithexploratory

testing,intheeventthatanomaliesappearthatrequireinvestigation.Thetestershall

knowhowtokeeptestrecordstocredible,professionalstandard.


25/462

Remember

Forskilledtesters,

goodtestingisntjustabout

passvs.fail.

Forskilledtesters,

testingisabout

problemvs.noproblem.

Where

Do

We

Look

For

Problems?

Coverage is

how much of the

product has been tested.


26/462

Coverageishowmuchoftheproductwehavetested.

WhatISCoverage?

Itstheextenttowhichwehave

traveledoversomemapoftheproduct.

MODELS

Models Amodelisanidea,activity,orobject

suchasanideainyourmind,adiagram,alistofwords,aspreadsheet,

aperson,atoy,anequation,ademonstration,oraprogram

suchassomethingcomplexthatyouneedtoworkwithorstudy

Amapisamodelthathelpstonavigateacrossaterrain.

2+2=4isamodelforaddingtwoapplestoabasketthatalreadyhastwoapples.

Atmosphericmodelshelppredictwherehurricaneswillgo.

Afashionmodelhelpsunderstandhowclothingwouldlookonactualhumans.

Yourbeliefsaboutwhatyoutestareamodelofwhatyoutest.

thatheuristicallyrepresents(literally,

represents)anotheridea,activity,orobject

wherebyunderstandingsomethingaboutthe

modelmayhelpyoutounderstandormanipulate

thething

that

it

represents.


27/462

Thereareasmanykindsoftestcoverageasthereare

waystomodelthesystem.

Intentionally OR Incidentally

OneWaytoModelCoverage:

ProductElements

(with

Quality

Criteria)

Capability

Reliability

Usability

Charisma

Security

Scalability

Compatibility

Performance

Installability

Supportability

Testability

Maintainability

Structure

Function

Data

Interfaces

Platform

Operations

Time


28/462

51

Totestaverysimpleproductmeticulously,

partofacomplexproductmeticulously,

ortomaximizetestintegrity

1. Startthetestfromaknown(clean)state.

2. Prefersimple,deterministicactions.

3. Traceteststepstoaspecifiedmodel.

4. Followestablishedandconsistentlabprocedures.

5. Makespecificpredictions,observationsandrecords.

6. Makeiteasyto reproduce(automationmayhelp).

GeneralFocusingHeuristics

usetestfirstapproachorunittestingforbettercode

coverage

workfrompreparedtestcoverageoutlinesandrisklists

usediagrams,statemodels,andthelike,andcoverthem

applyspecifictesttechniquestoaddressparticularcoverage

areas

makecarefulobservationsandmatchtoexpectations

Todothismorerapidly,makepreparationandartifactsfastandfrugal:

leverageexistingmaterialsandavoidrepeatingyourself.

Emphasizedoing;relaxplanning. Youllmakediscoveriesalongtheway!


29/462

53

Tofindunexpected problems,

elusive problemsthatoccurinsustainedfielduse,

ormoreproblemsquicklyinacomplexproduct

1. Start from different states (not necessarily clean).

2. Prefer complex, challenging actions.

3. Generate tests from a variety of models.

4. Question your lab procedures and tools.

5. Try to see everything with open expectations.6. Make the test hard to pass, instead of easy to reproduce.

Thats a

PowerPoint

bug!

GeneralDefocusingHeuristics

diversifyyourmodels;intentionalcoverageinoneareacanlead

tounintentionalcoverageinotherareasthisisaGoodThing

diversifyyourtesttechniques

bealerttoproblemsotherthantheonesthatyoureactively

lookingfor

welcomeandembraceproductivedistraction

dosometestingthatisnotorientedtowardsaspecificrisk

usehigh

volume,

randomized

automated

tests


30/46


31/462

ExtentofCoverage

Smokeand

sanity

Canthisthingevenbetestedatall?

Common,core,andcritical

Canthisthingdothethingsitmustdo?

Doesithandlehappypathsandregularinput?

Canitwork?

Complex,harsh,extremeandexceptional

Willthisthinghandlechallengingtests,complexdataflows,

andmalformedinput,etc.?

Willitwork?

HowMightWeOrganize,

Record,and

Report

Coverage?

automatedtools(e.g.profilers,coveragetools)

annotateddiagramsandmindmaps

coveragematrices

bugtaxonomies

MichaelHuntersYouAreNotDoneYetlist

JamesBachsHeuristicTestStrategyModel

describedatwww.satisfice.com

articlesaboutitatwww.developsense.com

MikeKellysMCOASTERmodel

productcoverage

outlines

and

risk

lists

sessionbasedtestmanagement

http://www.satisfice.com/sbtm

Seethreearticleshere:

http://www.developsense.com/publications.html#coverage


32/46


33/463

VisualizingTestProgress



34/463


See A Sticky Situation , Better Software, February 2012

What

IS

Exploratory

Testing?

Simultaneous test design, test

execution, and learning.

James Bach, 1995

But maybe it would be a good idea to underscore

why thats important


35/46


36/463

WhyExploratoryApproaches?

Systemsare

too

complexfor

individualsto

comprehendand

describe

Productsevolve

rapidlyinways

thatcannotbe

anticipated

Inthefuture,developerswilllikelydomoreverificationandvalidationatthe

unitlevelthantheyhavedonebefore.

Testersmustexplore,discover,investigate,andlearnaboutthesystem.

Why

Exploratory

Approaches? Developersareusingtoolsandframeworksthat

makeprogrammingmoreproductive,butthat

maymanifestmoreemergentbehaviour.

Developersareincreasinglyadoptingunittesting

andtestdrivendevelopment.

Thetraditionalfocusisonverification,validation,

andconfirmation.

Thenewfocusmustbeonexploration,discovery,

investigation,andlearning.


37/463

WhyExploratoryApproaches?

Wedont

have

time

to

waste

preparingwastefullyelaboratewrittenplans

forcomplexproducts

builtfrommanyparts

andinteractingwithmanysystems

(manyofwhichwedontcontrol

orevenunderstand)

whereeverything

is

changing

over

time

andtheressomuchlearningtobedone

andtheresult,nottheplan,isparamount.

QuestionsAbout

Scriptsarrows and cycles

Where do scripts

come from?

What happens when the

unexpected happens

during a script?

What do we do

with what we

learn?Will everyone fol low the same

script the same way?

(task performing)


38/463

QuestionsAboutExplorationarrows and cycles

(value seeking)

Where does

exploration

come from?

What happens when

the unexpected

happens during

exploration?

What do we do

with what we

learn?

Will everyone

explore the same way?

Explorationis

Not

Just

Actionarrows and cycles


39/463

Youcanputthemtogether!arrows and cycles

Youcanputthemtogether!arrows and cycles


40/463

WhatExploratoryTestingIsNot

Touring http://www.developsense.com/blog/2011/12/whatexploratorytestingisnotpart1touring/

AfterEverythingElseTesting http://www.developsense.com/blog/2011/12/whatexploratorytestingisnotpart2after

everythingelsetesting/

ToolFreeTesting http://www.developsense.com/blog/2011/12/whatexploratorytestingisnotpart3tool

freetesting/

QuickTests http://www.developsense.com/blog/2011/12/whatexploratorytestingisnotpart4quick

tests/

UndocumentedTesting http://www.developsense.com/blog/2011/12/whatexploratorytestingisnotpart5

undocumented

testing/ ExperiencedBasedTesting

http://www.satisfice.com/blog/archives/664

definedbyanyspecificexampleofexploratorytesting http://www.satisfice.com/blog/archives/678

Exploratory

Testing

ISNOTrandomtesting(orsloppy,

orslapdashtesting)

ISNOTunstructuredtesting

ISNOTprocedurallystructured

ISNOTunteachable

ISNOTunmanageable

ISNOTscripted

ISNOTatechnique

ISadhoc,inthedictionarysense,

tothepurpose

ISstructuredandrigorous

IScognitivelystructured

IShighlyteachable

IShighlymanageable

ISchartered

ISanapproach

Thewaywepracticeandteachit,exploratorytesting


41/463

ContrastingApproaches

ScriptedTesting Isdirectedfromelsewhere

Isdeterminedinadvance

Isaboutconfirmation

Isaboutcontrollingtests

Emphasizespredictability

Emphasizesdecidability

Likemakingaspeech

Likeplayingfromascore

ExploratoryTesting Isdirectedfromwithin

Isdeterminedinthemoment

Isaboutinvestigation

Isaboutimprovingtestdesign

Emphasizesadaptability

Emphasizeslearning

Likehavingaconversation

Likeplayinginajamsession

Exploratory

Testing

IS

Structured Exploratorytesting,asweteachit,isastructuredprocessconductedbya

skilledtester,orbylesserskilledtestersorusersworkingundersupervision.

ThestructureofETcomesfrommanysources: Testdesignheuristics

Chartering

Timeboxing

Perceivedproductrisks

Thenatureofspecifictests

Thestructureoftheproductbeingtested

Theprocess

of

learning

the

product

Developmentactivities

Constraintsandresourcesaffordedbytheproject

Theskills,talents,andinterestsofthetester

Theoverallmissionoftesting

Inotherwords,

itsnotrandom,

butsystematic.

Notprocedurally

structured,but

cognitivelystructured.


42/464

79

Inexcellentexploratorytesting,onestructuretendsto

dominatealltheothers:

Exploratorytestersconstructacompellingstoryof

their

testing.

It

is

this

story

that

givesETabackbone.

ExploratoryTestingISStructured

80

Totestistocompose,edit,narrate,andjustify

THREEstories.

A story about the status of the PRODUCT

about how it failed, and how it might fail...

in ways that matter to your various clients.

A story about HOW YOU TESTED it

how you configured, operated and observed it

about what you havent tested, yet

and wont test, at all

A story about how GOOD that testing was

what the risks and costs of testing are

what made testing harder or slower

how testable (or not) the product is

what you need and what

you recommend.


43/464

81

Whatdoestakingadvantageofresourcesmean?

Mission Theproblemweareheretosolveforourcustomer.

Information Informationabouttheproductorprojectthatisneededfortesting.

Developerrelations Howyougetalongwiththeprogrammers.

Team Anyonewhowillperformorsupporttesting.

Equipment&tools Hardware,software,ordocumentsrequiredtoadministertesting.

Schedule Thesequence,duration,andsynchronizationofprojectevents.

TestItems Theproducttobetested.

Deliverables Theobservableproductsofthetestproject.

82

Waystotest?

GeneralTest

Techniques

Functiontesting

Domaintesting

Stresstesting

Flowtesting

Scenariotesting

Claimstesting

Usertesting

Risktesting

Automaticchecking


44/464

83

HappyPath

TourtheProduct

SampleData

Variables

Files

Complexity

Menus&Windows

Keyboard&Mouse

Aquicktestisacheaptestthathassomevalue

butrequireslittlepreparation,knowledge,

ortimetoperform.

Interruptions

Undermining

Adjustments

DogPiling

ContinuousUse

FeatureInteractions

ClickonHelp

CostasaSimplifyingFactor

Tryquicktestsaswellascarefultests

84

InputConstraintAttack

ClickFrenzy

ShoeTest

BlinkTest

ErrorMessageHangover

Aquicktestisacheaptestthathassomevalue

butrequireslittlepreparation,knowledge,

ortimetoperform.

ResourceStarvation

MultipleInstances

Crazy

Configs CheapTools

CostasaSimplifyingFactor

Tryquick

tests

as

well

as

careful

tests


45/464

TouringtheProduct:

MikeKellysFCCCUTSVIDS

Featuretour

Complexitytour

Claimstour

Configurationtour

Usertour

Testabilitytour

Scenariotour

Variabilitytour

Interoperabilitytour

Datatour

Structuretour


46/46

88

SummingUp:

Themesof

Rapid

Testing

Putthetester'smindatthecenteroftesting.

Learntodealwithcomplexity andambiguity.

Learntotellacompellingtestingstory.

Developtestingskillsthroughpractice,notjusttalk.

Useheuristicstoguideandstructureyourprocess.

Replacecheckforwithlookforproblemsin

Beaservicetotheprojectcommunity,notanobstacle.

Consider costvs.

valuein

all

your

testing

activity.

Diversifyyourteamandyourtactics.

Dynamicallymanagethefocusofyourwork.

Yourcontextshoulddriveyourchoices,bothofwhichevolveovertime.

rapid testing presentation

Documents