Exploratory  Tes.ng  in  the  Enterprise  

CQAA  June  22,  2011  

Partner,  DeveloperTown   www.DeveloperTown.com    

Past  President,  Associa.on  for  SoFware  Tes.ng   www.Associa.onForSoFwareTes.ng.org    

Ar.cles  and  Blogs   www.MichaelDKelly.com      

Author  of  the  chapter  on  “Session  Based  Test  Management”  in  upcoming  book  How  to  Reduce  the  Cost  of  So0ware  Tes4ng    

                   

A  STORY  

•  3  testers  instead  of  15  •  66%  (or  $200,000)  cheaper  than  budgeted    •  two-‐week  shorter  window  for  tes.ng  •  delivered  on  .me  •  no  serious  defects  in  produc.on    

HOW?  

“Simultaneous  learning,  test  design,  and  test  execu.on.”  

                                   -‐  James  Bach    

pure scripted freestyle exploratory

charters vague scripts

fragmentary test cases (scenarios) roles

-‐image  from  Jon  Bach  

 the  tester,  the  way  they  work,    

and  their  ability  to  work  in  the  future,  is    as  important  as    

the  product  being  tested  

   What  makes  them  “exploratory?”  

Self  Management    

     

Developing  Ideas  

Examining  the  Product  

       There’s  always  resistance  

 A  desire  for  visible,    track-‐able,  progress  

       It’s  difficult  to  quan.ta.vely    measure  coverage  

There’s  extraordinary  pressure  for    “reuse”  and  “cost  control”  

           Session  Based  Test  Management  

•  charters  

•  charters  •  .me-‐boxed  

•  charters  •  .me-‐boxed  •  session  notes  

•  charters  •  .me-‐boxed  •  session  notes  •  debriefs  

•  charters  •  .me-‐boxed  •  session  notes  •  debriefs  •  team  priori.za.on  

•  charters  •  .me-‐boxed  •  session  notes  •  debriefs  •  team  priori.za.on  •  ad-‐hoc  test  documenta.on  

•  charters  •  .me-‐boxed  •  session  notes  •  debriefs  •  team  priori.za.on  •  ad-‐hoc  test  documenta.on  •  ad-‐hoc  test  automa.on  

•  charters  •  .me-‐boxed  •  session  notes  •  debriefs  •  team  priori.za.on  •  ad-‐hoc  test  documenta.on  •  ad-‐hoc  test  automa.on  •  dynamic  metrics  and  repor.ng  

       

     Focus  on  dealing  with  uncertainty  

             START  

   Begin  by  implemen.ng  a  specific  prac.ce    

Add  exploratory  tes.ng  days  to    the  end  of  exis.ng  test  cycles  

Carve  out  a  small  por.on  of  an  exis.ng  project  and  apply  exploratory  tes.ng  to  that  por.on  

Run  a  pilot  project  

Responding  to  concerns  

   the  Michael  Jordan  effect  

       

“We  don’t  do  agile.”    

   

               “What  about  regression  tes.ng?”  

“We  hand  our  test  scripts  off  for  test  automa.on…”  

 “How  do  you  provide  traceability?”  

“Your  percent  complete  keeps  changing!”  

 “Don’t  we  lose  the  ability  to  add  people  late  in  

the  project  if  we  get  sideways?”  

“Our  testers  aren’t  subject  mager  experts.”    

         

“We  can’t  afford  all  senior  testers.”    

   “How  do  you  make  charters  repeatable?”    

       

“What  about  es.ma.on?”    

 Ques.ons?  

Reading  Materials  

•  Bach,  James.  "Exploratory  Tes.ng  Explained."  First  published  as  a  chapter  in  The  Test  Prac..oner,  2002.  Available  online  at:  hgp://www.sa.sfice.com/ar.cles/et-‐ar.cle.pdf  

•  Bach,  James  and  Jon  Bach.  "Exploratory  Tes.ng  Dynamics."  Available  online  at:  hgp://www.sa.sfice.com/ar.cles/et-‐dynamics.pdf    

•  Bach,  Jon.  "Session  Based  Test  Management."  First  published  in  SoFware  Tes.ng  and  Quality  Engineering,  November  2000.  Available  online  at:  hgp://www.sa.sfice.com/ar.cles/sbtm.pdf  

•  Bach,  James.  "A  Low-‐Tech  Tes.ng  Dashboard."  Presented  at  STAREast,  1999.  Available  online  at:  hgp://www.sa.sfice.com/presenta.ons/dashboard.pdf  

•  Bach,  Jon.  "How  to  Manage  and  Measure  Exploratory  Tes.ng."  Quardev,  2006.  Available  online  at:  hgp://www.quardev.com/content/whitepapers/how_measure_exploratory_tes.ng.pdf  

•  Pegy,  Kenn.  "Reflec.ons  on  the  Use  of  Session-‐Based  Exploratory  Tes.ng  As  the  Primary  Test  Methodology  for  SoFware  in  an  Agile  Environment."  Presented  at  the  Indianapolis  Workshops  on  So0ware  Tes4ng,  April  2005.  Available  online  at:  hgp://www.indianapolisworkshops.com/docs/Reflec.ons_on_the_use_of_Session-‐Based_Exploratory_Tes.ng_in_an_Agile_Environment.doc  

•  Pegy,  Kenn.  "Transi.oning  from  Standard  V&V  to  Rapid  Tes.ng  Prac.ces  in  a  Chao.c  Project  Environment."  Presented  at  the  Conference  of  the  Associa4on  for  So0ware  Tes4ng,  2007.  Available  online  at:  hgp://www.associa.onforsoFwaretes.ng.org/documents/Kenn_Pegy_Transi.oning_from_Standard_V&V_to_Rapid_Tes.ng_Prac.ces.pdf  

•  Wood,  Bill  and  David  James.  "Applying  Session-‐Based  Tes.ng  to  Medical  SoFware."  Medical  Device  and  Diagnos.c  Industry  Magazine,  May  2003.  Available  online  at:  hgp://www.mddionline.com/ar.cle/applying-‐session-‐based-‐tes.ng-‐medical-‐soFware  

•  Kelly,  Michael.  “The  benefits  of  exploratory  tes.ng  in  agile  environments.”  SearchSoFwareQuality.com,  2009.  Available  online  at:  hgp://searchsoFwarequality.techtarget.com/.p/0,289483,sid92_gci1350741,00.html    

•  Kelly,  Michael.  “Using  session-‐based  test  management  for  exploratory  tes.ng.”  SearchSoFwareQuality.com,  2009.    Available  online  at:  hgp://searchsoFwarequality.techtarget.com/.p/0,289483,sid92_gci1352925_mem1,00.html    

•  Kelly,  Michael.  “Using  session-‐based  test  management  for  exploratory  tes.ng  coverage  problems.”  SearchSoFwareQuality.com,  2009.  Available  online  at:  hgp://searchsoFwarequality.techtarget.com/.p/0,289483,sid92_gci1355475_mem1,00.html    

•  Kelly,  Michael.  “Solving  problems  with  session-‐based  test  management.”  SearchSoFwareQuality.com,  2009.  Available  online  at:  hgp://searchsoFwarequality.techtarget.com/.p/0,289483,sid92_gci1355815_mem1,00.html