THE LOW-HANGING FRUIT OF TEST AUTOMATION - WHAT DO WE AUTOMATE FIRST?

RANDALL RICE, CTAL

WWW.RICECONSULTING.COM

March 19, 2014

2

BIO - RANDALL W. RICE •  Over 35 years experience in building and testing

information systems in a variety of industries and technical environments

•  ASTQB Certified Tester – Foundation level, Advanced level (Full)

•  Director, American Software Testing Qualification Board (ASTQB)

•  Chairperson, 1995 - 2000 QAI’s annual software testing conference

•  Co-author with William E. Perry, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems

•  Principal Consultant and Trainer, Rice Consulting Services, Inc.

•  www.riceconsulting.com & randallrice.blogspot.com

3

FEASIBILITY 1.  Is it easy (or even feasible) to automate the function/

component? •  Problematic issues

•  Custom objects •  Non-standard objects •  Objects with dynamic attributes (session IDs, etc.) •  Streaming, dynamic data (think stock tickers and other data

that is changing even while you are testing it) •  The tool doesn’t handle a certain type of object or component.

4

CONTROL 2. Who controls/develops the application?

•  Are any components (or even the entire application) vendor-developed?

•  How stable is the code? •  Are there a lot of hotfixes or maintenance releases •  If so, do you know when they occur?

•  Do you have prior knowledge of changes? •  Some vendors have problems in creating and distributing

release notes, even to their own testers.

5

OTHER QUESTIONS 3. Is the function easily repeatable?

•  Do you feel like a robot testing it? 4. Is the function easily performed?

•  Does it take a lot of set-up? •  Are there a lot of steps to perform?

•  Long scripts are hard to maintain? 5. How frequently is it used?

•  X times per day (or hour) 6. How long does the function take to test manually, with all

test conditions?

6

NUMBER OF TESTS Ti

me

per f

unct

iona

l exe

cutio

n

Number of times executed per test

Long term ROI (not so good)

7

TEST FREQUENCY Ti

me

per t

est s

uite

exe

cutio

n

Number of times tested per day, week, etc.

Good choice

Long term ROI (not so good)

Much depends on the length of the test. 5 test conditions vs. 500 obviously makes a big Difference!

8

REAL-LIFE EXAMPLE •  A financial services client had a simple

function that took about 5 minutes per condition to test.

•  However, it took the tester all day to test it due to the number of conditions.

•  We spent 3 days to automate it, with the result being 15 minutes to test all conditions!

9

HELPFUL TEST AUTOMATION METRICS (2) •  Effort saved by test automation (manual vs. automation) •  Effort needed to automate new tests •  Effort needed to analyze failed automated tests •  Effort to maintain automated tests •  Test coverage achieved by automated tests

10

EQUIVALENT MANUAL TEST EFFORT •  Also known as EMTE •  How much time would it take if the automated tests were

performed manually?

11

EMTE EXAMPLE •  Automated test takes 1 hour to run and are performed for

10 cycles. •  Manual version of test takes 4 days to run. •  EMTE = 40 days

•  4 days X 10 cycles

12

RETURN ON INVESTMENT (ROI)

Time

Effort Spent on

Automating Tests

Development Effort on

Production Code

Initial Effort

Increased Effort (Hump)

Reduced Effort

Source: xUnit Patterns by Gerald Meszaros

Saved Effort

13

RETURN ON INVESTMENT (2) - NOT SO GOOD

Time

Effort Spent on

Automating Tests

Development Effort on

Production Code

Initial Effort

Increased Effort (Hump)

Ongoing Effort

Source: xUnit Patterns by Gerald Meszaros

Saved Effort

14

MORE QUESTIONS 7. Does the function require:

•  Human judgment for outcomes? •  Creative types of testing

•  “Off the scripted path?” 8. Are you looking for new defects?

•  Test automation is more confirmatory than discovery in nature.

•  i.e., the defects found by automation are normally regression-type defects.

15

NEW OR MAINTENANCE? 9. Is this new software or maintenance?

•  Great debate: When should the automation be created – during the project or after?

•  Since functional changes ripple through automation, great care must be taken not to automate too early.

•  However, automation can be helpful during the final phases of a project to speed up regression testing.

•  Keep in mind it takes time to create test automation and the distraction can actually turn into a project risk!

16

WHO AND WHEN? 10. Who will create and use the automation?

•  Simple capture/playback (when it works) can be created by testers.

•  When problems are encountered, technical help at the developer level is often needed.

11. Which phase of testing?

•  Maintenance testing is ideal •  UAT is not good (you want humans testing) •  System testing is a possible win •  Unit test automation is ideal and essential

17

NEED FOR PRECISION 12. What is the need for precision?

•  Exact regression testing requires automation

•  One mistake can invalidate the entire test

•  Otherwise, you are doing “pseudo-regression” testing

•  What is the risk? •  If exact testing is needed, such as

medical devices, avionics, etc., automation can help mitigate risks

•  WARNING: test automation is “software testing software”, therefore the test scripts may have bugs

18

KNOWLEDGE AND STABILITY 13. Do you have a sufficient way to compare the test results

for pass/fail? •  You must have a source (non-human)

14. How stable is the function to be tested?

•  This is an irony •  Functional errors (which you want to find) will cause the

scripts to fail •  However, they will make test automation difficult because you

will need a way to deal with errors without stopping other tests in the suite.

19

20

CONTACT INFORMATION

Randall W. Rice, CTAL Rice Consulting Services, Inc. P.O. Box 892003 Oklahoma City, OK 73170 Ph: 405-691-8075 Fax: 405-691-1441 Web site: www.riceconsulting.com e-mail: rrice@riceconsulting.com