Top Ten Challenges of

Test Automation

Featuring

Bob Crews, Checkpoint

Technologies

presents the

www.qaiQUEST.org/2013

Join IT, QA, and Testing professionals from

across North America this spring!

One- and Two-Day

Classes and Workshops

Three-Day Leading Edge

Conference

World Class EXPO with Top

Vendors and Providers

Save up to $300 with Early

Registration Discounts

through January 31!

Addressing Mobile App

Testing Challenges

Featuring

Lee Barnes, Utopia Solutions

presents the

www.qaiQUEST.org/2013

QAI Global Institute specializes in

providing individuals and

organizations with expert-led training

solutions, internationally-recognized

software certifications, leading-edge

professional conferences, and

engaging networking opportunities.

It is this unique blend of services that

makes QAI the service provider of

choice for knowledge-intensive

organizations around the world.

Discover how QAI Global Institute can

help you and your organization to

achieve impressive results!

Expert-led Training

Solutions

Internationally-recognized

Software Certifications

Leading-edge Professional

Conferences

Engaging Networking

Opportunities

• Select “Use Mic & Speakers” or dial-in

by phone to listen to the presentation.

• Troubleshoot any technical problems

by visiting

http://support.citrixonline.com.

• Submit questions using the

“Questions” box in your GoToWebinar

application window.

• Webinar recording and slides will be

available on the QUEST 2013 website

24 hours after the presentation ends.

presents the

www.qaiQUEST.org/2013

• Training & Public Courses

• Software Certification & Exam Prep

• QUEST, TesTrek, & AgileTrek Conferences

• Free Webinars, White Papers, & Publications

• Local Chapter Announcements

presents the

www.qaiQUEST.org/2013

Top Ten Challenges of

Test Automation

Featuring

Bob Crews, Checkpoint

Technologies

presents the

www.qaiQUEST.org/2013

1/15/2013

1

TOP TEN CHALLENGES

WITH TEST AUTOMATION

Bob Crews

Checkpoint Technologies

Speaker Information

• Bob Crews

• President of Checkpoint Technologies, Inc.

• Experts in Quality Assurance and software testing

• President of TBQAA (Tampa Bay QAI Chapter)

• Co-leader of Florida Vivit Chapter

• HP’s Independent Software User Group

• Contact Information

• Email: bcrews@checkpointech.com

• Office: (813) 818-8324 ext. 101

• Direct: (813) 493-3678

2

1/15/2013

2

What we will cover…

• Top 10 challenges with Impact and Solutions • Note: Challenges are NOT ranked. The numbering is merely

sequential

• Impacts - Potential risks if challenges are not

overcome

• Solutions – Tips to overcome challenges

• Focus will be on automated functional testing

• Criteria of an Automation Assessment

What we will not cover…

• Technical challenges specific to specific

automated tools. Each solution has it’s own

limitations.

• Challenges pertaining to a programming

language

• Issues with hardware, security, networking, etc.

• Performance nor application security testing

1/15/2013

3

Challenge 1: Utilizing the full benefits of data-

driving automated tests

Description: Storing data that is relevant to a test

case in an external file. A set of data makes up a

“record” and, typically, a test is executed once for

each record within the file.

The challenge for most organizations

is getting the team to take full advantage

of the benefits of data-driving.

Impacts

Failure to realize maximum ROI

Greater challenge testing multiple conditions

Increased maintenance time

1/15/2013

4

Solutions

Train and educate your team

Implement guidelines & standards

Enforce guidelines & standards (peer reviews)

Implement strategy for test data creation and management

What Might You Validate?

Error message

Tabs

Date

Object state

Calculated value

Properties of list

Existence of

image

Values in edit

fields

1/15/2013

5

Challenge 2: Effectively implementing an

efficient framework

Description: In this challenge focus on “effectively

implementing” an “efficient framework”.

Organizations lean towards two extremes:

• No framework at all

• An overly complex framework that never seems

to be ready

Impacts

• No framework

• More difficult to create system test strategy

• Reduces ROI by not creating and taking advantage of reusable,

modular components

• Increases learning curve by not creating standards & guidelines

• Overly complex framework

• Increases learning curve

• Slows down use of automation

• Decreases ROI

• Insisting framework complete before executing

• Decreases ROI

• Delays realization of automation benefits

1/15/2013

6

Solutions

• Let the framework evolve

1. Start the creation of “Automation Standards & Guidelines”

2. Perform analysis and select tests that are best candidates for

automation

3. Automate those tests (and execute)

4. As automation occurs take note of common processes

5. Create modular, reusable common processes

6. Integrate common, automated processes with other automated

tests

7. Continue building framework – it’s an on-going process

• Monitor the creation and evolution of the framework

Challenge 3: Committing to the investment

required to realize an ROI

Description: Once implemented, it takes

effort, time, and money to realize (and

continue to realize) a return-on-investment

with automation. Such a commitment is

needed:

- At the company (managerial) level

- At the user level - -

1/15/2013

7

Impacts

• No commitment

• Education on the tool, and thus utilization, will suffer

• ROI not realized or, worse, a loss will be realized

• Automation tool will become “shelfware”

• Highest quality (of applications and systems) will not be

realized

• Frustration will occur

Solutions

• Understand the commitment needed early:

• Time

• Money

• Resources

• Be supportive of team and automation initiative

• Ensure team remains open-minded

• Provide training, mentoring, and on-going education

• Set guidelines and standards for test automation

• Involve team in decisions

1/15/2013

8

Challenge 4: Knowing what processes to

automate and what to leave manual

Description: It is important to know, and

heed, the criteria as to when to automate a

test and when to leave manual.

Impacts

• Automating tests that should not be automated

• NOT automating tests that should be automated

• Don’t fall victim to:

• Red Durango Mentality (Automate everything!)

• Analysis Paralysis (Automate nothing!)

• Framework Fumble (Slow to automate anything)

1/15/2013

9

Solutions

What to Automate

• Tests that validate

requirements that will exist

for multiple versions to come

• Tedious, error-prone tests

• Manually intensive

• Tests that validate numerous

conditions with same actions

but different data values (ex:

field edits)

What to Leave Manual

• Tests that validate

requirements soon to be

obsolete

• Exploratory tests

• Tests for which expected

results are unknown

• Tests for ad-hoc and

emergency processes

• Usability testing (subjective)

Tip: Set strict guidelines and conduct peer reviews to check for compliance.

Challenge 5: Planning and implementing an

automation strategy

Description: Automation needs to be planned and

well thought out. It presents a unique set of

challenges and requires a different approach. It

should NOT be treated as manual testing.

Automated testing:

• Requires different skills

• Has different considerations

• Provides different benefits

• Should be a strategy within a strategy

1/15/2013

10

Impacts

• Automation may grow out-of-control and become the

primary testing focus

• Automation may be “set aside” and not be fully utilized

• Approached with same strategy as manual testing

• Decreases benefit of re-usability

• Redundant tests

• Excessive number of tests

• Increase in development/maintenance time

Solutions

• Understand and accept that automated testing has a strategy very different from manual testing

• Do NOT believe that automated testing is realistic for 100% of your testing efforts

• Consider automation to be a PART of your overall testing strategy

• Early in planning stage have Automation Assessment performed

1/15/2013

11

Automation Assessment Criteria

• Environment Overview

• Automated Tool Experience and Training

Evaluation

• Test Automation Process Documentation

Evaluation

• Test Automation Environment Evaluation

• Automated Test Design Process Evaluation

• Automated Test Execution Process and Metrics

Evaluation

Challenge 6: Developing a versatile, skillful QA

team with automation skills

Description: In order to successfully implement

and utilize automation an organization must have,

or develop, a skillful automation team.

Skills must include:

• Utilization of the chosen automated testing

solution

• Implementation of an automation strategy

1/15/2013

12

Impacts

• Decrease in realization of ROI

• Not realizing high level of application quality that can be

obtained with better testing

• Automation tool becomes “shelfware”

• Longer automation implementation period

Solutions

• Assess the automation skill level of your existing team

early in the process

• Identify those team members that are, or can quickly

evolve, into automation leads and/or administrators

• Know your options in regards to:

• Improving the skills of your existing team

• Contracting experienced testers

• Monitor the skill level of your team to ensure that their

skills and knowledge are improving

1/15/2013

13

Basic Skill Assessment - Exercise

• Technical:

• Do you consider yourself to be technical?

• What is a “loop”?

• Name one conditional statement

• What is the difference between a “constant” and a “variable”?

• What is an “array”?

• Methodology:

• Can you describe an automation strategy?

• Describe a test that should NOT be automated

• Is it better to define expected results within a script or a data-table?

• What data in a script would you not parameterize?

• What are the benefits of modularization?

Level of Skill 10

9

8

7

6

5

4

3

2

1

1 2 3 4 5 6 7 8 9 10

T

E

C

H

N

I

C

A

L

S

K

I

L

L

AUTOMATION METHODOLOGY KNOWLEDGE

Automation

Lead

Automation

Technical

Advisor

Automation

Tester

Automation

Administrator

1/15/2013

14

Challenge 7: Executing automated tests

unattended and remotely

Description: Most automated solutions offer

capabilities to schedule tests to run at a specific

date/time on remote network machines

Many users fail to utilize this feature.

Why?

• Lack of confidence in automation

• Lack of secure environment

• Conflict with other processes (i.e., backups)

Impacts

• Won’t realize as significant an ROI with automation

• Will have greater challenges testing on different machines with different setups (software, OS, etc.)

• Can’t create automated tests on local machine while running tests

Solutions

• Set up a secure, dedicated testing environment with network access

• Always own at least two licenses of automated tool

• Ensure team has enough knowledge and skill (with tool) to instill confidence in automation

1/15/2013

15

Challenge 8: Calculation of ROI

Description: How can management

calculate the return-on-investment to justify

costs of automation?

Direct ROI

• Savings in man-hours

• Time for tester to perform other tasks

• Greater coverage

Indirect ROI

• Risk avoidance

• Decrease in errors

• Increase in defects reported

Impacts

• If ROI is not calculated or if it’s incorrectly

calculated

• Management may abandon automation or

• Too much might be automated

• Incorrect decisions may be made

1/15/2013

16

Solutions

• Report and track relevant data prior to, during, and after

automation effort for manual vs. automated testing

• Cost of resources

• Time to execute manually and automated

• Time to develop and maintain automated tests

• Number of conditions validated

• Number of defects reported

• Track “Cost of Failure” data

Actual R.O.I. Example

Real case scenario: Fortune 500 Financial Firm

B.A. validates random selection of 60 (out of 600)

mortgage records. Each record with 56 fields.

◦ Analysis performed monthly (no end in sight)

◦ High calculation

◦ 18 – 24 hours to perform analysis

◦ Tedious and error prone

◦ Perfect candidate for automation

1/15/2013

17

Automation Process and R.O.I.

30 hours to automate

Direct R.O.I. ◦ 30 minutes to execute and 1 – 3 hours to analyze for total of 1.5 –

3.5 hours monthly. ◦ Can be scheduled to run remotely and unattended! (Therefore not

requiring B.A.’s time.) ◦ Validates – not 60 random records but ALL 600 records!!!

Indirect R.O.I. ◦ B.A. can perform other tasks. ◦ Decrease in errors. ◦ More records validated – greater coverage!

R.O.I. Calculated

• The manual process validated 60 random records at an estimated

cost of $8,640

• To manually validate 600 records would cost an estimated $86,400

($8,640 * 10)

• The automated process validates 600 records at an estimated cost of

$4,780

$86,400 (manual costs)

$4,780 (automation

costs)

$81, 620 (First Year

ROI)

1/15/2013

18

Challenge 9: Integration of automated tests to

create a full system test

Description: Tests need to be integrated to

validate the system. Tests, standalone,

should validate specific, granular

requirements. Integrated they should

validate the “system”.

Biggest challenge typically seems to be test

data management.

Impacts

• System requirements fail to get validated for those

processes up and/or down stream

• Fail to maximize ROI (return-on-investment)

1/15/2013

19

Example

Enter Arrested Person

• Arrested person data entered into NCIC system

Undercover Agent Monitor

• If undercover agent arrested triggers warning to FBI

Undercover Agent Arrested Notification

• Display undercover agent is arrested notification

Background process

Solutions

• Perform integration tests by writing data created in “initial” (or prior) process to data table and using that data as input in subsequent process

• Conduct peer reviews

Enter Arrested Person

Undercover Agent Monitor

Undercover Agent Arrested

Notification

Last Name First Name SSN Offense

Malone Mugsy 345-67-9999 Armed

Robbery

Weiser Bud 654-88-7777 DUI

Output Input

1/15/2013

20

Challenge 10: Tendency to focus on scripting

(programming) features

Description: Focus on programming and fail

to take advantage of tool features.

• Recording features, packaged functions,

etc.

• People with strong development

experience tend to be most guilty

Impacts

• Increased automation

script development

and maintenance time

• Scripts become more

difficult to maintain

• More reliant on the

person (programmer)

that created automated

script(s)

Solutions

• Train all users on

automation and

automation tool

• Require all users to

learn features of

automation tool

• Conduct peer reviews

1/15/2013

21

Top Ten Challenges in Review

1. Utilizing the full benefits of data-driving automated tests

2. Effectively implementing an efficient framework

3. Committing to the investment required to realize an ROI

4. Knowing what processes to automate and what to leave

manual

5. Planning and implementing an automation strategy

6. Developing a versatile, skillful QA team with automation

skills

7. Executing automated tests unattended and remotely

8. Calculation of ROI

9. Integration of automated tests to create a full system test

10. Tendency to focus on scripting (programming) features

QUESTIONS

1/15/2013

22

QAI QUEST2013 Conference & Expo!

Stop and say “Hello”

• Location: Chicago, IL

• Hotel: Hyatt Regency McCormick Place

• Dates: April 15th – April 19th

• Website: www.qaiquest.org/2013/

Bob Crews

Checkpoint Technologies, Inc.

Email: bcrews@checkpointech.com

Office: (813) 818-8324 Ext. 101

Direct: (813) 493-3678

Thank you!

44