Evaluating UI and Mobile Automated Testing Solutions

5 K E Y C O N S I D E R A T I O N S F O R

www.testim.io | +1-415-535-6690

For more information contact us at info@testim.io

Or call +1-415-535-6690

R A J S U B R A M A N I A N

About the author

Raj Subramanian is a former developer who moved to testing to focus on his passion. Raj currently works as a

Developer Evangelist for Testim.io. He actively contributes to the testing community by speaking at

conferences, writing articles, blogging, making videos on his youtube channel and being directly involved in

various testing-related activities. He currently resides in Chicago and can be reached at raj@testim.io and on

twitter at @epsilon11. He actively blogs on www.testim.io and his website www.rajsubra.com . His videos on

testing, leadership and productivity can be found here – http://www.rajsubra.com/my-youtube-channel/

Why should you invest in test automation?

01 - Identify your needs and challenges

Situation 1: Never implemented a test automation

solution before

Situation 2: Underperforming test automation solutions

Situation 3: Already own a big legacy ALM suite

Define your business drivers

Identify your challenges

02 - What to look for in a vendor

03 - Cost considerations

05 - Prepare for launch

04 - Determine the specific features that you need

Automated testing tools are designed to support the software delivery process

in an iterative manner, continuously collecting feedback. The feedback will, in

turn, make developers more quality-conscious and encourage such practices

as static code analysis and code reviews, enabling CI/CD. The ultimate goal is

shipping high quality code, quickly to create exceptional user experiences!

A test automation platform is designed to speedup the testing process. The

benefits of a robust test automation platform includes:

Test automation allows teams to author and execute tests, reuse test data, and

report on status and quality.

Below is a 5-step process to help you identify the best test automation

platform that not only solves your challenges today but gives you flexibility to

change as your software development processes and tool stack evolve over

time.

S T E P 0 1

S T E P 0 2

S T E P 0 3

S T E P 0 4

S T E P 0 5

I D E N T I F Y Y O U R N E E D S A N D C H A L L E N G E S

W H A T T O L O O K F O R I N A V E N D O R

C O S T C O N S I D E R A T I O N S

D E T E R M I N E T H E S P E C I F I C F E A T U R E S T H A T Y O U N E E D

P R E P A R E F O R L A U N C H

There are several overarching needs that all organizations face regardless of

their unique challenges and budget restraints. Buyers of test automation tools

typically fall under three fundamental situations:

Many small and mid-sized organizations fall under this category. Most of them

have limited or no QA support staff. Their needs can often be adequately met

by on-demand, off-the-shelf test automation solutions that require minimal or

no customization. Such systems typically require only basic integration with

defect tracking tools and other solutions in the CI/CD stack.

Many mid-sized to large organizations fall under this category. Most have

experience using open source or new unproven tools that have limited

capabilities. They need a solution that can keep up with how they work and

integrate with new best of breed tools in their stack.

Rapidly growing mid-sized businesses or large companies with complex

development processes fall under this group. Such organizations often have

globally distributed teams, sophisticated internal IT teams as well as well-

established ALM suites which utilize a mix of methodologies and require

advanced data and systems integration.

Once you determine your particular situation, the next step involves defining

the exact business drivers that you are trying to resolve with a test automation

platform. These include, but are not limited to, needs such as:

The software industry is growing increasingly complex and applications are

becoming more sophisticated. Further, there's added pressure to get your

software to the market as soon as possible in a bid to stay ahead of the

competition. These escalating needs have been driving the need for automated

test solutions to help development teams to centralize, organize and prioritize

their testing processes.

It's therefore imperative to identify the challenges that your test automation

must address. You need to ask yourself questions like:

Once you have identified your needs and challenges, you are now ready to

start vetting potential vendors. On the surface, many test automation solutions

appear quite similar. A deeper look, however, reveals that different vendors

offer varying levels of performance, support and cost-effectiveness. You

should, therefore, look carefully under the hood to determine which test tools

are the best fit for your objectives.

Here are the top criteria when evaluating vendors:

It's quite tempting for organizations to only consider the license fee when

evaluating automated testing tools but fail to fully account for other ''hidden''

costs. The total cost of ownership (TCO) of any test automation platform is a

sum of the following costs:

The last step in the evaluation process is formal onboarding and becoming

completely operational. Will the vendor provide white-glove service to ensure

your success?

The purchase of software is a purchase of a relationship. You need to carefully

consider the following factors to make sure your up and running quickly to

realize the benefits of your investment:

At this stage you have already established your goals, challenges, costs and

vetted different vendors. It's now time to determine the exact features that you

require in the test solution. You can use the chart below to compare broad-

based features between different vendors:

How to use this table:

1. Read through all of the features/capabilities and then prioritize

them in order of importance

2. Then as you go through vendor demos and evaluations score

each feature 1-3

Now total up the scores. The vendor that has the lowest score has the the best

set of capabilities based on your prioritized needs.

The organization should determine their expected return on investment (ROI)

after setting realistic expectations. Many test automation solutions require time

before they can begin to demonstrate their true value. It's also equally

important for the business to develop specific metrics (both qualitative and

quantitative) for measuring the effectiveness of the test solution. These could

be aspects such as improvements in software quality or increased test

coverage.

Growing need for integration with newer development tools

Requires migration of some tests

Requires more functionality and less maintenance of their tests

Need to more reporting and visibility into status and quality

Need grid/cross browser testing capabilities

Lower your software development/deployment costs

Speed-up your time to market

Achieve better integration and alignment with your development tool stack

Reduce time to resolution

Improve your overall test coverage and traceability

Improve the quality of your software

Foster communication and collaboration between your teams

Fit to development methodology (Separate QA team, QA inside the dev

team, Dev authoring tests)

License fee

Platform customization and integration to tool-set

Ramp-up time to get fully productive

Integration, training and consulting

Data conversion and migration

Longevity - how long has the vendor been in business? Does the vendor

have a good track record?

Customer base - does the vendor offer industry-specific or catchall

solutions? Does the vendor have good customer reviews? You can verify these

through detailed references, case studies, testimonials and online reviews.

Credibility - does the vendor have a credible and well-established brand?

Support - does the vendor provide regular software updates? What other

kind of customer success and support does the vendor provide?

Training services - does the vendor provide any kind of training to help

customers learn how to navigate their way around the platform?

Do you have diverse teams across the globe leading to impaired visibility and

collaboration?

Are these teams expected to synchronize multiple releases on tight schedules?

Has your testing environment become too jumbled up with many operating

systems and different configurations?

Is your current test automation solution evolving fast enough to

accommodate new systems and assets?

Are your teams spending more time on test maintenance than actual testing?

Implementation process - Does the vendor have a well documented

30/60/90 day onboarding process which can be tailed specifically to your

team's needs and timelines?

Migration - does the vendor handle data migration and if so, how much do

they charge for the service?

Seamless integrations - will the vendor help you integrate with your dev

stack and assist with testing to ensure everything is working correctly?

Deployment - will the product be deployed exclusively in the cloud or does

the vendor provide an on-premise solution?

Training/support - will the vendor provide training and ongoing support to

ensure team product adoption? What is the vendor's support SLAs?

Backup/disaster recovery - what tools and mechanisms does the solution

employ for backup and recovery tasks?

Scalability - how easily can the solution be scaled to handle both your near-

and long-term workload requirements?

Usability

Maintenance

Customer

Support

Enterprise

Grade

Ramp up

time

Skiill set fit &

dev cycle fit

Pipeline

Integrations

Stability &

confidence

Data

Migration

Deployment

Multi-platform

support for

web & mobile

Reporting &

traceability

How easy is it to write automated tests? Your developers, testers, BAs and anyone who knows how to use your product should be able to do UAT. The tool should make it easy to create and execute tests.

How much time is spent maintaining flaky tests? The solution should make it easy to determine exactly why the test failed without having to worry about the stability of the test itself.

Does the vendor offer adequate customer support and training? What are their SLAs? How do they measure your success? What does their product roadmap look like and is it aligned with your future?

Enterprise-grade test automation tools are able to differentiate themselves by providing superior security features, easy data portability, superior audit trails including full versioning of test cases and and excellent disaster recovery capability.

How long does it take for users to become operational and start realizing the benefits of why you purchased the solution? Ask to speak with customers to get a real idea of their on-boarding experience.

Does the tool require additional skills in order to use it like coding or scripting?

How well does the tool integrate with your pipeline and release process? Does it integrate with Dev tools such as Git, collaboration tools like Slack, bug tracking tools like JIRA and CI tools like Jenkins and test management solutions?

Do the tests break? How well do the tests handle changes in code push? Are there any Issues with timeouts? The test automation solution should ideally have an easily accessible repository of error logs, network logs, screenshots, DOM of failed tests and application snapshots.

Does the tool allow you to reuse tests from previous solutions?

How easy is it to install and configure the test environment? Can the tool be used in the cloud via any web-browser?

What’s the process for running the tests continuously and on different browsers? Does the solution support different OS (e.g. iOS, Android, Windows Mobile) Browsers (e.g Chrome, Firefox, Internet Explorer) and application frameworks(e.g. .NET and HTML5) for software you would like to interface with for combined issue reporting? How much effort is required for cross browser testing?

The tool should not merely offer statistics but also visibility into the possible meaning of the results as well as enriched information (e.g logs, DOM, screenshots) to help trace why a test failed. Enriched information and traceability translates into shorter time to resolution and reduced cost.

FEATURE CAPABILITY VENDOR 1 VENDOR 2 PRIORITIZE