why invest in test automation?...2019/08/05 · evaluating ui and end-to-end automated testing...
TRANSCRIPT
Evaluating UI and End-to-End Automated Testing Solutions
5 K E Y C O N S I D E R AT I O N S F O R
www.testim.io | +1-415-535-6690
Why invest in test automation?
01 - Identify needs and challenges
Situation 1: No test automation solution
Situation 2: Underperforming test automation solution
Situation 3: Already own a big legacy ALM suite
Define your business drivers
Identify your challenges
02 - Consider vendor attributes
03 - Evaluate current costs and budget
05 - Analyze post-purchase needs
04 - Determine required features
Nearly every software-driven company is trying to innovate faster and
release code more frequently. Yet, delivering poor software quality can
lower customer satisfaction and hurt profitability. Software testing remains
a bottleneck in many organizations due to slow authoring and instable
tests. A robust test automation platform is designed to accelerate the
software testing process and ensure high-quality releases. The benefits
include:
Test automation allows teams to author and execute tests faster, easily
maintain tests, report bugs quickly and clearly show testing results.
Below is a 5-step process to help you identify the best test automation
platform that not only solves your challenges today but gives you flexibility
to change as your software development processes and tools evolve over
time.
S T E P 0 1
S T E P 0 2
S T E P 0 3
S T E P 0 4
S T E P 0 5
I D E N T I F Y N E E D S A N D C H A L L E N G E S
C O N S I D E R V E N D O R AT T R I B U T E S
E VA LUAT E C U R R E N T C O S T S A N D B U D G E T
D E T E R M I N E R E Q U I R E D F E AT U R E S
A N A LY Z E P O S T- P U R C H AS E N E E D S
Your test automation challenges and needs will vary depending on where
you are today. Most organizations will fall under one of three situations:
Organizations in this category tend to be small or midsized with limited or
no QA support staff. Their development processes and toolchains are
simple yet standardized with limited variability. To this point, they have met
their testing needs manually, though the cost and effort is starting to grow.
Such systems typically require only basic integrations with bug tracking
tools and CI solutions.
Often these organizations have tried an open source solution and found it
to be less automated than they had hoped, requiring special skills, long
test authoring times, or constant maintenance of tests. Others may have
tried a vendor tool that promised to solve the problems of open source
solutions, but didn't have the maturity or functionality to fully test their
applications.
Larger, often established organizations with complex development
processes fall under this group. These organizations have well-defined QA
processes and globally distributed teams. Standardized tools may exist,
but it's likely that there is a wide variety including legacy technologies that
require custom integrations.
Once you determine your particular situation, the next step involves
defining the exact business drivers that you are trying to resolve with a test
automation platform. These include, but are not limited to:
Software applications are becoming more sophisticated, leveraging open
source and third-party components that add functionality but also
increase complexity. Further, organizations are quickening release cycles to
stimulate innovation, learning, and growth. These trends place pressure on
QA teams to automate and streamline testing processes. Consider these
questions when evaluating test automation solutions:
Once you have identified your needs and challenges, you are ready to start
vetting potential vendors. On the surface, many test automation solutions
appear quite comparable, with vendors claiming similar features and
benefits. A deeper look, however, may reveal differences in product
maturity, fit with your tools and processes, customer support, and cost-
effectiveness.
Here are some key criteria to consider:
It's not uncommon for organizations to focus on the license fee when
evaluating automated testing tools and then fail to fully account for other
hidden costs. The total cost of ownership (TCO) of any test automation
platform includes the following costs:
The last step in the evaluation is understanding the onboarding process
and the steps to achieve operational effectiveness. Will the vendor provide
white-glove service to ensure your success?
The purchase of software is a start of a partnership. You need to carefully
consider the following factors to make sure you're up and running quickly
to realize the benefits of your investment:
At this stage you have already established your goals, challenges, costs
and vetted different vendors. It's now time to determine the exact features
that you require in the test solution. You can use the chart below to
compare broad-based features between different vendors:
How to use this table:
1. Read through all of the features/capabilities and then prioritize
them in order of importance.
2. Weight the importance of each feature to your organization using
percentages from 0% to 100%.
3. Score the vendors from 1 (worst) to 5 (best) for each feature/attribute.
Now calculate the weighted score (sum of weight X score for each line
item). The vendor with the highest weighted score has the best set of
capabilities that most closely match your priorities.
The organization should determine their expected return on investment
(ROI) after setting realistic expectations of benefits. Many test automation
solutions require time before they can begin to demonstrate their true
value. It's also equally important for the business to develop specific
metrics (both qualitative and quantitative) for measuring the effectiveness
of the test solution. These could be aspects such as improvements in
software quality or increased test coverage.
Accelerate time to market
Improve software quality
Integrate and align with your development tools
Reduce software development/deployment costs
Reduce time to identify and fix bugs
Increase test coverage and improve traceability
Foster better team communication and collaboration
Fit to development methodology (separate QA team, QA inside the dev
team, dev authoring tests)
License, maintenance, and support fees
Platform customization and integration to tool set
Ramp-up time to get fully productive
Data conversion and migration
Maturity - is their solution missing key features? Does the vendor provide
frequent feature updates? Is it old technology that makes it less usable
or inflexible?
Customer base - does the vendor support customers like you? Do they
have good online reviews? Can you live with reviewer's criticisms about
their solution?
Training services - does the vendor provide any kind of training to help
customers learn how to navigate their way around the platform?
Support - what kinds of technical support do they provide? Do they offer
dedicated customer success representatives to help you be successful?
Fit - does the solution integrate with the 3rd party tools your team uses?
Does the technology fit your team’s skills so they can work effectively?
Security - does the solution meet your security needs? Do they have any
3rd party verification such as SOC2 certification?
Is slow test authoring impeding your ability to effectively test new
functionality in a timely manner?
Do your QA teams spend too much time maintaining or updating tests?
How does that impact your costs?
Does your team spend a lot of time maintaining your testing environment
due to different operating systems and configurations?
Do you have distributed teams that require visibility into testing status for
better collaboration?
Does your test automation solution fit into your DevOps tools and
processes to simplify branching, test runs, status, and bug reporting?
Implementation process - Does the vendor have a well documented
30/60/90 day onboarding process which can be tailed specifically to your
team's needs and timelines?
Migration - does the vendor handle data migration and if so, how much
do they charge for the service?
Seamless integrations - will the vendor help you integrate with your dev
stack and assist with testing to ensure everything is working correctly?
Deployment - will the product be deployed exclusively in the cloud or
does the vendor provide an on-premises solution?
Training/support - will the vendor provide training and ongoing support
to ensure product adoption? What is the vendor's support SLAs?
Backup/disaster recovery - what tools and mechanisms does the
solution employ for backup and recovery tasks?
Scalability - how easily can the solution be scaled to handle both your
near- and long-term workload requirements?
Vendor lock-in - if you decide to leave the vendor, can you take your tests
with you? Can they be reused on another platform?
Test authoring
Maintenance
Customer support
Enterprise grade
Ramp up time
Skill set fit & dev cycle fit
Risk management
Toolchain integrations
Stability & confidence
Data migration
Deployment
Multi-platform support for web & mobile
Reporting & traceability
Product direction
How easy is it to write automated tests? Your developers, testers, BAs and anyone who knows how to use your product should be able to do UAT. Can you record or code stable tests? Can all members of your team participate in QA?
How much time is spent maintaining flaky tests? The solution should make it easy to determine exactly why the test failed without having to worry about the stability of the test itself.
Does the vendor offer adequate customer support and training? What are their SLAs? How do they measure your success?
Does the solution meet your requirements for security, data portability, user management, etc.? Enterprise-grade solutions should provide robust security features, easy data portability, audit trails, test case versioning, disaster recovery and more.
How long does it take for users to become operational and start realizing the benefits of why you purchased the solution? Ask to speak with customers to get a real idea of their on-boarding experience.
Does the tool require skills like coding or scripting? If so, are those skills accessible to your QA team?
Can you take your tests with you if you leave? Can you run them on another platform?
How well does the tool integrate with your pipeline and release process? Does it integrate with Dev tools such as Git, collaboration tools like Slack, bug tracking tools like JIRA and CI tools like Jenkins and test management solutions?
Do the tests break? How well do the tests handle code changes? Are there any Issues with timeouts? The test automation solution should help you troubleshoot quickly with error aggregation, screenshots, DOM of failed steps, network logs, and console logs.
Does the tool allow you to reuse tests from previous solutions? If not, is it easy to recreate old tests on the new platform?
How easy is it to install and configure the test environment? Can the tool be used in the cloud via any web browser?
What’s the process for running the tests continuously and on different browsers? Does the solution support different OS, browsers, and application for software you would like to interface with for combined issue reporting? Can you run tests on third-party test grids?
Does the tool offer visibility into the possible meaning of the results as well as enriched information (e.g. logs, DOM, screenshots) to help trace why a test failed?
Does their product roadmap align with your future anticipated needs? Do they release new features on a regular basis?
FEATURE CAPABILITY VENDOR 1 VENDOR 2PRIORITY WEIGHT
B U S I N E S S D R I V E R
D E V I N I T I AT I V E S TO AC C O M P L I S H D R I V E R S
S P EC I F I C G OA LS TO C O M P L E T E I N I T I AT I V E
H OW T E ST I M S U P P O RT S YO U R G OA LS
Increase RevenueTime
Quality
Cost
Deliver innovation faster
Shorten feedback loop, shift left
Coded or codeless tests, branching, parallel runs, grids
Customer Satisfaction Increase test coverage Build test automationFast authoring of AI-stabilized tests
Reduce Dev Costs Improve productivityCreate a stable test environment that aligns to developer workflows
Dev tools integrations, code or codedless, advanced debugging
For more information contact us at [email protected] or call +1-415-535-6690