develop a cloud testing strategy for today’s apps

Info-Tech Research Group 1

Info-Tech Research Group, Inc. Is a global leader in providing IT research and advice.Info-Tech’s products and services combine actionable insight and relevant advice with

ready-to-use tools and templates that cover the full spectrum of IT concerns.© 1997-2012 Info-Tech Research Group Inc.

Develop a Cloud Testing Strategy for Today’s AppsThe Cloud opens new opportunities for testing that were previously impossible.


Organizations are finding it difficult to execute test cases and monitor performance of a new breed of cloud applications. Leverage cloud testing to augment traditional practices and open the door to a higher level of software testing.

Introduction

Organizations that are refining testing methodologies and protocols to encompass cloud technology.

Application Testing Managers responsible for setting a standardized testing strategy.

Software Developers tasked with testing small units of newly written code.

Quality Assurance Managers looking to streamline QA processes.

Realize the changes of the software testing life cycle with the introduction of cloud technology.

Show you how the Cloud can significantly expand your testing capabilities.

Understand the costs and benefits of cloud testing.

Develop a standardized approach to testing in the Cloud.

This Research Is Designed For: This Research Will Help You:


Follow the Info-Tech Software Testing Roadmap

Develop a Cloud Testing Strategy for Today’s Apps

• This solution set will provide insight into how to incorporate cloud technology into your current software testing strategy.

• Learn how to align cloud testing initiatives with the development and traditional testing workflow.

Assess the Appropriateness of Automated Functional Testing

• Automating functional testing can improve software quality for the right shop, butthe cost of automation tools and ongoing maintenance can outweigh the benefits.

Vendor Landscape Plus: Tools for Functional Testing Automation

• There are dozens of tools out there that can automate functional testing. Select the right tool by understanding the features and requirements associated with functional testing tools.

Testing Strategy

Vendor Landscape: Software Test Management

• Gain insight into which vendor best aligns with your organization’s needs and which Software Test Management solution will provide the most efficiency within your infrastructure.

You are Here!Develop and Improve a Software Testing Strategy

• Traditional software testing processes provide a concrete foundation when developing and standardizing your software testing strategy.

http://www.infotech.com/research/ss/assess-the-appropriateness-of-automated-functional-testing


http://www.infotech.com/research/ss/select-a-tool-for-functional-testing-automation

http://www.infotech.com/research/ss/select-a-tool-for-functional-testing-automation

http://www.infotech.com/research/ss/it-vendor-landscape-software-test-management

http://www.infotech.com/research/ss/improve-or-develop-software-dev-qa-process



Executive Summary

The reality of today’s complex web and cloud applications demands a new approach to testing. • Increasingly connected apps using the Cloud at the back end along with highly fragmented endpoints at the front end

make traditional combinatorial testing almost impossible to execute; there are simply too many options.

• Traditional load and stress testing scaled down, simulated production environments, and forecasting the break point doesn’t hold with cloud virtualization and scalability. The focus on load and test needs to be done in the Cloud with scale if testing is to be reliable.

• Today’s increasingly global apps requires a global test lab with custom stacks. The Cloud offers a cost effective means to achieve this.

Your rollout strategy needs to be in three phases.• Business Case: Align cloud testing strategy and execution in order to ensure both the implementation and metrics are of

value to the business.

• Implementation: Thankfully much of the testing and development process remains the same. Organizations need a good vendor with a toolset aligned along the business strategy, and solid knowledge of the Cloud to help the implementation go smoothly.

• Metrics: Organizations need to measure testing metrics before and after cloud testing is rolled out. The ability to parallelize tests in volume with low cost can add considerable return on the investment.

Get cloud testing into your testing portfolio mix and start reaping the benefits.• Innovation is occurring in the in the young cloud testing domain. This includes such features as the ability to test gestures

for multiple touch devices – something almost impossible to achieve in volume for a broad array of devices.

• Cloud testing vendors are eager to gain market share in this young business model. This gives organizations a high degree of service as cloud testing vendors compete for revenue.


Testing should be where applications are hosted: in the CloudAs more applications move to the Cloud, testing with a limiting on-premise toolset introduces risk.

• With cloud testing services, organizations can now more accurately represent a production environment by deploying scalable infrastructure components and application stacks cost effectively.

• Long running tests can now be shortened by using cloud testing services to fan out a scalable number of functional testing units on demand during testing and executing in parallel.

• Today’s applications are hosted on the web or moving towards the Cloud. Organizations require a new framework around testing cloud applications that account for on-demand scalability and self provisioning.

• Expectations of application consumers (whether desktop, mobile, or touch) are continually increasing with speed-to-market and quality being the norm.

• Applications reside in different locations – on-premise, private cloud, public cloud, and hybrid cloud options – and testing against such diverse deployments requires reach and scale.

• Even if vendors have a cloud testing product, migrating would mean added costs of reconstructing existing test cases.

You need a lot servers to be able to simulate load. Cloud computing gave us access, availability, and affordability to lots of servers around the world to simulate that load.

- Tom Lounibos, President & CEO, SOASTA

Challenges?

Opportunities!


Adopt cloud testing or risk releasing defective applications due to budget constraintsTraditional testing is costly and time consuming. Remove these burdens with a cloud testing solution.

Issues with Traditional Testing

• Physical machines restrict the number of replicable test labs.

• This contributes to high capital costs when a large number of testing labs are needed.

• Mobile introduces functionalities, tools, and applications that testing teams are not prepared for with existing tools.

• Organizations are caught off guard by the growth of multiple platforms, particularly mobile.

• Tests are conducted on-premise in a closed environment with limited access.

• Off-shore locations are required to conduct their own tests locally, even if systems are identical.

Cloud Testing Solution

Benefits from Cloud Testing

• Systems can be scaled to a high factor at a low operational cost.

• Cloud testing solutions are compatible with multiple form factors.

• The Cloud enables accessible testing assets globally at any time.

Scale on Demand Multiple Form Factors Globalization

Cloud technology does not change the traditional testing processes that many organizations have accepted and adopted. It does, however, enhance traditional practices with increased replication capacity, execution speed, and compatibility through its virtual infrastructure and automated processes. For more information about traditional testing, refer to Info-Tech’s Develop and Improve Your Software Testing Strategy.



Leverage the benefits of cloud testing and avoid being stuck with traditional testing limitationsCloud testing offers two unique capabilities that are difficult to replicate with traditional testing: virtualization and compatibility with cloud services.

Virtualization is the foundation for automated, on-demand, and elastic testing. Organizations can:

• Create custom test labs, of both in-scope and out-of-scope systems, for various technology stacks through virtual machines and service virtualization.

• Closely replicate production environments. Traditional test labs were restricted to scaled versions for capital expense.

• Parallelize testing through creations of multiple test labs running concurrently such that each lab is focused on a subset of use cases.

Cloud testing is compatible with today’s web and cloud paradigm. Cloud testing can reside:

• In a public Cloud and spread globally to provide a more accurate picture of load testing.

• In a private Cloud with access to internal resources and capability to test software-as-a-service (SaaS) cloud services in the public Cloud.

Virtualization

Compatibility with Cloud Services


Use cloud testing to handle test cases for cloud or web apps and avoid incomplete test runs caused by scalability barriersCloud testing offers better options for tests that depend on scale and automation.• Cloud technology has shone a new light onto

traditional software testing strategies, enabling a number of unseen testing activities, such as:

◦ High quality testing environments.

◦ Parallel development and testing capabilities.

• Physical testing labs are replaced by virtual ones that can be scaled n-times.

• Testers are no longer required to make individual configurations to the testing infrastructure due to the Cloud’s automated process. Volatility is minimized.

• Stubbing is a practice of the past. Dependent systems are virtualized and replicated at a high degree of accuracy.

Testing Requirements

Testing Design

Code

Test Execution

Develop Test Case

Test Eval.

Deployment: UAT/SMOKE

FailFail

Succeed Succeed

Development

Maintenance

Cloud testing offers a new

testing avenue for certain tests.

Cloud Traditional

Code

Test Execution

Develop Test Case

Test Eval.

Source: Info-Tech Research Group


The only impact to your software development life cycle (SDLC) is within testing. The process for development and lower level unit and integration testing remain intact.

Integrate cloud testing into your current development lifecycle to strengthen quality

1. Requirements 2. Design 3. Build

6. Maintain 5. Deploy

4. Test

Development Workflow

Software Development Life

Cycle (SDLC)


Each stage before testing remains the same. Functional, load, and stress

testing is forked to the cloud while other

test processes remain as is.


Manage current technology trends through cloud testing to avoid stretching timelines or compromising test resultsCloud applications introduce new capabilities around JIT scalability and global reach. Your testing practices need to account for these new capabilities in light of recent trends.The application development and testing models are changing:

• Fixed infrastructure is now elastic.• Mandated and organization-issued devices are now

employee supplied through bring-your-own-device (BYOD).

• Data centers are now hosted and stored in the Cloud.

• Development times are shorter. Testing&

Development

Decrea

sing C

ycle

Time

Elastic Environment

Device

Fragmen

tation

Cloud Applications

Organizations need to adapt to these changes because current testing processes are limited. Today’s cloud testing needs to:

• Be elastic to scale up/down test clients.

• Accommodate test labs with unique stacks for multiple devices.

• Handle applications hosted on the web and in the Cloud (public and private) rather than client/server.

• Be available on-demand.


Develop a cloud testing strategy and avoid being left with buggy applications for the modern eraFollow a step by step process to ensure cloud testing satisfies the required business and IT goals.

Follow this diagram to navigate your way

through the storyboard.

1. Build a Business Case

3. Focus on the Metrics

2. Implement Cloud Testing

Traditional testing alone is no longer sufficient for

modern applications. Cloud testing fills the

gap.Cloud testing requires

new thought patterns to handle modern web and cloud integrated apps.

Create metrics to measure the success of

your cloud testing efforts.


What’s in this Section: Sections:

Build a Business Case

Build a Business CaseImplement Cloud Testing

Focus on the Metrics

• Understand organizational characteristics• Evaluate cloud testing readiness• List of business drivers• Cloud testing benefits and challenges• Determine the costs• Establish key performance indicators


Create a cloud testing business case for your organization to avoid downstream compliance hurdles Perform an analysis to assess operational and risk factors before embarking in cloud testing.

Cloud testing can have little value to an organization’s software testing needs if company policies and industrial standards restrict the full potential of the Cloud. Consider the following organizational attributes when developing a testing strategy:

• Cloud deployment – Applications can be hosted and tested on private, public, community, and hybrid cloud platforms. Each type of Cloud requires different testing access permissions: identify your cloud deployment on the cloud continuum.

• Hosting platforms – Applications can be hosted on the web, desktop (locally or virtualized), and in the Cloud.

• Application integration – Integration enables applications to communicate with each other directly or through a broker or bus.

• Application complexity – Applications with multi-tiered architecture introduces intricate application environments.

• Private Cloud – Cloud infrastructure provisioned for exclusive use by a single organization.

• Community Cloud – Cloud infrastructure provisioned for exclusive use by a specific community of consumers.

• Public Cloud – Cloud infrastructure provisioned for open use by the general public.

• Hybrid Cloud – Cloud infrastructure composed of two or more distinct cloud infrastructure.

Cloud Deployment Options

PublicCloud

PrivateCloud

CommunityCloud

HybridCloud

CLOUD CONTINUUM


Conduct a readiness assessment across people, process, and technology to avoid roadblocksSince cloud testing can affect people, process, and technology across the enterprise, build a cloud testing strategy designed around these.

ProcessCloud testing works best within the context of testing processes for applications: consider the level of business and application process orientation, automation, and standardization.

TechnologyCloud testing requires technology: IT needs to consider how the infrastructure, hardware, and software support match testing needs and complexity.

PeopleCloud testing requires people which means it will affect the development team: consider resourcing levels and the overall corporate culture, including possible change readiness.

• Level of development process maturity and standardization.• Level of testing automation.• Level and availability of documentation.• Willingness to implement and refresh IT policies and procedures.• Frequency of process change to meet changing development

strategies.

• End-user technical skill.• Willingness to learn new methods and technologies.• Frequency of communication between testing and development.• Acceptance of, and willingness to use, new tools.• Willingness to be a team player and align to corporate goals and

strategies for improving application quality.

• Age of current technology environment.• Available cloud solutions from vendors.• Type of application hosting platform.• Degree of standardization around new and existing technology.• Rate of application delivery.• Reliability, capacity, and availability of technology.• Degree of physical and virtualized testing labs.


Answer questions regarding people, process, and technology, and realize the areas that need improvement.

Use Info-Tech’s Cloud Testing Readiness Assessment Tool to determine your organization’s readiness

1. Analyze the people, process, and technology in your organization to determine cloud testing readiness.

2. Focus on the areas for improvements in this tool and plan to deal with them before fully engaging in your cloud testing strategy.

3. This tool will help you think about your current testing processes, which sets the stage for developing a cloud testing strategy.

Info-Tech Insight

http://www.infotech.com/research/it-cloud-testing-readiness-assessment-tool


Identifying business drivers is crucial to cloud testing success and is the initial point of your cloud testing strategy.

Steer your cloud testing strategy towards your business drivers to increase business buy-in

• Business drivers are the root of a cloud testing strategy. With well defined business drivers, your testing requirements, test design, test execution, and user and release assessments will also be well-defined – increasing cloud testing success.

• Involve all key business stakeholders when generating the list of important business drivers. These drivers will reveal the applications, technologies, patterns, and scenarios necessary to achieve the desired business goals.

• Keeping business stakeholders involved in the foundation of a cloud testing strategy drastically increases the software testing success rate.

• Reduce capital (CAPEX) and operational (OPEX) development expenditures.

• Faster time-to-market delivery.• Transparent reporting into development status.• Development and testing standardization.• Self-service capabilities.• Global application access.

Cloud technology is no longer a hype. Adoption has seen considerable growth in recent years and the impact of adoption will be experienced throughout IT. For more information refer to Info-Tech’s solution set, Prepare IT for Increased Cloud Adoption.

Info-Tech InsightCommon Business Drivers for Cloud Testing

http://www.infotech.com/research/ss/it-understand-the-cloud-to-make-informed-adoption-decisions


Development process and technology may be cemented based on proven best practices. Cloud testing should fit closely into those practices.

Realize technical hurdles early so that your cloud testing strategy can work around the risks

Identify any current hurdles within your organization’s boundaries:

Technological – Applications and test cases need to be migrated to the Cloud, systems need to replicated, and environments need to be provisioned. This will likely affect your current budgeting and resources allocated to IT.

Test Environment Issues – Test environments may need to be redesigned to scale up or scale down components n-times and across Cloud deployments.

Response Times – When applications are hosted on third-party platforms and infrastructure, latencies are beyond the tester’s control. Testing processes need to be adjusted to different testing runtimes.

Compliance and Regulations – Company standards and regulations may restrict test data from being hosted off-premise. Pay attention to privacy and compliance issues surrounding test data management.

• Firewall exceptions, remote configurations, and other security and privacy issues.

• Storage and bandwidth costs.• International requirements.• Systems that can not be virtualized.• Various approaches to cloud computing

from different cloud vendors.

Other Notable Obstacles


When executives see intangible benefits alongside dollar gains, they are more likely to commit to a formal cloud testing strategy, improving success.

Include hard benefits when making the cloud testing case to make the investment a well calculated risk

Benefits from a formal cloud testing strategy largely depend on how requirements are defined and how tests are designed. Include techniques that will bridge people, process, and technology gaps, and create quality test cases when making the strategy case.

In an action plan, include the preliminary people (testers, developers), processes (automation, migration), and technology (applications, Cloud, SaaS) that will be used to fill any foreseeable gaps.

Formal cloud testing strategies receive greater benefits, as displayed above. Formal cloud testing requires well defined business drivers defined by business leadership. Let the business know its involvement is crucial to maximizing these benefits.

Additional benefits include: rapid provisioning, advanced virtualization, scalable and elastic testing environments, multi-tenancies, real-time testing, accurate replication of production environment, and executing long-running tests.

Accurately set up test cases

Closely replicate production environment

Facilitate comm. between development & testing

Lower our overall development cost

Concurrently run multiple tests

Quickly simulate long-term effects

Decrease app delivery time

0% 20% 40% 60% 80% 100%

4%

9%

9%

8%

12%

12%

17%

96%

91%

91%

92%

88%

88%

83%

Ad-hoc Strategy Formal Strategy

Organizations With Formal Testing Strategies are Perceived To Have Greater Cloud Testing Benefits

Survey: Info-Tech Research Group; N = 39


Quantify how cloud testing improves the testing life cycle to surface the opportunity cost of not doing cloud testing Developers implement lower level tests in their tightly controlled workflow. Any long running test cycles will throw off the development timeline.

• Each software version is released for debugging and testing. When a software version passes this test, the update is added to the trunk, or parent, codebase.

• This serial process has two fundamental flaws:

◦ Development can only proceed when the prior version passes debugging and testing. Inadequate testing slows software development.

◦ Test cases need to be developed or updated when new versions branch from the trunk. Organizations with ad-hoc strategies will consume a significant amount of time and resources rerunning tests.

• Cloud testing enhances existing testing practices while leaving traditional development processes intact.

• The Cloud enables transferrable test cases, rapid test executions, and quick scalable resource provisions. This translates to shortened testing lifecycles.

With every software update, the testing designs and scripts must be redefined. Leverage a test case catalog to use old tests as frameworks and to identify the dependencies of the updates with out-of-scope systems.

Info-Tech Insight

Flaws of Traditional Testing

Added Value from Cloud Testing

More Bugs Found Per Test Cycle

Reusable Test Cases

Quick Scalable Resources

Fast Test Run-time

Significant Cost Savings from Cloud Testing


Be realistic and plan to deal with cloud testing challenges or get ready to suffer with disillusionmentOrganizations that had a harder time dealing with cloud testing challenges experienced lower levels of success.

Challenge Test cases must be transparent and capable of migrating between testing technologies. Cloud test cases should be independent from each other to see the benefits.

Cloud testing offers new capabilities and options that were unseen from traditional practices. This technology also brings new risks and concerns.

Virtualization and outsourcing tests does not necessarily lower implementation and operational costs. For example, virtualizing complex and dependent systems are time and resource consuming.

Recommendation Leverage existing testing standards with cloud testing.

Amend testing processes to reflect cloud testing capabilities.

Solicit multiple vendors to get the best pricing model.

Difficult Easy0%

20%

40%

60%

80%

43%

75%Cl

oud

Test

ing

Perc

eive

d Su

cces

s Ra

te (%

)

Difficult Easy

43%

73%

Difficult Easy

40%

80%

Migrating Test Casesto the Cloud

Redefining Testing Processes

Controlling Implementation &Operational Costs

Survey: Info-Tech Research Group; N = 36


Establishing benchmarks when making the case shows stakeholders the cloud testing strategy targets relevant and attainable business goals.

When making the case, outline KPIs relevant to the success of the business to avoid strategic misalignment

There is little motivation to continue IT projects if the performance and effectiveness are not evaluated. Measure your on-going key performance indicators (KPIs) against your benchmarks established by current IT standards and business drivers.

Identify the necessary criterion to exit the project. Continuously update these exit-conditions according to the changes in architecture and organizational needs.

At a minimum, the KPIs should be:

1. Relevant to your business strategy.

2. Based on your current IT metrics, at least in part.

3. Relevant to current and proposed testing services.

4. Metrics that exhibit software testing and development.

Choose metrics and pilot projects carefully. A successful pilot (as measured by your metrics) will gain political support for continuing the cloud testing initiative. Keep the business informed while establishing KPIs. Successful KPIs understood by the business are more likely to get the cloud testing initiative approved.

• Number of errors and defects• Testing latencies• Provisioning virtual machines and

services on-demand• Performance degradation• Server and storage utilization• Time to develop and migrate test

cases• Availability of SLA• Test run-time• Testing efforts• CAPEX and OPEX• Length of application development

and testing lifecycles

KPI Considerations


Benefit Scenario: An organization evaluates the cost of cloud testing

TEST/LAB CONSTRUCTION

TEST EXECUTION

TEST REPORTING

TEST DESIGN

TEST CASE

DEVELOPMENT FIXES

Company X, a small ecommerce business, is considering testing a new business application through a cloud testing solution.

Cloud TestingTraditional Testing

Total Cost: $2295

• Number of test cycles: 3• Cycle run time: 5 hours/cycle

• Avg. Testing Resourcing: $41/hour

• Bugs found per cycle: 210• Development Resourcing: $50/hour• Rate of bug fixing: 70 bugs per day

• Avg. Testing Resourcing: $83/hour

• Bugs found per cycle: 70• Development Resourcing: $50/hour• Rate of bug fixing: 70 bugs per day

Total Cost: $1091

Bottom Line: Company X will see a net benefit of over $1000 when executing a load and functional test in the Cloud. The benefit comes from the increased scalability and execution speed from cloud testing.

• Test Environment Cap.: 15 tests/cycle

• Number of Scenarios to test: 45

• Number of test cycles in parallel: 1• Cycle run time: 1 hour/cycle

• Test Environment Cap.: 45 tests/cycle

• Number of Scenarios to test: 45



Cloud testing is not just about an IT tool, it directly impacts business ROI. Your business case should incorporate four key components:

Build a business case on cloud testing so that benefits are clear to the business for the specific investment

1 32 4

Identify the functionalities and capabilities the

business can obtain from cloud testing. Business

drivers are the foundations of a cloud

testing strategy.

Analyze the hard and soft costs and benefits across

several years. The business will likely be unwilling to fund

the cloud testing project if monetary gains is not

immediate or significant.

Understand the challenges of cloud testing from a

business perspective. These challenges should be

prioritized to determine whether cloud testing

should be used for greenfield development or

across the board.

Establish an evaluation scheme to monitor the

performance of the cloud testing initiatives using KPIs. Business

owners can then determine if cloud

testing is risky.

Business Drivers

Cost/Benefit Challenges KPIs

[Business drivers] always starts from speed and urgency of the business problem… but then it eventually comes back to the old standard within the enterprises of ‘we would like greater control’.



Case Study: Formalize testing strategies based on business drivers to increase buy-in and improve planning

Drivers to Cloud TestingCurrent Testing ProcessSituation

Nicole VictorTest Manager

CSCwww.csc.com

CSC offers a range of innovative business solutions to help solve complex industry challenges.

Their application portfolio includes web applications. Some applications are located on hardware, i.e. client-based.

For one of their clients they used a standardized and documented manual testing process, and a formal software development life cycle process. Test cases are put into version control and are manually updated when changes are made.

The testing process is based on an incremental model, utilizing a change board process. Their change board process is a series of building, analysis, and approval activities between development and testing teams.

Test cases are manually executed in a step-by-step process, utilizing in-house test scripts. Their test scripts act as a pass/fail checklist, outlining how to conduct the test and what to look for in a successful test.

Their detailed test scripts provide frameworks to structure and implement automated processes. Cloud testing could help automate the manual documentation and testing process.

With a limited budget, maintaining testing quality would be difficult since implementing extensive regression and load tests would be limited.

They would like to maintain application quality when certain regulations and standards are amended without incurring additional costs.

A client of an IT consulting company is prepared to migrate to automated testing.

IT and Professional Consulting

Profile

http://www.csc.com/


Propose your testing initiatives to IT managers by outlining your testing methodologies, resource provisions, and testing schedule.

Use Info-Tech’s Cloud Testing Project Planning and Monitoring Template to create the project test plan

1. Describe the objectives and scope of your testing initiatives. Identify the functions and features being tested.

2. Specify the major activities, techniques, and tools which are used to test. Identify the major testing tasks and estimate the time required for each task.

3. This tool will help have a clear idea of testing dependencies.

Info-Tech Insight

http://www.infotech.com/research/it-cloud-testing-project-planning-and-monitoring-tool

http://www.infotech.com/research/it-cloud-testing-project-planning-and-monitoring-tool



Implement Cloud Testing


Implement Cloud TestingFocus on the Metrics

• Rollout cloud testing in a well defined process:

◦ Select a cloud testing vendor.

◦ Obtain proper trainings and support.

◦ Conduct a pilot.

◦ Measure results and KPI.

◦ Increase footprint.

◦ Standardize cloud testing.


Follow a well defined rollout process that keeps business objectives in mind while optimizing for cloud testing.

Implement cloud testing in a controlled manner and avoid retooling chaos

1 32 4

Choose the right cloud testing

vendor that will provide the right level of support and expertise to

help you get your cloud

testing initiative off the ground.

As with any new product, training and support are

essential. Build time into the

development cycle for retooling and moving down the learning curve.

Select a pilot project to start. Involve the right people from the

development team who can work with the

vendor to iron out any issues.

Collect relevant testing metrics

that show business value for the cloud

testing investment.

Select the cloud testing

vendor

Obtain proper training and

support

Conduct a pilot Measure results and KPIs

Increase footprint

Standardize cloud testing

5 6

Gradually roll out cloud testing to

include more applications.

Prioritize based on urgency and

importance.

Document and implement

cloud testing as the normative path for web

and cloud applications

development.

Introducing a new variable, like cloud testing, injects risk into the development mix which the containment counter measure is rooted in calculated steps along the journey.

Follow this icon to navigate your way

through this section.1


Select the right cloud testing vendor to avoid development debt during testing

Some key characteristics to look for in a cloud test vendor:

Horizontal breadth – Ability to offer both a cloud testing product as well as a testing service. In some cases, it can be more efficient to farm out an entire suite of test cases rather than have an internal team execute and record results.

Vertical depth – A full service model that includes extensive experience with application testing, migration of testing scripts, and training for new users of cloud testing. On the service side, this includes a well documented process for notification around outages, upgrades, and bug fixes. On the deployment side, this includes the ability to test web and cloud applications that include public Cloud, private Cloud, and hybrid Cloud.

Easy to use – Service is easy to migrate into for testers used to popular on-premise testing suites. Allows testers to increase productivity through simple point and click decisions and the ability to generate real time reports.

Global footprint – Ability to spin up test clients globally to properly conduct load and stress test from any region.

Work with your cloud testing vendor to realign your current testing process to incorporate their tool efficiently and avoid costly delays.

Horizontal breadth

Verticaldepth

Easy to use

Globalfootprint

A cloud vendor must be aligned with your business needs and requirements to ensure a sustainable partnership. A wrong partner can lead to missteps that increase technical debt and force IT to deploy quickly in the future without proper testing.

It may be necessary to prioritize the importance of each variable both in the short and long term.

Vendors in this space include: Soasta, LoadStorm, and 360logica.

1


Consider your short, mid, and long term strategies to determine how the importance of scoring variables may change over time.

Use a scoring criteria and select the right cloud testing vendor to avoid getting caught in overselling

Your scoring criteria may change over time as you evolve with cloud testing. Therefore, it is imperative to revisit your scoring criteria regularly to ensure vendor fit with business goals.

Horizontal breadth• Do they offer black box testing service?• Do they offer cloud testing training and tool specific

training?• Do they offer services to improve customer tests?• Do they offer integration services for various cloud

deployment options?• Do they offer advice on how to save expenses on cloud

testing?

Vertical depth• Do they have extensive experience with app testing? • Is there a process for notification around outages,

upgrades, and bug fixes?• Will they help you migrate your test cases to their

platform?• Does the tool integrate with current development

processes?• Is their tool continually being enhanced?

Ease of use• Is the vendor support quick and timely?• Is the tool interface simple to understand?• Does reporting enable export functionality?• Does the tool allow notifications to be sent

automatically?• Is the tool scalable on demand?

Global footprint• Are test endpoints available globally?• Is global support available to help troubleshoot testing

discrepancies?• Can testing be scheduled across various regions at

different times?• Does the tool accommodate multiple languages?• Can each region within regional testing be scaled

independently?

1


Case: An organization is looking for a cloud testing vendor with simple tools and deep vertical experience

Horizontal breadth

Verticaldepth

Easy to use

Globalfootprint

Horizontal breadth (Vendor Score: 2, Optimal: 1)Do they offer black box testing service? Do they offer cloud testing training and tool specific

training?

Vertical depth (Vendor Score: 5, Optimal: 5)Do they have extensive experience with app

testing? Is there a process for notification around outages,

upgrades, and bug fixes? Will they help you migrate your test cases to their

platform? Does the tool integrate with current development

processes? Is their tool continually being enhanced?

Ease of use (Vendor Score: 3, Optimal: 5) Is the vendor support quick and timely? Is the tool interface simple to understand? Is the tool scalable on demand?

Global footprint (Vendor Score: 2, Optimal: 1)Can testing be scheduled across various regions at

different times? Does the tool accommodate multiple languages?

Mapping each criteria to a score and plotting the result provides insight into the relative strengths and weaknesses of the vendor.

A multi-layered graph can be generated by adding multiple vendors as an overlay to determine commonalities amongst the vendors.

Optimal FitVendor Score

1


Obtain cloud testing training to ease the burden of retooling on test resources

To ensure ease of transition to cloud testing:

• Testing team should have access to training and support resources from the cloud testing vendor.

• Testing team should have access to a sandbox cloud testing environment where scripts can be generated and tested initially.

• The testing team does not change, just the tool changes.

Allocate sufficient time in the cloud testing plan rollout for test resources to get trained prior to a committed project.

Cloud Testing

Client Testing Team

Cloud Testing Support Team Cloud testing should not preclude the ability to test applications that reside in multiple cloud instances. The cloud testing support team should work closely with the organization’s testing team to ensure the necessary connectivity is in place.

PrivateCloud

HybridCloud

Internal to the organization

External to the organization

PublicCloud

CommunityCloud

Shared within community

OrganizationalBoundary

Clo

ud T

estin

g or

app

s be

ing

test

ed c

an

resi

de a

nyw

here

2

Insufficient time to settle a new tool could adversely affect the quality of the final application and increase overall costs caused by insufficient time to address workflow bugs.


Use the pilot to modify parts of the testing life cycle so that a full rollout will be more stable

• A different way to design test scripts. Some vendors rely on traditional scripting while others use a visual scripting tool built into the Cloud testing service.

• Different reports particularly if existing reports are localized within a region. Cloud testing introduces global reporting. In some cases the reports may come back much faster especially if parallelization of test cases is used. In such cases, the development team will need to handle results coming back from the cloud testing team much sooner.

• Lab setup time which was traditionally done by IT, but now is done through a cloud self-service portal.

• A triage of testing which delegates some test cases to the Cloud and others to traditional methods.

Expect to see some changes in workflow with the introduction of a new cloud testing tool. It is important to realize that the cloud testing workflow does not impact the entire application development process. Incoming and outgoing tasks for testing can remain intact. The only difference now is how the test team conducts their functional, load, and stress tests.

TEST/LAB CONSTRUCTION

TEST EXECUTION

TEST REPORTING

TEST DESIGN

TEST CASE

DEVELOPMENT FIXES

With cloud testing, setup and teardown of test labs may require its own governance to ensure test clients are properly configured. One approach to handle this is “Infrastructure as Code”.

3

Cloud testing leverages virtualized and automated techniques to minimize test runtime and reduce the risk of compromising live systems. Consider these capabilities in your pilot program.

Possible Workflow Changes


Take advantage of the virtualization under a cloud testing approach to improve test resultsTraditional testing can leverage virtualization techniques to scale the production environment.• Typically, but not always, a non-virtualizable system component in

a cloud infrastructure will bottleneck the testing throughput. The scalability factor is based on the least scalable component.

◦ Consider virtualizing the service rather than the actual testing component to eliminate the bottleneck.

• Include the capabilities of cloud testing with your design of a virtual testing environment, such as:

◦ Scale up and down test environments by a large factor.

◦ Rapidly provision resources.

◦ Pooling of testing assets and services.

◦ Maintain multiple test beds based for multiple release testing.

• Scalable by a large factor. Testing labs can be hosted on virtual servers off-premise, thereby eliminating the need for testing-specific hardware.

• Virtualized testing assets can be accessed globally. The Cloud provides easy access to off-site users.

Benefits from Virtualization Through Cloud Testing

• Scalable by a small factor. The total number of test labs is restricted by the total number of physical servers or memory.

• Virtualized testing assets are only accessed locally or through unique transfer protocols. Off-site users need special access to the servers of the virtualized assets.

Drawbacks from Virtualization Through Traditional Testing

Before After

On-Premise

fixed virtual serversn virtual servers

3


Simulate production systems through virtual data to eliminate the risk to live and inaccessible systemsCloud testing enables the parallel execution of development and testing activities.

• Cloud environments can be virtualized images of your in-scope and out-of-scope applications and systems.

◦ Develop an automation model to capture the relevant scenarios that are out-of-scope.

◦ Virtual data is used to simulate the behavior of out-of-scope systems with enough realism that the in-scope system believes it is interacting with a live system.

• Parallel testing solutions allows teams to execute against live systems where synchronized data and functionality is available. If a component of an updated application breaks or if a new testing scenario is needed, testers can immediately switch back to virtual services.

◦ This ability is only applicable to purely downstream live or virtual systems.

Security and regulatory concerns may prevent organizations from recording and storing live production data on third-party cloud applications, and public Clouds. Perform a risk assessment of expediting production data to external testing sites. In this case, consider developing an internal, private cloud testing strategy.

Input Data In-Scope App. Out-of-Scope

Test Data Management

Problem: Organizations may not have full access rights to the entire out-of-scope system or the input data of the in-scope system. The usual practice to alleviate this issue is stubbing.

Solution: Virtualization through cloud testing accurately captures the performance of systems by working around access and privacy hurdles.

3


Automate tests through cloud testing to avoid exhausting systems and memory resourcesCloud testing can run through numerous scenarios in a fraction of the time it would take to manually run the same tests using less resources.• Automation can reproduce some situations that would be

virtually impossible for a person to do, such as simulating hundreds of thousands of users. However, automation under traditional testing comes with costs. Automation:

◦ Consumes significant amounts of power from physical servers to execute.

◦ Requires sufficient memory to support immense testing data from scalable tests.

• Cloud testing incorporates automation, self-service, and cost accounting (metering) features into their management stacks.

◦ Infrastructure and platforms are virtual and off-premise. Immense computing power is not a requirement.

◦ Testing data is stored virtually; thereby, eliminating the need to provision large quantities of memory.

Automated tests can be shared between your test group and developers, and can be triggered to run automatically every time new or changed code is checked in. If the test fails, the developer can be notified automatically and avoid involving your testing resources. For more information about automation, refer to Info-Tech’s solution set, Assess The Appropriateness of Automated Functional Testing.

Virtual Server/User Generator

……1 2 3 n

In-Scope Applications

Cloud Testing Solution

Aut

omat

ed

Itera

tions

Repeated Requests

Repeated Simulations

3



Measure KPI results from the pilot to test effectiveness of a full rollout

Faster test execution

ROI

Keep business strategy in focus during the pilot phase.

Increased automation

Low cost of migration

More accurate testing

Decrease development

time

Lower testing cost

There are several metrics for measurement that are possible for cloud testing.

Reports from cloud testing can help determine the following metrics:

• Test execution times

• Test throughput

• Global results

Cloud Testing Reporting

Observation from the modified testing workflow can help determine the following metrics:

• Cost of migration

• Resource allocation for testing

• Development time

• Degree of automation

Testing Lifecycle

4

It is not always possible to see improvement in all KPI metrics. Often there is a trade-off involved. The best way to determine whether cloud testing should move forward for the business is to prioritize the KPI metrics and focus cloud testing on the most important metrics.

Info-Tech Insight


Take a gradual approach for migrating into cloud testing to allow time for processes to settleMove testing to the Cloud over time to allow benefits of elasticity and virtualization to integrate.

The benefits of a gradual rollout include:

• Controlled migration into cloud testing.• The ability to repurpose existing scripts and to set up test

templates in the Cloud for future reuse.• Conformance with business priorities around application

development thereby not wasting resources on those applications that will be phased out.

• Allowing modified testing workflow time to settle.

The risk of a gradual rollout include:

• Distributed applications may be difficult to partition thereby requiring a “big bang” transition to cloud testing.

• Benefits will be deferred for some applications potentially leading to higher opportunity costs of not moving to cloud testing.

• Test teams will have to use two different testing tools for functional, load, and stress testing in the interim.

Pilot 1

End goal

As the rollouts continue, the maturity of cloud testing skills should increase paving the way for cloud testing as the standard for functional, load, and stress testing.

5

Pilot 2

Process settlement

Measured im

provement in

testin

g

Additional pilots


Implement cloud testing as the standard to avoid relapse into traditional testingEnsure cloud testing meets business needs consistently throughout the organization to maintain business buy-in.Cloud testing has now been adopted across various teams and departments in the organizations. However, the implementation and monitoring strategies may be inconsistent within the organization. Consolidate the multiple cloud testing approaches into one focused strategy.

A formal, standardized strategy will achieve several key reusability and accessibility benefits. These benefits will reduce the cost of re-provisioning resources and creating new testing assets:

• Testing assets, such as virtual data, are easily accessible through the Cloud.

• Multiple teams can obtain, use, and amend test cases in the Cloud.

• Virtualized and simulated systems can be shared.

• Consistent compliance across departments.

• Standardized communication approaches with vendors.

6

Monitor the executions of each test. Cloud testing vendors structure their performance metrics based on common factors in an industry, which may or may not be representative of your organization’s standards. Develop a performance model based on your expectations and compare your results with your vendor’s. The next section of this solution set will help you identify and utilize key performance metrics.

Clo

ud T

estin

g Im

plem

enta

tion

Cloud TestingStandardization

Inconsistent Testing Approaches

Initial Adoption

Standard Practice


Case Study: Standardize traditional testing practices to minimize risks in transitioning to the Cloud

Cloud Testing DeploymentCurrent Testing ProcessSituation

Lead Testing AnalystTesting Analyst Group

Power Generation Industry

This company strives to be the leading provider of electricity, natural gas, and related services across the United States.

They are a high end, high security facility which restricts the use of live data for external testing or public cloud deployment.

About half of their system is virtualized in the Windows space and about a third is virtualized in the Linux space. They operate their virtualized environment as a private Cloud.

Each team and department team use a different testing and development model, such as waterfall and agile. They use whichever model works best for that particular group.

A testing center solution is used for their testing needs. They also use the vendor’s services to mimic on-premise services and use it as a comparison with tests conducted off-premise.

Internal application tests are executed in-house. They use an internal toolset from the testing center for performance testing.

All test scripts are easily portable to the cloud testing vendor because the Cloud and internal toolsets are similar.

Testing scripts are written internally on their hardware and transferred to the cloud vendor site. This vendor gives them complete control of running the tests. The vendor leases them the resources to execute the test internally.

There is no need to reach out to external testing solutions for production environment tests, due to risk management issues.

“Partner with a vendor and deliver a proof-of-concept, then decide to buy something after that experience.” – Lead Testing Analyst

Profile







• Ensure cloud vendor complies with testing requirements

• Amend applications according to the results of large testing datasets

• Evaluate the performance of cloud vendors• Be aware of the changing cloud testing landscape


Cloud testing excels in two areas: scalability and rapidity. Load/stress tests and functional tests for web and cloud apps are ideal candidates to benefit from these features. Focus your metrics around scale and speed.

Evaluate the effectiveness of cloud testing over the limitations of traditional testing

Load & Stress Testing Functional Testing

Traditional (non-Cloud) Testing

• Limited to a few test clients and an estimated projection is done for the live environment.

• Results are limited to regional test clients with no easy way to obtain global data for today’s web and cloud apps.

• Conducted against a test lab infrastructure that is a fraction of the true production infrastructure.

• Testing is limited to few test clients because the need to spin up browser clients is memory intensive.

• Difficult to shorten long running tests due to capital expense of additional virtual or physical clients.

• Parallelization has an upper limit based on test clients.

• Provisioning test clients can take time.

Cloud Testing

• Testing enables global reach with performance benchmarked from different test clients around the world resulting in better test spread globally.

• Rapid infrastructure scale up enables faster and more accurate test results through faster test client deployment.

• Cloud vendors willing to port existing tests into their environment thereby decreasing migration costs.

• Enables massive parallelization of independent test cases resulting in the generation of more test results faster.

• Ability to scale up large numbers of test clients in cases where memory constraints limit scalability per client which leads to lower capital cost.

• Ability to more accurately gauge the effects of functional performance during heavy load thereby better anticipating load threshold.

• Can replicate large portions of production environment to more accurately test functional requirements thereby reducing long term maintenance costs for integration.


Ensure cloud vendors continuously comply with your standards and testing requirementsRegular meetings with cloud vendors are your opportunity to address known and unknown issues as they emerge. Measure vendor response times.• You are giving a critical part of your software

development life cycle to a third-party testing solution provider. As such, you want to make sure that the vendor has a clear picture of your testing needs.

• Facilitate continuous communication between your organization and your cloud testing vendor throughout the execution of testing initiatives.

• Understand that dealing with consequences of failures beyond your control is the trade-off you make when leveraging testing-as-a-service (TaaS) and other cloud services.

• Make sure the vendor provides notification and transparency into service levels and disruptions.

• Make sure there is a point person responsible for receiving updates and notifications for each service (and delegating that responsibility as needed).

• Have clear plans and processes in place so information gets from the point person to those who need it.

Info-Tech Insight

For additional information of cloud vendor management practices refer to Info-Tech’s

Effectively Manage Cloud Vendors.

Cloud testing has shifted the roles and responsibilities of the testing and development teams due to its added capabilities

As a result, operational, technical, installation, and support touch points with business stakeholders and cloud vendors have changed from traditional practices.

Determine the locations of the touch point changes, and identify the new contacts at the various stages of cloud testing.

• Satisfaction• Response times• Knowledge gaps• Resolution rate• Support documentation

Some Cloud Testing Vendor Relationship Factors

http://www.infotech.com/research/ss/it-effectively-manage-outsourcers-cloud-vendors


Adapt test reporting to anticipate larger data sets and realize how this benefits the overall application qualityDebug, Fix, and Update Your Application

Repeat Your Tests, If Necessary

• Realize that testing is now an analytics project. Testers receive numeric test results due to the black-box nature of testing practices, whether done locally or in the Cloud.

o Due to the growth of integrated applications and systems, the importance of load and system compatibility significantly increases.

o Cloud testing produces large testing datasets that have a higher degree of usage accuracy, and identifies more bugs than traditional testing. This translates to less testing and maintenance cycles, and fewer bug updates; thereby, reducing costs and improving application quality.

• Document and measure any changes to your application and test cases in a catalog or library for future reference. Adjust test reporting to include the diverse testing scenarios from Cloud testing.

• Testing is not a purely downstream workflow. If the testing needs are not satisfied in the first iteration of the testing cycle, then the application should be updated with the latest fixes, test cases redesigned, testing scripts rewritten, and the tests executed again.

Design and Develop Implement

The [application] issues aren’t always in the design of the app. or the functionality of the app.; it’s how that app. or that workload relates or correlates to an infrastructure, a network, and millions of users around the world.



Measure performance of cloud testing to calculate benefits for ROIEvaluate the overall performance of your cloud testing solution and establish benchmarks.Some performance benchmarks and benefits include:

• Volume of tests capable of running with cloud testing versus without cloud testing. Larger volume shortens testing time translating to a reduction of resource cost.

• Number of maintenance release cycles required using cloud testing versus without cloud testing. Fewer maintenance release cycles translates to faster time to market on subsequent product versions while reducing testing costs for each maintenance release iteration.

• Speed of testing cycle when using cloud testing versus without cloud testing. Parallelizing translates to shorter testing cycles translating to a reduction of resource cost.

• Number of memory leaks (through long running tests) found using cloud testing versus without cloud testing. Shortening long running tests translates to reduction in maintenance cycles and improved productivity of end users.

• Resources required to run functional tests using cloud testing versus without cloud testing. Shorter testing cycle times translates to fewer resources required over the testing lifecycle.

Having a clear and consistent testing strategy is key to managing ROI. Those areas important to the business should be measured to determine the benefit derived.

Info-Tech Insight

25%

22%

Resource

Volume

15%Time

35% ROI

5% ROI

30% ROI

30% ROI



Be aware of cloud testing limitations so that you play on its strengths and avoid disillusionmentCloud testing is green and expanding. Prepare for the following four factors to stay ahead in the game.Factor What to Expect: How to Prepare:

Dynamic Vendor Landscape

• New cloud testing vendors and traditional testing vendors are entering the cloud testing landscape.

• Perform a due-diligence of your cloud testing vendors periodically.

• Establish a fall-back plan if your cloud testing vendor no longer provides support.

Various Cloud Testing Process Models

• A lack of consensus of what is the optimal cloud testing process model.

• As systems become virtual and complex, cloud testing models among organizations begin to vary.

• Develop a cloud testing strategy that works best for your organization.

• Identify areas in your process that needs improvements.

Rapid Evolution of Cloud Testing Technology

• Cloud services are becoming more affordable and compatible with various public/private and hybrid cloud solutions.

• Cloud vendors are developing a methodology for Cloud-to-Cloud interoperability.

• Evaluate emerging testing solutions and determine if it will have value in your organization.

• Conduct a pilot test to gain confidence and then roll out with cloud testing across the board.

Compliance and Industrial Standards

• In some organizations, security and privacy protocols are incorporating access of the Cloud and virtualizing production data.

• Ensure your cloud testing strategy is compliant with company standards.

• Avoid practices and technology that can present a risk.


Summary


• Formal cloud testing strategies consider business drivers, and key business and IT stakeholders. Engaging in a formal cloud testing strategy and involving the business when aligning business goals with cloud testing values increases the success rate.

• When making the case for cloud testing include a list of business drivers that support the cloud testing strategy, the benefit and costs, your organization’s unique characteristics, and the KPIs that will be used to gauge success.

• Assess your cloud testing readiness by taking an inventory of the people, process, and technology your organization has access to. Take note of the hurdles and strategies to overcome them before beginning your strategy.


• Implement cloud testing solutions in a well-defined rollout process to ensure business objectives are realized and cloud testing initiatives are optimized.

• The rollout process consists of six steps: select a cloud testing vendor, obtain proper training and support, conduct a pilot, measure results and KPIs, increase footprint, and standardize cloud testing.


• Evaluate the impact of cloud testing over traditional testing load/stress and functional tests. These metrics should be focused on scale and speed.

• Realize the changes in the touch points from cloud testing and measure cloud vendor performance accordingly.


Appendix: Survey Results


Survey Respondents - Industry


Survey Respondents - Country


Survey Respondents – Full Time Employees


Survey Respondents – IT Employees


Survey Respondents – Revenue


Survey Respondents – Job Title


Survey Respondents – Department


What best describes your organization’s approach to software testing?


What is your organization’s timeline for adopting a cloud testing platform?


How would you rate the difficulties of the following testing scenarios?


Do you believe cloud testing can help with the following at your organization?


How would you rate the following challenges of adopting cloud testing?

develop a cloud testing strategy for today’s apps

Documents

cloud testing strategy

benefits of cloud testing

cloud testing initiatives

leverage cloud testing

stress testing

testing methodologies

testing capabilities

functional testing tools