improve your quality assuran 238115

8/12/2019 Improve Your Quality Assuran 238115

1/8

This research note is restricted to the personal use of [email protected]


G00238115

Improve Your Quality Assurance When Using

SaaSPublished: 15 November 2012

Analyst(s): Thomas E. Murphy, Daniel Sholler, Christian Hestermann

Many users hope that moving to SaaS will help to reduce cost of ownership

and shorten time to market. However, loss of control over changes requires

users to spend additional effort improving their quality assurance to avoid

business disruptions.

Key Challenges Based on marketing messages by vendors, users are lured to assume that using a software as

a service (SaaS) solution requires less testing and quality assurance (QA) efforts than using an

on-premises solution, which is not true.

Users of a SaaS solution have less control over the changes applied to their solutions; in many

cases, they might not even know in detail what was changed, nor do they have visibility into the

QA effort (or lack thereof) that is applied when changes occur.

The changes can theoretically appear at any time, although more vendors offer options to

postpone changes for a limited period of time.

Recommendations Do not simply expect the vendor to be more accurate in its QA and testing than you are, but

understand where risk areas lie and the contractual or test mitigations you will need to apply.

Make sure you have a good communication channel to vendors to notify them of issues, and to

ensure early access to changes.

Automate regression testing to run as often as possible, or at least as necessary based on your

provider's change patterns.

Establish error-handling and problem resolution processes as a part of the communication plan

when negotiating the SaaS agreement.


2/8



IntroductionSoftware quality and testing are a constant source of grief in software production. When QA finds a

defect, it delays the delivery of the software, and when a defect slips through, QA didn't get the job

done. All in all, everyone looks for ways to reduce the cost of testing and it seems as though they'd

like to forget about it all together. This is driving solid growth in offshore outsourcing of testing, andhas been one of the desired values of utilizing packaged software. Now, SaaS proposes to reduce

the operating cost of software on top of the development and testing savings of packaged software.

However, there is a distinct difference in current SaaS offerings the loss of connection to the

change stream (see "The Impact of the Cloud on ERP and Business Application Planning" and "The

Impact of the Cloud on Architecting an ERP Solution").

While the focus on software testing varies greatly across organizations, with traditional software

delivery the consumer (or implementation partner) has some level of control over changes to the

system. In a traditional upgrade, users should understand that they have to extensively test new

versions before putting them into production. In a SaaS application, it is easy to be misled into

neglecting testing because the upgrade happens in a transparent way. Indeed, vendors tend tomake prospects believe that with a SaaS solution, testing the stability and performance of a newer

version is no longer needed. However, depending on the nature of the changes or how the SaaS or

platform as a service (PaaS) solution is extended or integrated into other systems, customers may

have to spend the same amount of diligence and prudence, or more, as with any other business-

critical application (see "Ten Challenges for Successful Composite Multienterprise SaaS

Integration"). But, without access to the change stream, instead of potentially having changes (and,

therefore, bugs) introduced on a periodic basis, they can be introduced daily (and without your

knowledge). As a whole, traditional package software vendors have not had a great record when it

comes to change management and upgrade support, which does not bode well for the transition

into cloud computing platforms.

Application managers and business users of SaaS-provided business applications face significant

risks when their mission-critical application breaks and no longer delivers correct results.

Depending on the functional scope that is deployed, these risks can affect every area of the

business. In severe cases, this can affect a company's ability to serve its customers or process their

orders.

Analysis

Understand the Costs and BenefitsTesting is a cost-benefit exercise. You can never test everything, because everything would literally

be everything. A combinatorial number of possible use cases, failures, data inputs, etc., would

theoretically need to be tested. The majority of organizations instead test based on the risks they

are trying to control. With SaaS there are hidden costs that you must be aware of which result from

the loss of quality and change management control. Therefore, the question in the SaaS model is

that since you do not know the change stream, is the risk that a vendor-applied change will create

an error in your process something that you want to spend time and money to control, or not? Most

Page 2 of 8 Gartner, Inc. | G00238115


3/8



people choose not to make these expenditures, because it means not accepting the change until it

is tested. More SaaS providers give their tenants the option to postpone the activation of a new

version for a limited period of time, but because all tenants in a multitenant SaaS application are

expected to use the same code, no guarantee is given that any found errors will be removed before

the new version is finally deployed. With traditional software packages, you have some access and

control over what is changing and an ability, through your own knowledge or a consulting party, to

understand the risks and how to mitigate them. In SaaS, we are led to believe that all is well, but no

software is bug-free and changes to vendor-provided software have traditionally caused headaches

for users. Thus, make sure you carefully measure the impacts and value of customization and

integration, and how changes will affect the cost of using the software.

Don't Lose QA Control

In a SaaS environment, your organization is paying for outcomes. You are paying the vendor to take

responsibility for many of the common challenges in running applications, at the cost of a limited

ability to modify the deployment and configuration choices. For this reason, it is vital that the focus

of testing shift from feature/function behavior to outcomes. Obviously, in the context of a new

release from a SaaS provider, it would be prudent to test the new things in that release, but that

testing should be done to ensure that the results are what they need to be. Even if a particular

feature does not work as expected, it may not compromise the ability of the system to create

usable results. For example, if tax calculation was done internally in the previous version and is now

done by reaching out to an external service in the new version, the same input data needs to

consistently deliver the same output data, regardless of how the data is derived and regardless of

any additional features provided. Nevertheless, even if the underlying code is provided as a service

and the focus shifts from function to outcome, your focus in QA control still has to be on the

functionality of the solution. You still need to ask: "Does the code provided as a service, when

applied to my data, still produce the expected outcome?"

Critical aspects of remaining in control require you to demand certain behaviors from your SaaS

vendor. The vendor must notify you when changes have occurred, and preferably give you the

option to roll those changes into your system (and back them out) during a transition period.

Second, they must provide some impact analysis for new capabilities, either functional or

nonfunctional. Finally, new capabilities need to be very clearly detailed, so that the "failures" that

are discovered are not failures of expectations. Finally, the vendor must have a clear policy that

allows you to remain operational during a time when an error has been discovered but not yet

remedied.

Ensure ComplianceVendors will have to take on a greater responsibility for the U.S. Sarbanes-Oxley (SOX) Act and

other regulatory compliance issues in SaaS delivery. However, anywhere you can change a

calculation or a workflow, or anywhere data comes out of the system and then re-enters it, requires

end-user validation. Users will need to add regulatory issues as part of the buying criteria, e.g., has

the vendor been certified? Certification of the vendor's practices, security, load/stress, and

compliance should all be part of your decision on whether to choose that vendor (see "Look Before

You Leap Into Cloud Computing"). You must focus on the fact that purchasing a SaaS solution is

Gartner, Inc. | G00238115 Page 3 of 8


4/8



buying a service, not software. This means that you are relying on the vendor to manage the

software correctly, as well as its operation, support and change control. The issue that you are

testing is the suitability of the service as offered to your business and its requirements.

Another potential area of concern is the quality of the vendor's change control practice and how the

provider notifies users. When the large offshore outsourcing vendors started up, they sought outCapability Maturity Model Integrated (CMMI) Level 5 and International Organization for

Standardization (ISO) 9001 compliance as a way to show they were trustworthy and reliable. If you

are expending the effort to be ITIL-compliant in your own operations, would you not also expect

similar levels of certifications from your service provider, like the Cloud Security Alliance GRC Stack

or Oasis' TOSCA?

With financial systems, another challenge occurs when the fundamental nature of the application is

correct, but something about the specific dataset utilized causes an issue. In software testing,

100% path coverage may be obtained, but it is impossible to test all valid/invalid data inputs. In

general, you are responsible for your data and for ensuring that it works correctly with the software,

not the other way around. Auditors will not accept excuses or claims that you don't know whathappened to the application.

Automate Regression Tests to Run Whenever Change Might Have Occurred

With traditional business application software in the field, the tools to manage change sets and

testing are improving in response to those changes creating upgrade issues for customers. For

instance, SAP Solution Manager or other SAP change tools identify what has changed, which tests

need to be rerun, etc. In SaaS, we will either need to get to that point, or strongly limit customization

and integration. It also means that the test team should have a good set of automation around the

key transactions, integrations and customizations, and should run those potentially every day.

These tests become an internal change notification system. But that still leaves a big window whena change can occur and your software breaks. This is a risk you are agreeing to live with, unless you

contractually protect yourself from vendor change without notice.

This risk is currently mitigated by service-level agreements (SLAs). Most companies are not

concerned about software quality; they are concerned about risk management, and the goal is to

spend the least you can to have a level of risk you are willing to live with. In moving to SaaS, firms

are trying to reduce testing costs, not increase them. Costs can be controlled by limiting

customization and integrations, although this will limit the application's business value. Limiting

these will curtail the need for concern that vendor updates will break your system and reduce the

need for concern about change management or testing. However, in all cases where the software is

enterprise-deployed, or mission-critical, understanding the vendors' change management andoperational procedures and building a selected set of validation tests around core functionality is

critical.

Be Even More Diligent When Using Application PaaS

Note that PaaS/application platform as a service (aPaaS) means that you take more responsibility

back, and this complicates issues. Rather than just being software that you run (e.g.,

salesforce.com) with minor configuration changes, it is a hosted development platform (e.g.,

https://cloudsecurityalliance.org/


5/8



Force.com). In this case, you have software developed on top of the platform, and platform version

changes could break functionality. In a traditional development world, this is the same as the

relationship with an application server. The only difference is that in the aPaaS case, you are not

responsible for the deployment and configurations of the application server. This creates the

potential for breaking existing integrations as well as user skills if there isn't a provision for

backward compatibility or support from the vendor to run an older version of the application. There

are connections between this and testing for service-oriented architecture (SOA) and SaaS. This is

especially true if services are primarily externally provided, such as credit authorization, payroll,

shipping, etc. When the vendor changes a service (maybe not even a functional service), the

performance characteristics change, and the system must deal with this (see "APaaS: A Step to a

'Killer App' for Cloud Computing?"). Programs deployed in aPaaS environments must be managed

the same way as any custom program, with the exception of the issues around platform

configuration and deployment.

Take Ownership of Change Discovery and Defect Management

The issues of defects in off-the-shelf software and SaaS-provided solutions are similar in that the

vendor is responsible for the defects in its software, and the user is responsible for the defects in

any customization or integration. However, the difference with SaaS is that the vendor now has

responsibility for platforms and configuration, which means that they are responsible for most of the

nonfunctional requirements of the systems. Because of this transition in responsibility, it is critical

that these vendors provide a method of informing their customers about what defects have been

detected, and what is being done (or can be done) to mitigate them.

Many SaaS providers are more willing and able to provide transparency regarding defect

information than traditional vendors. Because these vendors are trying to disrupt the market and

tend to make use of more-agile practices, and because they need to evolve solutions to manage the

impact of change on software users, it is in the vendors' best interest to make use of a more publicstream of information. However, this will still require user resources to monitor and understand the

impact on help desk trouble tickets, defects and feature requests, and to ensure that changes don't

negatively impact running solutions.

Application managers and key users need to diligently study the information on changes delivered

by the SaaS provider and figure out which of these changes might affect their application. Because

of the fact that all changes are installed into their application, they also need to pay attention to

changes that they do not even intend to activate and use, because any change can potentially have

a negative impact on the correct execution of the deployed software. Some more extensive

changes will require the adaptation and extension of the test cases and test data to ensure that

these changes are being tested in future regression tests.

Should you find that your provider is not providing the needed information or the necessary level of

detail, you must pressure them to correct this. If necessary, the corresponding SLAs need to be

completed in the course of negotiating the next renewal of your subscription.



6/8



An Example

SaaS seems to offer the benefit of helping you avoid upgrade pain; you just gain new features

without all the planning and deployment effort involved in on-premises solutions (see "Top Five

Ways to Make Your ERP Upgrade a Success"). For example, you don't worry about your Google

docs working tomorrow, but you hope for improved functionality. One advantage that vendors gainin Web applications is knowing exactly how many people use specific functionality. Microsoft has

been able to do this with its applications whenever you select the "help provide feedback" option

on product installation. This feedback lets the product manager understand how many people use a

function, how often it is utilized, the potential problems that may occur, etc. This feedback may

show the need to change a feature for usability, or show a feature that isn't widely used (but is still

critical to you), and is not profitable in the vendor's ROI assessment.

A major issue occurred when Microsoft introduced version 4 of Excel. Prior versions supported

macros in foreign languages, e.g., German, so you could say "Berechne " instead of

"Calculate ." But in the next version, this support was dropped, so users had to manually go

through and redo all their macros. Because the software was locally installed, users could controlthe rollout of the change and potentially revert to an earlier version while the broken formulas where

updated. Returning to the example of Google docs, basic functionality should be unaffected after a

change. However, if you can use macros, formulas etc., then you hope they still deliver exactly the

same results as before because you don't have a way to revert to a previous version. Thus, when

you move to SaaS, the pace of change is no longer yours to control, and your organization has to

be able to respond to that change as quickly as possible. Because you can't protect yourself from

change (beyond postponing the activation of a new version), a plan should be put in place to ensure

a quick response in addressing issues that arise due to vendor changes. A critical element of this

plan should be the establishment of a key internal contact person who works directly with the

vendor when issues arise.

These mechanisms have evolved. We have seen, in the world of constant betas the Web has

delivered, that there are ways to opt into early access to changes. Vendors need the ability to

understand the effects of these changes, and the only way to get the combinatorial advantage is to

put users on the software. In SaaS business applications, more providers now offer sandboxing

environments where users get access to the next version before it is deployed. Do not simply

expect the vendor to be more accurate than you are, but understand where risk areas lie and the

contractual or test mitigations you will need to apply. In addition, make sure you have a good

channel to vendors to notify them of issues, and ensure early access to changes. Think of SaaS as

a partnership where you are helping the vendor with early feedback and reports, and where they are

building enhancements and resolving issues.

Bottom Line

When moving to SaaS, don't expect defects to magically disappear. Even if you don't know exactly

what has changed and cannot avoid using the defective version until it is corrected, a moderate

testing program will mitigate risks and help avoid damage.



7/8



Recommended ReadingSome documents may not be available as part of your current Gartner subscription.

"Software as a Service: Component Development Challenges"

"Look Before You Leap Into Cloud Computing"

"SaaS ERP Only Reduces Part of the Effort Needed to Implement and Operate Your ERP"

"Top Five Ways to Make Your ERP Upgrade a Success"

"Case Study: Deploying SaaS-Based ERP"

"ERP/Business Applications and the Public Cloud: A Life Cycle Assessment Methodology and Key

Focus Areas"

"Oracle, Salesforce.com and SAP Acquisitions Will Impact Your SaaS Strategy"

"The Impact of the Cloud on Architecting an ERP Solution"

"The Impact of the Cloud on ERP and Business Application Planning"



8/8


GARTNER HEADQUARTERS

Corporate Headquarters

56 Top Gallant RoadStamford, CT 06902-7700

USA+1 203 964 0096

Regional Headquarters

AUSTRALIABRAZILJAPANUNITED KINGDOM

For a complete list of worldwide locations,visit http://www.gartner.com/technology/about.jsp

2012 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This

publication may not be reproduced or distributed in any form without Gartners prior written permission. If you are authorized to accessthis publication, your use of it is subject to the Usage Guidelines for Gartner Servicesposted on gartner.com. The information contained

in this publication has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy,

completeness or adequacy of such information and shall have no liability for errors, omissions or inadequacies in such information. Thispublication consists of the opinions of Gartners research organization and should not be construed as statements of fact. The opinions

expressed herein are subject to change without notice. Although Gartner research may include a discussion of related legal issues,

Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company,and its shareholders may include firms and funds that have financial interests in entities covered in Gartner research. Gartners Board of

Directors may include senior managers of these firms or funds. Gartner research is produced independently by its research organization

without input or influence from these firms, funds or their managers. For further information on the independence and integrity of Gartnerresearch, see Guiding Principles on Independence and Objectivity.

http://www.gartner.com/technology/about/ombudsman/omb_guide2.jsphttp://www.gartner.com/technology/about/ombudsman/omb_guide2.jsphttp://www.gartner.com/technology/about/policies/usage_guidelines.jsphttp://www.gartner.com/technology/about.jsp

improve your quality assuran 238115

Documents