An Automated approach towards Dev-Ops based

continuous Regression Testing for SAP Landscapes

Navaneetha Kowdle – Associate Director

Akshay Mullick - Consultant

Cognizant

Abstract

Executive Summary

Wedged between an increasingly convoluted digital landscape and the obligation to expedite

releases, the world around today’s QA Organization has been turbulent. QA Managers face a

daunting task of assuring complex business applications yet maintaining a pre-meditated

balance between quality and speed. This solution approach manuscript provides an

introduction to Business Process Change Analyser (BPCA) as an effective tool that increases

Test Coverage and Test Case Optimization for large and medium-sized SAP releases. The

document explores a two-phased approach towards a robust regression suite to perform BPCA

based impact analysis and analyses the benefits realized by one our major Customer.

Correlating Impact analysis to Business Process Change Analyser (BPCA)

The surge in organizations ‘going’ digital has augmented the complexity of the ERP landscape.

With a mission to deploy features at a shorter period, it made sense for SAP deployment

managers to consider a viable impact analysis tool during design and before the testing phase

(especially regression testing) to optimize test effort in addition to unearthing possible gaps in

coverage. Initially limited to increasing Test Scope Optimization (TSO) for SAP, the impact

analysis has now become a viable tool of choice for many SAP consultants and deployment

managers. Applying impact analysis during the design phase enables easy effort estimation for

change while a post development impact analysis aids in the identification of test cases that

were impacted by change.

Its primary focus is to identify affected modules in the system that were introduced through

new features or upgrades. By quickly identifying and focusing on regression testing of

impacted areas, impact analysis reduces regression cycle time, decreases cost and at the same

time ensures a focused approach to testing.

Business Process Change Analyser (BPCA) is an in-house tool provided by SAP that exposes

modules impacted due to modifications made through new features or change requests. It

helps compare the changes brought in through the transports with the current system and

identifies all the impacted areas (Objects). With the right configuration and a positive link

between the test management solution and the system, BPCA guides test scoping and reduces

the cost to testing by enabling businesses in identifying scenarios impacted by change. It

supports a wide range of areas of SAP modules including ECC, CRM, APO, SCM etc. BPCA also

has the advantage of being able to map impact changes to specific test scenarios. This is

necessary as sometimes it may be required to identify process and data variants to be tested

due to changes.

The Business Situation

The need for impact analysis began when one of our customers began investigating the list of

Urgent incidents that had an impact on their business. The analysis was conducted to

recognize Regression Testing Gaps that required attention despite the thoroughness and rigor

in the customer’s regression testing capability.

The analysis considered all the Urgent Incidents that were raised throughout the year 2014

against its IT Services. Based on these findings the customer wanted to:

1. Improve the quality of regression testing by increasing the granularity of transaction

coverage i.e. Identify & Test scenarios that can test object coverage instead of

transaction code

2. Increase Test Coverage by identifying gaps within the existing regression test suite and

designing new scenarios to plug the gaps

Considering the above, the customer worked collaboratively with cognizant to come up with a

viable solution. Post an exercise that compared multiple commercially available tools, it was

decided that BPCA will be the tool of choice to enable impact based regression testing.

Execution Approach

The deployment of BPCA was conducted in 2 distinct phases

Phase 1 – Set-up and Pre-requisites

BPCA relies on 2 key inputs for its analysis. A comprehensive Business Blueprint (BBP) –

solution snapshot and a Technical Bill of Materials (TBOMS) which is a list of objects assigned

to various Business Process in SAP. The quality of the analysis conducted and the ability of

BPCA to analyse is dependent on the breadth of the Business Blueprint and depth of the

TBOMS.

SAP automatically provides a means to build business blueprints and compile static or semi-

dynamic TBOMs. However, the maximum potential towards test scope optimization or test

coverage gap identification could only be realized if TBOMs generated from existing test assets

(Dynamic TBOMs) and the BBP is constructed as per the structure in the test management

tool.

Since the customer was pre-equipped with a robust set of artifacts and SAP TAO automated

scripts were readily made available, creating Dynamic TBOMs and structuring the BBP was

easier than starting from scratch. The Dynamic TBOM build went through a two stage process

-

a) Manually building a business blueprint that mirrored the E2E test scenario folder

structure in the test management tool (e.g. QC-ALM). BBP entries and test cases were

then mapped together which further allowed for impacted test scenarios to be

categorized on the basis of time and effort before the regression test cycle

b) Enable the Solution Manager to generate TBOMs while the test cases were being run

from the test management tool. This generated corresponding dynamic TBOMs for

each TCodes /Test scenario in the regression suite.

Phase 2 – Deployment and usage

To cater to the monthly release cadence that comprised of medium and major releases, the

BPCA tool was deployed to perform impact analysis. The tool empowered QA Managers to

focus on testing areas that were impacted by the changes, thereby reducing effort and cost.

Improving Test Scope Optimization by implementing BPCA

Conventionally, regression testing requires test scenarios in a particular landscape to be re-

tested – manually or through automation. However, the implementation of BPCA and the

results obtained through Impact Analysis empowered the customer to sort and rank test

scenarios as per preference.

Immediate effects of BPCA are illustrated in the diagram below:

BPCA analysis conducted over multiple monthly medium releases have indicated that the

effort and elapsed time savings in regression testing can be around 30 - 80%, depending how

BPCA becoming parameterized and on the risk-appetite of decision-makers.

Increasing Test Coverage for SAP Testing through BPCA

Although BPCA does not provide information on test coverage but provides data for TSO, the

analysis and data points that are made available supports the Testing team in recognizing the

adequacy/inadequacy in coverage for critical changes.

BPCA is executed on a set of transports that are involved in code or configuration changes that

could have impacted objects.

Dynamic TBOMs created by using the regression suite will have complete knowledge of objects

covered by the test scenarios – one might call it the “Coverage DNA” of a scenario. Therefore,

to increase the Test Coverage, the tangible way forward was to identify Objects/Scenarios that

weren’t covered by the TBOMs list but are present in the set of transports being analysed for

impact analysis. By comparing the list of known Objects with new Objects discovered in the

transports, new scenarios that covered the new objects were added to the regression suite and

this resulted in additional test coverage.

A comparison between the lists of objects changed by transports as against those provided by

TBOM analysis facilitates in the discovery of objects that aren’t covered by the regression test

suite. As Standard BPCA does not provide gap identification in the granularity level as

required by the client, we requested a SAP partner to develop an Add-On that fully utilizes the

granularity level as provided by TBOMs and also identify gaps in a more granular approach

than what BPCA provides in SAP standard.

Implementation of the above has shown an increase of an additional 5-10% unique test

scenarios qualifying as opposed to former baseline approach.

Phase 3 – Moving towards Dev-Ops based continuous regression testing

Implementation of BPCA on the SAP landscape and consistent usage opened up a host of

opportunities to take it to the next level. Traditionally SAP landscapes are complex and are not

a first choice candidate to consider for continuous testing. But the advantages BPCA provided

to implement focused regression testing pushed the team to consider a method to establish

just that. The result was a home grown automation tool that performs an End to End

automated regression testing on all changed transports with zero human intervention. The

tool integrates ChaRm, SolMan and HP-ALM to provide an end-to-end mechanism to identify

changed transports, perform impact analysis, identify regression scope and complete

automated regression testing daily. Not only has this helped the team identify defects earlier in

the lifecycle but the Regression Testing window that typically used to happen right at the end

of the release cycle over 1 week has been reduced to 2 days.

Conclusion

The implementation of BPCA and continuous testing has provided the customer with early

benefits such as reduced cycle time and increased test coverage by identifying optional

scenarios and new scenarios. And although the project continues to proceed as usual, it is

estimated that the customer stands to significantly gain through savings in efforts and cost.

Client’s comment:

After more than four years of executing Regression Testing in a so-called Baseline approach,

i.e. executing per release a fixed set of test scenarios mirroring our most critical business

processes, we started investigating into a more intelligent approach helping us to focus our test

execution on the change-impacted parts of the system and more importantly to strengthen the

RT capability regarding coverage.

Cognizant as our Testing Services provider was supporting us in market screening, in tool

evaluation, identifying our key demands and finally also in planning and implementing the

chosen toolset.

By end 2016, we have been successfully implementing BPCA in the European region whilst we

are still working on rollout to two more regions (AAR and Americas).

The benefits, especially the strengthening of our RT capability has turned out as successful for

Europe. Second stage in 2017 will be to leverage the focused RT capability and move into

continuous Regression Testing rather than it manifesting into a dedicated phase per release

cycle. This will become one key enabler for moving into a more agile Continuous Deployment

mode even for ERP Releases.

Note: Second stage as described the customer has already happened and is covered in the

paper above as “Phase 3 - Moving towards Dev-Ops

Client Testimonial from Dec 2016

References & Appendix

Author Biography

Navneetha Kowdle is an Associate Director with the Process &

Quality consulting division (a part of Cognizant Business Consulting

group). In this capacity, his role entails overall management of PQC

practice in the Communication, Media and Technology space, in

addition to personally getting engaged with customers in providing

consultancy services in niche areas of Quality Assurance such as

DevOps, Business Process Testing, QA transformation strategy etc

Akshay Mullick is working as a Consultant with the Process &

Quality consulting division (a part of Cognizant Business Consulting

group). My role is to act as a bridge between business and

technology for client and propose solutions which address their

problem statement. My area of expertise is Agile, Dev-Ops, Change

Management, and Tools & Automation around Impact based testing.

Main Author

Co - Author

THANK YOU!