techmahindra - telcograde_testautomationsolution

8/6/2019 Techmahindra - TelcoGrade_TestAutomationSolution

http://slidepdf.com/reader/full/techmahindra-telcogradetestautomationsolution 1/9

© Tech Mahindra Limited 2011 © Tech Mahindra Limited 2010

Telco Grade – Test Automation SolutionProven & Cost Effective Approach

Author: Rajiv Ranjan

Feb 2011

Abstract:

Test Automation has been around but has not yet become part of the testing DNA for both Telco’sand Equipment Manufacturers. Tech Mahindra has been providing automation solution to both usingtools. It has also developed an in-house automation framework to provide a cost effective automationsolution. The learning’s of 200+ person years of experience has enabled Tech Mahindra to providevalue enhancing, vendor agnostic automation solutions.

This paper describes the approach needed to make automation a successful and cost effectivesolution.



1 © Tech Mahindra Limited 2011

Table of Contents

Executive Summary ......................................................................................................................... 2

Business Need .................................................................................................................................. 3

Network Automation Approach ................................................................................................. 4

Automation Assessment ............................................................................................................ 4

Arriving at the solution.............................................................................................................. 5

Design............................................................................................................................................ 6

Planning .......................................................................................................................................... 6

Benefit Measurement...................................................................................................................... 7

Conclusion ........................................................................................................................................ 8



2


Executive Summary

Automation is a much clichéd expression in today’s business environments. In the

networks area, automation has been notorious in soaking up expensive investments andproducing little in terms of the ballyhoo of the benefits promised. Telco’s and

communication equipment providers (CEP) therefore tread with extreme caution when the

‘A’ word is flaunted as the solution to all its problems. In the end automation in most

organizations ends up being nothing more than an ‘also-ran’ and in many cases a pariah

which is unlikely to be invoked in the testing scheme of things.

Tech Mahindra (TechM) understands this hesitation faced by the Telecom industry in

adopting automation as a viable approach in solving many of the latter’s Achilles heel.

This white paper intends to carefully describe the approach needed to make automation

succeed at Telco’s and CEP’s. Tech Mahindra’s assertion to this approach is drawn from

its expertise in providing bespoke and turnkey automation solutions to its Telecom

customers. Tech Mahindra has more than 200 person years of Test Automation

Experience which spans across various Testing tools, Network devices/equipments &

languages used for automating network test scenarios.

The stages of TechM’s automation approach covered as part of this wh itepaper include

assessment, solution formulation, and automation implementation strategy. The first stage

will probe into the need for automation assessment and will delve into TechM’s automation

methodology. The automation solution development stage will cover both the tool

evaluation & selection strategy and the automation design aspects. The planning stage will

examine the key aspects necessary to operationalize the automation plan. The narrative,

at each stage, is strengthened with examples from a successful engagement and will provide

legitimacy to the approach recommended through this white paper.

Tech Mahindra’s vendor and technology agnostic position allows it to provide value

enhancing automation solutions to its customers. Tech Mahindra, as part of the

assessment strategy, also evaluates, identifies & provides recommendations of the tools and

technologies that will bring in the desired benefits. The approach presented in this

whitepaper together with the right experience will enable unlocking some of the utopian

benefits promised by automation.

Less than 40% of Telco’s

and CEP’s use some kind

of automation currently

Less than 15% of them

have actually claimed

that automation has

provided tangible

benefits




Business Need

Telecom industry is highly dynamic both in terms of technology & competition. Changing

business needs & market compulsions make Telco’s and CEP’s to perform more and more

testing in order to ensure that the desired levels of quality and performance are met.

Time-to-market pressures often could mean deploying new features or devices into live

without adequate testing which invariable leads to increased support costs. These costs

associated with fixing problems from field are exponentially high when compared with

fixing them during pre-deployment stages. Yet, many a times Telco’s and CEP’s are forced

to cut corners during the test cycles in order to avoid missing out on bringing a product or

service to market.

In the absence of Test Automation the pre deployment testing is done manually. This

methodology has its own pros & cons. The human element involved in it brings its own

intelligence but also brings along its human limitation in terms of getting tired & bored

with repetitive work. This leads to a time consuming human incentive process which

comes out with a not-so-fully tested system which again is not-so-dependable.

Network test automation can help Telco’s and CEP’s to fasten the test cycle times

thereby enabling them to meet market opportunities while maintaining quality.

Regression tests are a ‘low-hanging fruit’ when it comes to selecting automation

candidates owing primarily to their repetitive nature and wider test coverage. Network

test automation can be used primarily for

Regression / repetitive tests for a specific test setup

Sanity Tests prior to upgrades or deployments

Stress tests to measure performance of the desired service, feature or

product

Interoperability tests to validate desired features or product behavior

1 in every 4 dollars of

R&D/Design spend goes

towards Testing

Over 45% of defects in

live is attributed to

seepage from testing

Automation could

potentially help plug this

defect seepage upto 90%



4


Network Automation Approach

While a lot of emphasis is seen around the execution aspects of automation, not much is

written or mentioned regarding the stages prior to execution, namely the assessment,

design & planning stages. Ironically, it is these pre-execution stages that govern the

effectiveness of the automation outcome. While there is no cookie-cutter approach to

assess and build automation solutions, a robust guiding framework will go a long way in

helping automation planners to achieve the desired benefits. The automation framework

described in the sections below follows a 3-step approach which can be applied by the

QA/Test teams to formulate their automation strategy.

A s s e s s m e n t - Existing Test

Plan & Test Bed

Analysis- Arrive at Pointsof Automation

- AssessmentReport

S o l u t

i o n- Identify

Automation Tool

- Identify clearlythe inclusions andExclusions

- Design thesolution using thetool

P l a n n i n g- Define roles and

responsibilities

- Setup testenvironment

- GenerateAutomation TestPlan

Fig 1. Automation Assessment Stages

Automation Assessment

Absence of an objective assessment before automation implementation leads to several

pitfalls. Limited understanding of the automation tools and the requirements leads to a half

baked approach resulting in a costly solution which fixes only a part of the problem. Fact

remains that automation is not the answer to all the functionalities which a Tester

performs. A detailed assessment is a must to evaluate the present state objectively and in

totality to arrive at the automation recommendations.

The assessment begins with the detailed analysis of the existing test plan & test case

documents to determine whether or not they can be automated (fully/partially automated).

Assessment involves understanding and defining the various elements of an existing system

like Base line Configuration, Interoperability, Failover, Capacity Validation, Security, Application

Test, System Upgrades, Regression Tests, New and Enhanced features, protocols etc.

In this stage, specific points of assessment (POA) are created, which are specific devices

and interfaces that currently require manual intervention and therefore need to be

assessed for automation. These points of assessment cover the three areas of test

operations: pre-execution test configuration, test case execution and test validation. If

Case Analysis

Customer Profile: Tier 1 USTelco

Business Need: E2Eautomation of its NextGeneration Mobile AccessGateway platform

Test Setup: Multiple network elements and platformsa. Network layer – RAN,

SGSN, GGSN, Cisco loadbalancer, load sharingNGG & MAG

b. Content – Web,application servers, IMgateways, MMSC, demosetup of third partyservices like MediaNet

c. Provisioning - MIPS+ R2,Radius Server, MINDLDAP subscriber database

d. User devices & trafficgenerators – 2.5G/3GHandsets, Laptop with 3Gdata card

Case Analysis

TechM identified 5 POA’s for theclient.




these points of assessment can be automated then these points are termed as points of

automation.

In addition the entire Test Bed setup is taken into consideration and evaluated. Test

Configurations are identified & analyzed for automation. Finally a summary report with a

breakup of fully automatable, partial and manual test case counts is prepared which then is

grouped logically into test categories that can be implemented as discrete test suites.

Finally, automation test case candidates are prioritized based on customer business

requirements.

Arriving at the solution

The next task, once the assessment of the current environment is complete, is to identify

the right automation tool that would fit the requirement. Three options are typically

available:

1. COTS (Commercial Off The Shelf) Tools

2. Open Source Tools

3. Bespoke or Custom Built Tools

All the options have their inherent advantages and disadvantages as shown in the table

below:

- Offers 'a l l i n o ne go ': S aves co s t & ti me - Li cens e & s u pp ort co sts a re exp en si ve

- Be tte r tra ce abi li ty a nd ma na ge abi li ty - Di ffi cul t to s ca le

- Availa bili ty of drivers & plugins for larger COTS - Integration chal lenges with other products

- Generally upto date with dependable quality - Unreliable support

- Cost effective - Tool roadmap not well defined

- Community contribution - Poor traceabil i ty and management functions

- Offers s ca la bi li ty & fl exi bi lity - Ti me cons umi ng tool devel opment cycl e

- Al lows better cost control - Robustness and stabil i ty i ssues

- Ta i l or ma de to fi t exa ct requ irements - S up po rt i s s ue on ce co re deve l op ers l ea ve

COTS Tools

Open

Source

Tools

Custom

Built Tools

Advantages Disadvantages

Fig 2. Automation Tool Analysis

Recommendation: Perform a PoC using the identified tool on a subset of automatable

features in order to validate the automation feasibility and the tool selection.

Once the right automation tool is identified, the solutioning aspect comes into play.

Besides the network or device under test, the automation environment variables need to

be chosen. These include elements such as simulators, protocol analyzers, platforms for

hosting automation suites & management applications etc. Inclusions & exclusion of the

automation are clearly stated and agreed upon. The Library procedures to be automated

Case Analysis

Tools EvaluatedHandset Automation Tools

1. AToM2. TestQuest3. Custom Application basedHandset Simulator

1. Silk Test2. Selenium (open source)3. Watir (open source)4. Custom simulators (three) Automation Platforms

1. Fanfare iTest2. TestShell

3. Quick Test Pro (QTP)

Case Analysis

Proposed Solution:

2-tiered automation solutionwas identified whereindividual automation tools,driving the POA and POS will

be centrally controlled by anautomation platform.



6


are identified & categorized as fully/partially automated. Response Maps (templates for

responses from the Network Elements & Test Equipment), Test Suites & Test cases are

identified & finalized during this stage.

DesignDesign considerations vary based upon the test tool selected for automation. It entails the

following:

Identification of all common procedures by examining existing test cases for

generic execution and validation steps

Building a library of common procedures

Maximizing design of analysis rules in the tool, instead of having this logic reside

on other automation elements

Ensuring that the test cases are test bed and user independent

Building modular test cases using library procedures, test beds, session profiles,

global and local variables

Design hierarchy bases test suites to allow flexibility

The Key design features of the automation solution should be

Reusability – generic & reusable library procedures reduce development time

Data Driven – input data stored in external data files eliminates need for hard

coding data in script

Variation Testing – variation testing for any number of pre-configured handset

user agents

Test Bed Portability – test bed data saved in separate configuration files for

easy script portability

Flexible Validation – user configured criterion for protocol & content validation

In addition, the design phase may involve exploring ways to integrate the automation

management system with the existing tools available in the organization for Test

Management, Defect Tracking & Reporting Tools etc.

Planning

Businesses that don't build a careful plan for automated testing may end up wasting

expensive resources: time, effort, and money. The automation planning phase defines




roles and responsibilities, project test schedule, test planning and design activities. The key

activities that should be part of this phase are:

Test environment setup

Preliminary test schedule

Define roles and responsibilities

Define performance measures

Identify risks and develop contingencies

Develop the automation acceptance criteria

As part of the test environment setup the test team schedules and track environment

activities. The planning phase should also involve installation and integration of test

environment hardware, software, and network resources. All these inputs are captured as

part of the automation test plan that will be used by the Test teams during the execution

phase. The test manager can use the plan to ensure that tests are executed according to

schedule, and test personnel are allocated and redirected when necessary to handle

problems that arise during the test execution process.

Recommendation: TechM suggests a phased approach to Test Automation

implementation to help progressively derive the benefits of Automation and also allowing

you to measure the benefits in a phased manner.

Benefit Measurement

The basic economic premise of network test automation is to automate if cost of test

automation and cost of automation execution is below cost of manual test execution. The

business case for automation will require that the benefits are measurable. A few

quantitative methods to measure automation benefits include :

Savings derived from cost of automating versus cost of manually executing test

cases in a given period of time

Savings derived from reduction in cycle time

Cost savings from avoidance of defects found in field

Case Analysis

Benefits

- Execution time per test

case reduced from 96 min

to 1.5 min

- Test Cycle time Reductionfrom 4 weeks to 8 hrs

- Enabled multiple device

testing as compared to

max of 5 device testing

earlier

Case Analysis

Planning

24 week plan covering- Dev & test

- Requirement analysis- Design of automation

test suits

- Coding and unit tests

- Perform integration and

acceptance tests



8

© Tech Mahindra Limited 2011© Tech Mahindra Limited 2011

Conclusion

Network test automation has the potential to significantly reduce OPEX of CEP’s and

Telco’s. Currently the network automation uptake in the telecom industry is patchy which

is attributed to inadequate assessment and planning. Detailed automation assessment in

combination with proper design and planning are a must to enable an efficient automation

strategy and to unlock the true potential of network test automation. The white paper

provides test planners with the framework required to make the right automation decision.

The white paper identified critical success factors needed for making network test

automation successful -

Shoring-up the right skills required to perform assessment of current Test

Methodology

Understanding the customers existing test management framework

Analysis & validation of test scripts & procedures- which all procedures can be

automated (if yes to what extent).

Identification of right Test Automation tool, which can interact with all

network Elements.

Designing flexibility in Test scripts, to test in the ever changing & varied

business requirements.

Understanding various interface aspects (of network elements, network test

tools, traffic simulators, existing defect and test management tools etc.).

This framework was validated by depicting instances where it was successfully applied and

the resulting benefits highlighted.

In the near future as testing in networks gets more complex, there will be an increasing

demand to improve test efficiencies and effectiveness in test cycles through automation.

The key to success here will be to adopt winning strategies realized through better

automation planning that will enable users to keep testing costs and quality under control.

How to reach us:

AmericasTech Mahindra (Americas), [email protected]: +1 972 991-2900

Europe

Tech Mahindra [email protected] : +44 01908 553400Fax : +44 01908 553499

Middle-east & AfricaTech Mahindra [email protected] : +002 02 5806608Fax : +002 02 5806601

Asia-PacificTech Mahindra Ltd.

[email protected]: +91 80 40243000Fax: +91 80 25527027

mailto:[email protected]








techmahindra - telcograde_testautomationsolution

Documents