Open Source Tool Based

Automation solution with Continuous

Integration and end to end BDD

Implementation

Arun Krishnan, Automation Manager Maria Afzal, Automation Test Analyst

Cognizant Technology Solutions

Abstract

Until October 2014 a top US bank customer was following a 3 month release cycle and in the proposed agile model, the release window was planned to reduce to 6 weeks and with a further scope to optimize the release every 4 weeks. The only way to accommodate this change was to bring about a path breaking change in the automation landscape by implementing lifecycle automation in every possible avenue. In order to achieve this, QA team implemented Behavioral Driven Development (BDD) with Open Source Tool stack of Ruby, Watir and Cucumber to embrace agile and improve Speed to Market. Implementation of Continuous Integration(CI) with Jenkins for Unattended Execution helped to achieve Speed to Market, the team was able to reduce regression cycle time by 80% and deliver zero defect leakage in production.

Challenges

Frequent application deployment changes under stringent

timelines

High defect leakage to UAT/Production

100K+ test runs to be executed per quarterly release

Minimal automation coverage

Speed to Market

Isolated teams (BA,Dev & QA)

Our Solution –

Traditional Automation

Until September 2011, 700+ Legacy scripts designed in Ruby,

WATiR & Cucumber setup. Client not only wanted to fix them &

own it but also wanted to enhance them by doing Gap analysis.

With the new BDD Methodology along with agile practice, client

wanted to create a Regression pack of 6 UI applications as well

as Service layer application

As shown in (Fig-1) the model follows a 3 month release cycle

and in the agile model multiple iterations are covered. Manual

testing commences from 1st Iteration while Automation scripts

can be designed as soon as the design phase begins.

Regression Automation was conducted for all technologies for UI

application, SOA test and ETL/Database applications

Smoke suite for all the applications was formed containing all the

important scenarios and covering all the possible business flows.

Smoke Automation was done for Build verification automation,

such that any change in build can be captured at its earliest, and

depending on the build result , analysis can be done to execute

all scripts or not.

Specialized Automation

As shown in (Fig-1), BDD in Iteration Automation: Implement

Automation in Iteration using BDD concept. Business and IT are

not aligned all the times. Requirements often interpreted

differently by IT and time is lost in back-forth communications.

This issue is resolved by implementing BDD Agile automation

approach, where manual testing is minimized and automated

tests developed in parallel with application development, till date

which has resulted in execution of over 680,000 automated test

iterations, while identifying 1259 defects

Continuous automation and execution of suite of 125000+ UI

scenarios, 192 web services and 46 automated ETL workflows

for each release resulting in faster automation execution using

data independent scripts and Service layer Automation.

Reducing SIT cycle by providing more test coverage in BDD

Suite in each release cycle helps to detect any old or new

defects.

Advanced Automation

Test Design Automation

As shown in (Fig-1), reuse of BDD Objects in SIT automation to

save the time and effort to identify new objects and prevent

redundancy of objects. Shared object repository created for

Ruby framework such that it can be reusable by all QA Testers

QA has developed a tool ‘Excel-to-Feature Converter’ which

automatically converts test case written in excel to feature files

which saves effort to explicitly write feature. These feature files

can be shared with business to maintain quality.

Once the feature is ready, step definitions can be easily written

using shared objects repository and parallel automation of new

tests is done in same iteration and executed in same release

and finding bugs early saves effort and time. Execution of BDD

Suite on Code Check in Headless mode results in reduction in

defect leakage in UAT/Production from 10% to less than 1%

Continuous Integration

As shown in (Fig-1), Implementation of Continuous Integration

(CI) for unattended execution along with improved leverage in

SIT and UAT environment. To ensure low defect density and

effective functional testing, smoke test suite is required to be

executed after every build deployment

Execution report is shared immediately with developer teams on

the quality, functionality, or system-wide impact of build.

Full regression suite execution is also required multiple times

within short release execution window. To provide immediate

feedback post build deployment, Cognizant proposed to extend

Jenkins setup to test team for continuous integration of

Automation smoke and regression suite.

Jenkins machines are configured to fetch latest test

automation scripts (Cucumber) and object repository from

SVN subversion as per latest build path

Daily smoke test and planned regression execution is also

configured to auto execute through Jenkins

Periodic executions are also scheduled from Jenkins

configuration for planned regression runs

Module with dependency to run on specific machine are

configured to be executed on same machine

Process Automation

As shown in (Fig-1), achieved effective Continuous Integration -

Jenkins to trigger automation suite after every build deployment

in an unmonitored way – 1000+ scenarios executed regularly.

With executions happening over Jenkin machines, improved

local machine and automation tester time utilization in

maintenance and new scripting

Execution report is shared immediately with developer teams on

the quality, functionality, or system-wide impact of

build.Centralized result logs over Jenkin (latest, historical logs),

also auto email is triggered to notify failure or completion of

execution to dev teams

How BDD Works-

As shown in Fig. 2

Initiation Phase:

Requirement Gathering- BA gathers all the requirements

from Client

Business Epics – BA writes all the requirements in the form

of Business Epics in VersionOne

Story Test Writing – BA & QA writes story tests in Gherkin

format (using Given, When, Then) in QA stories in

VersionOne

Test Scenarios Review - Review will be held by Business

Team on the basis of story tests written in QA story on

VersionOne

Scripting & Implementation:

Automation Script Development – Complete Automation of

the story tests using Excel Input to BDD Feature Converter

(Cognizant Value Add)

Development & Deployment – Story wise Development

completion by developers / deployment of code in Testing

environment & parallel Automation Test Execution

Use of Object Repository to create/store reusable functions

Defect Triage – Defects raised, resolved and retested

Customized test results in HTML

Analysis of generated test execution report

Sign Off the Story – All defects resolved, all story tests

passed in Automation

Continuous Integration with Jenkins to trigger automation

suite after every build deployment in an unmonitored way

UAT Testing – Testing of new functionality in UAT

environment and identification of Gap JIRAs (UAT JIRAs

raised for the functionality remained as a GAP i.e. not

covered in Regression)

BDD Technical Implementation-

As shown in Fig. 3

Feature File:

Features – a text file with a .feature extension, house the

tests. Feature files are written in simple language i.e.

Gherkin which is easy to understand by QA, business and

developers and can be shared across.

Writing of Test Scenarios in Excel Sheet using Auto

Suggest is done to avoid the redundancy & increase test

reusability

Auto conversion of Excel sheet into feature file using Excel

to Cucumber tool

Features are made up of Scenarios – scenarios are

synonymous with a user story or test case(s) i.e. Behaviour

The Scenarios are made up of Individual Steps – Steps are

the component interactions (or Test Steps) for a given

Scenario or Test Case(s), i.e. the given... When... Then

description

Step Definitions:

The Step Definitions essentially convert the simple English

“Steps” into code that can be executed to perform the step.

Shared object repository created for Ruby framework helps

prevent redundancy of objects and ease the process of

object identification

Support Code will be executed to perform the specific Step

in the Feature file

WATiR & Selenium are all examples of pre-built Support

Libraries that can be used in conjunction with Cucumber to

conduct Automated tests

Jenkins

To provide immediate feedback post build deployment,

Cognizant proposed to extend Jenkins setup to test team

for continuous integration of Automation smoke and

regression suite

Tools involved: Jenkins (CI orchestration), Cucumber

(Gherkins for Behaviours), WATiR (execution engine),

Ruby (Scripting Lang), SVN (Version)

WMQA Jenkins setup includes master - slaves

configuration with provision to configure additional slaves

based on execution requirement

Jenkins machines are configured to fetch latest test

automation scripts (Cucumber) and object repository from

SVN subversion as per latest build path

Daily smoke test and planned regression execution is also

configured to auto execute through Jenkins

Periodic executions are also scheduled from Jenkins

configuration for planned regression runs

Module with dependency to run on specific machine are

configured to be executed on same machine

Benefits

• 350 + Business Scenarios automated

• 80% Reduction of in-sprint testing time

• 30% defect detected via BDD automation

• 300% Increase in test case execution count post BDD

implementation

• > 30% reduction in overall team size over a year

• 0% defect leakage and improved test coverage

References & Appendix

Reference

Topic Link

BDD

https://en.wikipedia.org/wiki/Behavior-

driven_development

http://behaviourdriven.org/

https://github.com/cucumber/cucumber/wiki/Gherkin

Fig-1 Our Solution

Fig-2 How BDD Happens?

Fig.3 BDD Technical Implementation

Author Biography

Arun Krishnan is an automation manager with more than 11 years of extensive experience working with automation tools, such as HP UFT, Selenium, Cucumber, Watir, SeeTest, Perfecto Mobile, and Mobile Labs. His core expertise is in functional and mobile automation with traditional and nontraditional tools and techniques. Engaged with customer teams to design, implement, and deliver solutions that address specific pain points. Maria Afzal is an automation expert with more than 3 years of experience in automation testing. Maria has extensive experience on key automation tools and platforms and has worked on Hybrid framework for Selenium,Ruby. Maria has expertise in developing automation framework for multiple technologies like Web, and Mobile and has worked on NextGen methodologies like BDD, CI and Agile Maria is currently working with a leading US banking client

THANK YOU!