Open Source Tool Based
Automation solution with Continuous
Integration and end to end BDD
Implementation
Arun Krishnan, Automation Manager Maria Afzal, Automation Test Analyst
Cognizant Technology Solutions
Abstract
Until October 2014 a top US bank customer was following a 3 month release cycle and in the proposed agile model, the release window was planned to reduce to 6 weeks and with a further scope to optimize the release every 4 weeks. The only way to accommodate this change was to bring about a path breaking change in the automation landscape by implementing lifecycle automation in every possible avenue. In order to achieve this, QA team implemented Behavioral Driven Development (BDD) with Open Source Tool stack of Ruby, Watir and Cucumber to embrace agile and improve Speed to Market. Implementation of Continuous Integration(CI) with Jenkins for Unattended Execution helped to achieve Speed to Market, the team was able to reduce regression cycle time by 80% and deliver zero defect leakage in production.
Challenges
Frequent application deployment changes under stringent
timelines
High defect leakage to UAT/Production
100K+ test runs to be executed per quarterly release
Minimal automation coverage
Speed to Market
Isolated teams (BA,Dev & QA)
Our Solution –
Traditional Automation
Until September 2011, 700+ Legacy scripts designed in Ruby,
WATiR & Cucumber setup. Client not only wanted to fix them &
own it but also wanted to enhance them by doing Gap analysis.
With the new BDD Methodology along with agile practice, client
wanted to create a Regression pack of 6 UI applications as well
as Service layer application
As shown in (Fig-1) the model follows a 3 month release cycle
and in the agile model multiple iterations are covered. Manual
testing commences from 1st Iteration while Automation scripts
can be designed as soon as the design phase begins.
Regression Automation was conducted for all technologies for UI
application, SOA test and ETL/Database applications
Smoke suite for all the applications was formed containing all the
important scenarios and covering all the possible business flows.
Smoke Automation was done for Build verification automation,
such that any change in build can be captured at its earliest, and
depending on the build result , analysis can be done to execute
all scripts or not.
Specialized Automation
As shown in (Fig-1), BDD in Iteration Automation: Implement
Automation in Iteration using BDD concept. Business and IT are
not aligned all the times. Requirements often interpreted
differently by IT and time is lost in back-forth communications.
This issue is resolved by implementing BDD Agile automation
approach, where manual testing is minimized and automated
tests developed in parallel with application development, till date
which has resulted in execution of over 680,000 automated test
iterations, while identifying 1259 defects
Continuous automation and execution of suite of 125000+ UI
scenarios, 192 web services and 46 automated ETL workflows
for each release resulting in faster automation execution using
data independent scripts and Service layer Automation.
Reducing SIT cycle by providing more test coverage in BDD
Suite in each release cycle helps to detect any old or new
defects.
Advanced Automation
Test Design Automation
As shown in (Fig-1), reuse of BDD Objects in SIT automation to
save the time and effort to identify new objects and prevent
redundancy of objects. Shared object repository created for
Ruby framework such that it can be reusable by all QA Testers
QA has developed a tool ‘Excel-to-Feature Converter’ which
automatically converts test case written in excel to feature files
which saves effort to explicitly write feature. These feature files
can be shared with business to maintain quality.
Once the feature is ready, step definitions can be easily written
using shared objects repository and parallel automation of new
tests is done in same iteration and executed in same release
and finding bugs early saves effort and time. Execution of BDD
Suite on Code Check in Headless mode results in reduction in
defect leakage in UAT/Production from 10% to less than 1%
Continuous Integration
As shown in (Fig-1), Implementation of Continuous Integration
(CI) for unattended execution along with improved leverage in
SIT and UAT environment. To ensure low defect density and
effective functional testing, smoke test suite is required to be
executed after every build deployment
Execution report is shared immediately with developer teams on
the quality, functionality, or system-wide impact of build.
Full regression suite execution is also required multiple times
within short release execution window. To provide immediate
feedback post build deployment, Cognizant proposed to extend
Jenkins setup to test team for continuous integration of
Automation smoke and regression suite.
Jenkins machines are configured to fetch latest test
automation scripts (Cucumber) and object repository from
SVN subversion as per latest build path
Daily smoke test and planned regression execution is also
configured to auto execute through Jenkins
Periodic executions are also scheduled from Jenkins
configuration for planned regression runs
Module with dependency to run on specific machine are
configured to be executed on same machine
Process Automation
As shown in (Fig-1), achieved effective Continuous Integration -
Jenkins to trigger automation suite after every build deployment
in an unmonitored way – 1000+ scenarios executed regularly.
With executions happening over Jenkin machines, improved
local machine and automation tester time utilization in
maintenance and new scripting
Execution report is shared immediately with developer teams on
the quality, functionality, or system-wide impact of
build.Centralized result logs over Jenkin (latest, historical logs),
also auto email is triggered to notify failure or completion of
execution to dev teams
How BDD Works-
As shown in Fig. 2
Initiation Phase:
Requirement Gathering- BA gathers all the requirements
from Client
Business Epics – BA writes all the requirements in the form
of Business Epics in VersionOne
Story Test Writing – BA & QA writes story tests in Gherkin
format (using Given, When, Then) in QA stories in
VersionOne
Test Scenarios Review - Review will be held by Business
Team on the basis of story tests written in QA story on
VersionOne
Scripting & Implementation:
Automation Script Development – Complete Automation of
the story tests using Excel Input to BDD Feature Converter
(Cognizant Value Add)
Development & Deployment – Story wise Development
completion by developers / deployment of code in Testing
environment & parallel Automation Test Execution
Use of Object Repository to create/store reusable functions
Defect Triage – Defects raised, resolved and retested
Customized test results in HTML
Analysis of generated test execution report
Sign Off the Story – All defects resolved, all story tests
passed in Automation
Continuous Integration with Jenkins to trigger automation
suite after every build deployment in an unmonitored way
UAT Testing – Testing of new functionality in UAT
environment and identification of Gap JIRAs (UAT JIRAs
raised for the functionality remained as a GAP i.e. not
covered in Regression)
BDD Technical Implementation-
As shown in Fig. 3
Feature File:
Features – a text file with a .feature extension, house the
tests. Feature files are written in simple language i.e.
Gherkin which is easy to understand by QA, business and
developers and can be shared across.
Writing of Test Scenarios in Excel Sheet using Auto
Suggest is done to avoid the redundancy & increase test
reusability
Auto conversion of Excel sheet into feature file using Excel
to Cucumber tool
Features are made up of Scenarios – scenarios are
synonymous with a user story or test case(s) i.e. Behaviour
The Scenarios are made up of Individual Steps – Steps are
the component interactions (or Test Steps) for a given
Scenario or Test Case(s), i.e. the given... When... Then
description
Step Definitions:
The Step Definitions essentially convert the simple English
“Steps” into code that can be executed to perform the step.
Shared object repository created for Ruby framework helps
prevent redundancy of objects and ease the process of
object identification
Support Code will be executed to perform the specific Step
in the Feature file
WATiR & Selenium are all examples of pre-built Support
Libraries that can be used in conjunction with Cucumber to
conduct Automated tests
Jenkins
To provide immediate feedback post build deployment,
Cognizant proposed to extend Jenkins setup to test team
for continuous integration of Automation smoke and
regression suite
Tools involved: Jenkins (CI orchestration), Cucumber
(Gherkins for Behaviours), WATiR (execution engine),
Ruby (Scripting Lang), SVN (Version)
WMQA Jenkins setup includes master - slaves
configuration with provision to configure additional slaves
based on execution requirement
Jenkins machines are configured to fetch latest test
automation scripts (Cucumber) and object repository from
SVN subversion as per latest build path
Daily smoke test and planned regression execution is also
configured to auto execute through Jenkins
Periodic executions are also scheduled from Jenkins
configuration for planned regression runs
Module with dependency to run on specific machine are
configured to be executed on same machine
Benefits
• 350 + Business Scenarios automated
• 80% Reduction of in-sprint testing time
• 30% defect detected via BDD automation
• 300% Increase in test case execution count post BDD
implementation
• > 30% reduction in overall team size over a year
• 0% defect leakage and improved test coverage
References & Appendix
Reference
Topic Link
BDD
https://en.wikipedia.org/wiki/Behavior-
driven_development
http://behaviourdriven.org/
https://github.com/cucumber/cucumber/wiki/Gherkin
Fig-1 Our Solution
Fig-2 How BDD Happens?
Fig.3 BDD Technical Implementation
Author Biography
Arun Krishnan is an automation manager with more than 11 years of extensive experience working with automation tools, such as HP UFT, Selenium, Cucumber, Watir, SeeTest, Perfecto Mobile, and Mobile Labs. His core expertise is in functional and mobile automation with traditional and nontraditional tools and techniques. Engaged with customer teams to design, implement, and deliver solutions that address specific pain points. Maria Afzal is an automation expert with more than 3 years of experience in automation testing. Maria has extensive experience on key automation tools and platforms and has worked on Hybrid framework for Selenium,Ruby. Maria has expertise in developing automation framework for multiple technologies like Web, and Mobile and has worked on NextGen methodologies like BDD, CI and Agile Maria is currently working with a leading US banking client
THANK YOU!