Achieving the Right Testing Pyramid

Background

5%

95% No Automation

Automation Coverage

Background

1.5 Months4.5 Months

0 1 2 3 4 5 6 7

Months

Typical Release Cycle

Development Regression

Background

Critical Issues reported from Production

0

2

4

6

8

10

12

Critical Data Issues Cosmetic Localization

Issues

Background

After Automation Introduced

Selenium Tests 45%

Unit & Integration Tests 20%

No Automation 35%

Automation Coverage

After Automation Introduced

2.5 Months 3 Weeks

0 0.5 1 1.5 2 2.5 3 3.5

Months

Typical Release Cycle

Development Regression

Critical Issues reported from Production

0

2

4

6

8

10

12

Critical Data Issues Cosmetic Localization

After automation

Before automation

State of the Product

Tightly coupled with UI

Browser Dependent Data tests

False fails

Difficult to pinpoint real culprit

Difficult to maintain and time consuming as test are fragile

Lagging with current release development

Image Courtesy: Alex Yakyma

State of the Product

Manual Checking

End- to End Selenium Tests 70%

Integration Tests20%

Unit Tests10%

Analysis of Bugs Report

10%

90% Data Failures

Bug Report Analysis

UI Navigation Failures

Our transition story…

• We decided to "Test things right , at right place on right time" which would result in "right testing pyramid“

Our transition story…

Application UI

Service Layer

DAO Layer

Automated UI Tests

Tightly Coupled

Fragile

Lag with current development

Automate One Level Below UI

UI Independent

Robust & Stable

With current dev cycle

Fast Execution

Introduced BDD with Cucumber

Current Status

Selenium Tests 10%

Workflow Tests: 40%

No Automation 20%

Unit & Integration Tests 30%

Test Automation Status

After Right Testing Pyramid

1 Week2.75 Months

0 0.05 0.1 0.15 0.2 0.25 0.3

0 0.5 1 1.5 2 2.5 3 3.5

Months

Typical Release Cycle

Regression Time Development

Pain points

• Learning curve for the team

• Difficult to implement in case where no API is exposed or tight coupling with UI

• Heavy dependency on pre-populated database

Looking ahead

• Expose RESTful APIs for all upcoming features

• Move BDD specs to lower layers

• Moving away from baseline database