inverting test pyramid - a first hand experience report

Inverting Test PyramidExperience Report

Sachin Natu @natusachinNaresh Jain @nashjain

25YEARS 6000+CLIENT

S

Nr 1M ROOMS 94COUNTRIES 95%

+ago we started

helping clients

manage their

revenue

later we’re still finding

the revenue opportunity

in every situation

around the world are

priced every single day

using IDeaS

on 6 continents have

hotels that now run on

IDeaS Revenue

Management

Systems

More than 95

percent of all

our clients are

still working

with us today.

About IDeaS

Our Worldwide Clients :

Hospitality + Travel + Distribution

Agenda

● Where were

we?

● Problem

● How we tried to

solve?

● How we

optimized?

Status of regression time

Page

No Automation !

Problems

● Thousands of

regression tests

● Cross cutting defects

found were more.

● Repeat Execution

Moving to shorter release cycle - a dream?

Regression period is incompressible

Manual and repetitive tasks are error prone.

Issue leakages

Page

Releases

Let’s automate regression

Page

UI Test

Page

After Automation Introduced

Production feedback

Page

Duration…

2

Yrs

Team of 2 full time Automation Testers

State of the Product

Manual

Checking 45%

End- to End UI Tests 40%

Integration Tests10%

Unit Tests5%

Always Catching Up

No pinpointed feedback

Silos

Tests are slow

Costly to maintain

Is there way out?

Page

Analysis of Bugs Report

Why are we testing from far?

Manual

Checking

End- to End Selenium Tests 70%

Integration Tests20%

Unit Tests10% Unit Tests 70%

Domain Logic Acceptance

Tests 10%

Integration Tests 9%

Workflow API

Tests 6%

End to

End Flow

Tests 4%

UI

1

%

Typical test strategies lead to an inverted testing pyramid

This is the need of the hour….Courtesy: Naresh Jain @AgileFAQs.com

The Test Pyramid

Our transition story…

• We decided to "Test things right , at right place on right time" which would result in "right test pyramid“

Unit Tests 70%

Domain Logic Acceptance

Tests 10%

Integration Tests 9%

Workflow API

Tests 6%

E to E

4%

UI

1%

Business logic test

Current Status

Benefits

Automation is part of development

• TDD

• BDD

• UI Tests only worry for UI

Improved collaboration

Developer QA

UI Tests – Grouping Functionality

80 Scenarios - 300 + Min!

Correct automation improved time.

80 Scenarios < Min!

Workflow - 4 Min

UI Tests - 4 Min

Regression timelines

Page

Months

Feels like Agile now...

Page

Challenges

• Legacy code - “Unit test unfriendly”

• Mapping acceptance tests with various slices of pyramid

• Building team competencies

• Pairing/ collaboration is backbone

Key learning

• Use UI automation only where really necessary

• It’s team responsibility

• Testability is important criteria.

Unit Tests 70%

Domain Logic Acceptance

Tests 10%

Integration Tests 9%

Workflow API

Tests 6%

End to

End Flow

Tests 4%

UI

1

%

Naresh Jain @nashjainSachin Natu @natusachin

Page

Some Statistics...