inverting test pyramid - a first hand experience report
DESCRIPTION
Test automation is extremely crucial in adoption of an agile delivery. However, it can take one for a ride, if the approach is not correct. In this sensational, heart throbbing, experience report, we'll share our story of how we turned around an inefficient, expensive automation style to lean, efficient style. In addition to sharing a real-world example, we'll also share some of the key challenges we faced and how we solved them. If you are convinced about the Testing Pyramid, but are struggling to invert it, then this session is for you. Business Impact: * Earlier Defect Detection - Higher test coverage at Unit/Intermediate layers lead to earlier defect detection. Reduced number of issues found on higher test environments/Production. Reduced cost of defect fixing. * Reduced maintenance cost - UI tests are fragile and costlier to maintain Vs backend tests. No of changes in services layer are comparatively less. Reduced test execution time - Backend tests are much faster. Almost 7-10 times faster than UI Tests - improved build certification time. * Test feedbacks are naturally distributed across layers of application. Test feedbacks are more pin pointed/ granular. Details: http://confengine.com/agile-pune-2014/proposal/471/inverting-test-pyramid-a-first-hand-experience-report Conference: http://pune.agileindia.org/TRANSCRIPT
Inverting Test PyramidExperience Report
Sachin Natu @natusachinNaresh Jain @nashjain
25YEARS 6000+CLIENT
S
Nr 1M ROOMS 94COUNTRIES 95%
+ago we started
helping clients
manage their
revenue
later we’re still finding
the revenue opportunity
in every situation
around the world are
priced every single day
using IDeaS
on 6 continents have
hotels that now run on
IDeaS Revenue
Management
Systems
More than 95
percent of all
our clients are
still working
with us today.
About IDeaS
Our Worldwide Clients :
Hospitality + Travel + Distribution
Agenda
● Where were
we?
● Problem
● How we tried to
solve?
● How we
optimized?
Status of regression time
Page
No Automation !
Problems
● Thousands of
regression tests
● Cross cutting defects
found were more.
● Repeat Execution
Moving to shorter release cycle - a dream?
Regression period is incompressible
Manual and repetitive tasks are error prone.
Issue leakages
Page
Releases
Let’s automate regression
Page
UI Test
Page
After Automation Introduced
Production feedback
Page
Duration…
2
Yrs
Team of 2 full time Automation Testers
State of the Product
Manual
Checking 45%
End- to End UI Tests 40%
Integration Tests10%
Unit Tests5%
Always Catching Up
No pinpointed feedback
Silos
Tests are slow
Costly to maintain
Is there way out?
Page
Analysis of Bugs Report
Why are we testing from far?
Manual
Checking
End- to End Selenium Tests 70%
Integration Tests20%
Unit Tests10% Unit Tests 70%
Domain Logic Acceptance
Tests 10%
Integration Tests 9%
Workflow API
Tests 6%
End to
End Flow
Tests 4%
UI
1
%
Typical test strategies lead to an inverted testing pyramid
This is the need of the hour….Courtesy: Naresh Jain @AgileFAQs.com
The Test Pyramid
Our transition story…
• We decided to "Test things right , at right place on right time" which would result in "right test pyramid“
Unit Tests 70%
Domain Logic Acceptance
Tests 10%
Integration Tests 9%
Workflow API
Tests 6%
E to E
4%
UI
1%
Business logic test
Current Status
Benefits
Automation is part of development
• TDD
• BDD
• UI Tests only worry for UI
Improved collaboration
Developer QA
UI Tests – Grouping Functionality
80 Scenarios - 300 + Min!
Correct automation improved time.
80 Scenarios < Min!
Workflow - 4 Min
UI Tests - 4 Min
Regression timelines
Page
Months
Feels like Agile now...
Page
Challenges
• Legacy code - “Unit test unfriendly”
• Mapping acceptance tests with various slices of pyramid
• Building team competencies
• Pairing/ collaboration is backbone
Key learning
• Use UI automation only where really necessary
• It’s team responsibility
• Testability is important criteria.
Unit Tests 70%
Domain Logic Acceptance
Tests 10%
Integration Tests 9%
Workflow API
Tests 6%
End to
End Flow
Tests 4%
UI
1
%
Naresh Jain @nashjainSachin Natu @natusachin
Page
Some Statistics...