building test automations the agile way
DESCRIPTION
An alternate approach towards building Test Automations for the Agile development environments. Traditional Test Automation approach isn't a great fit in fast pace dynamic ecosystem, hence there exists a dire need for an alternate approach. This is very well suited to an e-commerce platform, and can be easily extended to similar environments.TRANSCRIPT
Bringing Agility in Test Automations
Hemant Kumar JaggiSenior Manager, MakeMyTrip.com
LET’S BE
AGILE !
Heard this before?
Thoughts on test automations the “Agile” way
To keep up the pace with agile application development
Traditional Test Automation approach isn’t working
Best practices must be incorporated everywhere
Ideally suited to an Ecommerce (or similar) environment
An Ecommerce looks like
Only thing that doesn’t change is “change”
Building Test Automations the “Agile” way
WHY?
BECAUSE TRADITIONAL
AUTOMATION APPROACH JUST DIDN’T
WORK FOR US
WHY? Challenges we face @ MakeMyTrip
Fast PaceReleases in a week?Monthly unique visitors?Monthly page views?
High Maintenance Cost & Test cases not always in “shape”
Challenges (contd.)
Complex Business RulesHotel promotion combinations?Cheapest Flights search?
Multiple GDS connections
Hotel Search engine?~10,500 Domestic Hotels~250,000 hotels across world
It’s impossible to reverse-engineer!
Challenges (contd.)
Execution issues:Automation execution may take 2-3
days to completeGolden data set takes ~7 days to
buildAutomation teams don’t know
domainFinally .. W
e don’t
trust
Autom
ation c
ode
Life of a QA Engineer is like
WHAT?
WHAT it should be?
Faster time to develop Automation
LOW maintenance costAgnostic to business logic“QA team” friendlyLesser complexityFaster execution time
“It just works ! ”
HOW?
HOW? In lines with Agile Manifesto..
Break into smaller storiesPrioritize them basis
Functional ImportanceFrequency of executionsDependencies
Work “with” QA teamDevelop a story and deliver
Create Buckets
Sanity testsEnd to end user flowsRegression tests*
Miscellaneous flows
Sanity Test
Breadth wise test coverage.MATS – Minimal Acceptance Test
SuiteRelease checklistDetermines whether build is
stable.
User Flows
Covers end to end user flowsDetailed breadth wise coverageE.g. different combinations of
booking flows.
But how to simplify regression tests?
Break them further ..
Into smaller flows, e.g. page wise test suites
Comparative TestingInteractive Testing
Comparative Testing
Comparative Testing
Works on two builds:“QA” build (i.e. build under test)“Production” build
Deploy them in same environment.
Read all ‘to-be-tested’ values from front end
Comparative Testing
Execute automation script on both builds
Dump them into XML (or excel)Excel template has formulae to
compare values from two buildsOr use an XML comparator (in
case of XML output format)
Comparative Testing
IF (Actual == Expected) THEN PASS
ELSE FAIL* Actual Value = value(QA build)* Expected Value = value(live build)
Take XML(or excel) as input to reporting framework, generate HTML reports
Let’s discuss
Pros & Cons of
Comparative Testing
Advantages of Comparative Testing
Faster turn around to developAutomation script does just what it is
supposed to doAutomation becomes a “dumb”
executionVery LOW maintenance costKeep it simple
Advantages of Comparative Testing
Agnostic to business layersChanges in business rules don’t
matter
High level of accuracyMakes it “thorough”Enhances confidence in QA team
Advantages of Comparative Testing
Overall progress is fasterMore dependency on Test Scenarios
than Test CasesLess/No domain knowledge required
for Automation team“Win-Win” for Automation and QA
team
Disadvantages of Comparative Testing
Any “live” issues won’t be caught
Execution requiresEither double the time (if sequential)Or double the resources (if parallel)
Interactive Testing
Interactive Testing
Complements comparative testing
Features that require user interaction
E.g. Filters, Sorting etc.
Comparative + Interactive equals “Regression”
Automation Coverage Tracking
Assign weights to each bucket, say
Automation Score = sum([(Tn/An)*100]*[Wn/(W1+W2+W3+W4+W5)]
Tn=Total Automated TC’s for a bucketAn=Total Automation feasible TC’s for a bucketWn=Weighted score of a bucket
Automation Coverage Tracking (Sample)
Computing efficiency gains (Sample)
Take Away
The “Agile” way of building Test Automations
We resolved “challenges” of traditional automation approach
Simplifying regressionRegression=Comparative + Interactive
Its working @ MakeMyTrip
