build the ‘right’ regression suite using behavior driven testing (bdt)
TRANSCRIPT
BUILD THE “RIGHT” REGRESSION SUITE USING BEHAVIOR DRIVEN TESTING (BDT) vodQA
WHAT DO YOU EXPECT FROM THIS SESSION?
CASE STUDY
• Multiple, long running projects
• Legacy applications
• Integrated
• Limited / flaky / long-running Automation
• Long regression cycle
• Huge cost of fixing defects
FUNCTIONAL TESTING CHALLENGES
• Brittle
• Little / less value
• No visibility into what is tested
• Maintenance nightmare
• Outdated
• Expensive
AS A RESULT
• Trust deficit on the team • In existing Automation
• In manual testing
• Finger pointing / blame game
BIGGEST PROBLEM
My Case Study
Concept #1
Given -> When -> Then
Expected Functionality:
The customer should be able to withdraw money from his account via an ATM machine
BDD Example:
Given the account has balance of 5000
When the customer requests 1000
Then the account is debited by 1000
13
Workshop Begins – Iteration 1
Case Study
PHASE 1 REQUIREMENTS
• Guest User is able to search for a flight for a single traveller
• Select option from Search Results page
• Specify contact information for person booking the flight
A Guest User can …
WORKFLOW
Search Search Results
Contact Information
SEARCH MODULE
SEARCH RESULTS
CONTACT DETAILS
CONTACT DETAILS
PHASE 1 REQUIREMENTS
• Guest User is able to search for a flight for a single traveller
• Select option from Search Results page
• Specify contact information for person booking the flight
A Guest User can …
WORKFLOW
Search Search Results
Contact Information
LETS GET STARTED … WRITE 2 TEST SCENARIOS
Iteration #1
Showcase
Concept #2
Test Specification Styles
• Imperative
• Declarative
TEST SPECIFICATION STYLES
A Guest User can …
AN EXAMPLE
Search Search Results
Contact Information
IMPERATIVE STYLE
Given I am a guest user on the home page
And I choose “round” trip option
And I select “Chicago” from the origin dropdown
And I select “San Francisco” from the destination dropdown
And I select departure date as “5 December 2013”
And I select returning date as “25 December 2013”
When I click on Search
Then I should see the search results page
And I should see at least 1 option for my criteria
...
IMPERATIVE STYLE …
…
When I select the “first” option
Then I am on the Contact Information page
When I enter first name as “Anand”
And I enter last name as “Bagmar”
…
…
And I click the “Select and Continue” button
Then I should be on the next page
DECLARATIVE STYLE
Given I am a guest user
When I search for flight options for a “one-way” trip for “1” “Adult” from “Chicago” to “San Francisco”
And I select the “first” flight
And I enter “valid” contact details for “traveller1”
Then I am able to Save and Continue
• Ben Mabey - http://benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories.html
• Alister Scott - http://watirmelon.com/2010/12/10/cucumber-imperative-or-declarative-that-is-the-question/
REFERENCES
Workshop – Iteration 2
Case Study Continues
• User registration from Contact information page
• Registered user can login from home page (below search criteria)
• For logged-in user, when booking ticket, contact information is pre-filled with logged-in user information
PHASE 2 REQUIREMENTS
Login Search Search Results
Contact Information
• Pre-filled
WORKFLOW … ENHANCED
REGISTERED USER
• User registration from Contact information page
• Registered user can login from home page (below search criteria)
• For logged-in user, when booking ticket, contact information is pre-filled with logged-in user information
PHASE 2 REQUIREMENTS
Login Search Search Results
Contact Information
• Pre-filled
WORKFLOW … ENHANCED
LETS GET STARTED … WRITE 2 TEST SCENARIOS
Iteration #2
Showcase
WHY IS THIS IMPORTANT?
Concept #3
The Test Pyramid
AUTOMATION – A SAFETY NET
• “Succeeding with Agile” – Mike Cohn
• Martin Fowler – Test Pyramid
martinfowler.com/bliki/TestPyramid.html
THE TEST PYRAMID
Time Value Cost
IDEAL TEST AUTOMATION PYRAMID
Manual / Exploratory
UI
Web Service
View
JavaScript
Integration
Unit
Technology-facing Tests
Business-facing Tests
REALITY
UI
Web Service
View
JavaScript
Integration
Unit
Manual / Exploratory
ICE CREAM CONE ANTI PATTERN
Developers Test Pyramid QA Team Test Pyramid
Web Service
View
JavaScript
Integration
Unit
Manual / Exploratory
UI
Web Service
DUAL TEST PYRAMID ANTI-PATTERN
Concept #4
Behavior Driven Testing (BDT)
BEHAVIOR DRIVEN TESTING (BDT)
• How is this functionality going to be used by the end-user?
• What is their thought process going to be when using this product?
• What “core-business-value” should be delivered to the end-user?
THE BIG PICTURE
Manual / Exploratory
UI
Web Service
View
JavaScript
Integration
Unit
Technology-facing Tests
Business-facing Tests
Identify the “right” type of tests that sit on top of the Test Pyramid
My Case Study
HOW TO SOLVE THE PROBLEM?
• Remove the ambiguity
• Same visibility for all
SOLUTION
• Whiteboards
• Mind maps
• Flow charts
• Collaboration!!
TOOLS TO ENABLE BDT
HOW DID WE ACCOMPLISH THIS?
IDENTIFY BUSINESS FLOWS
• Identify business flows based on personas, and how the personas use the business functionality, or,
• Identify business flows, and how different user personas can use the business flows
NOTE: Business flows are different from UI flows
HOW TO IDENTIFY BUSINESS FLOWS ?
PRIORITIZE BUSINESS FLOWS
1st candidate • Automate! • High Value, Low Cost
2nd candidate • Automate! • High Value, High Cost
Can Automate? • Manual Regression? • Low Value, Low Cost
Do Not Automate • Manual Regression! • Low Value, High Cost
High Value
High Cost
Low Value
Low Cost
AUTOMATE THE TOP LAYER USING
• BDD Tools, or,
• Directly in any appropriate programming language / tools
• Identification of Tests
• Automate it!
WHAT’S NEXT?
AUTOMATE TESTS THAT ARE VALUABLE, NOT BECAUSE YOU CAN OR LIKE TO!
CAUTION!
Who is doing the action?
What is the business functionality?
Depicts a user flow / user journey
Actions & Verifications happen implicitly
Given auctioneer creates an auction And 5 vehicles are added to the auction When auctioneer starts the auction And auctioneer sets a starting bid of 5000 And buyer_1 bids And buyer_2 bids Then auctioneer Sells the vehicle And buyer_2 wins the vehicle
EXAMPLE
Delete profile
UPDATE BUSINESS FLOWS
1st candidate • Automate! • High Value, Low Cost
2nd candidate • Automate! • High Value, High Cost
Can Automate? • Manual Regression? • Low Value, Low Cost
Do Not Automate • Manual Regression! • Low Value, High Cost
High Value
High Cost
Low Value
Low Cost
Project Management tool Mingle
Test case repository (manual + automated)
Cucumber .feature files with appropriate tagging
Feature files / Scenarios 33 / 65
# of Manual Tests 0
Tests running on CI (Jenkins) Since the 1st test was automated
Smoke tests execution time 15 minutes
Regression tests execution time
45 minutes
FINAL STATE … IN MY CASE STUDY
• Builds and deployments were automated
• Tests ran on demand, automatically! • Smoke / Regression / Javascript / Unit
• Functional Performance
• Concurrent Testing
ADDITIONALLY …
How did this Technique (BDT) become successful?
CONTEXT IS KING!
Domain
Distributed Teams
Skills
Capabilities Product-under test
Tech Stack
Process
Tools
NOT work in isolation
BDT WILL …
BDT REQUIRES …
Lower layers of Test Pyramid to be well automated!
BDT REQUIRES …
Dedicated and Focused testing (manual / exploratory)
BDT helps identify the “right” type of regression tests!
Executable Specs
BDT Scenarios
Business Idea
Features
Epics
Stories
Iterations
Releases
BDT VALUE PROPOSITION
BDT VALUE PROPOSITION …
Incrementally build
Business Flows
Regress Business Value
Test in-sync with Product Functionality
Effective Tests
BDT VALUE PROPOSITION …
Remove ambiguity
Understand new functionality
Living Documentation
Onboarding
