tutorial: implementing specification-by-example with gherkin
DESCRIPTION
1/2 day Tutorial held at XP 2013 conference in Vienna Many teams struggle with the implementation of user story acceptance criteria and having a shared understanding about the expected story outcomes. This often results in missed stakeholder expectations, ad-hoc assumptions made by the team during implementation and conflict between team members and the product owner around testing. In this tutorial, you will learn how specification-by-example and acceptance test driven development will address team conflict, missed stakeholder expectations and overall increasing the level of clarity on the project end-to-end. The presentation will cover the theory behind ATDD, case-studies and practical experience from real projects and several hands-on exercises to try out the presented concepts. You will leave this tutorial with a fundamental understanding of specification-by-example and its benefits, as well as concrete pointers on how to get started using it in your own projects.TRANSCRIPT
CHRISTIAN HASSA ([email protected])
TWITTER: @CHRISHASSA
XP 2013 Vienna, June 3rd 2013
Specification-By-Example with Gherkin
About me
• Managing Partner at TechTalk• Working as/coaching Product Owners
TechTalk office Vienna/Austria
• Agile consulting and delivery• Offices: Vienna, Budapest, Zurich
TechTalk Team
About my Team
4
RefiningUser Stories
5
Impact Mapping
Story Mapping
Specification-By-Example
Establishing a shared understanding
Why?
How?
Code
AcceptanceCriterion
Epic
Capability
Impact, Goal
Easier to define upfront Harder to define upfront
Bugreport
Isolated,formalizedexample
Feature
User Story
Example
Reminderfor adiscussion
6
Collecting Acceptance Criteria
“I would try to put a book into the shopping cart …”
“I would try to remove a book from the shopping cart…”
“I’d check whether the shopping cart is empty, when I enter the shop …”
“I would try to add the same book again to the shopping cart …”
Books can be placed into shopping cart.
Books can be removed from shopping cart.
Shopping cart should be empty when entering the shop.
Adding same book again to shopping cart should increase quantity.
As a potential customerI want to collect books in a shopping cartSo that I can order several books at once.
“Imagine this story is already implemented:
how would you verify it?”
7
Example:Mobile Ticket for Wiener Linien
8
Exercise: Smart Alarm
• What would you try out when the following user story was done?
• Collect and write down the list of discussed acceptance criteria• Work in groups of 2-3
As a regular travelerI want to setup a smart alarmso that I can be warned about service disruptions
I want to setup multiple smart alarmsso that I can plan ahead for different travel plans I have
9
Usingexamples
10
UI wire frames,existing UI
rules, key examples
existing artifacts,samples
Examples for user stories
11
Specification-by-Example
Examples …• make abstract descriptions
better understandable
However …• examples are usually not formally
exchanged or documentedBrian Marick
Examples Tests
Requirements
consist of
describe verifyfulfillment of
12
Discussion of acceptance criteria
public void TestInitialOrderDiscount()
{
Customer newCustomer = new Customer();
Order newOrder = new Order(newCustomer);
newOrder.AddBook(
Catalog.Find(“ISBN-0955683610”)
);
Assert.Equals(33.75,
newOrder.Subtotal);
}
Register as “bart_bookworm”Go to “/catalog/search”Enter “ISBN-0955683610”Click “Search”Click “Add to Cart”Click “View Cart”Verify “Subtotal” is “$33.75”
We would like to encourage new users to buy in our shop.Therefore we offer 10% discount for their first order.
Original idea for the illustration: George Dinwiddiehttp://blog.gdinwidiee.com
13
… illustrated with formalized examples
Given the user has not ordered yet
When the user adds a book with the price of EUR 37.5
into the shopping cart
Then the shopping cart sub-total is EUR 33.75.
Original idea for the illustration: George Dinwiddiehttp://blog.gdinwidiee.com
14
Discover hidden assumptions
Actually, this is not quite right:Books on sale should be excluded.
Original idea for the illustration: George Dinwiddiehttp://blog.gdinwidiee.com
15
Collaboration: 3 amigos
“HappyPath”
Technical feasability
Exceptions, border cases
Original idea for the illustration: George Dinwiddiehttp://blog.gdinwidiee.com
16
Abstract acceptance criteria
As a shop visitorI want to collect books in my shopping basketso that I can purchase multiple books at once.
Books can be added to the shopping basket
Books can be removed from the shopping basket
Shopping basket is initially empty
The same book can be added multiple times to the shopping basket
17
Examples for acceptance criteria
As a shop visitorI want to collect books in my shopping basketso that I can purchase multiple books at once.
Books can be added to the shopping basket
Given my shopping basket is empty
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 1 copy of “Harry Potter”
18
As a shop visitorI want to collect books in my shopping basketso that I can purchase multiple books at once.
Books can be added to the shopping basket
Examples for acceptance criteria
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
The same book can be added multiple times to the shopping basket
19
The same book can be added multiple times to the shopping basket
Structure of examples
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Title: Describes intention/abstract acceptance criterion
Arrange: Context, initial state of the system
Act: Execution of the feature
Assert: Assertion of observable behaviour
And I should see the warning: “Book already existed in basket”
Triple-Aconstraint“Checks”
Chainingup steps
20
Gherkin
21
Feature: Description of feature or user story
Szenariogrundriss: Beschreibung des Akzeptanzkriteriums
Szenario: Beschreibung des Akzeptanzkriteriums
Gherkin Feature Files
Background: context for all scenarios in the feature file
Scenario: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario SchritteGiven/When/Then: automated scenario steps
Scenario Outline: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario SchritteGiven/When/Then: automated scenario steps with <place holder>
Examples: table with examples for <place holders>Examples: table with examples for <place holders>
Given: automated scenario steps
@tagname
@tagname
@tagname
@tagname
22
Feature
Feature: Description of feature or user story
#language:en-en@doneFeature: Shopping cartAs visitor of the web shopI want to collect books in a shopping cartso that I can- prepare a short list of books I want to order- combine multiple books in one shipment
@Tagname
23
Scenario
Scenario: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario SchritteGiven/When/Then: automated scenario steps
@Tagname
@inprogressScenario: The same book can be added multiple times to shopping cartGiven my shopping cart contains 1 copy of "Harry Potter"
When I add the book "Harry Potter" to the shopping cart
Then my shopping cart contains 2 copies of "Harry Potter"And a warning is displayed: "You have added the same book again"
24
Scenario OutlineScenario Outline: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario Schritte
Examples: table with examples for <place holders>
Given/When/Then: automated scenario steps with <place holder>
Examples: table with examples for <place holders>
@Tagname
@Tagname
Scenario Outline: Simple searchWhen I search for books by the phrase '<search phrase>'Then the list of found books should contain only: <books>
Examples:| search phrase | books | Explanation || Domain | 'Domain …' | whole words are matched || Analysis Communication | 'Bridging…', 'Analysis …' | multiple words are matched with OR |
Scenario Outline: Candidate profile can only be saved with <mandatory field>
When I enter a valid candidate profile, without filling in '<mandatory field>'Then I see the following validation error: 'field <mandatory field> missing'
Examples: Main fields| mandatory field || email || name || first name |
Examples: Extended fields| mandatory field || birthdate |
25
Background
Background: context for all scenarios in the feature file
Given: automated scenario steps
Background: Given the catalog contains the following books| Title | Author || Specification-By-Example | Gojko Adzic || Agile Testing | Lisa Crispin, Janet Gregory || Scrum Field Guide | Mitch Lacey |
26
Scenario step arguments
Given the catalog contains the following books| Title | Author || Specification-By-Example | Gojko Adzic || Agile Testing | Lisa Crispin, Janet Gregory || Scrum Field Guide | Mitch Lacey |
When I add the book "Harry Potter" to the shopping cart
Scalar arguments:
Table arguments:
27
Exercise
• Work in same previous groups of 2-3• Pick an acceptance criteria from the
previously compiled list• Outline usage examples with the following
structure• Given: Context, initial state• When: utilization of the feature• Then: assertions of observable behaviour
• Validate that scenario title (acceptance criterion) still describes intention of formalized scenario
28
Life time of
examples
29
Purpose of the examples
• Shared understanding:acceptance criteria
•Documentation:system details
• Regression-tests:violated assumptions
30
Continuous validation with automation
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
System
„Step Definitions“ are binding individual stepsto an automatable interface of the application.
Automatableinterface
UIAutomation
Automation does not necessarily have to bind to the UI.
Automatability of system is supported/evolving with development.
31
Demo
http://www.specflow.org
Gherkin automation for .NET• Visual Studio plugin (VS-Gallery)•NuGet Package
32
Some more complex exercises
Choose between• Best-fare option ticket• Smart Alarm
33
Best-fare option ticket
• Assumptions• Ticket price is always adjusted during
validation to the best option (considering all previous trips)
• Feedback about cost of travel and ticket option (type) after each validation
• Valid in the core zone 100
• Scope of the story• Different travel scenarios and how the
price adjusts• Billings resulting from the user’s travel• Feedback to traveler (cost of travel, status
of ticket) when validating ticket
• Out of scope for this story• Credit limit of traveler• Authentication of ticket (validity)
As an ad-hoc travelerI want my ticket to adjust to the best available fare whenever I validateSo that I don’t need to study upfront which ticket I need
Ticket Type Price
Single trip 2,10
24 hours 7,10
48 hours 12,40
72 hours 15,40
8 days 35,80
Week 15,00
Month 45,00
Year 365,00
34
Smart Alarm
• Assumptions• User can specify route to monitor for
disruptions• User can specify when disruptions
should trigger the alarm earlier (minutes before alarm)
• Scope of the story• Different possibilities of disruptions
provided by GTFS and their impact on activating the alarm
• Out of scope for this story• Configuring routes (the user is
selecting an already defined route)• Setting up alarms (the user has
already setup an alarm)
As a regular travelerI want to have a smart alarm that activates earlier when there are traffic disruptions on my routeSo that I can travel earlier to be still on time for my appointment
35
Exercise
Choose between• Best-fare option ticket• Smart Alarm
Collect and discuss scenarios•What would you try out?• Collect examples• Limit scope of the story
36
Exercise
Refine scenarios and formalizeinto Gherkin• Start with a “warm-up” scenario• Increase complexity with additional
scenarios• Reconsider structure of scenarios as you
refine them• Validate scenario title/description after
refinement
37
Impact on
testing
38
Test automation becomes expensivewhen …
• trying to automate
manual tests
• making tests
unreadable when
automating them
• automating after
completing
development
structure
readability
point in time
39
StructureManual tests
Asserts Multiple combined features
Structure ACT-ASSERT-ACT-ASSERT-ACT-ASSERT-…
Dependent featuresLong test path with high chance to breakCause and impact of error hard to trace
Automated Check
Single aspect of a single feature
ARRANGE –ACT –ASSERT
Independent featuresShort test path with lower chance to breakCause and impact of error easy to relate
40
Test automation pyramid
Userjourneys
Acceptance-criteria
Units
exploratory testing
Source: Mike Cohn
many
few
hard
easy
Automatability
41
// Go to web page 'http://localhost:40001/' using new browser instanceBrowserWindow localhostBrowser = BrowserWindow.Launch(
new System.Uri(this.RecordedMethod1Params.Url));
// Click 'Register found item' linkMouse.Click(uIFundstückerfassenHyperlink, new Point(56, 9));
// Click 'Save' buttonMouse.Click(uISpeichernButton, new Point(44, 14));
int fundNr1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9));
// Click 'Register found item' linkMouse.Click(uIFundstückerfassenHyperlink, new Point(63, 7));
// Click 'Save' buttonMouse.Click(uISpeichernButton, new Point(34, 11));
int fundNr2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9));
Assert.IsTrue(fundNr1 + 1 == fundNr2);
// Click 'Close' buttonMouse.Click(uICloseButton, new Point(26, 11));
Readability
42
A readable test caseScenario: New found items should receive a consecutive number for the current year
Given the previous found item of thecurrent year had the number 145
When I register a new found item
Then the last found item of thecurrent year should have the number 146
43
When to test (point in time)
Acceptance criteria(ATDD, BDD)
Unit Tests(TDD)
business view
technical view
Exploratory tests Workflow tests
Performance, Scalability, Usability,Security, …
def
inin
g th
e p
rod
uct
criticizing th
e p
rod
uct
New dimension: defining the productSynergy: Specification of requirements and tests
Agile Testing Quadrants: Brian Marick
44
Manual Testing is always necessary!
Userjourneys
Acceptance-criteria
Units
exploratory testing
Source: Mike Cohn
many
few
harder
easier
Automatability
ManualCheckafterStoryDone
Mainsuccesspathes
Undiscoveredacceptancecriteria
No/(few)manualregressionchecks
Few pathesare enough
More timefor exploration
47
Impact on
development
48
Workflow - TDD
49
Extending TDD for business
50
Transparency for stakeholders
In Progress
51
Current sprint report: all sprint scenarios
52
Starting with first scenario (AC)
53
Finishing the first scenario (AC)
54
Progressing scenario after scenario
55
Progressing scenario after scenario
56
Progressing scenario after scenario
57
Implementing user stories in parallel
58
First user story ready for testing
59
Manual testing can start even earlier
60
Work done already can break again
61
Work done already can break again
62
See what is temporarily not working
64
Livingdocumentation
65
Living documentation
Drill into Details(Gherkin scenarios)
Overview(Story Map)
66
User Stories vs. Features
Product/Sprint Backlog
User Story 1AccCrit 1
AccCrit 2
User Story 2AccCrit 3
AccCrit 4
Living Documentation
Feature 1
AccCrit 1
AccCrit 2
Feature nAccCrit 4
AccCrit mUser Story n
AccCrit 5
AccCrit m
AccCrit 3AccCrit 5
„Done“
• Future options of the system• Organized/refined according to
priority, value, effort, risk, ...• Next possible increments of
the product (units of work)
• Current state of the system• Organized/refined for
functional overview• Versioned and maintained
together with source code
67
Lessonslearned
68
Collaboration over specification
Documentation after conversation• Collaborative discovery• Should trigger new questions• Infinite number of examples• Readability and ability to automate
69
Collaboration smells
• Silos and too formal hand-overs• Formalization cannot follow implementation• Scenarios replace communication• Scenarios inhibit consideration of solutions• Scenarios inhibit discovery of new things
•Duty after development is done• Unreadable scenarios without value• High cost for creation and maintenance• Technical problems with automation
70
Level of automation
Controller
Business Layer
Data Layer
Model
View
Browserautomation
Trigger behaviourthrough controller
Assert behaviouron model, db, ..
Setup pre-conditionsthrough serviceinterfaces
Out-of-process
In-process
71
Test execution performance
• Grouping tests
• Current WIP
• Completed work• Database
• In-memory
• Templates for setup• Parallel execution• Smart execution order
72
Internal vs. external DSL
Example Source:Liz Keoghhttps://github.com/lunivore/tictactoe-java/blob/master/scenarios/com/lunivore/tictactoe/scenarios/Three_in_a_row_wins.java
73
Non-functional acceptance criteria
Given there are 100,000 users registered on the systemWhen I create a new accountThen I should be taken to my dashboard within 5ms
Given 1000 users are hitting the homepage simultaneouslyThen each user should get a response within 2ms
Matt Wynnehttp://blog.mattwynne.net/2012/03/13/using-cucumber-for-load-testing
74
Tools
75
Gherkin automation tools
www.cukes.info
www.behat.org
Ruby, Java, JavaScript, C++
www.specflow.org
.NET, Mono, Silverlight, WP7
PHP
76
77
Books
Gojko AdzicBridiging theCommunication Gap
Gojko AdzicSpecification byExample
Elisabeth HendricksonExplore IT!
78 Christian Hassa: [email protected] - @chrishassa