specification-by-example with gherkin

Specification-By-Example with Gherkin

Christian Hassa, TechTalk (ch@techtalk.at, @chr99ha)

Almost 4 years later …

#106 in Visual Studio Gallery (30.1.2013)

#135 in NuGet (30.1.2013)

~800 visits daily

~ 4000 active users > 30 contributors

Goals

Make you curious about Specification-By-Example

New perspective for SpecFlow users

About me • Managing Partner at TechTalk • Agile consulting and delivery • Based in Europe:

Vienna, Budapest, Zurich

Terminology

• Specification-By-Example

• Acceptance Test Driven Development (ATDD)

• Behaviour Driven Development (BDD)

Agile requirements

Purpose of user stories?

• Describe user needs

• Unit of prioritization

• Unit of planning

• Reminder for a discussion

• Mechanism to defer detail

Requirement levels Impact Mapping

Story Mapping

Specification-By-Example

Why?

How?

Acceptance Criterion

Epic

Deliverable, Outcome

Impact, Goal

Easier to define upfront Harder to define upfront

Reminder for a discussion

Bug report

Isolated, formalized example

User Activity

User Story

Example

Code

Collecting acceptance criteria

“I would try to put a book into the shopping cart …”

“I would try to remove a book from the shopping cart…”

“I’d check whether the shopping cart is empty, when I enter the shop …”

“I would try to add the same book again to the shopping cart …”

Books can be placed into shopping cart.

Books can be removed from shopping cart.

Shopping cart should be empty when entering the shop.

Adding same book again to shopping cart should increase quantity.

As a potential customer I want to collect books in a shopping cart So that I can order several books at once.

“Imagine this story is already implemented: What would you try out?”

Using examples

Examples for user stories

UI wire frames, existing UI

rules, key examples

existing artifacts, mock-ups

Discussion of acceptance criteria

Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com

public void TestInitialOrderDiscount()

{

Customer newCustomer = new Customer();

Order newOrder = new Order(newCustomer);

newOrder.AddBook(

Catalog.Find(“ISBN-0955683610”)

);

Assert.Equals(33.75,

newOrder.Subtotal);

}

Register as “bart_bookworm” Go to “/catalog/search” Enter “ISBN-0955683610” Click “Search” Click “Add to Cart” Click “View Cart” Verify “Subtotal” is “$33.75”

We would like to encourage new users to buy in our shop. Therefore we offer 10% discount for their first order.

Specification-By-Example Examples …

– make abstract descriptions better understandable

– are usually not formally exchanged or documented

Brian Marick

Examples Tests

Requirements

consist of

describe verify fulfillment of

Discussion of acceptance criteria

Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com

public void TestInitialOrderDiscount()

{

Customer newCustomer = new Customer();

Order newOrder = new Order(newCustomer);

newOrder.AddBook(

Catalog.Find(“ISBN-0955683610”)

);

Assert.Equals(33.75,

newOrder.Subtotal);

}

Register as “bart_bookworm” Go to “/catalog/search” Enter “ISBN-0955683610” Click “Search” Click “Add to Cart” Click “View Cart” Verify “Subtotal” is “$33.75”

We would like to encourage new users to buy in our shop. Therefore we offer 10% discount for their first order.

Illustrated with shared examples

Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com

Given the user has not ordered yet When the user adds a book with the price of USD 37.50 into the shopping cart Then the shopping cart sub-total should be USD 33.75.

Discover new aspects

Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com

Actually, this not quite right: Books on sale should be excluded.

Collaboration: 3 amigos “Happy Path”

Technical feasability

Exceptions, border cases

Abstract acceptance criteria As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once.

Books can be added to the shopping basket

Books can be removed from the shopping basket

Shopping basket is initially empty

The same book can be added multiple times to the shopping basket

Formalizing examples As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once.

Books can be added to the shopping basket

Given my shopping basket is empty

When I add the book “Harry Potter” to my shopping basket

Then my shopping basket should contain 1 copy of “Harry Potter”

Formalizing examples As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once.

Books can be added to the shopping basket The same book can be added multiple times to the shopping basket

When I add the book “Harry Potter” to my shopping basket

Then my shopping basket should contain 2 copies of “Harry Potter”

Given my shopping basket contains 1 copy of “Harry Potter”

Example structure

The same book can be added multiple times to the shopping basket

When I add the book “Harry Potter” to my shopping basket

Then my shopping basket should contain 2 copies of “Harry Potter”

Given my shopping basket contains 1 copy of “Harry Potter”

Title: Describes isolated, focused intention=abstract acceptance criterion

Arrange: Context, initial state of the system

Act: Execution of the feature Assert: Assertion of observable behavior

And I should see the warning: “Book already existed in basket”

Triple-A constraint “Checks”

Steps can be chained up

Evolution of a domain specific test and specification language.

Life time of examples

Purpose of examples

• Shared understanding: acceptance criteria

• Documentation: system details

• Regression-tests: violated assumptions

Continuous validation

When I add the book “Harry Potter” to my shopping basket

Then my shopping basket should contain 2 copies of “Harry Potter”

Given my shopping basket contains 1 copy of “Harry Potter”

„Step Definitions“ are binding individual steps to an automatable interface of the application.

Automation does not necessarily have to bind to the UI.

Automation of system is supported/evolving with development.

System UI Automation

Demo

Gherkin automation for .NET

• Visual Studio plugin (VS-Gallery)

• NuGet Package

http://www.specflow.org

Impact on testing

Test automation becomes expensive, when …

• trying to automate manual tests

• making tests unreadable when automating them

• automating only after completing implementation

structure

readability

when to test

Structure Manual tests

Asserts Multiple combined features

Structure ACT-ASSERT- ACT-ASSERT- ACT-ASSERT- …

Dependent on existence of other features Long test path with high chance to break Cause and impact of an error hard to trace.

Automated Check

Single aspect of a single feature

ARRANGE – ACT – ASSERT

Independent of other features Short test path with lower chance to break Cause and impact of an error easier to trace.

Test automation pyramid

User journeys

Acceptance-criteria

Units many

harder

easier

Au

tom

ati

on

few

exploratory testing

Source: Mike Cohn

Readability // Go to web page 'http://localhost:40001/' using new browser instance BrowserWindow localhostBrowser = BrowserWindow.Launch( new System.Uri(this.RecordedMethod1Params.Url)); // Click 'Register found item' link Mouse.Click(uIRegisterFoundItemHyperlink, new Point(56, 9)); // Click 'Save' button Mouse.Click(uISaveButton, new Point(44, 14)); int foundItemNo1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9)); // Click 'Register found item' link Mouse.Click(uIRegisterFoundItemHyperlink, new Point(63, 7)); // Click 'Save' button Mouse.Click(uISaveButton, new Point(34, 11)); int foundItemNo2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9)); Assert.IsTrue(foundItemNo1 + 1 == foundItemNo2); // Click 'Close' button Mouse.Click(uICloseButton, new Point(26, 11));

A readable test case Scenario: New found items should receive a consecutive number for the current year Given the previous found item of the current year had the number 145 When I register a new found item Then the last found item of the current year should have the number 146

When to test

Acceptance criteria (ATDD, BDD)

Unit Tests (TDD)

business view

technical view

Exploratory tests Workflow tests

Performance, Scalability, Usability,Security, …

de

fin

ing

th

e p

rod

uct

criticizing

the

pro

du

ct

New dimension: defining the product Synergy: Specification of requirements and tests

Agile Testing Quadrants: Brian Marick

Manual testing is still necessary!

User journeys

Acceptance-criteria

Units

exploratory testing

Manual Check after story done

Main success paths

Undiscovered acceptance criteria

No/(few) manual regression checks

Few paths are enough

More time for exploration

Cross-functional work

Sprint 1 Sprint 2 Sprint 3

short iteration

US4 Plan

Implement & autom. test

US5 Plan

Implement & autom. test

US2 Plan

Implement & autom. test

US3

Plan

Implement & autom. test

US6 Plan

Implement & autom. test

US1 Plan

Implement & autom. test

US7 Plan

Implement & autom. test

US8 Plan

Implement & autom. test

US9 Plan

Implement & autom. test

Explo

ratory Tests

Specification and test

Collaboration for defining acceptance criteria

Collaboration for automation

Preventing bugs instead of finding them!

Extension of “Test Cases”

Limit WIP

Collaboration in manual

testing

Impact on development

TDD workflow

Extending TDD for business

Transparency for stakeholders

In Progress

Current sprint report: all scenarios

Starting with first scenario

Finishing first scenario

Progressing on scenarios

Progressing on scenarios

Progressing on scenarios

Progressing on scenarios

First user story ready for testing

Earlier start of manual testing

Done work can break again

Done work can break again

Business readable error reports

Living documentation

Higher level scenario structure

Story Mapping

Why?

How?

Acceptance Criterion

Epic

Deliverable, Outcome

Impact, Goal

User Activity

User Story

Example

Code Easier to define upfront Harder to define upfront

Story Maps

• Original concept by Jeff Patton

• Support iterative product design

• Optimized for desired outcome or deliverable the system should support

Building story maps Get popular books delivered fast and

conveniently

Find book

Collect books

Order Wait for delivery

Receive delivery

time

browse best

sellers

shopping basket

enter address

delivery slip

delivery notification

pay with credit card

search book by

title wish list

inquiry order status

desired outcome or deliverable

user activities

system features

nec

essi

ty

Walking skeleton

Enabling build–measure-learn Get popular books delivered fast and

conveniently

Find book

Collect books

Order Wait for delivery

Receive delivery

time

browse best

sellers

enter address

delivery slip

pay with credit card

search book by

title wish list

inquiry order status

nec

essi

ty

shopping basket

delivery notification

manual work-

around

omitted steps

desired outcome or deliverable

user activities

system features

Story Map example

Sprint 1

Sprint 2

Sprint 3

Sprint 4

Dropped Scope

Added scope

User Stories vs. Features Product/Sprint Backlog

User Story 1 AccCrit 1

AccCrit 2

User Story 2 AccCrit 3

AccCrit 4

Living Documentation

Feature 1

AccCrit 1

AccCrit 2

Feature n AccCrit 4

AccCrit m User Story n

AccCrit 5

AccCrit m

AccCrit 3 AccCrit 5

„Done“

• Future options of the system • Organized/refined according to

priority, value, effort, risk, ... • Next possible increments of

the product (units of work)

• Current state of the system • Organized/refined for

functional overview • Versioned and maintained

together with source code

Living documentation • Link business

readable automation from source code to story maps

• Feed execution results into story maps

• Sync Story Story Map requirements with TFS work items

Lessons learned

Collaboration over specification

Documentation after conversation • Collaborative discovery • Should trigger new questions • Infinite number of examples • Readability and ability to automate

Level of automation

Controller

Business Layer

Data Layer

Model

View

Browser automation

Trigger behavior through controller

Assert behavior on model, db, ..

Setup pre-conditions through service interfaces

Out-of-process

In-process

Test execution performance

• Grouping tests – Current WIP

– Completed work

• Database – In-memory

– Templates for setup

• Parallel execution

• Smart execution order

• Level of automation

Internal vs. external DSL

Example Source: Liz Keogh https://github.com/lunivore/tictactoe-java/blob/master/ scenarios/com/lunivore/tictactoe/scenarios/Three_in_a_row_wins.java

Non-functional acceptance criteria

Given there are 100,000 users registered on the system

When I create a new account

Then I should be taken to my dashboard within 5ms

When 1000 users are hitting the homepage simultaneously Then each user should get a response within 2ms

Matt Wynne http://blog.mattwynne.net/2012/03/13/using-cucumber-for-load-testing

Tools

Gherkin automation tools

www.cukes.info

www.behat.org

www.specflow.org

Ruby, Java, JavaScript, C++

.NET, Mono, Silverlight, WP7

PHP

Books

Bridiging the Communication Gap Gojko Adzic

Specification by Example Gojko Adzic

Explore It! Elisabeth Hendrickson

Exploratory Software Testing James Whittaker

Summary

Specification-by-example

• Collaborative discovery and specification

• Impact on development and test

• Combine with other agile reqmt methods

• Can be supported by tools

Questions

Got curious about Specification-By-Example?

Learned something new about using SpecFlow?

Christian Hassa, TechTalk (ch@techtalk.at, @chr99ha)

Join the BoF Session „Specification-By-Example (ATDD, BDD) in practice“ Today at 15h15 (right after this talk)