AUTOMATEDWEB ACCEPTANCE TESTINGWITH BEHAT

Fabian KissYMC AG | May 2012

Picture by viZZZual.com, licensed under the CC BY 2.0

TESTINGIN AGILE

Pict

ure

by S

heff

ield

Tig

er, l

icen

sed

unde

r the

CC

BY 2

.0

MIKE COHN'S AGILE TESTING PYRAMID

Automated

tests

Succeeding with Agile: Software Development Using Scrum, Addison-Wesley

MIKE COHN'S AGILE TESTING PYRAMID

Succeeding with Agile: Software Development Using Scrum, Addison-Wesley

UI

Service

Unit

MIKE COHN'S AGILE TESTING PYRAMID

Succeeding with Agile: Software Development Using Scrum, Addison-Wesley

UI

Service

Unit

least investment

most investment

MIKE COHN'S AGILE TESTING PYRAMID

Succeeding with Agile: Software Development Using Scrum, Addison-Wesley

UI

Service

Unit

least investment

most investment

BRIAN MARICK'S AGILE TESTING MATRIX

http://www.exampler.com/old-blog/2003/08/21/

Manual + automated tests

BRIAN MARICK'S AGILE TESTING MATRIX

http://www.exampler.com/old-blog/2003/08/21/

AcceptanceTesting

...

Unit Testing...

Usability Testing

...

Performance Testing

…

BRIAN MARICK'S AGILE TESTING MATRIX

http://www.exampler.com/old-blog/2003/08/21/

AcceptanceTesting

...

Unit Testing...

Usability Testing

...

Performance Testing

…

Business-facing

Technology-facing

Supp

ort p

rogr

amm

ing Critiqu e pro duct

BRIAN MARICK'S AGILE TESTING MATRIX

http://www.exampler.com/old-blog/2003/08/21/

AcceptanceTesting

...

Unit Testing...

Usability Testing

...

Performance Testing

…

Business-facing

Technology-facing

Supp

ort p

rogr

amm

ing Critiqu e pro duct

Manual + automated tests

Manual + automated tests

BRIAN MARICK'S AGILE TESTING MATRIX

http://www.exampler.com/old-blog/2003/08/21/

AcceptanceTesting

...

Unit Testing...

Usability Testing

...

Performance Testing

…

BRIAN MARICK'S AGILE TESTING MATRIX

http://www.exampler.com/old-blog/2003/08/21/

AcceptanceTesting

...

Unit Testing...

Usability Testing

...

Performance Testing

…

BDD

TDD/BDD

BDD

BDD

Specification-oriented BDD

Scenario-oriented BDD

BDD

Specification-oriented BDD

Scenario-oriented BDD

RSpec

BDD

Specification-oriented BDD

Scenario-oriented BDDCucumber

Behat

RSpec

BDD

Specification-oriented BDD

Scenario-oriented BDDCucumber

Behat

Gherkin DSL

RSpec

BDD

Specification-oriented BDD

Scenario-oriented BDDCucumber

Behat

Gherkin DSL

RSpec

BDD

Specification-oriented BDD

Scenario-oriented BDDCucumber

Behat

Gherkin DSL

RSpec

...

.. .

GHERKIN SYNTAX + SEMANTICS

http://docs.behat.org/guides/1.gherkin.html

Scenario: Some determinable business situation    Given some precondition      And some other precondition     When some action by the actor      And some other action     Then some testable outcome is achieved      And something else we can check happens too 

GHERKIN SYNTAX + SEMANTICS

http://docs.behat.org/guides/1.gherkin.html

Scenario: Some determinable business situation    Given some precondition      And some other precondition     When some action by the actor      And some other action     Then some testable outcome is achieved      And something else we can check happens too 

GHERKIN SYNTAX + SEMANTICS

http://docs.behat.org/guides/1.gherkin.html

Scenario: Some determinable business situation    Given some precondition      And some other precondition     When some action by the actor      And some other action     Then some testable outcome is achieved      And something else we can check happens too 

GHERKIN SYNTAX + SEMANTICS

http://docs.behat.org/guides/1.gherkin.html

Scenario: Some determinable business situation    Given some precondition      And some other precondition     When some action by the actor      And some other action     Then some testable outcome is achieved      And something else we can check happens too 

GHERKIN SYNTAX + SEMANTICS

http://docs.behat.org/guides/1.gherkin.html

Scenario: Some determinable business situation    Given some precondition      And some other precondition     When some action by the actor      And some other action     Then some testable outcome is achieved      And something else we can check happens too 

GHERKIN SYNTAX + SEMANTICS

http://docs.behat.org/guides/1.gherkin.html

Scenario: Some determinable business situation    Given some precondition      And some other precondition     When some action by the actor      And some other action     Then some testable outcome is achieved      And something else we can check happens too 

GHERKIN EXAMPLE

http://docs.behat.org/guides/1.gherkin.html

Scenario: Wilson posts to his own blog  Given I am logged in as Wilson  When I try to post to "Expensive Therapy"  Then I should see "Your article was published."

Scenario: Wilson fails to post to somebody else's blog  Given I am logged in as Wilson  When I try to post to "Greg's anti­tax rants"  Then I should see "Hey! That's not your blog!"

GHERKIN EXAMPLE

http://docs.behat.org/guides/1.gherkin.html

Scenario: Wilson posts to his own blog  Given I am logged in as Wilson  When I try to post to "Expensive Therapy"  Then I should see "Your article was published."

Scenario: Wilson fails to post to somebody else's blog  Given I am logged in as Wilson  When I try to post to "Greg's anti­tax rants"  Then I should see "Hey! That's not your blog!"

Feature

STEPS & STEP DEFINITIONS

http://docs.behat.org/quick_intro.html#basic-usage

Given some precondition

/** * @Given /^some precondition$/ */public function somePrecondition(){    //...}

STEPS & STEP DEFINITIONS

http://docs.behat.org/quick_intro.html#basic-usage

Given some precondition

/** * @Given /^some precondition$/ */public function somePrecondition(){    //...}

Step

Step definition

STEPS & STEP DEFINITIONS

http://docs.behat.org/quick_intro.html#basic-usage

Given some precondition

/** * @Given /^some precondition$/ */public function somePrecondition(){    //...}

Step

Step definition

Matching

Run

STEP ARGUMENTS

http://docs.behat.org/quick_intro.html#some-more-behat-basics

When I try to post to "Expensive Therapy"

/** * @When /^I try to post to "([^"]*)"$/ */public function iTryToPostTo($site){    //...}

STEP ARGUMENTS

http://docs.behat.org/quick_intro.html#some-more-behat-basics

When I try to post to "Expensive Therapy"

/** * @When /^I try to post to "([^"]*)"$/ */public function iTryToPostTo($blog){    //...}

Argument

SCENARIO BACKGROUNDS

http://docs.behat.org/guides/1.gherkin.html#backgrounds

Background:  Given I am logged in as Wilson

Scenario: Wilson posts to his own blog  When I try to post to "Expensive Therapy"  Then I should see "Your article was published."

Scenario: Wilson fails to post to somebody else's blog  When I try to post to "Greg's anti­tax rants"  Then I should see "Hey! That's not your blog!"

SCENARIO BACKGROUNDS

http://docs.behat.org/guides/1.gherkin.html#backgrounds

Background:  Given I am logged in as Wilson

Scenario: Wilson posts to his own blog  When I try to post to "Expensive Therapy"  Then I should see "Your article was published."

Scenario: Wilson fails to post to somebody else's blog  When I try to post to "Greg's anti­tax rants"  Then I should see "Hey! That's not your blog!"

SCENARIO OUTLINES

http://docs.behat.org/guides/1.gherkin.html#scenario-outlines

Scenario Outline: Wilson posts to a blog  Given I am logged in as Wilson  When I try to post to "<blog>"  Then I should see "<authorization>"

  Examples:   | blog                  | authorization               |   | Expensive Therapy     | Your article was published. |   | Greg's anti­tax rants | Hey! That's not your blog!  |

SCENARIO OUTLINES

http://docs.behat.org/guides/1.gherkin.html#scenario-outlines

Scenario Outline: Wilson posts to a blog  Given I am logged in as Wilson  When I try to post to "<blog>"  Then I should see "<authorization>"

  Examples:   | blog                  | authorization               |   | Expensive Therapy     | Your article was published. |   | Greg's anti­tax rants | Hey! That's not your blog!  |

x 2

STEP TABLES

http://docs.behat.org/guides/1.gherkin.html#tables

Scenario: Wilson posts to a blog  Given I am logged in as Wilson  When I try to post to:   | blog                  |   | Expensive Therapy     |   | Greg's anti­tax rants |  Then I should see an authorization outcome

STEP TABLES

http://docs.behat.org/guides/1.gherkin.html#tables

Scenario: Wilson posts to a blog  Given I am logged in as Wilson  When I try to post to:   | blog                  |   | Expensive Therapy     |   | Greg's anti­tax rants |  Then I should see an authorization outcome

x 2

ACCESSING THE WEB

Pict

ure

by M

isse

rion,

lice

nsed

und

er th

e CC

BY

2.0

MINK API

Driver object

$client = new \Selenium\Client($host, $port);$driver = new \Behat\Mink\Driver\SeleniumDriver(    'firefox', 'base_url', $client);

Session object

$session = new \Behat\Mink\Session($driver);$session­>start();$session­>visit('http://my_project.dev/some_page.php')

Exemplary interaction

$page = $session­>getPage();$elementByCss = $page­>find('css', 'h3 > a');$elementByCss­>click();

http://mink.behat.org/

MINK API

Driver object

$client = new \Selenium\Client($host, $port);$driver = new \Behat\Mink\Driver\SeleniumDriver(    'firefox', 'base_url', $client);

Session object

$session = new \Behat\Mink\Session($driver);$session­>start();$session­>visit('http://my_project.dev/some_page.php')

Exemplary interaction

$page = $session­>getPage();$elementByCss = $page­>find('css', 'h3 > a');$elementByCss­>click();

http://mink.behat.org/

MINK API

Driver object

$client = new \Selenium\Client($host, $port);$driver = new \Behat\Mink\Driver\SeleniumDriver(    'firefox', 'base_url', $client);

Session object

$session = new \Behat\Mink\Session($driver);$session­>start();$session­>visit('http://my_project.dev/some_page.php')

Exemplary interaction

$page = $session­>getPage();$elementByCss = $page­>find('css', 'h3 > a');$elementByCss­>click();

http://mink.behat.org/

Sahi, Goutte, Zombie.js

MINK API

Driver object

$client = new \Selenium\Client($host, $port);$driver = new \Behat\Mink\Driver\SeleniumDriver(    'firefox', 'base_url', $client);

Session object

$session = new \Behat\Mink\Session($driver);$session­>start();$session­>visit('http://my_project.dev/some_page.php')

Exemplary interaction

$page = $session­>getPage();$elementByCss = $page­>find('css', 'h3 > a');$elementByCss­>click();

http://mink.behat.org/

Sahi, Goutte, Zombie.js

MINK API

Driver object

$client = new \Selenium\Client($host, $port);$driver = new \Behat\Mink\Driver\SeleniumDriver(    'firefox', 'base_url', $client);

Session object

$session = new \Behat\Mink\Session($driver);$session­>start();$session­>visit('http://my_project.dev/some_page.php')

Exemplary interaction

$page = $session­>getPage();$elementByCss = $page­>find('css', 'h3 > a');$elementByCss­>click();

http://mink.behat.org/

Sahi, Goutte, Zombie.js

BEHAT REPORT

~/Behat$ behat features/blogpost.feature

Feature: Blog posting  As a blogger  I want to post an article to a blog

  Scenario: Wilson posts to his own blog            # features/blogpost.feature:5    Given I am logged in as "Wilson"                # FeatureContext::iAmLoggedInAs()    When I try to post to "Expensive Therapy"       # FeatureContext::iTryToPostTo()    Then I should see "Your article was published." # FeatureContext::iShouldSee()

  Scenario: Wilson fails to post to somebody else's blog # features/blogpost.feature:10    Given I am logged in as "Wilson"                     # FeatureContext::iAmLoggedInAs()    When I try to post to "Greg's anti­tax rants"        # FeatureContext::iTryToPostTo()      Blog "Greg's anti­tax rants" does not exist    Then I should see "Hey! That's not your blog!"       # FeatureContext::iShouldSee()

2 scenarios (1 passed, 1 failed)6 steps (4 passed, 1 skipped, 1 failed)

BEHAT REPORT

~/Behat$ behat features/blogpost.feature

Feature: Blog posting  As a blogger  I want to post an article to a blog

  Scenario: Wilson posts to his own blog            # features/blogpost.feature:5    Given I am logged in as "Wilson"                # FeatureContext::iAmLoggedInAs()    When I try to post to "Expensive Therapy"       # FeatureContext::iTryToPostTo()    Then I should see "Your article was published." # FeatureContext::iShouldSee()

  Scenario: Wilson fails to post to somebody else's blog # features/blogpost.feature:10    Given I am logged in as "Wilson"                     # FeatureContext::iAmLoggedInAs()    When I try to post to "Greg's anti­tax rants"        # FeatureContext::iTryToPostTo()      Blog "Greg's anti­tax rants" does not exist    Then I should see "Hey! That's not your blog!"       # FeatureContext::iShouldSee()

2 scenarios (1 passed, 1 failed)6 steps (4 passed, 1 skipped, 1 failed)

default formatter (“pretty”)

all formatters:http://docs.behat.org/guides/6.cli.html#format-options

­­format pretty

CHALLENGES...

DEVELOPERS WRITESCENARIOS

Pict

ure

by M

arja

n Kr

ebel

j, lic

ense

d un

der t

he C

C BY

2.0

DEVELOPERS WRITESCENARIOS

AN ARTIFACT ASKED FOR?UNDERSTANDABILITY?

Pict

ure

by M

arja

n Kr

ebel

j, lic

ense

d un

der t

he C

C BY

2.0

Pict

ure

by C

onor

Law

less

, lic

ense

d un

der t

he C

C BY

2.0

SCENARIO ABSTRACTION LEVEL

Pict

ure

by C

onor

Law

less

, lic

ense

d un

der t

he C

C BY

2.0

SCENARIO ABSTRACTION LEVELWhen I fill the form

When I fill my first name in

to field “first name”

When I fill my last name int

o field “last name”

When I fill ...VS.

REDUNDANCY

Pict

ure

by R

ob F

aulk

ner,

licen

sed

unde

r the

CC

BY 2

.0

REDUNDANCY

LIMITED: GHERKIN'S REUSE CAPABILITIESLIMITED: UNDERSTANDING OF THOSE

Pict

ure

by R

ob F

aulk

ner,

licen

sed

unde

r the

CC

BY 2

.0

TIME-CONSUMING REFACTORING

Pict

ure

by E

arls

37a,

lice

nsed

und

er th

e CC

BY

2.0

TIME-CONSUMING REFACTORING

EXECUTION IN REAL-TIME

Pict

ure

by E

arls

37a,

lice

nsed

und

er th

e CC

BY

2.0

COMPLEX INFRASTRUCTURE

Pict

ure

by D

omen

ico

Nar

done

, lic

ense

d un

der t

he C

C BY

2.0

COMPLEX INFRASTRUCTURE

Pict

ure

by D

omen

ico

Nar

done

, lic

ense

d un

der t

he C

C BY

2.0

Behat Mink Driver Browser Emulator Browser

FURTHER REFERENCES: http://www.softwarequalityconnection.com/2011/08/how-to-improve-communication-between-qa-and-development/

http://dannorth.net/introducing-bdd/

http://www.infoq.com/news/2011/02/BDD-ATDD

http://www.slideshare.net/headrevision/the-concept-of-behaviordriven-development

http://www.slideshare.net/everzet/bdd-in-symfony2

SOME BEST PRACTICES: http://www.agileengineeringdesign.com/2012/01/7-deadly-sins-of-automated-software-testing/

http://elabs.se/blog/15-you-re-cuking-it-wrong

http://www.engineyard.com/blog/2009/15-expert-tips-for-using-cucumber/

http://eggsonbread.com/2010/09/06/my-cucumber-best-practices-and-tips/