browser-level testing

Browser-level testing

Martin KleppmannGo Test It

Email: martin@go-test.itBlog: yes-no-cancel.co.uk

Twitter: @martinkl

“But I have unit tests!”

?

Array literals?

var x = [1,];

x.length == 1 // Firefox 3.5x.length == 2 // IE 7

Uninitialised vars?

<div id="hello"></div>

typeof(hello) == 'undefined' // Firefox 3.5typeof(hello) == 'object' // IE 7

Attempt #1:Simulate events in JavaScript.

(Selenium, Windmill, Sahi)

Selenium example.com

Attempt #1: Simulate events in JavaScript.

Seleniumclick

example.com

Attempt #1: Simulate events in JavaScript.

Selenium

http://test.example.comhttp://localhost:8080

clickexample.com

Attempt #1: Simulate events in JavaScript.

Selenium

http://test.example.comhttp://localhost:8080

clickexample.com

Cross-domain security policy

Attempt #1: Simulate events in JavaScript.

Selenium example.com

Attempt #1: Simulate events in JavaScript.

Selenium example.com

test.example.com proxy

Attempt #1: Simulate events in JavaScript.

Selenium example.com

test.example.com proxy/selenium-server/ /*

Attempt #1: Simulate events in JavaScript.

clickSelenium example.com

test.example.com proxy/selenium-server/ /*

Attempt #1: Simulate events in JavaScript.

Limitations

• Redirect to other domain or https

• Outside of DOM: pop-ups, Flash, ...

• Slow

Attempt #2:Browser-specific automation

APIs.

(WebDriver, Watir)

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

WebDriver

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

WebDriver

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

WebDriver

Watir

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

WebDriver

Watir

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

WebDriver

Watir

Selenium

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

WebDriver

Watir

Selenium

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

Selenium

WebDriver

Watir

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

Selenium WebDriver Watir

Selenium WebDriver Watir

HTA/Proxy ✔ COM ✔ COM

Ext/Proxy ✔ Extension ✔ Extension

Proxy ✘ ✔ AppleScript

Proxy ✔ Extension ✔ V8 Debug

Proxy Dragonfly? ✘

✘ ✔ ✔ Celerity

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

Watir

require 'watir'Watir::Browser.default = 'firefox'

describe 'Google' do before(:each) { @browser = Watir::Browser.new } after(:each) { @browser.close }

it 'should return search results for "hello world"' do @browser.goto "http://www.google.co.uk" @browser.text_field(:name, "q").set("hello world") @browser.button(:name, "btnG").click @browser.contains_text( "Hello world program - Wikipedia").should be_true endend

WebDriver

require 'selenium-webdriver'

describe 'Google' do before(:each) { @browser = Selenium::WebDriver.for :firefox } after(:each) { @browser.quit }

it 'should return search results for "hello world"' do @browser.navigate.to "http://www.google.co.uk" @browser.find_element(:name, "q").send_keys("hello world") @browser.find_element(:name, "btnG").submit @browser.find_element(:partial_link_text, "Hello world program - Wikipedia") endend

Selenium

gem 'selenium-client'; require 'selenium/client'

describe 'Google' do before(:each) { @browser = Selenium::Client::Driver.new :browser => "*firefox", :url => "http://www.google.co.uk" @browser.start } after(:each) { @browser.stop }

it 'should return search results for "hello world"' do @browser.open "/" @browser.type "q", "hello world" @browser.click "btnG", :wait_for => :text, :text => 'Results' @browser.is_text_present("Hello world program - Wikipedia"). should be_true endend

Go Test It (≈Selenium)

require 'gotest'

gotest "http://www.google.co.uk" do open "/" type "q", "hello world" click "btnG" verify_text_present "Hello world program - Wikipedia"end

Cucumber

Feature: Google search In order to learn something new As an enthusiast for obscure programming languages I want to find "Hello world" programs on Google

Scenario: Search for "hello world" Given that I am on google.co.uk When I enter "hello world" into the search box And I submit the form Then I should be shown a list of results And the page should contain "Hello world program - Wikipedia"

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

WebDriver

Watir

Selenium

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

WebDriver

Watir

Selenium

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

WebDriver

Watir

Selenium

IE

Firefox

Safari

Chrome

Opera

HTMLUnit

WebDriver

Watir

Selenium

The future?

Watir API

Selenium API

Ok.

Fundamental goal:

Robust tests

Choose your locators carefully

Choose your locators carefully

id = searchInputname = search

xpath = //form[@id='searchform']/input[2]

link = EU makes 7bn euro climate pledge

link = EU makes 7bn euro climate pledge

xpath = //td[@class='text']/div[1]/a

link = EU makes 7bn euro climate pledge

css = a.tsh

xpath = //td[@class='text']/div[1]/a

People are complicated

⇒Frontend tests are

messy

2 approaches:

Disposable tests(record/playback, quick&dirty)

Domain-specific abstractions(carefully crafted)

Which approach will

you take?

Thank you!

Martin KleppmannGo Test It

Email: martin@go-test.itBlog: yes-no-cancel.co.uk

Twitter: @martinkl

Fundamental goal:

Parallel tests

Don’t assume a clean database

Don’t assume a clean database

Unit tests:

Load fixtures

Given...

When...

Then...

Reset database

Don’t assume a clean database

Unit tests:

Load fixtures

Given...

When...

Then...

Reset database

Functional tests:

Set up example data

Do stuff

Checkoutcome

Do stuff

Checkoutcome

Tolerate partially run tests

Tolerate partially run tests

(Assume there is no frob)

Create frob

Check frob was created

Delete frob

Tolerate partially run tests

(Assume there is no frob)

Create frob

Check frob was created

Delete frob

Tolerate partially run tests

(Assume there is no frob)

Create frob

Check frob was created

Delete frob

Boom!

Bad:

Tolerate partially run tests

(Assume there is no frob)

Create frob

Check frob was created

Delete frob

Boom!

Bad:

Delete frob if it exists

Create frob

Check frob was created

Delete frob (optional)

Better:

Avoid race conditions: isolation

Avoid race conditions: isolation

Delete frob if it exists

Create frob

Check frob was created

Delete frob if it exists

Create frob

Check frob was created

Avoid race conditions: isolation

Delete frob if it exists

Create frob

Check frob was created

Delete frob if it exists

Create frob

Check frob was created

Log in as User A Log in as User B

Avoid race conditions: isolation

Delete frob if it exists

Create frob

Check frob was created

Delete frob if it exists

Create frob

Check frob was created

Log in as User A Log in as User B

User A and User B: independent & not used concurrently

Make test runs unique

Make test runs unique

Make test runs unique

Take care with external services

Thank you!

Martin KleppmannGo Test It

Email: martin@go-test.itBlog: yes-no-cancel.co.uk

Twitter: @martinkl