spa 2009 - acceptance testing ajax web applications through the gui

© 2 0 0 9 C A P L I N S Y S T E M S

Testing Ajax ApplicationsSPA 2009

2© 2 0 0 9 C A P L I N S Y S T E M S

INTRODUCTIONS

Testing AJAX Applications

3© 2 0 0 9 C A P L I N S Y S T E M S

Who we are

Patrick MylesHead of Engineering, Caplin Systems

Andrew MacLeodJavaScript Architect, Caplin Systems

4© 2 0 0 9 C A P L I N S Y S T E M S

How this talk came about

We have a big Ajax application We use Agile We needed a way to test each release

This is the overview we wish

we’d had 2 years ago

5© 2 0 0 9 C A P L I N S Y S T E M S

This is not a Selenium HOWTO

Just enough Selenium to: understand the grammar promote a discussion on frameworks see how good Selenium is!

Not enough Selenium to: lose non-programmers (e.g. QA engineers) get bogged down in Selenium nuts-and-bolts see how frustrating Selenium is!

6© 2 0 0 9 C A P L I N S Y S T E M S

Frustration

Browsers were never designed as

platforms for automated tests...

...so browser test tools are a hack!

7© 2 0 0 9 C A P L I N S Y S T E M S

We would like you to leave with...

Confidence automated testing hands-on experience

Ideas designing tests tool selection building a framework planning a test programme

8© 2 0 0 9 C A P L I N S Y S T E M S

THE PROBLEM

Testing AJAX Applications

The Problem

How can we ensure that each release of our application works everywhere?

10© 2 0 0 9 C A P L I N S Y S T E M S

Manual Testing

Repeat for each new feature Repeat for all supported platforms

BrowserApplication

DO

M

Tester(acting as the User)

TestPack

.

11© 2 0 0 9 C A P L I N S Y S T E M S

Automated Testing

Robot(acting as the User)

BrowserApplication

DO

MTestScript

How intelligent is the robot? How human readable is the script? How much effort does the robot consume? .

12© 2 0 0 9 C A P L I N S Y S T E M S

DOM browser control

COM browser control

Screen coordinates

TestFramework

Behaviour DrivenDevelopment

Domain SpecificLanguage (DSL)

Tools

FIT

TestNG

Watir

WebDriverMercury

Selenium

Windmill

LoadRunnerAuto-Hotkey

WindowLicker

Cucumber

rspec

hamcrest

WebTest

JSUnit

Fitnesse

HttpUnit

13© 2 0 0 9 C A P L I N S Y S T E M S

Workshop

Why are browser applications

so difficult to test?

14© 2 0 0 9 C A P L I N S Y S T E M S

CAPLIN: 5 MINUTE CASE STUDY

Testing AJAX Applications

Caplin Trader

Quick demonstration

16© 2 0 0 9 C A P L I N S Y S T E M S

200920082007

Caplin’s Timeline

VM F

arm

Auto

Hot

-Key

Web

Driver

Fire

fox

1.5

IE6/

FF2

x ht

tps

IE6,

FF2

IE6/

FF2

x ht

tps

IE7,

IE8

Smoke Test (continuous integration)

Selenium Framework v1

Performance Tests

Functional Tests (nightly)

Qua

lity

Gat

e

Selenium Framework v2

Development

QA

Reference Implementation

Bespoke Implementations (up to 3 in parallel)

Sco

pe

Ap

ps

Au

tho

rs

Fra

mew

ork

Our first test...

... a more recent test

goal: Use product finder portletdo Login (${username}=”user1", ${password}="tester")do CreateNewLayoutdo Insert (${portlet}=“Product Finder”)do AssertPortletPresent (${portlet}=“Product Finder”)do Logout

19© 2 0 0 9 C A P L I N S Y S T E M S

EXERCISE 1:SELENIUM IDE: SEND AN EMAIL

Testing AJAX Applications

Objective

To get familiar with the Selenium IDE To produce a simple test script To review the script afterwards Experience some Ajax testing pitfalls

Schema

record

playback

interact

22© 2 0 0 9 C A P L I N S Y S T E M S

User Story

“As a user,I want to send an e-mail messageso that I can stay in touch with my friend”

23© 2 0 0 9 C A P L I N S Y S T E M S

Golden Rule

Put yourself in the user’s position

write the test first watch it fail fix it

24© 2 0 0 9 C A P L I N S Y S T E M S

Tools: Firefox Add-ons

Firefox Selenium IDE easy authoring tool

Firebug debugger reverse engineering tool

XPather locates UI elements

25© 2 0 0 9 C A P L I N S Y S T E M S

Demo

Open Firefox IDE Record Save Playback

26© 2 0 0 9 C A P L I N S Y S T E M S

Setup… Disconnect from LAN Connect to CaplinSpa2009 wireless

network Plug USB key Open Firefox + Selenium IDE (set ?:\Examples\user-extensions.js) Navigate to http://spa2009.com/webmail

27© 2 0 0 9 C A P L I N S Y S T E M S

Your Turn… Login:

user: spa1 password: caplin2009

Send mail To: patrick@spa2009.com Insert appropriate actions:

pause waitForElementPresent waitForEditable

Save in /Exercise1-Basic/

28© 2 0 0 9 C A P L I N S Y S T E M S

Selenium IDE Review

Pros easy to set up easy to write easy to run little Selenium knowledge

required generates code!

Cons no explicit verification locators are unintelligible brittle unstructured desktop orientated feature must exist

29© 2 0 0 9 C A P L I N S Y S T E M S

SELENIUM: 5 MINUTE TUTORIAL

Testing AJAX Applications

30© 2 0 0 9 C A P L I N S Y S T E M S

Selenium Test Script

One stepper line...

action thing to act on

optionalargument

Actions

SeleniumSelenium.prototype.doType(sInputLocator, sInputText)Selenium.prototype.doClick(sLocator)

Custom FrameworkSelenium.prototype.doClickControl(sLocator)

31© 2 0 0 9 C A P L I N S Y S T E M S

Locators

Seleniumxpath

id

link

...

Custom Framework

PageBot.prototype.locateElementByMenuBarItem(sLocatorText, oDocument)

32© 2 0 0 9 C A P L I N S Y S T E M S

33© 2 0 0 9 C A P L I N S Y S T E M S

Accessors

Selenium

Selenium.prototype.isElementPresent(sLocator)

Custom

Selenium.prototype.getMessageCount(sLocator)

waitForstoreassertverify

ElementPresent

waitForstoreassertverify

MessageCount

34© 2 0 0 9 C A P L I N S Y S T E M S

SCRIPTS:INTERPRETERS AND RUNNERS

Testing AJAX Applications

35© 2 0 0 9 C A P L I N S Y S T E M S

Script: Machine vs. Human readable

Pianola(acting as pianist)

Piano Roll

36© 2 0 0 9 C A P L I N S Y S T E M S

After the break...

BrowserApplication

DOM

LinguisticConstructs

ApplicationAbstractions

COMRunner

SourceCode

Interpreter

• structure• functions• control flow

• locatorstrategies

• applicationoperations

• statetransitions

compiler

37© 2 0 0 9 C A P L I N S Y S T E M S

COFFEE BREAK (10 MINUTES)

Testing AJAX Applications

38© 2 0 0 9 C A P L I N S Y S T E M S

Design PrinciplesTesting Ajax applications

39© 2 0 0 9 C A P L I N S Y S T E M S

Design Principles

Domain Hierarchies how to make my tests more readable? modeling vs. maintainability? how can I isolate application changes?

Test structure and composition how do I avoid duplication? how do my tests map to my requirements? how can I make my framework scalable?

40© 2 0 0 9 C A P L I N S Y S T E M S

DOMAIN HIERARCHIES

Testing AJAX Applications

41© 2 0 0 9 C A P L I N S Y S T E M S

System Domain(OO language, may model business domain)

Browser Domain(DOM, text nodes, clicks, etc.)

?

Verification Domain(Locate, input, verify, etc.)

Application

Test tool

Test framework

42© 2 0 0 9 C A P L I N S Y S T E M S

What to use for framework domain?

1. Don’t bother! ...

43© 2 0 0 9 C A P L I N S Y S T E M S

System Domain(OO language, may model business domain)

Browser Domain(DOM, text nodes, clicks, etc.)

Verification Domain(Locate, input, verify, etc.)

Application

Test tool

Test framework

?Test framework

44© 2 0 0 9 C A P L I N S Y S T E M S

1. Don’t bother...Quick to get going

Can use off-the-shelf tools (e.g. Selenium IDE)

Lots of duplication in tests

Maintenance nightmare

Brittle tests

Rapidly become hard to read and author

What to use for framework domain?

45© 2 0 0 9 C A P L I N S Y S T E M S

What to use for framework domain?

2. Mirror of system domain...

46© 2 0 0 9 C A P L I N S Y S T E M S

??

System Domain(OO language, may model business domain)

Browser Domain(DOM, text nodes, clicks, etc.)

Verification Domain(Locate, input, verify, etc.)

Application

Test tool

Test framework

System Domain (again)

Browser Domain(DOM, text nodes, clicks, etc.)

Verification Domain(Locate, input, verify, etc.)

47© 2 0 0 9 C A P L I N S Y S T E M S

2. Mirror of system domain...Usually in same language as System

Seems attractive – particularly if System is well modelled on problem domain

Can “share” domain objects

Often ends up with either tight coupling...

...or duplicated code / abstractions on both sides of the Browser/Verification divide

Using same language as System doesn’t reinforce the domain separation

What to use for framework domain?

48© 2 0 0 9 C A P L I N S Y S T E M S

What to use for framework domain?

3. Model or DSL that reinforces user’s interactions and use-cases...

49© 2 0 0 9 C A P L I N S Y S T E M S

?Business (User) Domain / DSL(Inbox, compose, check mail, logout, etc.)

System Domain(OO language, may model business domain)

Browser Domain(DOM, text nodes, clicks, etc.)

Verification Domain(Locate, input, verify, etc.)

Application

Test tool

Test framework

50© 2 0 0 9 C A P L I N S Y S T E M S

3. Model or DSL that reinforces user’s interactions and use-cases...

Easier to understand for non-programmers

Less brittle when underlying domains change

More natural abstractions

Requires more maintenance of different abstractions

Can make it hard to implement complex interactions

What to use for framework domain?

51© 2 0 0 9 C A P L I N S Y S T E M S

EXERCISE 2: USING A FRAMEWORK

Testing AJAX Applications

52© 2 0 0 9 C A P L I N S Y S T E M S

Objectives

Move test script closer to application domain Introduce logical names for screen controls Remove physical references to screen controls

Show Selenium extensions Make tests more readable

53© 2 0 0 9 C A P L I N S Y S T E M S

Selenium Framework

BrowserApplication

DOM

ApplicationAbstractions

COMRunner

SourceCode

Interpreter

• actions

• locators

54© 2 0 0 9 C A P L I N S Y S T E M S

Exercise 2 - Framework

Demo

55© 2 0 0 9 C A P L I N S Y S T E M S

UCD TEST STRUCTURE

Testing AJAX Applications

56© 2 0 0 9 C A P L I N S Y S T E M S

User Centered Design (UCD)

A powerful way to model and design a system

Like Agile, focuses on Users – their needs and goals

Typically used for UI/UX design, but provides great models for test frameworks

We use with a simple text DSL

57© 2 0 0 9 C A P L I N S Y S T E M S

User Centered Design (UCD)

Brings focus back to users and their Goals Tasks

• Actions

Fits well with user story structure:

As a [type of user]

I want to [a task or sequence of tasks]

so that I can [achieve some goal]

58© 2 0 0 9 C A P L I N S Y S T E M S

Use-Cases and UCD

Goal

Task

Action

UCD concepts map well to use-case levels

59© 2 0 0 9 C A P L I N S Y S T E M S

Story Structure

Send Email Login Action

Action

Send Message Action

Action

Action

Logout Action

Action

Goal Tasks Actions

60© 2 0 0 9 C A P L I N S Y S T E M S

EXERCISE 3:RUNNING A UCD TEST

Testing AJAX Applications

61© 2 0 0 9 C A P L I N S Y S T E M S

Objectives

Demonstrate UCD structure Show task re-use Implement a new task

62© 2 0 0 9 C A P L I N S Y S T E M S

Old Structure

Test Action

Action

Action

Action

Action

Action

Action

Action

63© 2 0 0 9 C A P L I N S Y S T E M S

UCD Structure

Goal Task Action

Action

Task Action

Action

Action

Task Action

Action

64© 2 0 0 9 C A P L I N S Y S T E M S

Demonstration

Goal/Task/Action hierarchy Task level convenient for

Reuse Maintenance Communication

Action level too small to be reusable

65© 2 0 0 9 C A P L I N S Y S T E M S

UCD Structure

Send Email Login Action

Action

Send Message Action

Action

Action

Logout Action

Action

Verify Sent Action

Action

66© 2 0 0 9 C A P L I N S Y S T E M S

A new story…

“As a user,I want to receive an emailso that I can prove I have a friend!”

67© 2 0 0 9 C A P L I N S Y S T E M S

Your Turn...

Receive Email Login Action

Action

Logout Action

Action

Verify Inbox non-empty

Action

Action

68© 2 0 0 9 C A P L I N S Y S T E M S

SUMMARY

Testing AJAX Applications

Summary

Selenium is not a perfect testing tool, but… Selenium extensions are effective

actions locators accessors

UCD is a good way to structure test cases goals tasks actions

Taking things forward…

-- goal: use product finder portletdo Login (${username}=”user1", ${password}="tester")do CreateNewLayoutdo Insert (${portlet}=“Product Finder”)do AssertPortletPresent (${portlet}=“Product Finder”)do Logout

Develop a simple DSL for testers to use invoke tasks with parameters get rid of ugly HTML “Selenese” use a PATH mechanism to locate tasks