spa 2009 - acceptance testing ajax web applications through the gui
DESCRIPTION
These are the slides that Andrew MacLeod and Patrick Myles presented at the SPA conference in London, April 2009 http://www.spaconference.org/spa2009/sessions/session219.htmlTRANSCRIPT
© 2 0 0 9 C A P L I N S Y S T E M S
Testing Ajax ApplicationsSPA 2009
2© 2 0 0 9 C A P L I N S Y S T E M S
INTRODUCTIONS
Testing AJAX Applications
3© 2 0 0 9 C A P L I N S Y S T E M S
Who we are
Patrick MylesHead of Engineering, Caplin Systems
Andrew MacLeodJavaScript Architect, Caplin Systems
4© 2 0 0 9 C A P L I N S Y S T E M S
How this talk came about
We have a big Ajax application We use Agile We needed a way to test each release
This is the overview we wish
we’d had 2 years ago
5© 2 0 0 9 C A P L I N S Y S T E M S
This is not a Selenium HOWTO
Just enough Selenium to: understand the grammar promote a discussion on frameworks see how good Selenium is!
Not enough Selenium to: lose non-programmers (e.g. QA engineers) get bogged down in Selenium nuts-and-bolts see how frustrating Selenium is!
6© 2 0 0 9 C A P L I N S Y S T E M S
Frustration
Browsers were never designed as
platforms for automated tests...
...so browser test tools are a hack!
7© 2 0 0 9 C A P L I N S Y S T E M S
We would like you to leave with...
Confidence automated testing hands-on experience
Ideas designing tests tool selection building a framework planning a test programme
8© 2 0 0 9 C A P L I N S Y S T E M S
THE PROBLEM
Testing AJAX Applications
The Problem
How can we ensure that each release of our application works everywhere?
10© 2 0 0 9 C A P L I N S Y S T E M S
Manual Testing
Repeat for each new feature Repeat for all supported platforms
BrowserApplication
DO
M
Tester(acting as the User)
TestPack
.
11© 2 0 0 9 C A P L I N S Y S T E M S
Automated Testing
Robot(acting as the User)
BrowserApplication
DO
MTestScript
How intelligent is the robot? How human readable is the script? How much effort does the robot consume? .
12© 2 0 0 9 C A P L I N S Y S T E M S
DOM browser control
COM browser control
Screen coordinates
TestFramework
Behaviour DrivenDevelopment
Domain SpecificLanguage (DSL)
Tools
FIT
TestNG
Watir
WebDriverMercury
Selenium
Windmill
LoadRunnerAuto-Hotkey
WindowLicker
Cucumber
rspec
hamcrest
WebTest
JSUnit
Fitnesse
HttpUnit
13© 2 0 0 9 C A P L I N S Y S T E M S
Workshop
Why are browser applications
so difficult to test?
14© 2 0 0 9 C A P L I N S Y S T E M S
CAPLIN: 5 MINUTE CASE STUDY
Testing AJAX Applications
Caplin Trader
Quick demonstration
16© 2 0 0 9 C A P L I N S Y S T E M S
200920082007
Caplin’s Timeline
VM F
arm
Auto
Hot
-Key
Web
Driver
Fire
fox
1.5
IE6/
FF2
x ht
tps
IE6,
FF2
IE6/
FF2
x ht
tps
IE7,
IE8
Smoke Test (continuous integration)
Selenium Framework v1
Performance Tests
Functional Tests (nightly)
Qua
lity
Gat
e
Selenium Framework v2
Development
QA
Reference Implementation
Bespoke Implementations (up to 3 in parallel)
Sco
pe
Ap
ps
Au
tho
rs
Fra
mew
ork
Our first test...
... a more recent test
goal: Use product finder portletdo Login (${username}=”user1", ${password}="tester")do CreateNewLayoutdo Insert (${portlet}=“Product Finder”)do AssertPortletPresent (${portlet}=“Product Finder”)do Logout
19© 2 0 0 9 C A P L I N S Y S T E M S
EXERCISE 1:SELENIUM IDE: SEND AN EMAIL
Testing AJAX Applications
Objective
To get familiar with the Selenium IDE To produce a simple test script To review the script afterwards Experience some Ajax testing pitfalls
Schema
record
playback
interact
22© 2 0 0 9 C A P L I N S Y S T E M S
User Story
“As a user,I want to send an e-mail messageso that I can stay in touch with my friend”
23© 2 0 0 9 C A P L I N S Y S T E M S
Golden Rule
Put yourself in the user’s position
write the test first watch it fail fix it
24© 2 0 0 9 C A P L I N S Y S T E M S
Tools: Firefox Add-ons
Firefox Selenium IDE easy authoring tool
Firebug debugger reverse engineering tool
XPather locates UI elements
25© 2 0 0 9 C A P L I N S Y S T E M S
Demo
Open Firefox IDE Record Save Playback
26© 2 0 0 9 C A P L I N S Y S T E M S
Setup… Disconnect from LAN Connect to CaplinSpa2009 wireless
network Plug USB key Open Firefox + Selenium IDE (set ?:\Examples\user-extensions.js) Navigate to http://spa2009.com/webmail
27© 2 0 0 9 C A P L I N S Y S T E M S
Your Turn… Login:
user: spa1 password: caplin2009
Send mail To: [email protected] Insert appropriate actions:
pause waitForElementPresent waitForEditable
Save in /Exercise1-Basic/
28© 2 0 0 9 C A P L I N S Y S T E M S
Selenium IDE Review
Pros easy to set up easy to write easy to run little Selenium knowledge
required generates code!
Cons no explicit verification locators are unintelligible brittle unstructured desktop orientated feature must exist
29© 2 0 0 9 C A P L I N S Y S T E M S
SELENIUM: 5 MINUTE TUTORIAL
Testing AJAX Applications
30© 2 0 0 9 C A P L I N S Y S T E M S
Selenium Test Script
One stepper line...
action thing to act on
optionalargument
Actions
SeleniumSelenium.prototype.doType(sInputLocator, sInputText)Selenium.prototype.doClick(sLocator)
Custom FrameworkSelenium.prototype.doClickControl(sLocator)
31© 2 0 0 9 C A P L I N S Y S T E M S
Locators
Seleniumxpath
id
link
...
Custom Framework
PageBot.prototype.locateElementByMenuBarItem(sLocatorText, oDocument)
32© 2 0 0 9 C A P L I N S Y S T E M S
33© 2 0 0 9 C A P L I N S Y S T E M S
Accessors
Selenium
Selenium.prototype.isElementPresent(sLocator)
Custom
Selenium.prototype.getMessageCount(sLocator)
waitForstoreassertverify
ElementPresent
waitForstoreassertverify
MessageCount
34© 2 0 0 9 C A P L I N S Y S T E M S
SCRIPTS:INTERPRETERS AND RUNNERS
Testing AJAX Applications
35© 2 0 0 9 C A P L I N S Y S T E M S
Script: Machine vs. Human readable
Pianola(acting as pianist)
Piano Roll
36© 2 0 0 9 C A P L I N S Y S T E M S
After the break...
BrowserApplication
DOM
LinguisticConstructs
ApplicationAbstractions
COMRunner
SourceCode
Interpreter
• structure• functions• control flow
• locatorstrategies
• applicationoperations
• statetransitions
compiler
37© 2 0 0 9 C A P L I N S Y S T E M S
COFFEE BREAK (10 MINUTES)
Testing AJAX Applications
38© 2 0 0 9 C A P L I N S Y S T E M S
Design PrinciplesTesting Ajax applications
39© 2 0 0 9 C A P L I N S Y S T E M S
Design Principles
Domain Hierarchies how to make my tests more readable? modeling vs. maintainability? how can I isolate application changes?
Test structure and composition how do I avoid duplication? how do my tests map to my requirements? how can I make my framework scalable?
40© 2 0 0 9 C A P L I N S Y S T E M S
DOMAIN HIERARCHIES
Testing AJAX Applications
41© 2 0 0 9 C A P L I N S Y S T E M S
System Domain(OO language, may model business domain)
Browser Domain(DOM, text nodes, clicks, etc.)
?
Verification Domain(Locate, input, verify, etc.)
Application
Test tool
Test framework
42© 2 0 0 9 C A P L I N S Y S T E M S
What to use for framework domain?
1. Don’t bother! ...
43© 2 0 0 9 C A P L I N S Y S T E M S
System Domain(OO language, may model business domain)
Browser Domain(DOM, text nodes, clicks, etc.)
Verification Domain(Locate, input, verify, etc.)
Application
Test tool
Test framework
?Test framework
44© 2 0 0 9 C A P L I N S Y S T E M S
1. Don’t bother...Quick to get going
Can use off-the-shelf tools (e.g. Selenium IDE)
Lots of duplication in tests
Maintenance nightmare
Brittle tests
Rapidly become hard to read and author
What to use for framework domain?
45© 2 0 0 9 C A P L I N S Y S T E M S
What to use for framework domain?
2. Mirror of system domain...
46© 2 0 0 9 C A P L I N S Y S T E M S
??
System Domain(OO language, may model business domain)
Browser Domain(DOM, text nodes, clicks, etc.)
Verification Domain(Locate, input, verify, etc.)
Application
Test tool
Test framework
System Domain (again)
Browser Domain(DOM, text nodes, clicks, etc.)
Verification Domain(Locate, input, verify, etc.)
47© 2 0 0 9 C A P L I N S Y S T E M S
2. Mirror of system domain...Usually in same language as System
Seems attractive – particularly if System is well modelled on problem domain
Can “share” domain objects
Often ends up with either tight coupling...
...or duplicated code / abstractions on both sides of the Browser/Verification divide
Using same language as System doesn’t reinforce the domain separation
What to use for framework domain?
48© 2 0 0 9 C A P L I N S Y S T E M S
What to use for framework domain?
3. Model or DSL that reinforces user’s interactions and use-cases...
49© 2 0 0 9 C A P L I N S Y S T E M S
?Business (User) Domain / DSL(Inbox, compose, check mail, logout, etc.)
System Domain(OO language, may model business domain)
Browser Domain(DOM, text nodes, clicks, etc.)
Verification Domain(Locate, input, verify, etc.)
Application
Test tool
Test framework
50© 2 0 0 9 C A P L I N S Y S T E M S
3. Model or DSL that reinforces user’s interactions and use-cases...
Easier to understand for non-programmers
Less brittle when underlying domains change
More natural abstractions
Requires more maintenance of different abstractions
Can make it hard to implement complex interactions
What to use for framework domain?
51© 2 0 0 9 C A P L I N S Y S T E M S
EXERCISE 2: USING A FRAMEWORK
Testing AJAX Applications
52© 2 0 0 9 C A P L I N S Y S T E M S
Objectives
Move test script closer to application domain Introduce logical names for screen controls Remove physical references to screen controls
Show Selenium extensions Make tests more readable
53© 2 0 0 9 C A P L I N S Y S T E M S
Selenium Framework
BrowserApplication
DOM
ApplicationAbstractions
COMRunner
SourceCode
Interpreter
• actions
• locators
54© 2 0 0 9 C A P L I N S Y S T E M S
Exercise 2 - Framework
Demo
55© 2 0 0 9 C A P L I N S Y S T E M S
UCD TEST STRUCTURE
Testing AJAX Applications
56© 2 0 0 9 C A P L I N S Y S T E M S
User Centered Design (UCD)
A powerful way to model and design a system
Like Agile, focuses on Users – their needs and goals
Typically used for UI/UX design, but provides great models for test frameworks
We use with a simple text DSL
57© 2 0 0 9 C A P L I N S Y S T E M S
User Centered Design (UCD)
Brings focus back to users and their Goals Tasks
• Actions
Fits well with user story structure:
As a [type of user]
I want to [a task or sequence of tasks]
so that I can [achieve some goal]
58© 2 0 0 9 C A P L I N S Y S T E M S
Use-Cases and UCD
Goal
Task
Action
UCD concepts map well to use-case levels
59© 2 0 0 9 C A P L I N S Y S T E M S
Story Structure
Send Email Login Action
Action
Send Message Action
Action
Action
Logout Action
Action
Goal Tasks Actions
60© 2 0 0 9 C A P L I N S Y S T E M S
EXERCISE 3:RUNNING A UCD TEST
Testing AJAX Applications
61© 2 0 0 9 C A P L I N S Y S T E M S
Objectives
Demonstrate UCD structure Show task re-use Implement a new task
62© 2 0 0 9 C A P L I N S Y S T E M S
Old Structure
Test Action
Action
Action
Action
Action
Action
Action
Action
63© 2 0 0 9 C A P L I N S Y S T E M S
UCD Structure
Goal Task Action
Action
Task Action
Action
Action
Task Action
Action
64© 2 0 0 9 C A P L I N S Y S T E M S
Demonstration
Goal/Task/Action hierarchy Task level convenient for
Reuse Maintenance Communication
Action level too small to be reusable
65© 2 0 0 9 C A P L I N S Y S T E M S
UCD Structure
Send Email Login Action
Action
Send Message Action
Action
Action
Logout Action
Action
Verify Sent Action
Action
66© 2 0 0 9 C A P L I N S Y S T E M S
A new story…
“As a user,I want to receive an emailso that I can prove I have a friend!”
67© 2 0 0 9 C A P L I N S Y S T E M S
Your Turn...
Receive Email Login Action
Action
Logout Action
Action
Verify Inbox non-empty
Action
Action
68© 2 0 0 9 C A P L I N S Y S T E M S
SUMMARY
Testing AJAX Applications
Summary
Selenium is not a perfect testing tool, but… Selenium extensions are effective
actions locators accessors
UCD is a good way to structure test cases goals tasks actions
Taking things forward…
-- goal: use product finder portletdo Login (${username}=”user1", ${password}="tester")do CreateNewLayoutdo Insert (${portlet}=“Product Finder”)do AssertPortletPresent (${portlet}=“Product Finder”)do Logout
Develop a simple DSL for testers to use invoke tasks with parameters get rid of ugly HTML “Selenese” use a PATH mechanism to locate tasks