selenium rc: automated testing of modern web applications

Selenium RC: Automated Testing Selenium RC: Automated Testing of Modern Web Applicationsof Modern Web Applications

Thomas Herchenröder, 1&1 Internet AG

OverviewOverview

● Testing Web Apps● Selenium RC● Extending Selenium RC

Web Testing: Goals and Web Testing: Goals and ChallengesChallenges

Goals:

● create tests● run automatically, repetitively● regression testing● compare results


Challenges:

● GUI app testing● OS/browser/platform● JS engines/versions/levels, plug-ins

Web Testing: ApproachesWeb Testing: Approaches

● GUI Automation● locating by coordinates● using generic OS mouse/keyboard events● suitable for any GUI app

● HTML Automation● parsing HTML● evaluating HTML elements (forms, ...)● “batch browsers”

● DOM Automation● DOM as presentation of application● DOM traversal (getElementByName, ...)● interacting with DOM tree (target.dispatchEvent,...)

A Simple Example: GUI A Simple Example: GUI AutomationAutomation

A Simple Example: HTML A Simple Example: HTML AutomationAutomation

A Simple Example: DOM A Simple Example: DOM AutomationAutomation

A Complex Example: GUI A Complex Example: GUI AutomationAutomation

A Complex Example: HTML A Complex Example: HTML AutomationAutomation

A Complex Example: DOM A Complex Example: DOM AutomationAutomation

Web Testing: Approaches (Rev.)Web Testing: Approaches (Rev.)

● GUI Automation● constant complexity● platform-dependent● fragile

● HTML Automation● insufficient content capture

● DOM Automation● DOM tree represents application● sufficient content capture

Web Testing: DOM AutomationWeb Testing: DOM Automation

● full access to app elements● full capture of dynamic content● issue: locating elements (reliably)● issue: non-interference with AUT

● you can embed test code with AUT● issue: reliability of results● (i)frames● cross-frame interaction

OverviewOverview


Selenium: selenium.openqa.orgSelenium: selenium.openqa.org

● Selenium Core● Selenium IDE● Selenium on Rails● Selenium RC

● Selenium Grid● open source, OpenQA hosted, ThoughtWorks

Selenium RC: ArchitectureSelenium RC: Architecture

Selenium RCSelenium RC

● overcome SOP restriction● drive tests programmatically● language bindings (Java, Python, Perl, ...)

● access to OS features (timers, threads, screen shots, ...)

● user-extensions.js

Selenium RC: The BrowserSelenium RC: The Browser

Selenium RC: An Interactive Selenium RC: An Interactive SessionSession

● Starting the server:● java -jar selenium-server.jar -interactive

● Interactive commands:● cmd=getNewBrowserSession&1=*firefox&2=http://d

emo.qooxdoo.org● cmd=open&1=http://demo.qooxdoo.org/current/sho

wcase● cmd=qxClick&1=//div[text()="Tab"]● cmd=testComplete● quit

OverviewOverview


Selenium and qooxdooSelenium and qooxdoo

● crucial issues:● object identification● event generation

● qooxdoo rendering: a world of <div>s● Selenium commands● Selenium locators● the need for (and feasibility of) qooxdoo specific extensions

userextensions.js: commandsuserextensions.js: commands

Selenium.prototype.doQxClick = function(locator, eventParams)

{ var element = this.page().findElement(locator);

this.clickElementQx(element, eventParams);

};

userextensions.js: locatorsuserextensions.js: locators

PageBot.prototype.locateElementByQx = function(qxLocator, inDocument, inWindow)

{ LOG.info("Locate Element by qooxdoo UserData

Locator”);

var qxObject = this._findQxObjectInWindow(qxLocator, inWindow);

if (qxObject) { return qxObject.getElement(); }};

userextensions.js: language userextensions.js: language bindingsbindings

● new custom extensions:● support for qx locator

● just a string passed as parameter of command

● no change to client driver API● support for qxClick command

● must be supported in language bindings

● so client drivers can use it● Sample:

● qxClick(“qx:foo/bar/baz”)


● Python:● def qxClick(self, locator, opts=None):

return self.do_command("qxClick", [locator,]) ● sel.qxClick(r'//div[text()="Form"]')

● Java● public String qxClick(String locator) {

return this.commandProcessor.doCommand("qxClick", new String[] {locator,}); }

● sel.qxClick('qxh=*/[@label="Tab"]/[@page]/*/[@label="Find"]');

● JavaScript using Rhino● uses Java infrastructure, so no additional overhead

A Client Driver ScriptA Client Driver Script

importClass(Packages.com.thoughtworks.selenium.QxSelenium);

var sel = new QxSelenium("localhost",4444,"*firefox", "http://demo.qooxdoo.org");sel.setSpeed("1000");sel.start();sel.open("http://demo.qooxdoo.org/0.7.3/showcase/");Packages.java.lang.Thread.sleep(2000);sel.qxClick('//div[text()="Tab"]');sel.qxClick('qxh=*/[@label="Tab"]/[@page]/*/[@label=

"Find"]');sel.stop();

ConclusionConclusion● Interacting with the DOM seems to be the best way to test Web applications

● Selenium RC provides a distributed environment to run tests on a broad range of platforms

● Various language bindings exist, so tests can be written as stand-alone, client-server programms in a preferred language

● Custom commands and locators can be added to the Selenium environment, to suite specific testing needs

Selenium RC: Automated Testing of Selenium RC: Automated Testing of Modern Web ApplicationsModern Web Applications

Thank you!


Selenium RC: Automated Testing Selenium RC: Automated Testing of Modern Web Applicationsof Modern Web Applications


This talk is concerned with automated testing of Web applications. It looks at testing Web apps in general, its goals and challenges; it will present Selenium and Selenium RC in particular as a testing platform; and will then focus on adaptions made to Selenium to ease the effort to test apps made with qooxdoo, a JavaScript framework.

OverviewOverview


OverviewOverview


Part I: An Introduction to testing Web applications and its approaches.


Goals:

● create tests● run automatically, repetitively● regression testing● compare results

●When we speak of creating automated tests for Web applications (“application” in the broadest sense – from a static HTML page through CGI-driven applications to full-blown single-page apps), what do we mean by the term 'test'?!●All test start by retrieving a specific web page.●The simplest test might then just look at its static contents, inspecting the page and checking for specific properties (e.g. the correct title, presence of keywords or phrases, etc.).●More advanced tests will want to look at the dynamic aspects of the web page, and therefore need some means to interact with it. This encompasses detecting and following hyperlinks, automatically filling in and submitting forms, or use other interactive elements. In all cases, the outcome of such interaction (new page, change of current page content, ...) has to be checked against some expected result.●Depending on these requirements, test methodology and effort can vary a lot.●Once you have created a test in a suitable form, its purpose is to be run repetitively and, hopefully, automatically, i.e. without user intervention. Results need to be verified, and probably captured, to be compared wit subsequent test runs.●This allows for regression testing and test-driven development, to ensure existing functionality is not broken while the web app is further developed.


Challenges:

● GUI app testing● OS/browser/platform● JS engines/versions/levels, plug-ins

●Every tester loves APIs! A clear defined interface that can be driven programmatically and in a defined and reproducible way. Results can be compared, collected, logged, standardized, or put in a database. The scope of testing is clearly defined (list of API calls), and though parameter ranges and call protocols (sequencing of calls) make things more complex, there often is hope for test coverage (getting all – important – test cases).●GUI apps evade from such a close grip. Although they often have a number of distinct interaction elements (buttons, boxes, fields, ...), they rely on event handling that in most cases defies protocols and a systematic way of testing. Events can be triggered in arbitrary order, and might – or might not – lead to the same results.●Moreover, any GUI can rely on one of the most versatile interpreters available – the human brain. Pictures with colored dots are recognized as numbers, technically unrelated geometrical elements are recognized as text, context is maintained across pages or parts of a page, across time and across any change happening to a page. With this kind of semantic processing, correlations can be established and information can be provided that is very hard to capture by any known automaton. How can 'correctness' or its absence be detected in such a context?●To worsen things, in the realm of web applications the tester is confronted with the 'plague of platforms': multiple OSs, multiple desktops, multiple browsers, multiple configurations, multiple ... To devise tests in this scattered landscape is hard, to devise tests that are maintainable and provide reliable results in all situations is very hard.

Web Testing: ApproachesWeb Testing: Approaches

● GUI Automation● locating by coordinates● using generic OS mouse/keyboard events● suitable for any GUI app

● HTML Automation● parsing HTML● evaluating HTML elements (forms, ...)● “batch browsers”

● DOM Automation● DOM as presentation of application● DOM traversal (getElementByName, ...)● interacting with DOM tree (target.dispatchEvent,...)

●There are mainly three approaches to tackle the problem of web testing:●GUI Automation:

● uses the least common denominator of GUIs: generic mouse and keyboard events.

● uses OS-level I/O event generation.● doesn't care whether it is Web browser, Chat client, File

browser, MP3 player, etc.● but is entirely platform-dependent, since it relies on OS-level

APIs.● result verification is hard (comparing bitmaps?!)● fragile: depends on correct coordinates for mouse actions,

keyboard layouts, themes in place, current key bindings, ...

A Simple Example: GUI A Simple Example: GUI AutomationAutomation

Click to add text

●Locate input field, push search button - GUI automation would certainly work●But: what about result verification?!

A Simple Example: HTML A Simple Example: HTML AutomationAutomation

Click to add text

●fair reflection of the rendered page:● opening “body” tag● navigational links (Images, Maps, ...), with embedded menu

(“more”)● “form” element with input field, buttons● further information links

●functional “meat” is present: input field, buttons

A Simple Example: DOM A Simple Example: DOM AutomationAutomation

Click to add text

●Interesting elements: HTML->Body->Center->Form->...->TD:● Input: hidden● Input: text field for query string● Input: submit button● Input: I'm feeling lucky button

●uses name attribute for input fields ('q' for text field, 'btnG' for submit etc.)●makes it easy to locate (getElementByName(), Xpath, ...)●Conclusion:

● good “semantic” representation of the page and its interactive elements (not layout dependend!)

● easy and robust to drive through DOM manipulations

A Complex Example: GUI A Complex Example: GUI AutomationAutomation

Click to add text

●Complex GUI:● buttons, tabs, panes● multiple embedded “windows”, independently movable● check-/radio-buttons, pop-ups

●Tip: Try imagining this in HTML!●GUI Automation:

● easy, doesn't care about complexity● problematic: exact locating● (influence of screen resolution?)● problematic: re-arranging of elements (e.g. tabs)

●(Next Slide: corresp. HTML - “Prepare for disappointment!”)

A Complex Example: HTML A Complex Example: HTML AutomationAutomation

Click to add text

●HTML: That's all there is to it! :-(●nothing to work with●all HTML-processing programms will fail●Conclusion: completely unusable

A Complex Example: DOM A Complex Example: DOM AutomationAutomation

Click to add text

●again, full access to page elements●harder to relate DOM elements to visual result

● DOM tree as “assembler”●problematic: locating elements (“flood of divs”)●remedy: “id”, “name” etc. attributes●remedy: Xpath queries for “robust” features (unique labels, images etc.)

● still not comfortable, but works

Web Testing: Approaches (Rev.)Web Testing: Approaches (Rev.)

● GUI Automation● constant complexity● platform-dependent● fragile

● HTML Automation● insufficient content capture

● DOM Automation● DOM tree represents application● sufficient content capture

●GUI Automation: ● use the least common denominator of GUIs: generic mouse and

keyboard events.● Use OS-level I/O event generation.● Doesn't care whether it is Web browser, Chat client, File

browser, MP3 player, etc.● But entirely platform-dependent, since it relies on OS-level APIs.● Result verification is hard (comparing bitmaps?!)● Fragile: Depends on correct coordinates for mouse actions,

keyboard layouts, themes in place, current key bindings, ...● Impossible to write one test that runs the same on

Win/Linux/OSX/... in IE/FF/Safari/...●HTML Automation:

● just not feasible for modern applications●DOM Automation:

● provides everything needed● (more next slide)

Web Testing: DOM AutomationWeb Testing: DOM Automation

● full access to app elements● full capture of dynamic content● issue: locating elements (reliably)● issue: non-interference with AUT

● you can embed test code with AUT● issue: reliability of results● (i)frames● cross-frame interaction

Overview of some features of the DOM automation approach:●everything in the app is accessible through its DOM representation●even across dynamic changes●finding the element to interact with can be a challenge, esp. across various revisions of the same app●non-interference: if testing code changes DOM, the app is no longer the same; danger of introducing subtle changes in behaviour

● test code could be part of the app (and this is actually used!)● will AUT behaviour be the same as when driven by a real user?● one solution: separation of AUT and testing logic in different

frames● issue here: cross-frame interaction

OverviewOverview


Part II: An Introduction to Selenium and Selenium RC in particular.

Selenium: selenium.openqa.orgSelenium: selenium.openqa.org

● Selenium Core● Selenium IDE● Selenium on Rails● Selenium RC

● Selenium Grid● open source, OpenQA hosted, ThoughtWorks

●Selenium Core:● JavaScript engine● runs in browser● controls AUT in other frame● exposes API

●Selenium IDE:● FF plug-in● directly communicates with Selenium Core in the browser● record/replay features● great to get started (interactively)

●Selenium RC:● remote control● server component, proxy● allows test automation in various ways● this is what we will focus on● has a capability called Selenium Grid, to drive multiple instances

of the SRC server●project frame: open source (Apache 2.0 lic.), OpenQA run by collaboration of a few companies

Selenium RC: ArchitectureSelenium RC: Architecture

●Selenium RC server (blue)● starts browser instance with modified profile● promotes itself as proxy for all requests● injects Selenium Core with startup page● all requests go through server, dispatches on URL

●browser instance (orange)● initial page installs Core, multi-(i)frame● Core maintains connection with server (comet-like, long polling)● receives commands and performs them in browser (load URL in

AUT frame, click, type, check results, ...)●web server hosting the AUT (brown)

● SRC server appears as normal client●client drivers (green)

● multiple language bindings (Java, Python, Perl, ...)● completely control SRC server

● starting of browser● loading of AUT● any (test) interaction● ending of browser session

Selenium RCSelenium RC

● overcome SOP restriction● drive tests programmatically● language bindings (Java, Python, Perl, ...)

● access to OS features (timers, threads, screen shots, ...)

● user-extensions.js

●SOP: same-origin-policy: ● JS can only make XHR requests to domain it was loaded from● browser only sees a single domain:

http://<domain_of_aut>/selenium-server/core/RemoteRunner.html?...

● SRC server dispatches on URL path● even handles all requests to

http://<domain_of_aut>/<aut_path> (due to proxy)● control ends when requesting a page in another domain

●user-extensions.js● main topic of part III

●Hint: some more 'hands-on' material before

Selenium RC: The BrowserSelenium RC: The Browser

Click to add text

●default appearance●three parts:

● AUT frame● Log frame (command history)● Selenium frame

● “show log” button, for message log● important for debug messages

●Hint: next: log of interactive session to bring this up

Selenium RC: An Interactive Selenium RC: An Interactive SessionSession

● Starting the server:● java -jar selenium-server.jar -interactive

● Interactive commands:● cmd=getNewBrowserSession&1=*firefox&2=http://d

emo.qooxdoo.org● cmd=open&1=http://demo.qooxdoo.org/current/sho

wcase● cmd=qxClick&1=//div[text()="Tab"]● cmd=testComplete● quit

●Starting the server:● Selenium RC is a Java application (built atop of Jetty)● So invoke Java, pass Selenium Server Jar as argument● SRC takes arguments itself (-interactive for interactive shell,

others for AUT window handling, proxying, ... see SRC command line reference)

●Interactive Commands:● First, create a new browser session for this test.

● SRC can handle multiple sessions● Yields SessionID● (which is mandatory as a parameter for later commands

“...&ssessionId=4711”, but can be omitted in single-session mode)

● is automatically kept in the Selenium object in client driver code

● This will create a new browser instance running on the local machine (FF: creating profile, IE: manipulating the Registry).

● Then, open the AUT in this browser session.● This will load the web app in iframe/window, controlled by

Selenium Core.● Perform various actions on the AUT, possibly checking outcomes

(mouse clicks, typing, loading pages, ...)● Finally, end the browser session.● Eventually, end SRC server.

OverviewOverview


Part III: Extending Selenium RC through its user-extensions.js and language bindings.

Selenium and qooxdooSelenium and qooxdoo

● crucial issues:● object identification● event generation

● qooxdoo rendering: a world of <div>s● Selenium commands● Selenium locators● the need for (and feasibility of) qooxdoo specific extensions

●event generation:● Selenium: synthetic events on elements● Sel events on an element (think 'div') might not match qooxdoo's

expected events on the element● 'click' vs. 'mousedown', 'mouseup'● background:

● qooxdoo establishes unified event model over AUT● only top-most widget registers for all events● qooxdoo handles event capture and bubbling internally

●Selenium commands:● tell the browser what to do

●Selenium locators:● determine where to apply a command

●qooxdoo specific extensions:● exactly in these two areas● new commands● new locators

●Rest of talk: practical examples for these

userextensions.js: commandsuserextensions.js: commands

Selenium.prototype.doQxClick = function(locator, eventParams)

{ var element = this.page().findElement(locator);

this.clickElementQx(element, eventParams);

};

●doQxClick:● naming convention for extension commands: “doXxYy” if

command is “xxYy”●locator: locator string from command●eventParams: Selenium-provided eventParam object, filled from command or defaults (think: mouse button, shift key, ...)●findElement(): Sel function, returning DOM object reference●clickElementQx(): custom method that triggers mouseover, mousedown, mouseup, ... events on element (element.dispatchEvent(), element.fireEvent())

userextensions.js: locatorsuserextensions.js: locators

PageBot.prototype.locateElementByQx = function(qxLocator, inDocument, inWindow)

{ LOG.info("Locate Element by qooxdoo UserData

Locator”);

var qxObject = this._findQxObjectInWindow(qxLocator, inWindow);

if (qxObject) { return qxObject.getElement(); }};

●locateElementByQx:● naming conventions for locators: “locateElementByXxx” if

locator is “xxx:”●qxLocator: locator string from Selenium command, specific to locator “qx:”●LOG: another Selenium interface object●_findQxObjectInWindow():

● custom method● parses qxLocator● supports parent/child relation between widgets and “userData”

property of widgets● depends on qooxdoo object hierarchy:

qx.ui.core.ClientDocument etc.● returns JS object (not DOM!)

●qxObject.getElement(): stand. qooxdoo object method to return corresp. DOM object

● to satisfy Sel locator interface


● new custom extensions:● support for qx locator

● just a string passed as parameter of command

● no change to client driver API● support for qxClick command

● must be supported in language bindings

● so client drivers can use it● Sample:

● qxClick(“qx:foo/bar/baz”)

●you want to support the extended features in your test scripts●new locator: just use in locator strings●new command: need to define in language-specific API, in order to use it in test scripts●then use as method of Selenium test object


● Python:● def qxClick(self, locator, opts=None):

return self.do_command("qxClick", [locator,]) ● sel.qxClick(r'//div[text()="Form"]')

● Java● public String qxClick(String locator) {

return this.commandProcessor.doCommand("qxClick", new String[] {locator,}); }

● sel.qxClick('qxh=*/[@label="Tab"]/[@page]/*/[@label="Find"]');

● JavaScript using Rhino● uses Java infrastructure, so no additional overhead

●here are two examples of doing this:●Python:

● the first item shows how to define the new API method● the second how to use it in testing code

●Java:● define the new method● use it in testing code

● this one uses yet another locator that provides XPath-like syntax to traverse the qooxdoo object hierarchy

●you can use JavaScript as implementation language for your tests, atop of the Java language binding, through Rhino (www.mozilla.org/rhino/)●Hint: complete JS example on next slide

A Client Driver ScriptA Client Driver Script

importClass(Packages.com.thoughtworks.selenium.QxSelenium);

var sel = new QxSelenium("localhost",4444,"*firefox", "http://demo.qooxdoo.org");sel.setSpeed("1000");sel.start();sel.open("http://demo.qooxdoo.org/0.7.3/showcase/");Packages.java.lang.Thread.sleep(2000);sel.qxClick('//div[text()="Tab"]');sel.qxClick('qxh=*/[@label="Tab"]/[@page]/*/[@label=

"Find"]');sel.stop();

●importClass() Rhino built-in to load Java classes; QxSelenium is derived class from standard Selenium class, to hold user extensions●create interface object●set properties●start() method will launch browser●open the AUT●use a standard platform (Java) call (“waiting” is inherently difficult in browser-side JavaScript)●invoke test commands with locators●stop() brings down the browser

ConclusionConclusion● Interacting with the DOM seems to be the best way to test Web applications

● Selenium RC provides a distributed environment to run tests on a broad range of platforms

● Various language bindings exist, so tests can be written as stand-alone, client-server programms in a preferred language

● Custom commands and locators can be added to the Selenium environment, to suite specific testing needs

●draw some conclusions

Selenium RC: Automated Testing of Selenium RC: Automated Testing of Modern Web ApplicationsModern Web Applications

Thank you!


selenium rc: automated testing of modern web applications

Technology