node.js and selenium webdriver, a journey from the java side
TRANSCRIPT
Node.js and Selenium WebDriver
A journey from the Java side
19th Nov SF Selenium Meetup @ Airware
Mek Srunyu Stittri
Volkan Gurel
2
Agenda
● Background● Challenges, overcoming the asynchronous● Page object implementation● Applitools screenshot validation integration● Fitting in http request for REST API tests● After tests: Continuous deployment with
Hashicorp Kubernetes and Docker
4
Background
Back in June start looking at node.js for selenium E2E functional test framework.
● Kept Node.js adoption in mind● More and more company moving to node and
going full stack.● Share code with developers and get help
5
Problem statement
Disconnected engineering stack
QA, Automation engineers
Frontend engineers
Backend engineers
Java
Javascript
Java
Python
Javascript
Java
Ruby
Javascript
Node.js
Company A Company B Company C
Node.js
Javascript
Node.js
8
Googling node.js and selenium
Results● NightwatchJS● WebdriverIO ● WD.js● The Intern● webdriver-sync● Cabbie● Selenium-Webdriver● Protractor ● And many more..
10
Javascript 101
Javascript is AsynchronousExample
var source = ['foo', 'bar', 'baz'];
var result = [];
setTimeout(function () {
for (var i = 0 ; i < source.length ; i++) {
console.log('Stepping through : ' + source[i]);
result.push(source[i]);
console.log('Current result: ' + result);
}
}, 1000); // Wait 1000 ms to finish operation
console.log('Result: ' + result);
console.log('Finished!!');
Output:
Result: ←------- Empty array ?!?!
Finished!!
Stepping through : foo
Current result: foo
Stepping through : bar
Current result: foo,bar
Stepping through : baz
Current result: foo,bar,baz
12
Callbacks
Callback Pattern
driver.get("http://www.google.com", function() {
driver.findElement(By.name("q"), function(q) {
q.sendKeys("webdriver", function() {
driver.findElement(By.name("btnG"), function(btnG) {
btnG.click(function() {
driver.getTitle(function(title) {
assertEquals("webdriver - Google Search", title);
});
});
});
});
});
});
Equivalent Java code
driver.get("http://www.google.com");
driver.findElement(By.name("q")).sendKeys("webdriver");
driver.findElement(By.name("btnG")).click();
assertEquals("webdriver - Google Search", driver.
getTitle());
Pyramid of doomhttps://en.wikipedia.org/wiki/Pyramid_of_doom_(programming)
13
Callback example
Marcel Erz’s Southbay Selenium Slides - NodeJs based selenium
Work on your webelement here
14
Promises
Promise Pattern
driver.get("http://www.google.com"). then(function() { return driver.findElement(By.name("q")); }). then(function(q) { return q.sendKeys("webdriver"); }). then(function() { return driver.findElement(By.name("btnG")); }). then(function(btnG) { return btnG.click(); }). then(function() { return driver.getTitle(); }). then(function(title) { assertEquals("webdriver - Google Search", title); });
Equivalent Java code
driver.get("http://www.google.com");
driver.findElement(By.name("q")).sendKeys("webdriver");
driver.findElement(By.name("btnG")).click();
assertEquals("webdriver - Google Search", driver.
getTitle());
15
Promise example
Marcel Erz’s Southbay Selenium Slides - NodeJs based selenium
Work on your webelement here
16
Back to the list
● NightwatchJS● WebdriverIO formerly WebdriverJS● WD.js● The Intern● webdriver-sync● Cabbie● Selenium-Webdriver now WebDriverJs● Protractor ● And many more…
WebDriverNode jwebdriver ot-webdriverjs burnout testium yiewd nwd co-nwd selenium-node-webdriver nemo.js taxi etc...
???
17
Experimenting with Nightwatch
What Nightwatch offers● Convenient chain APIs
○ A workaround for dealing with callbacks● Wraps Selenium JsonWireProtocol● Some form of page object support● Some extendability custom
commands● Saucelabs / Browserstack integration
out of the box● Pretty good documentation
Nightwatch test
module.exports = {
'Demo test Google' : function (browser) {
browser
.url('http://www.google.com')
.waitForElementVisible('body', 1000)
.setValue('input[type=text]', 'nightwatch')
.waitForElementVisible('button[name=btnG]', 1000)
.click('button[name=btnG]')
.pause(1000)
.assert.containsText('#main', 'Night Watch')
.end();
}
};
18
Experimenting with WebdriverIO
WebdriverIO test
client
.init()
.url('https://duckduckgo.com/')
.setValue('#search_form_input_homepage', 'WebdriverIO')
.click('#search_button_homepage')
.getTitle().then(function(title) {
console.log('Title is: ' + title);
// outputs: "Title is: WebdriverIO
})
.end();
What WebdriverIO offers● Convenient chain APIs● A+ Promise support● Wraps Selenium JsonWireProtocol● Saucelabs / Browserstack
integration out of the box● Some form of visual testing capability
○ Based on WebdriverCSS○ Limited support for Applitools
● But.. pageobject ??
19
WebdriverIO & Pageobject
https://github.com/webdriverio/webdriverio/issues/356https://github.com/webdriverio/webdriverio/issues/583
20
Chain based api - Nightwatch
Pageobject
this.clickLogout = function() {
browser
.waitForElement(USERNAME_DROPDOWN_TRIGGER)
.click(USERNAME_DROPDOWN_TRIGGER)
.waitForElement(LOGOUT_BUTTON)
.click(LOGOUT_BUTTON);
return browser;
};
Test code
testLoginProjectOwner: function (browser) {
browser
.page.Login().enterUserInfo(OWNER_USER,
DEFAULT_PASSWORD)
.page.Login().clickSignIn()
.page.Jobs().isJobListPresent()
.page.TopNavBar().verifyUserName("Project Owner")
.page.TopNavBar().clickLogout()
.page.Login().waitForLoginLoad();
}
The nice part
But.. starting to see a pattern forming : a chain within a chain
21
Chain based api - NightwatchThe not so nice..
browser .page.Login().enterUserInfo(OWNER_USER,DEFAULT_PASSWORD) .page.Jobs().getNumberOfJobs(function (result) { var numberOfJobsBefore = result.value.length; browser .page.JobConfig().createJob(jobName) .page.Jobs().getNumberOfJobs(function (result) { var numberOfJobsAfter = result.value.length; Assert.equal(numberOfJobsAfter, numberOfJobsBefore + 1); browser.page.Jobs().getJobs(function (result) { for (var i = 0; i <= result.length; i++) { if (result[i].name === jobName) { jobInfo = result; break; } } }); }).perform(function(client, done){ Assert.equal(jobInfo.name, expectedJobName, 'Job name is correct'); Assert.equal(jobInfo.creator, expectedJobCreator, 'Job creator is correct'); browser.page.TopNav().clickLogout() .end(); }); });}
Chain breaks once you start to do something complex that is not supported in the api
● Datastructure ● Iterating
22
Chain based api - NightwatchThe not so nice..
function getJobRow (index) {
var deferred = new Q.defer();
var jobInfo = {name: '', creator: '', date: ''};
browser.getText(JOB_LIST_ROW + ':nth-child(' + index + ') > td:nth-child(1)', function (result) {
jobInfo.name = result.value;
console.log('Retrieved job name ' + jobInfo.name);
browser.getText(JOB_LIST_ROW + ':nth-child(' + index + ') > td:nth-child(2)', function (result) {
jobInfo.creator = result.value;
console.log('Retrieved job name ' + jobInfo.creator );
browser.getText(JOB_LIST_ROW + ':nth-child(' + index + ') > td:nth-child(3)', function (result) {
jobInfo.date = result.value;
console.log('Retrieved job date ' + jobInfo.date);
deferred.resolve(jobInfo);
});
});
});
return deferred.promise;
}
23
Lessons learned
● Chain based api - a particular bad pattern when async call are involved. As soon as you try to do something complex (dealing with an array of WebElements) you end up having to break the chain.
● Page Object pattern and chained APIs don’t get along well.○ Methods end up containing another chain which does not help
with code composition. Also still prone to pyramid of doom● Most selenium chain based libraries gives you just one main object and
all interaction commands are tied to that object’s chain○ NightwatchJS : browser○ WebdriverIO : client
● Ignore Github Stars when choosing which projects to use...
25
selenium-webdriver WebDriverJs
Then we took a deep look at selenium-webdriver the current WebDriverJshttps://code.google.com/p/selenium/wiki/WebDriverJs#Writing_Tests
WebDriverJs uses a promise manager● Coordinate the scheduling and execution of all commands. ● Maintains a queue of scheduled tasks, executing each once the one before it in the queue is
finished. The WebDriver API is layered on top of the promise manager.
Provided Mocha Framework Wrapper with a built in promise manager
There is a built in wrapper for mocha methods that automatically handles all the calls into the
promise manager which makes the code very sync like.
http://selenium.googlecode.com/git/docs/api/javascript/module_selenium-webdriver_testing.html
26
Achieving sync-like code
Code written using Webdriver Promise ManagerJavascript selenium tests using promise manager
driver.get("http://www.google.com");
driver.findElement(webdriver.By.name('q')).sendKeys('webdriver');
driver.findElement(webdriver.By.name('btnG')).click();
driver.getTitle().then(function(title) {
console.log(title);
});
Equivalent Java code
driver.get("http://www.google.com");
driver.findElement(By.name("q")).sendKeys("webdriver");
driver.findElement(By.name("btnG")).click();
assertEquals("webdriver - Google Search", driver.getTitle());
Hey we look similar now!
27
Mocha with selenium wrapperAll callbacks can be omitted and it just works which makes the code very “synchronous” like.
Specifically, you don’t have to chain everything and each individual line of code can do only one
ui action and then some assertion if necessary.
var test = require('selenium-webdriver/testing');var webdriver = require('selenium-webdriver');var By = require('selenium-webdriver').By;var Until = require('selenium-webdriver').until;
test.it('Login and make sure the job menu is there', function() { driver.get(url, 5000); driver.findElement(By.css('input#email')).sendKeys('[email protected]'); driver.findElement(By.css('input#password')).sendKeys(password); driver.findElement(By.css('button[type="submit"]')).click(); driver.wait(Until.elementLocated(By.css('li.active > a.jobs'))); var job = driver.findElement(By.css('li.active a.jobs')); job.getText().then(function (text) { assert.equal(text, 'Jobs', 'Job link title is correct'); });});
28
Comparison
Library API structure Underlying implementation
NightwatchJs chain Its own JsonWireProtocol 3457 stars on github
WebDriverIO chain/promise Its own JsonWireProtocol 1322 stars on github
The Intern chain leadfoot 3095 stars on github
WebDriver-sync sync JsonWireProtocol ?? 72 stars on github
WD.js chain/promise Its own JsonWireProtocol 890 stars on github
Officialselenium-webdriver
promise & built-in promise manager
Webdriver API with native WebElement & Driver objects
2016 stars on github
30
Designing frameworks
The magic number 7https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two
The human brain can only focus on 7 ± 2 things at once. ● Handle all UI interactions and nuances in a common location
○ Stale elements, retries and etc.○ Mimicking an actual human in the UI
● Keep tests dry, more business facing methods and logic in page-objects● Easy to add tests
31
Object Oriented Javascript
● The world's most misunderstood prog language● There are no real classes● Inheritance - different from Java
○ prototype-oriented (has a) vs class-oriented inheritance (is a)○ http://www.crockford.com/javascript/javascript.html
● Recommended reading○ The Principles of Object-Oriented Javascript
Nicholas C. Zakas
32
BasePage.js
var driver;
/**
* Base constructor for a pageobject
* Takes in a WebDriver object
* Sets the Webdriver in the base page surfacing this
to child page objects
* @param webdriver
* @constructor
*/
function BasePage(webdriver) {
this.driver = webdriver;
}
...
LoginPage.js
var BasePage = require('./BasePage');
/**
* Constructor for the Login Page
* Hooks up the Webdriver holder in the base page allowing to
call this.driver in page objects
* @param webdriver
* @constructor
*/
function LoginPage (webdriver) {
BasePage.call(this, webdriver);
this.isLoaded();
}
// Hooking up prototypal inheritance to BasePage
LoginPage.prototype = Object.create(BasePage.prototype);
// Declaring constructor
LoginPage.prototype.constructor = LoginPage;
...
Javascript pageobjects
Kinda like calling super(); in Java
33
BasePage.js con’t
BasePage.prototype.waitForLocated = function(locator, timeout) { var MAX_RETRIES = 5; var retry = 0; timeout = timeout || WAIT_TIME_PRESENT; var _this = this; // The actual wait, but we handle the error return _this.driver.wait(Until.elementLocated(locator),timeout).thenCatch(function (err) { if (err.name !== 'StaleElementReferenceError') { throw new Error(err.stack); } // fail after max retry if (retry >= MAX_RETRIES) { Logger.error('Failed maximum retries (' + MAX_RETRIES + '), error : ' + err.stack); throw new Error('Failed after maximum retries (' + MAX_RETRIES + '), error : ' + err.stack); } //retry retry++; Logger.debug('Element not located with error : ' + err.stack + ' retrying... attempt ' + retry); return _this.waitForLocated(locator, timeout, retry); });};
Javascript pageobjectsHandle most of the UI interaction in a common place● Takes care of stale elements exceptions● Retries● WaitForLocated();● WaitForVisible();● WaitForEnabled(); ● ...
34
LoginPage.js con’t
/*** Page load definition* @returns {LoginPage}*/LoginPage.prototype.isLoaded = function() { this.waitForDisplayed(By.css(EMAIL)); this.waitForDisplayed(By.css(PASSWORD)); this.waitForDisplayed(By.css(LOGIN_BUTTON)); return this;};/*** Enter the user information and login* @param username* @param password* @returns {LoginPage}*/LoginPage.prototype.enterUserInfo = function(username, password) { this.waitForEnabled(By.css(EMAIL)); this.driver.findElement(By.css(EMAIL)).sendKeys(username); this.driver.findElement(By.css(PASSWORD)).sendKeys(password); this.waitForEnabled(By.css(LOGIN_BUTTON)); return this;};
Javascript pageobjects
Pageobject methods work seamlessly with mocha promise manager wrapper● Each line is a promise that gets added to the queue● Everything runs top down just like java
a synchronous language
35
var test = require('selenium-webdriver/testing');var assert = require('chai').assert;var LoginPage = require('./../../src/pageobjects/LoginPage');var SideNav = require('./../../src/pageobjects/SideNav');var TopNav = require('./../../src/pageobjects/TopNav');var url = Constants.launch_url;
//Login Tests
test.describe('Login tests', function() { var driver; test.beforeEach(function() { driver = DriverBuilder.build(); });
test.it('Login with an invalid password @smoke', function() { var login = new LoginPage(driver); login.enterUserInfo(Constants.MANAGER_USER, 'foobar'); login.clickLogin(); login.getLoginErrorText().then(function(result){ assert.include(result, 'Your email or password was incorrect. Please try again.'); }); });});
Putting it together
Sample project : https://github.com/mekdev/mocha-selenium-pageobject
Import statements● pageobjects / other libs
TestNG @beforeTest looks familiar ? :)
Look ma no WebElements or Locators
37
Visual Validation - Applitools
Went with Applitools● Proven track record of prior implementation in Java
http://www.slideshare.net/MekSrunyuStittri/visual-automation-framework-via-screenshot-comparison
● Made Applitools integration a criteria when building the framework
● 3 implementation choices○ WebdriverIO’s WebdriverCSS○ Official selenium-webdriver WebdriverJs (Driver instance)○ Protractor○ Native eyes.images (manage your own uploads and imgs)
38
Trial runs with WebDriverCSSWebdriverCSS
webdrivercss.init(client, {key: 'your key here'});
client.init()
.url(url)
.webdrivercss('Check #1', {
name : 'Login'
}, function(err, res) {
assert.ifError(err)
})
.setValue('input#email', MANAGER_USER)
.setValue('input#password', PASSWORD)
.click('button[type="submit"]')
.webdrivercss('Check #2', {
name : 'Job page'
}, function(err, res) {
assert.ifError(err)
})
Challenges● Still stuck in chain API world● Cannot choose match level
○ Defaults to strict
● One screenshot eqs 1 test not 1 step● Even harder to do pageobjects
○ .webdrivercss() needs to be chained in order to capture the screenshot
39
Applitools test
test.it("test with login page and applitools", function() { var eyes = new Eyes(); var driver= DriverBuilder.build();
eyes.setApiKey("<your key here>"); eyes.setMatchLevel('Content'); eyes.open(driver, "Airware", "Simple Airware main page") .then(function(eyesDriver) { driver = eyesDriver; });
var login = new LoginPage(driver); login.open(url); eyes.checkWindow("Main Page"); login.enterUserInfo(USERNAME, PASSWORD); login.clickLogin(); eyes.checkWindow("Jobs Page"); eyes.close();});
Tests written using the promise manager fits with Applitools and Pageobjects perfectly.● Maintains app context while allowing the insertion of checkpoints
Applitools JS SDK : https://eyes.applitools.com/app/tutorial.html
Visual Checkpoints
Looked at REST frameworks
Supertesthttps://github.com/visionmedia/supertest
● Built on mocha● Chain API based● Asserts are built in
Chakramhttp://dareid.github.io/chakram/
● Runs on mocha● Promise based● Asserts are built in● Needs to return chakram.wait()
describe('GET /users', function(){ it('respond with json', function(done){ request(app) .get('/user') .set('Accept', 'application/json') .expect(200) .end(function(err, res){ if (err) return done(err); done(); }); });});
describe("HTTP assertions", function () { it("Should return 200", function () { var response = chakram.get("your.api/get"); expect(response).status(200); expect(response).header("application/json"); expect(response).comprise.of.json({...}); return chakram.wait(); });});
Request libraryRequesthttps://github.com/request/request
● Standard http request library● Callback syntax - request(options, callback)
it("A series of requests", function (done) { var request = require('request'); request({ method: 'POST', uri: '/login', form: { username: 'username', password: 'password' }, }, function (error, response, body) { request({ method: 'GET', ... }, function (error, response, body) { request({ method: 'PUT', ... }, function (error, response, body) { done(); }); } }); });});
Then around the same time..● Share code with UI devs
○ Generators and Coroutines● Node v4.0.0 (Stable)
2015-09-08
○ Official support for ES6!○ Yield statements!
Generator based callsCo-Requesthttps://github.com/denys/co-request
● wraps http request library but yieldable
it("Login", function *() { var request = require('co-request'); var cookieJar = request.jar();
response = yield request({ method: 'GET', url: BASE_URL, jar: cookieJar, followRedirect: false });
response = yield request({ method: 'POST', url: BASE_URL + '/login', jar: cookieJar, form: { username: '[email protected]', password: 'foobar', }, followAllRedirects: true });});
Generator based requests● Became the base for our WebClient● Same principles as a page object
but for REST APIs● Yield blocks until execution is done● The Magic number 7
45
JSON and Javascript
The hidden power of working in javascript○ JSON stands for JavaScript Object Notation
JSON is a subset of the object literal notation of JavaScript. Since JSON is a subset of JavaScript, it can be used in the language with no muss or fuss. dto jackson
Actual response{ "type": "forbidden", "message": "You do not have permissions in this project"}
Codevar webClient = new WebClient();yield webClient.login(VIEWER_USER, VIEWER_PASSWORD);
// Try to view usersvar projectUsers = yield webClient.getProjectUsers(qeProject.id);assert.strictEqual(projectUsers.type, TYPE_FORBIDDEN, 'Return type should be forbidden');assert.strictEqual(projectUsers.message, 'You do not have permissions in this project');
46
Fitting it with selenium framework● Adapting yield calls to work with the promise manager
○ Selenium Control Flows https://code.google.com/p/selenium/wiki/WebDriverJs#Framing
○ Allows execution order framing and supports generators from the manager queue● Functional programing - high order functions are your friends
test.it('Verify data from both frontend and backend', function() { var webClient = new WebClient(); var projectFromBackend; // API Portion of the test var flow = webdriver.promise.controlFlow(); flow.execute(function *(){ yield webClient.login(Constants.FORSETI001_EMAIL, Constants.FORSETI_PASSWORD); var projects = yield webClient.getProjects(); projectFromBackend = projectutil.getProjectByName(projects, Constants.QE_PROJECT); }); // UI Portion of the test var login = new LoginPage(driver); login.enterUserInfo(Constants.FORSETI001_EMAIL, Constants.FORSETI_PASSWORD); var topNav = new TopNav(driver); topNav.getProjects().then(function (projects){ Logger.debug('Projects from backend:', projectsFromBackend); Logger.debug('Projects from frontend:', projects); assert.equal(projectsFromBackend.size, projects.size);});
Utility module that heavily uses underscore
47
What the final stack looks like
Data
Backend : node.js
Browser
Input / update data
Get data
Frontend : javascript
Microservice 1
Microservice 2 .. n
Rest APIs
Input / update data
Get data
UI Framework : node.js● selenium-webdriver● mocha + wrapper● Applitools● co-wrap for webclient● chai (asserts)
Rest API Framework : node.js● co-requests● mocha● co-mocha● chai (asserts)● json, jayschema
WebClient
Pageobjects
Webclient adaptor
48
Introduction to deployments
Cloud Deployment Culture● Weekly deploys to production● Gated deploys to preprod● Automatic deploys to staging and tests
staging preprodprod
Clouds envs
After selenium testsContinuous deployment with
Kubernetes and Docker
Volkan Gurel - Engineering Manager
50
Deployment Dashboard (Vili)
Problem:● Manage and deploy:
○ Many microservices○ In many environments○ With many versions
● Access control for deployments● QA gating for production deployments
Solution: Vili
51
Vili Overview
Kubernetes- Controllers for apps- Pods for jobs- Rolling deploys
Docker Repo- Many apps- Many versions
Environments- Different variables- Need source control
Notifications- Slack
Authentication- Okta- Extensible
Approvals- Only QA can approve- Required for prod deploy
VILI
53
Follow the Airware github repo to be notified:
https://github.com/airware
(About to be) Open Sourced
The Team● Bj Gopinath - Guidance and support● Lucas Doyle, Nick Italiano - co and Node.js generators, locator sharing strategy with frontend● Phil Kates - Countless nights/weekends on infrastructure work● Eric Johnson - Guidance and support. On coming from Java to Javascript :
“Yeah, probably some unlearning going on. JS is crazy, but I’ve rarely had more fun”
Meetup Folks● Marcel Erz, Yahoo - Feedback on implementations of Webdriver-sync● Mary Ann May-Pumphrey - Nightwatch feedback
Special Thanks
SaucelabsInitial feedback on selenium node.js
● Neil Manvar● Kristian Meier● Adam Pilger● Christian Bromann - WebdriverIO
ApplitoolsTrial POC with Applitools Javascript bindings and performance
● Moshe Milman● Matan Carmi● Adam Carmi● Ryan Peterson