software test automation with open source...
TRANSCRIPT
Software Test Automation
With Open Source Software
Itai Agmon
• CTO of Top-Q
• Main contributor of the JSystem project
• Creator of the Difido project
• My laptop runs Linux
Who Am I?
1
Why do I like OSS Projects?
• It’s built by passionate communities of developer
• Real solutions to real problems
• Rate of innovation is faster.
• There’s no vendor lock-in. Remember SilverLight
• You can “look under the hood”
• No back doors. _NSAKEY
• Oh, and it’s free
“Given enough eyeballs, all bugs are shallow.”Eric S. Raymond
2
3
Automation tools and libraries
4
So, Selenium?
5
Inversion of Control
Parameters
Reporting
Setup and Teardown
Assertion and Validation
Layered Architecture
Test Automation Frameworks
Infrastructure Layer
Could beProgrammatically, Text Based, GUI
Tests Layer
Common Interface
Let’s select A framework
6
First - Functional Tests or NFT
• Non Functional
– Performance
– Security
– Usability
– Stability
– Availability & Resilience
– Configuration
– Data migrating
• Functional
– Black box testing
– Describes what the
system does
– Feed input,
examine the output
Experimental manual tests
End to End/GUI
Integration/API
Unit Tests
The higher we go
• More unreliability
• Greater cost
• Slower feedback
First - What is the type of the functional tests?
QA/
Automation
Teams
R&D
7
What is the programming language?
Who Are The Automation Customers?
QAAutomation Eng. R&D Developer
Cod
ing
Skill
Cod
ing
Skill
Cod
ing
Skill
• Common in small organization, startups
• Automation engineer write, execute and debug
tests
• It is better to use the code based approach
8
private String firstName = "Zhang";private String lastName = "Wei";
@Testpublic void testEditAccount(){
DashboardPage dashboardPage = web.getDashboardPage();AccountsPage accountsPage = dashboardPage.getTopMenuPage().clickOnCustomersItm().clickOnAccountsItm();
AccountDetailsPage accountDetailsPage = accountsPage.clickOnAccountByIndex(0);
EditAccountDetailsMainTab mainTab = accountDetailsPage.clickOnEditAccountLnkAndGoToEditAccountsPage();mainTab.enterTextToFirstName(firstName);mainTab.enterTextToLastName(lastName);
EditAccountDetailsAdditionalInfoTab additionlInfoTab = mainTab.clickOnAdditionInfoTab();additionlInfoTab.selectCountry(Country.Hungary);additionlInfoTab.clickOnOkBtn();
}
Code Based Approach
Code Based Approach Frameworks
MSTest
9
Who Are The Automation Customers?
QAAutomation Eng. R&D Developer
Cod
ing
Skill
Cod
ing
Skill
Cod
ing
Skill
• Common in large organization
• Automation engineers write the infrastructure
• QA are familiar with the system more then any other person
• QA are using GUI or any other UI to write, execute and debug tests
Layered Architecture
Infrastructure Layer
Text Based, GUI
Tests Layer
Common Interface
QA
Automation
Eng.
10
UI Based Approach Frameworks
Robot
Framework
GaugeGauge
11
Behavior Driven Development (BDD)
BDD Automation Frameworks
12
Who Are The Automation Customers?
QA Automation Eng. R&D Developer
Cod
ing
Skill
Cod
ing
Skill
Cod
ing
Skill
• Developers can write tests for each new feature
• Watch out! They always have their own opinion
• They will probably prefer to write code
• Need to enforce discipline
What services do I need from my
framework?
13
Not only must Automation be done; it must also be seen to be done
System Under
Test
Automation Tests
Report Manager
XMLHTML
DB
Excel/CSV
Console
• Start test
• End test
• Fail test
• Log message
Parameterization
@Parameters({ "first-name" })@Testpublic void testSingleString(String firstName) {
System.out.println("Invoked testString " + firstName);assert "Cedric".equals(firstName);}
<suite name="My suite"><parameter name="first-name" value="Cedric"/><test name="Simple example">
<-- ...-->
private File file = new File(".");private String str = "Some string";private int i = 5;private Date date = new Date();private String[] strArr;private Account account;
**/
* Test with different parameters/*
@Test@TestProperties(name = "Test with file '${file}' string '${str}' integer ${i} date ${date}", paramsInclude = {"file", "str", "i", "date", "strArr" })public void testWithParameters() {
report.report("File: " + file.getAbsolutePath());report.report("Date: " + date.toString());report.report("String: " + str);report.report("Integer: " + i);
}
14
Additional Framework Services
Inversion of Control
Parameters
Reporting
Fixtures
Assertion and Validation
Concurrency
Environment Configuration
peripheral systems
15
Peripheral Systems Types
• Scheduler (CI/CB)
• Test Management Systems (QC, TestLink, MTM)
• Resource Manager
• Report Server
• Reports BI provider
• Source Control
• Remote Agents Manager
• Report Generator
Peripheral Systems Example
16
Q&A
17
Matan Goren QA Automation Tech Lead
Automations in a
Continuous Deployment
Environment
Who we are
• More than 82 million registered users
• 28 million family trees
• Website, mobile apps (iOS and Android), desktop
apps (windows and Mac)
• 2.6 billion profiles (names in family trees)
• 7 billion historical documents and records. Including
the world’s largest collection of newspapers
• 200 million photos
• 42 languages
• 275 employees
18
• All manual QA
• Repetitive tasks
• Time consuming
• Two service packs a week
• Holding back development
• SPC size limits error handling
Where we were
• Number of tests
• Desktop Chrome: 650 (469)
• Desktop IE: 552 (0)
• Mobile-web 90 (0)
• Initial test suites for iOS and Android apps
• Runtime of full suite: 35 min for chrome + 32 min for all mobile-web devices (51 min)
• Grids: 4 (1)
• Tests run on prod and semi staging env
• Continuous Deployment
Where we are today
19
“In software, when something is painful, the way to reduce the pain is to do it more frequently, not less.” - David Farley
• We moved to CD a year ago
• On average we distribute code to production 25 times per day
• Comes with challenges
• Automated tests on every code release
• Only minimal tests on each release
• Dev is responsible to run appropriate tests PRE-RELEASE in staging
The CD mindset
20
• Risk reduction to production
• Small incremental changes are easier to monitor and revert in necessary
• Has lower impact on the system
• Increasing R&D velocity
• Avoiding wasted time on merges and complex coordination before dist
• Allow R&D and product to experiment and innovate more frequently
RND Goals
• Provide a safety net
• Avoid the bottleneck of manual sanity testing
• Reliability
• No flaky tests
• No false failures
• QA E2E test should not be the new bottleneck
QA Goals
21
• Continuous Deployment Applied at MyHeritage– SlideShare
Run Unit Tests
Canary Server
Pre-dist E2E tests
Dist
Post-dist E2E tests
Start CD flow
CD Flow (simplified)
• QA are responsible for E2E tests only
• Dev write and maintain Unit tests
• E2E tests are written during development process
• All QA members monitor CD suite results
• CD E2E tests are NOT acceptance tests
Responsibilities
22
• CD flow from commit to prod is ~25 minutes
• Current QA CD suite takes 3 minutes on average
• Runs in parallel to some of the jobs in the CD flow
• Limited by time frame of parallel jobs
• Still have room for further growth
Timing
• Minimal suite – No full regression on commit
• Each Dev is responsible to run the desired Automated regression tests
• Reliability
• Monitor a new test before adding it to the CD suite
• Velocity of tests
• Run parallel tests (currently up to 16)
• Write new CD dedicated tests if existing tests are too long
• Add hardware if needed
Challenges of CDAnd how we face them
23
• Failure visibility
• Slack messages for individual failed tests
• Browser console logs in report
• Screenshots in report
• Post dist – production server logs in the report (Using Elasticsearch)
• Large scale tests refactoring due to changes
• Ability to run tests with a patch instead of committing code that doesn’t correspond to production
Challenges of CD cont.
• CD halt on failure
• Fast response time for failure. All QA members monitor the relevant channel in slack
• Fast adaptation to feature flags
• Implemented feature flags for tests
Challenges of CD cont.
24
• Handling A\B tests
• All AB tests can be controlled with a cookie.
• All tests start with control-group variants set in the cookies
• If needed, set a desired variant by replacing the cookie.
Challenges of CD cont.
Gray Area for full image
“You don't have to be a genius or a visionary or even a college graduate to be successful. You just need a framework and a dream.”- Michael Dell
25
• Ruby
• Cucumber.io
• Page-Object Gem
• Grid Grouper
• Cuke_Slicer gem
• Jenkins
• Watir-webdriver
• Selenium Grid/Grid Extras
• Appium
• Appium_lib
• Xcode simulator
Framework
• Easy language for inexperienced developers
• Open source community is very active
• It ain’t Java!
Ruby
26
• Native to Ruby
• Cleaner than Selenium
• Adds methods not native to Selenium
• “Being able to select an element by a explicit identifier” –watirmelon.com
Watir-webdriver
• Provides a simple interface to define and interact with elements on a page
• Works with both Watir and Selenium
• Simple way of introducing OOP without a lot of technical knowledge required
• Page factory module handles common page classes action
Page-Object gem
27
• BDD tool
• Tests are written in plain English
• Makes for easy debugging
• Easy move from Manual test to automatic test
• Helps soften the introduction to programming
• “Top down” programming
• Write a step
• Define the step
• Make it work
• Pre and post test hooks
Cucumber.io
Directly with Watir-WebdriverUsing page-object
Tags specify the suites
Cucumber Feature
28
Directly with Watir-Webdriver
Using page-object
Step Definitions
Using page-object
Defining page-object
Verify page loaded with page-object
Page-object example
29
• Tests run nightly on the build or on demand
• Using two Mac minis as test machines
• Device parameters are passed from Jenkins (or configuration if running locally)
• The apps are built on the tests machines and deployed to the simulator\emulator
• Support for iOS and Android within existing infra
• iOS simulator, Android emulator and Appium server are opened by command from within the code
Native Mobile Apps
• Support for iOS and Android within existing infra
Native Mobile Apps
30
• iOS simulator, Android emulator and Appium server are opened by command from within the code
Native Mobile Apps
Native Mobile Test Flow
• Start E2E flow
• Build App
• Start test flow
• Open simulator\emulator
• Open Appium server
• Start Appium driver
• Perform test
• Close Appium driver
• Close Appium server
• Build Report
• Send slack notifications and emails
31
• Homegrown tool
• Takes a given suite and distributes the test across several groups
• Utilizes cuke_slicer gem
• Checks the feature flags and filters out tests that are turned off
• Multiple reports problem is solved by a report merger tool that was developed in house
Grid Grouper
• Chuck Norris!
• Was already in use by RND
• Open source (lot’s of available plugins)
• Schedule automated builds
• Kick off manual builds with a few clicks
• Control build parameters (server, suite, etc)
• Can build in any of our environments with any combination of tags
• Holds build history for a specified amount of time
• Merged report is stored in each build
• Slack and email notifications
Jenkins
32
• Comprised of Selenium Grid and Selenium-Grid-Extras
• Selenium-Grid-Extras makes the basic setup streamlined
• Gives extra features such as:
• Pulling in updated drivers for all browsers
• Also for Selenium itself
The Grids
• We have 4 grids
• Each grid has 4 VMs
• The VMs include 1 hub-node and 3 other nodes
• The hubs are Jenkins slaves
• Each node can run 4 parallel tests on Chrome
• One grid is dedicated to CD
• We can pull a grid “offline” for testing infra upgrades
• Ability to have multiple test runs in parallel on different environments and tags
• Only runs through Jenkins
• Jenkins selects grids automatically based on availability and run history
How we use the grids
33
“A challenge only becomes an obstacle when you bow to it.” - Ray A. Davis
• Maintaining production data integrity
• Reliance on pre-existing data
• Running on local staging env
• Identifying failure reasons
• Need a true BDD/TDD mindset
Challenges QA faces
34
Questions?
AA Milne
Thanks!
We are hiring
35
Mobile TestingOpen(ish) Source Solutions
https://www.youtube.com/watch?v=p8A4yq
N4_9A
Mobile TestingOpen(ish) Source Solutions
We ain’t gonna talk about ringtones…
111
What’s ahead
• Mobile evolution
• Motivation
• Define UI Testing (with mobile emphasis)
• UI Mobile testing – Than VS Now
Mobile evolution
112
Mobile evolution
Mobile evolution
113
Mobile evolution
Mobile evolution
114
Motivation
Motivation
115
Motivation
Motivation
116
Motivation
Motivation
117
UI Testing
End to End/GUI
Integration/API
Unit Tests
The higher we
go
• More unreliability
• Greater cost
• Slower feedback
Experimental manual tests
QA/
Automation
Teams
R&D
Define UI Testing
• Functional Black box testing
• Running on actual (or as close as possible)
device
• Manual Repetitive/Time/Cost
118
UI Testing – Then
• Image based
UI Testing – Then
119
UI Testing – Then
UI Testing – Then
• Baseline per resolution
• Non adaptive for changes
• Restriction to single IDE
• Many devices
• Hybrid apps treated the same
120
UI Testing – Now
• Instrumentation based
• Virtualization support, better simulators
• Android UiAutomator/IOS UIAutomation
Frameworks
• Requires signature/permission
UI Testing – Now
• UI Automator Viewer
121
UI Testing – Now
• Appium
– Allows to locate UI elements and perform
various operations on them
– Multiple OS Support (Win, Lin, Mac)
– Supports native and hybrid applications
– Supports iOS and Android
– Supports real devices and simulators
UI Testing – Now
122
UI Testing – Now
DesiredCapabilities capabilities = new DesiredCapabilities();capabilities.setCapability("platformName", "Android");capabilities.setCapability("app","src/test/resources/com.twitter.android_5.5.0.apk");capabilities.setCapability("app-package", "com.twitter.android");capabilities.setCapability("app-activity", ".LoginActivity");
AppiumDriver driver = new AppiumDriver(new URL("http://127.0.0.1:4723/wd/hub"), capabilities);
//Find the element in page using a selector locating strategyWebElement element = driver.findElement(MobileBy.className("android.widget.EditText"));
//Perform operation on the found elementelement.sendKeys("jsystemdemo");
UI Testing – Now
123
UI Testing – Experitest
• Utilize cloud Devices
• Write code and run it using SeeTest on
hosted devices
• Build your own Device lab
• Network Virtualization
https://youtu.be/fWM_DBQ95p8
UI Testing – SauceLabs
https://youtu.be/kCJwwRXUTdo
• Appium Hub
• Run your code against multiple devices
online
• No need to configure prior testing
• Manual Testing
124
UI Testing – GenyMotion
https://www.youtube.com/watch?v=50PR7
ocy_IU
• Local Running android devices
• Faster emulations thanks to usage of GPU
• Must have devices configured
• Demanding
Page Object Design Pattern
• Within your application UI there are areas that your
tests interact with.
A Page Object simply models these as objects within
the test code.
• The Page Object’s methods represent the services
offered by a particular page.
• The page object encapsulate the UI elements. if the UI
changes, the fix need only be applied in one place.
125
Resources
http://powerlisting.wikia.com/wiki/File:Devolution_of_Man.jpg
http://www.geek.com/mobile/evolution-of-the-cell-phone-depicted-in-papercraft-1309666/
http://www.hexapolis.com/2015/07/28/12-video-game-franchises-then-and-now-the-graphics-evolution/
App Annie, BI Inteligence
http://worthstuff.blogspot.com/2010/05/lg-ally-mobile-phone-with-slide-out.html
http://www.kinvey.com/how-long-to-build-an-app-infographic/
http://www.kinvey.com/wp-content/uploads/2013/01/how-long-does-it-take-to-build-an-app-kinvey-1500x834.png
http://www.factslides.com/s-Mobile-Phones
http://jwtyler.blogspot.co.il/2010_10_01_archive.html
http://www.androidauthority.com/nokia-qwerty-keyboar-107145/
http://www.formotus.com/14018/blog-mobility/figuring-the-costs-of-custom-mobile-business-app-development
http://decor10blog.com/design-decorate/decorating-ideas/apple-design-team-abandoning-sketchbooks-for-the-apple-
pencil-says-jonathan-ive.html
https://www.quora.com/Why-Microsoft-failed-at-mobile
126
The Road To Automated TestingTal Tsach-Cohen
QA Manager Hexagon Technologies
About Me
• In QA 6 Years
• Studied .Net 2yrs at SELA while working
• Introduced automation to company when QA
was 2 members. Today I manage a team of
14.
• Previously was a chef in San Francisco for 8
years. This experience was more relevant than
you would think.
127
Who Are We?
• Development division for leading Forex
Company.
• 70 Employees – 15 QA
• B2B, B2C, CRM, CMS, Web, Mobile
(iOS, Android) currently 2 brands for all
applications.
• All developed in house.
What Does QA Look Like?
• 15 Members.
• 5 Teams
• 7 Automation, 7 Manual including team
leaders.
• 4000 tests run nightly.
128
Components
•.Net C#
•Selenium, Appium, Applitools, Git, NUnit
•All frameworks developed in house page object model for GUI.
•2 Solutions Server and Client systems.
•Same tests for both web applications, and same tests for iOS and
Android only element mapping is changed via configuration files.
•Jenkins for deploy to QA and running tests.
•Custom reporter including video, HTML Report and integration with
Elastic Search.
•Test analysis with Kibana and Elastic Search.
•Notifications via Slack.
Why C#?
•At time of birth of automation .Net was 95% of
existing platform.
•Existing in house knowledge and libraries in C#.
•Built in load testing framework with visual studio
ultimate edition.
•In retrospect would probably go with Java.
129
Why Jenkins?
•Open Source, easy to use.
•QA was first to implement automated deployment in Jenkins.
•Efficient for creating automated test jobs which can be
executed by manual testers/developers as needed.
•Integration with email, slack, Jira.
Challenges Specific To Hexagon
•Off the clock POC was needed to demonstrate need for investing in
automation.
•System is extremely complex, upwards of 50 components working
together making stability an issue.
•Core components have no unit testing so E2E testing was needed to
make up for deficit.
•Due to system complexity and need for known system wide state running
parallel tests on core business component (Trading Platform) is almost
impossible.
•Multitude of technologies used in system require high competency of
automation developers. Oracle DB, redis, kafka, couchDB, mongoDB,
NodeJS, Cassandra, Linux, ZMQ and more…
130
More Challenges Specific To Hexagon
•Server side automation team (No GUI) needs hard to find skill set of high
level developer with QA orientation.
•Automation developers are responsible for writing code as well as
defining tests. Finding testers with aptitude for both disciplines has been a
challenge.
•Inability to use external solutions due to lack of access to QA
environments from outside LAN.
Where We Are Today•Server side testing
– ~ 2500 tests written
– 2000 part of nightly regression
– 900 run parallel in first 15 minutes
– Regression done in 6 hours
– 95% coverage
•B2B – CRM Testing
– 1300 Tests all run nightly
– 89% First run pass 22% in second pass on failed tests
– 80% coverage of component
– We are working on parallelizing the tests should be 4 hours when done.
•Web – Trading and Website
– 800 tests nightly regression
– 6 hours chrome 75% first run pass 36% pass on failed run pass
– 11 hours firefox 63% first run pass 39% pass on failed run pass
– 6 hours explorer 45% first run pass 52% pass on failed run pass
131
Where We Are Today
•Mobile testing started this week
– 62 tests 30% regression coverage
– 88% pass rate, 50% on second pass.
– 1.5 hours
132
133
134
135
What Is Next?
• Image Recognition for GUI design validation.
• Improve Explorer browser stability, Edge
testing.
• Jmeter load testing, and browser performance
testing. We use ZMQ protocol between
servers, Created plugin for our purposes.
136
137
Page Object Design Pattern – De-facto
Standard for Test Automation Projects
Itai Agmon
What is software architecture?
“Software application architecture is the process of defining a
structured solution that meets all of the technical and
operational requirements, while optimizing common quality
attributes such as performance, security, and
manageability. It involves a series of decisions based on a
wide range of factors, and each of these decisions can
have considerable impact on the quality, performance,
maintainability, and overall success of the application.”
Microsoft
138
139
Image courtesy of Stuart Miles at FreeDigitalPhotos.net
First Things First
• I am not going to talk about Selenium
• The examples are in Java
• There is a lot of code involved. Can be
downloaded from here
• https://github.com/itaiag/page-objects-
training
140
So, What is That Page Object Thingy Anyway?
• Architecture for GUI test automation
• Mostly used for web application testing
• I use it for every GUI (Desktop, Web, Mobile
and even cli menus)
• Requires OO language
• De facto industry standard.
• It is a design pattern, not recipe.
141
@Testpublic void testRegisterAndLogin() throws Exception {
// Register new userdriver.findElement(By.linkText("Register")).click();driver.findElement(By.id("firstName")).sendKeys(FIRST_NAME);driver.findElement(By.name("lastName")).sendKeys(LAST_NAME);String userName = FIRST_NAME + String.valueOf(System.currentTimeMillis());driver.findElement(By.id("username")).sendKeys(userName);driver.findElement(By.id("password")).sendKeys(PASSWORD);driver.findElement(By.tagName("button")).click();wait.until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//div[text()='Registration successful']")));
// Logindriver.findElement(By.id("username")).sendKeys(userName);driver.findElement(By.id("password")).sendKeys(PASSWORD);driver.findElement(By.tagName("button")).click();
wait.until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//h1[text()='Hi " + FIRST_NAME + "!']")));driver.findElement(By.linkText("Logout")).click();
}
How About Those Record and Playback technologies?
142
@Testpublic void testRegisterAndLogin() throws Exception {
String userName = registerNewUser(FIRST_NAME, LAST_NAME, PASSWORD);Login(userName, PASSWORD);logout();
}private void logout() {
driver.findElement(By.linkText("Logout")).click();}
private void Login(String userName, String password) {driver.findElement(By.id("username")).sendKeys(userName);driver.findElement(By.id("password")).sendKeys(password);driver.findElement(By.tagName("button")).click();wait.until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//h1[text()='Hi " + FIRST_NAME + "!']")));
}
private String registerNewUser(String firstName, String lastName, String password) {// Register new userdriver.findElement(By.linkText("Register")).click();driver.findElement(By.id("firstName")).sendKeys(firstName);driver.findElement(By.name("lastName")).sendKeys(lastName);String userName = FIRST_NAME + String.valueOf(System.currentTimeMillis());driver.findElement(By.id("username")).sendKeys(userName);driver.findElement(By.id("password")).sendKeys(password);driver.findElement(By.tagName("button")).click();wait.until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//div[text()='Registration successful']")));return userName;
}
No Biggy, Let’s Use Functions
Functions Are Words
Image courtesy of Stuart Miles at FreeDigitalPhotos.net
143
Granularity – Let’s Talk Negative
Image courtesy of artur84 at FreeDigitalPhotos.net
//I can do thisloginAndSuccess(user, password);loginAndFail(user, password);
//Or thislogin(user, password, expectSuccess);
//Or maybe thislogin(user, password);assertInHomePage();assertInLoginPage();
Image courtesy of Bill Longshaw at FreeDigitalPhotos.net
144
Page Object Pattern
public class LoginPage extends AbstractPage {
public LoginPage(WebDriver driver) {super(driver);
}
public void typeToUserNameTb(String userName) {driver.findElement(By.id("username")).sendKeys(userName);
}
public void typeToPasswordTb(String password) {driver.findElement(By.id("password")).sendKeys(password);
}
public void clickOnLoginBtn() {driver.findElement(By.tagName("button")).click();
}
public void clickOnRegisterLnk() {driver.findElement(By.linkText("Register")).click();
}
}
The Test
@Testpublic void testRegisterAndLogin() throws Exception {
LoginPage loginPage = new LoginPage(driver);loginPage.clickOnRegisterLnk();
RegisterPage registerPage = new RegisterPage(driver);registerPage.typeToFirstName(FIRST_NAME);registerPage.typeToLastName(LAST_NAME);String userName = FIRST_NAME + String.valueOf(System.currentTimeMillis());registerPage.typeToUserName(userName);registerPage.typeToPasswordTb(PASSWORD);registerPage.clickOnRegisterBtn();
// OH MY GOD... DON'T USE SLEEP!!!Thread.sleep(1000);
loginPage = new LoginPage(driver);loginPage.typeToUserNameTb(userName);loginPage.typeToPasswordTb(PASSWORD);loginPage.clickOnLoginBtn();
HomePage homePage = new HomePage(driver);homePage.clickOnLogoutBtn();
}
145
Navigation
Image courtesy http://www.webseoanalytics.com/blog/link-structure-analyzing-the-most-important-methods/
…Adding Navigations
@Testpublic void testRegisterAndLogin() throws Exception {
LoginPage loginPage = new LoginPage(driver);RegisterPage registerPage = loginPage.clickOnRegisterLnk();
registerPage.typeToFirstName(FIRST_NAME);registerPage.typeToLastName(LAST_NAME);String userName = FIRST_NAME + String.valueOf(System.currentTimeMillis());registerPage.typeToUserName(userName);registerPage.typeToPasswordTb(PASSWORD);loginPage = registerPage.clickOnRegisterBtn();
loginPage.typeToUserNameTb(userName);loginPage.typeToPasswordTb(PASSWORD);HomePage homePage = loginPage.clickOnLoginBtn();
homePage.clickOnLogoutBtn();
}
public LoginPage clickOnRegisterBtn() {driver.findElement(By.tagName("button")).click();return new LoginPage(driver);
}
RegisterPage
TestRegistration
And what about
negative tests?
146
Negative Test
@Testpublic void testLoginWithWrongPassword(){
LoginPage loginPage = new LoginPage(driver);loginPage.typeToUserNameTb(FIRST_NAME);loginPage.typeToPasswordTb("WRONG PASSWORD");loginPage.clickOnLoginBtnAndStayInLoginPage();
}
@Testpublic void testSuccessfullLogin(){
LoginPage loginPage = new LoginPage(driver);loginPage.typeToUserNameTb(FIRST_NAME);loginPage.typeToPasswordTb(PASSWORD);HomePage homePage = loginPage.clickOnLoginBtnAndGoToHomePage();
}
Image courtesy of graur codrin at FreeDigitalPhotos.net
What do you actually test
here ?
There Are Two Kinds of Assertions
@Testpublic void testLoginWithWrongPassword() {
LoginPage loginPage = new LoginPage(driver);loginPage.typeToUserNameTb(FIRST_NAME);loginPage.typeToPasswordTb("WRONG PASSWORD");loginPage.clickOnLoginBtnAndStayInLoginPage();
String text = loginPage.waitForAlertMessage();Assert.assertEquals("Failed login message is incorrect",
"Username or password is incorrect", text);}
@Testpublic void testLoginWithWrongPassword() throws Exception {
LoginPage loginPage = new LoginPage(driver);loginPage.typeToUserNameTb(FIRST_NAME);loginPage.typeToPasswordTb("WRONG PASSWORD");loginPage. clickOnLoginBtnAndStayInLoginPage();
loginPage.assertAlertMessage("Username or password is incorrect");}
Image courtesy of Stuart Miles at FreeDigitalPhotos.net
147
We Sure Like Things Organized
public class LoginPage extends AbstractPage {
private By userNameBy = By.id("username");private By passwordBy = By.id("password");private By loginBtnBy = By.tagName("button");private By registerLnkBy = By.linkText("Register");private By alertDivBy = By.className("alert");
public LoginPage(WebDriver driver) {super(driver);
}
public HomePage clickOnLoginBtnAndGoToHomePage() {driver.findElement(loginBtnBy).click();return new HomePage(driver);
}
public void clickOnLoginBtnAndDoNotLogin() {driver.findElement(loginBtnBy).click();
}}
If You Don’t Have Anything to Return, Return Yourself
public LoginPage typeToUserNameTb(String userName) {driver.findElement(userNameBy).sendKeys(userName);return this;
}
public LoginPage typeToPasswordTb(String password) {driver.findElement(passwordBy).sendKeys(password);return this;
}
public HomePage clickOnLoginBtnAndGoToHomePage() {driver.findElement(loginBtnBy).click();return new HomePage(driver);
}
public LoginPage clickOnLoginBtnAndDoNotLogin() {driver.findElement(loginBtnBy).click();return this;
}
LoginPage
148
We Now Have a Fluent Test
@Testpublic void testRegisterAndLogin() throws Exception {
LoginPage loginPage = new LoginPage(driver);RegisterPage registerPage = loginPage.
clickOnRegisterLnkAndGoToRegisterPage();
String userName = FIRST_NAME + String.valueOf(System.currentTimeMillis());loginPage = registerPage.
typeToFirstName(FIRST_NAME).typeToLastName(LAST_NAME).typeToUserName(userName).typeToPasswordTb(PASSWORD).clickOnRegisterBtnAndGoToLoginPage();
HomePage homePage = loginPage.typeToUserNameTb(userName).typeToPasswordTb(PASSWORD).clickOnLoginBtnAndGoToHomePage();
homePage.clickOnLogoutBtnAndGoToLoginPage();
}
Binding Functions Together
@Testpublic void testRegisterAndLogin() {
LoginPage loginPage = new LoginPage(driver);
RegisterPage registerPage = loginPage.clickOnRegisterLnkAndGoToRegisterPage();
String userName = FIRST_NAME + String.valueOf(System.currentTimeMillis());loginPage = registerPage.doRegistration(FIRST_NAME, LAST_NAME, userName, PASSWORD);
HomePage homePage = loginPage.doLogin(userName, PASSWORD);homePage.clickOnLogoutBtnAndGoToLoginPage();
}
149
Different Browsers, Different Problems
• Not always click
• Can’t delete cookies
• Doesn’t scroll to links out of view
Layered Architecture
Page Objects Layer
Tests Layer
ActionBot
150
Meet ActionBot
Image courtesy of Simon Howden at FreeDigitalPhotos.net
public class ActionBot {
private final WebDriver driver;
public ActionBot(WebDriver driver){this.driver = driver;
}
public void clickOnElementLocatedBy(By by){WebElement element = driver.findElement(by);element.click();
}
public void sendKeysToElementLocatedBy(By by,String keys){WebElement element = driver.findElement(by);element.clear();element.sendKeys(keys);
}
}
Q&A
151
152
Tips and Tricks Appium Via Selenium
June 2016
On The Agenda
Intro of Gett & Automation FrameWork
Intro of Selenium Page Object Design Pattern
Examples of use
Web-Locators Tricks for Object Repository
Intro for mobile Client Automation infra
Drill Down into ios Platform using Appium
Example of Use for webDriver on AppiumDriver
153
Some Numbers
$520Million
In Founding
700Employees
50,000+Taxis RunningOn Gett Technology
70Cities
4000+Enterprise clients
Around the globe
30,000,000+PassengersOn all platforms
Automation Challenges
★ Multi platforms under same test
★ Challenge to manage small testable units
★ Challenge managing test run time
★ Location based product
154
Gett Automation FrameWork
Automation Platforms
GUI Platform:
→ Android - client and Driver app
→ Iphone DBX and client
→ Web -
● Company web
● Customer Care
● Private Web
● Mobile Web
Headles
Hybrid
155
Automation periodic runs
•Headless E2E - Main Server - content related - every pull request
•Headless Regression - Nightly run
•Api tests - micro services related - every pull request
•GUI Regression - every regression phase in Sprint
•API Regression - every regression phase in Sprint
Load Zone
●Load Infra inherit / based On Automation API infra
●Automation script with Jmeter adaptations run on Jmeter tool
Advantages:
→ Maintenance in automation zone reflect automatically on the load zone
→ Easy to create test scenarios
→ Long terms of use
→ Load is no longer belong to the load professionals …
→ Inherit Jmeters strength to load the system
156
Gett Page Object Design pattern within interfaces
Page object is a design pattern that can be implemented as a selenium best practices.
• The Page Object pattern represents the screens of your web app as a series of objects and
encapsulates the features represented by a page.
• It allows us to model the UI in our tests.
• A page object is an object-oriented class that serves as an interface to a page of your AUT.
• Using common wrapper class for basic web functionality is recommended
• When we have page with the same functionality/ Object in two different pages use interfase
Gett Page Object Design pattern Tips
• AbstractPage Class is the place to holds all the wrappers for selenium methods • Abstract Methods serves all webPages and contain only baseWeb Functionality regardless Buisness logic
Using - AbstractPage
Inside - AbstractPage class
157
Gett Web Page Design pattern within interfaces
inherit
AbstractPage()
inherit
AbstractPage()
Gett Page Object Design pattern within interfaces
• The Use in Abstract class - implement the interface for common Functionality && common objects
Two Implements for TariffService{}
The Middle abstract serves the Common TariffAbstract{}
inherit
AbstractPage()
158
Gett Page Object Design pattern within interfaces
The Interface - for Tariff functionality that implement by two different pages
TariffService class
Gett Page Object Design Pattern within interfaces -inside TariffAbstract class
• The method FillFixChargeForm() is not common
• The method setFixTariffrFromAddressDetailes() is common and also part of the calls in the above
Not a common Method
A Common Method
TariffAbstract class
159
Gett Page Object Design Pattern - within interfaces
• The method FillFixChargeForm() contain its unique functionality and also use setFixTariffrFromAddressDetailes()
PhoneUserTariffServiceImpl class
Gett Page Object Design Pattern Tip - isInModule
• Every Page holds Sync point to be sure that we in the correct place and the page was loaded • Wait contain polling - until object found limit by X time !!!
• In case of Failure throws exception
160
Tip - Web Locator and Object Repository - mapping the objects
WebLocator ClassWeb Locators Business Logic
Repository
Hold one focal point that capture the locators Differentiate by logicalZone and name for objects
Page object Design Pattern -Main Rules
• Every page implement in a class
• Every Page get the selenium driver and use it to perform actions on objects
• Every Page pass it’s Driver to the new Page in the flow
PhoneUserTariffServiceImpl class
Transition to the Next class FixPriceDetailsPage
161
Mobile Client Automation FrameWork
Appium is “cross-platform”: it allows you to write tests against multiple platforms (iOS, Android), using the same API , based on Selenium API .
Tip - One interface for Business logic per client
• The test use one interface object that implements differents platform - Main interface obj.
• The main business logic object perform all the business logic calls in the system
162
Trick -The Implementor for IOS platform - Appium Init
iphoneDriver = getTaxiAppium.connect();
driver = iphoneDriver;
// AbstractPage by passing to it the webDriver ;
pageImpl= new AppiumDriverPageImpl(driver)
Appium Driver PageImpl class
163
The Implementor for IOS platform - Appium
• Example of using the obj “pageImpl” to click on btn using selenium Api
• The Method iphoneButtons.get() transform the the getTaxiButton into our the WebLocatore Object
pageImpl.clickOnButton.
(iphoneButtons.get(getTaxiButton.name()));
Inside AbstractPage Class
The Implementor for IOS platform - Appium
• Example of using the obj “pageImpl” to click on Mouse using Events selenium Api
pageImpl.selectAutoCompleteOptionWithMouse
(iphoneButtons.get(GetTaxiButton.OK.name()));
164
● QA Testers
● Automation developers
● Architect
● Fullstack Web Developer
● iOS / Android
● Product Manager
● Product Analyst
● BI
● DevOps
● Big Data
● UX/UI designers
WE ARE HIRING
165
Reports
Get the most out of your tests
“Not only must automation be
done; it must also be seen to be
done” someone, somewhere
166
Who Cares About Reports?
167
JSystem
• My favorite test automation framework
• I’m the main project contributor
• Used in most of my company’s projects
168
• Ugly
• Confusing
• Slow
• Unmaintainable
• Hard to customize
It has some drawbacks
169
JUnit Example
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<testsuite errors="1" failures="1" hostName="agmon-PC" name="JSystem Reporter" tests="9" time="1022.419" timeStamp="Sun May 29
11:13:29 IDT 2016">
<testcase classname="il.co.topq.auto.tests_project.TestsExamples" name="reportError" time="0.227">
<error message="Error" type="class java.lang.Exception">java.lang.Exception: Error
at jsystem.framework.report.RunnerListenersManager.addError(RunnerListenersManager.java:355)
at jsystem.runner.remote.RemoteExecutorImpl$ReaderThread.run(RemoteExecutorImpl.java:368)
</error>
</testcase>
<testcase classname="il.co.topq.auto.tests_project.TestsExamples" name="controlParametersAttributes" time="0.173" />
<testcase classname="il.co.topq.auto.tests_project.TestsExamples" name="testWithParameterProvider" time="0.205" />
<testcase classname="il.co.topq.auto.tests_project.TestsExamples" name="reportWithLevels" time="0.224" />
<testcase classname="il.co.topq.auto.tests_project.TestsExamples" name="testWithReturnParameter" time="0.2" />
<testcase classname="il.co.topq.auto.tests_project.TestsExamples" name="testWithParameters" time="0.161" />
<testcase classname="il.co.topq.auto.tests_project.TestsExamples" name="reportWarning" time="0.136" />
<testcase classname="il.co.topq.auto.tests_project.TestsExamples" name="reportFailure" time="0.142">
<failure message="Fail report was submitted" type="class
junit.framework.AssertionFailedError">junit.framework.AssertionFailedError: Fail report was submitted
at jsystem.framework.report.RunnerListenersManager.addFailure(RunnerListenersManager.java:466)
at jsystem.runner.remote.RemoteExecutorImpl$ReaderThread.run(RemoteExecutorImpl.java:373)
</failure>
</testcase>
<testcase classname="il.co.topq.auto.tests_project.TestsExamples" name="reportSuccess" time="0.105" />
</testsuite>
Cucumber Example
170
TestNG Example
ReportNG Example
171
So I decided to write something on my own
• Real time!
• Easy on the eye
• Fast
• Flexible – Add screenshot, files, tables.
• No server
• It should have a weird name
It should be:
Frameworks and Languages
I also work with a few frameworks and
languages:
• C# with NUnit and MSTest
• Python with PYTest and Nose
• Java with TestNG and JSystem
• Javascript with Jasmine
172
Difido Architecture
• At that point, I didn’t know much about
Javascript.
• But I did know that MVC is the way to go
Controller
ModelView
Difido Architecture
173
Demo of HTML report
Now I want a server
• An addition to the local reports
• Central web interface
• Keep history
• Join results from different stations
• Real time, real time, real time
• Damn fast
• Easy to deploy
Requirements:
174
Difido Server Architecture
Difido Server Demo
175
How about BI?
• We gathered a lot of data. Can I do
something with it?
Elasticsearch
• Open source
• Java Based
• NoSQL
• Very Scalable (Peta bytes of data)
• Includes a text search engine (Lucene)
• Part of the ELK stack
Shay Banon
176
Difido Server Architecture
Difido Server Architecture
177
More Features
I needed some more features
• Send mail at the end of the run
• Delete old reports
Success Stories
178
You Can Use It too
• Site: http://top-q.github.io/difido-reports/
• Documentation: https://github.com/Top-Q/difido-
reports/wiki
• Github: https://github.com/Top-Q/difido-reports
Q&A
179
180