Boston

Best Practices using open source for testing

Eran Kinsbruner, @ek121268Amir Rozenberg, @Amir_Rozenberg

SHIFTING TO OPEN SOURCE, BENEFITS

MOBILE OSS MARKET LANDSCAPE, TOOLS, COMPARISON

DEMO!!!

OSS: BEYOND TESTING

Q&A

What is automated

Mobile WebAutomation target

Cannot be automated

60%-90%20% 30% 100%

The gap we want to address

Degree of automation

Increased complexity and no timeMany teams with varying set of skills

2/8/2017

TestersDevTestersCoders

A Day in A Life –Real User Environments Is EssentialCan Any OSS Framework Handle Such Requirements?

Automation ought to cover the full user environment not just the Application Under Test!

Which Tool is the best fit for you?

SELENIUM APPIUM CALABASH ESPRESSO XCTEST UI

When looking at today’s open-source mobile test automation landscape, there are five

highly-adopted test frameworks.

Each tool has advantages for your mobile and web testing depending on your needs.

Making the DecisionA Comparison

Detailed analysis of each test

automation tool:

Sometimes More Than A

Single Framework is The Right

Answer

Selenium

• The best choice for web test automation teams testing for responsive web design or stand-alone web sites

• Less suitable for developing unit testing, making this framework less appealing for developers

• On its own, Selenium core test reports are not highly informative and lack unique insights

Appium

• Best suited for QA teams testing the functionality of native, mobile web and mobile hybrid apps across iOS and Android

• Less suitable for developers who wish to develop and perform unit testing

• Appium reports are a bit limited from a debugging and fast feedback loop perspective, and do not include videos, network logs and key vitals information

Calaba.sh

• Designed for organizations that work in behavior-driven development workflows

• Does not cover the entire mobile system level events

• Offers an easy path to both develop and test features in parallel, all in an easy user-flow based language

• Appealing for both dev and QA practitioners

• Provides solid insights and reports to both dev and QA teams

Espresso & XCTest UI• Both requires a separate Test App executable

• Espresso is for Android and XCTest is for iOS – not a single script

• Both tools are fully integrated into development IDEs such as Android Studio/Xcode, and offer very easy-to-develop techniques, including test recorders

• Fully maintained by Google and Apple, which assures they always support the latest OS features so developers can stay ahead of the market and test accordingly

• Support both unit testing types and functional UI testing

• Both are app-context only, which limits their abilities to test for user condition scenarios

XCTEST UIESPRESSO

Things to Consider• Lab: Support for variety of frameworks is important for enterprises

• Flexibility• Autonomy• Complete E2E coverage• Unattended reliable testing

• Automation: support for multiple OSS frameworks becomes important• Different “persona’s” need different tools under 1 lab• Cross platform scripting• Robust framework to support high test coverage

• Analysis: Digital reporting is necessary for teams to take action and fix issues faster & earlier

When choosing a test automation tool

Area Components Quantum

Management

Reporting

Execution Manager

Test Development

Test Authoring (IDE+Language)

Object Repository

Object Locator/Spy

Appium/Selenium✓OSS

=

✓OSS

✓OSS

http://projectQuantum.io

Digital – Responsive Web Application

DEMO – BDD OSS Framework

Few other OSS-made mobile testing toys

Production Insight/ ”Testing in Production”

Network features: HAR file & NETEM

Some others:

• Security scans:

https://github.com/MobSF/Mobile-

Security-Framework-MobSF

• Deep visual analysis

• Etc.

Q&A @Amir_Rozenberg@ek121268

@Amir_Rozenberg@ek121268

22

Open &

Integrated

Xcode

Selenium