testdroid: tips and tricks with calabash for mobile app testing

webinar

Tips & Tricks with Calabash for Mobile App Testing

13 March 2014

Ville-Veikko HelppiTechnical Product [email protected]

mailto:[email protected]

Agenda• Prerequisites for Mobile App Testing• Calabash Introduction and How It Compares to

Other Test Automation Frameworks• Inside the Calabash – Building Those Scripts• Tips & Tricks with Calabash• Demonstration• Q&A

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

2


Other Test Automation Frameworks• Inside the Calabash – Building Those Scripts• Tips and Tricks with Calabash• Demonstration• Q&A

webinar


3

Why Apps Fail?webinar


4

Prerequisites for Mobile App Testing• Testing environment as authentic as possible– Real devices <-> emulators/simulators

• Test coverage needs to be adequate– Software is not identical in all Android/iOS devices– Hardware is even more fragmented landscape

• Surrounding testing conditions (e.g. network) • Try not to build in repetitive, manual processes• Test automation framework that works for you

webinar


5

What Framework Works You The Best?• High-level & declarative test automation

frameworks work well when app is under dev• Easy to integrate with continuous integration• Distance between use case and actual test• What do you build: Native, Hybrid or Web app?• The technical competence in your organization• Desired outcome from test automation

webinar


6

What Improves the App Quality?• Automate generic things as much as you can• During the development your app changes –

the same must apply for testing!• Carefully select (testing) technology & partner• Use all 24 hours per day with test automation• Cloud-based platform is the only solution to

get you quickly covered in the global markets

webinar


7

Why Real Devices are Must-to-Have?• Emulators/simulators won’t help you to test…– User Experience and Usability– Hardware– Software– Infrastructure

webinar

0 % = the percentage of your app users that use emulator to run your app!


8

Basics: Different Ways of Doing Test Automation

webinar

Automatic test exercisers Record and PlaybackHand written test scripts

Benefits:

Accurate, specific to your testing needs, plenty of options with frameworks, tools

Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools avail’ty

Fastest & extremely automated, excellent for smoke testing/quick testing, availability

Tradeoffs:

Takes a lot of time, ties resources to write test cases/scripts, error-prone (humans)

Compelling Recorder+Playback tools available for only few test automation frameworks

Not accurate as real test cases


9

Agenda• Prerequisites for Mobile App Testing• Calabash Introduction and How It Compares

to Other Test Automation Frameworks• Inside the Calabash – Building Those Scripts• Tips and Tricks with Calabash• Demonstration• Q&A

webinar


10

Introduction to Calabashwebinar

• Behavior driven test framework for native Android & iOS, and for hybrid web apps (embedded webviews)

• Tests are described in Cucumber and then converted to Robotium or “similar-to-Frank“ in run time

• Supports currently about 80 different natural language commands (controllers). New controllers can be implemented in Ruby or Java

• If your app uses custom UI Widgets you have to create controllers for those in order to test your app

• Calabash tools include a command line inspector for finding right UI element names/ids


11

Android Example: The Family Tree of Test Automation Frameworks

webinar

JUnit

Android Instrumentation Framework

Robotium Espresso

UI Automator

Calabash

Appium


12

Why Calabash is A Good Choice? • Cross-platform test automation framework• Human-readable syntax/tests• Not much changes needed for tests when the

mobile app changes (e.g. UI components)• Pretty easy to integrate with any CI• Great choice for teams that have limited

resources for development and/or testing

webinar


13

Calabash-iOS: Architecturewebinar

• Calabash installs an HTTP server as an instrumentation package that listens commands from Calabash server

• Tests are executed on server side• Each test scenario is described in

Cucumber• Ruby Client library converts Cucumber

commands to ”Frank” method calls• Webview support is implemented with

Javascript injection to the Webview

iOS device

Your app

Calabash HTTP server

Test server

Step definitions, Ruby client

library

Features


14

Calabash-Android: Architecturewebinar

• Calabash uses Android instrumentation test server (a separate app)

• Based on ActivityInstrumentationTestCase2

• Tests are executed on server side• Predefined and custom steps• Ruby Client library converts Cucumber

commands to Robotium method calls• Webview support is implemented with

Javascript injection to the Webview

Android device

Your app

Instrumentation Test Server

Test server

Step definitions, Ruby client

library

Features


15

Calabash: Code Examplewebinar

Feature: Login feature

Scenario: As a valid user I can log into my app

I wait for text "Hello" Then I press view with id "Sign in" Then I enter text "username" into "login_username" Then I enter text "password" into "login_password" Then I wait for activity "HomeTabActivity" Then I press view with id "menu_compose_tweet" Then I enter text "Testdroid" into field with id "edit" Then I press view with id "composer_post"


16

How Calabash Compares to Others?webinar

Calabash Robotium uiautomator Espresso Appium

Android Yes Yes Yes Yes Yes

iOS Yes No No No Yes

Mobile web Hybrid(webviews)

Yes (Android)

Limited to x.y clicks

No Yes (Android & iOS)

Scripting Language

Ruby Java Java Java Almost any

Test creation tools

CLI Testdroid Recorder

UI Automator viewer

Hierarchy Viewer

Appium.app

Supported API levels

All All 16 => 8, 10, 15-19 All

Community Active Contributors Google Google Active


17

Example #1: Calabash with Cloudwebinar

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Users / Consumers

Rele

ase

Com

mit


18

Example #2: Calabash with CIwebinar

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Users / Consumers

Rele

ase

Com

mit


19



webinar


20

Step Definitions webinar

• Fast and easy to modify definitions and features• Predefined and Custom steps– Predefined steps offer the way to start instantly– Custom steps provide more options & can be easily

tailored for your app• Works for both, Android and iOS, when added:

– calabash.framework– ActivityInstrumentationTestCase2

• Custom steps extremely easy to modify & create


21

Test Script – called Featureswebinar


22

Feature: Login feature

Scenario: As a valid user I can log into my app

I wait for text "Hello" Then I press view with id "Sign in" Then I enter text "username" into "login_username" Then I enter text "password" into "login_password" Then I wait for activity "HomeTabActivity" Then I press view with id "menu_compose_tweet" Then I enter text "Testdroid" into field with id "edit" Then I press view with id "composer_post"

Example: Predefined Stepswebinar


23

Given /Î press the "([^\"]*)" button$/ do |buttonText| performAction('press_button_with_text', buttonText)end

Then /Î press button number (\d+)$/ do |buttonNumber| performAction('press_button_number', buttonNumber) end

Then /Î press image button number (\d+)$/ do |buttonNumber| performAction('press_image_button_number', buttonNumber) end

Then /Î press view with id "([^\"]*)"$/ do |view_id| performAction('click_on_view_by_id',view_id)end

From Definition to Featurewebinar


24

Then /^I press view with id "([^\"]*)"$/ do |view_id| performAction('click_on_view_by_id',view_id) end

Then I press view with id "menu_compose_tweet"

Definition:

Feature:


Other Test Automation Frameworks• Building the Calabash Scripts• Tips and Tricks with Calabash• Demonstration• Q&A

webinar


25

#1: Use Only Real Deviceswebinar


26

#2: Calabash is Perfect for “Test Early, Test Often” Approach

webinar


27

#3: Integrate Dev & Testing -> CI

webinar


28

Com

mit

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Rele

ase

Users / Consumers



webinar


29



webinar


30

webinar


31

testdroid: tips and tricks with calabash for mobile app testing

Technology