automated ui test on mobile - with cucumber/calabash

Mobiltest automation - experiences from the trenches

Let’s Talk Apple, 11. sep, 2014 Niels Frydenholm, ebay Classifieds

eBay Inc. Confidential2

Agenda

• Intro

• Tools

• Examples / Demo

• Continuous Integration / Follow up

• Challenges

• Latest improvements

• Questions?


eBay Classifieds, Denmark

• DBA, BilBasen og Bilinfo

• DBA (a classifieds site/app) has approx. 1 mill downloads

• BilBasen (cars) has approx. 450.000 downloads

• Both brands have apps for iPhone, iPad and Android

• Traffic from mobile doubled during 2013

• Strive for rapid releases


How we are organised

• Developers • QA

• Product Owner • UX • Other stakeholders


Tools

Jenkins (CI)


Cucumber -‐ more than a test tool



• Behaviour-‐Driven Development

• Adds “power” to Gherkin syntax

• A great way to communicate and cooperate between “the business people”, developers and QA.

• Written in (your) business/domain language

• Easy to both read and write








Feature: Simple search As a user I want to be able to do simple searches









Scenario: I can search without any parameters Given I am on search When I perform the search Then I see the result









Scenario: I can search without any parameters Given I am on search When I perform the search Then I see the result

Step-definition!!Then(/^I see the result$/) do! unless query("view marked:'SearchResultCell'").count > 0! screenshot_and_raise "No result was shown"! end!end!!


Calabash

• Enables the use of Cucumber for Native apps

• iOS

• Android

!• The bits and pieces that interact with the app

• touch, scroll, swipe etc

• Query-‐syntax (iOS examples)

• query(“label marked:’label text’”)

• query(“button marked:’Ok’”)


Calabash -‐ architecture


Scenario implementation example

eBay Inc. Confidential

Scenario: Check that change classification clears matrixdata with warning! Given I am logged in as "Buyer"! And I am on the SYI hub! And I select classification "Hovedtelefoner"! And I set "Type" to “Nakkekontur”! And I set price to "250"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Fortryd"! Then I see cells containing "Nakkekontur"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Skift"! Then I see the SYI classification search!

9


SYI = Sell your item

eBay Inc. Confidential

Scenario: Check that change classification clears matrixdata with warning! Given I am logged in as "Buyer"! And I am on the SYI hub! And I select classification "Hovedtelefoner"! And I set "Type" to “Nakkekontur”! And I set price to "250"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Fortryd"! Then I see cells containing "Nakkekontur"!! When I press "Kategori"! Then I wait for "Du er ved at skifte kategori" to appear!! When I touch "Skift"! Then I see the SYI classification search!

9

Scenario: Check that change classification clears matrixdata with warning Given I am logged in as "Buyer" And I am on the SYI hub And I select classification "Hovedtelefoner" And I set matrixdata "Type" to “Nakkekontur" And I set price to "250"!! When I try to change classification Then I see an alert with title "Du er ved at skifte kategori" ! When I cancel the alert Then the classification is "Nakkekontur" ! When I try to change classification Then I see an alert with title "Du er ved at skifte kategori" ! When I accept the change classification alert Then I see the SYI classification search




Step from “sell your item feature”!!And I set price to "250"



!And /^I set price to "(.*?)"$/ do |price|! macro 'I swipe up'! sleep(1)! touch("view marked:'Price'")! sleep(0.5)! set_text("view marked:'Price'", price)! sleep(0.5)! touch("view marked:'OK'")! sleep(1)!end





!And /^I set price to "(.*?)"$/ do |price| @page.write_price(price) end





!And /^I set price to "(.*?)"$/ do |price| @page.write_price(price) end


SellYourItemPage (page object) def write_price(price) scroll_and_wait_for_row_with_mark("priceCell") touch("view marked:'Price'") keyboard_enter_text price close_keyboard end


PageObject pattern

Feature

Scenario

Step


PageObject pattern

Feature

Scenario

Step

Page object

iPad iPhone Android


PageObject pattern

Feature

Scenario

Step

Page object

iPad iPhone Android

SyiHubPage (PageObject) !• go_to_picture_center !• go_to_classification_search !• go_to_value_selection_for_matrix

data(label) !• write_price(price)


DEMO


Continuous Integration

• Jenkins on a couple of Mac Mini’s

• Runs multiple builds/tests in parallel (Master/Slave Jenkins setup)

• The tests only runs in the simulator

• Unit tests —> Integration tests —> Accept/UI tests

• A subset of the Accept tests run after each commit

• Controlled by @tags in cucumber

• All Accept tests runs once every evening (or manually)


The “control room”


Cucumber reports


When it fails…


Ensure high quality tests

• Review and/or Pair program tests with QA

• (Only) use “Custom steps”

• Test code is (also) production code

• Let the quality reflect that!

• Refactor, Refactor, Refactor!

• Make sure the tests actually covers what is expected

• Do it with facts -‐ no guessing/assumptions


Code coverage

• Do not (necessarily) focus on the coverage percentage

• Find inspiration to new test scenarios

• Ensure every feature in the app is well covered


Keys to success

• Team effort

• Definition of Done (DoD) includes automated tests

• Know that it costs time -‐ it’s not a silver bullet

• Maintenance of setup -‐ new versions of iOS & Calabash

• Fix broken builds ASAP!


Future improvements (as seen in January)

Run a subset of tests in Xamarin test cloud a couple of times each week

Do more BDD/TDD

Automatic feature builds on Jenkins

Faster feedback cycle

Select a subset (with @tags) to run after each commit

Use “backdoor” to do e.g. login

Avoid resetting after each scenario

Make all tests run stable all the time!


Xamarin Cloud


Xamarin Cloud

• Currently only as proof of concept on DBA

• 5 scenarios on 3 devices

• Waiting for a proxy solution to take full advantage

• Create listings (and complete payments)

• BilBasen is also running in the cloud

• Trying out “Series” for iPad and iPhone and other cloud experiments to find out what works best for us


Calabash backdoor


Calabash backdoor

• Invoke logic in the app “behind the scenes”

• Reduce scenario test-‐time

• Avoid testing the same UI over and over in each scenario

• Stable

• Easy to use


Calabash backdoor

• Invoke logic in the app “behind the scenes”

• Reduce scenario test-‐time

• Avoid testing the same UI over and over in each scenario

• Stable

• Easy to use

• We use it for

• Login (and create user)

• Create listings

• Clean up data

• (Code coverage)


Unstable tests


Unstable tests

• UI Automation / iOS Simulator often fails

• I thought it was Calabash being unstable :-‐)

• We would start tests over and over and over…

• …and overlook real failures

• It was almost impossible to get our “All scenarios” to run green


Stable tests

• Cucumber Rerun formatter to the rescue


Stable tests


A simple script that will….


Stable tests


Run tests - use rerun formatter



Stable tests



If rerun.txt contains tests - run them



Stable tests




Inform jenkins how it all went



Stable tests




Inform jenkins how it all went


Replace rerun result in original result


Wrap up!

• Test scenarios follows the code = the truth

• Enables rapid releases / fast time to market

• A necessity to deliver high quality

• Spend less time doing (boring) trivial regression tests

• Happy QA´er

• Brave developers -‐ higher code quality

• Quality product

• Happy users


QUESTIONS?


Ressourcer

• The Cucumber Book

• Calabash @github (https://github.com/calabash/)

• Calabash google groups • https://groups.google.com/forum/#!forum/calabash-‐ios

• https://groups.google.com/forum/#!forum/calabash-‐android

• Calabash overview (http://docs.xamarin.com/guides/cross-‐platform/test_cloud/calabash/intro_to_calabash/)

• Jenkins (http://jenkins-‐ci.org/) • plugins: Cucumber-‐reports, Cobertura (code coverage), Build monitor view

• RubyMine (http://www.jetbrains.com/ruby/)

• Uncle Bobs Clean Code

• because it’s a great book that promotes craftsmanship

https://github.com/calabash/

https://groups.google.com/forum/#!forum/calabash-ios

https://groups.google.com/forum/#!forum/calabash-android

http://docs.xamarin.com/guides/cross-platform/test_cloud/calabash/intro_to_calabash/

http://jenkins-ci.org/

http://www.jetbrains.com/ruby/

automated ui test on mobile - with cucumber/calabash

Technology