ios and android acceptance testing with calabash - xcake dublin

Acceptance Testing with Calabash

@roland9 roland@intercom.io

Xcake Dublin, 9th Dec 2014

What is Calabash?

What is Calabash?• Automated acceptance testing of mobile apps

What is Calabash?• Automated acceptance testing of mobile apps

• Cross-platform (Android & iOS)

What is Calabash?• Automated acceptance testing of mobile apps

• Cross-platform (Android & iOS)

• Open source - Company called Xamarin backing it

What is Calabash?• Automated acceptance testing of mobile apps

• Cross-platform (Android & iOS)

• Open source - Company called Xamarin backing it

• https://github.com/calabash/calabash-ios

What is Calabash?• Automated acceptance testing of mobile apps

• Cross-platform (Android & iOS)

• Open source - Company called Xamarin backing it

• https://github.com/calabash/calabash-ios

• https://github.com/calabash/calabash-android

Others

• Frank, KIF, Zucchini, UIAutomation, iCuke, …

Why would I do it?

Why would I do it?

Why would I do it?

Other Testing Methods

• Unit Tests

• Integration Tests

• Acceptance Tests

Unit Tests• Tests smallest unit of functionality

public void TestPhoneValidator(){ string goodPhone = "(123) 555-1212"; string badPhone = "555 12"

PhoneValidator validator = new PhoneValidator();

Assert.IsTrue(validator.IsValid(goodPhone)); Assert.IsFalse(validator.IsValid(badPhone));}

• Mocks and Stubs to simulate behaviour of other objects

Integration Tests

• Verify that combined modules function correctly

Acceptance Tests

• Performing tests on the full system (e.g. using your web page via a web browser)

• Described in plain English

How does it work?

• http://blog.lesspainful.com/2012/03/07/Calabash-iOS/

Install

Install• ruby -v -> recommended version 2.0+

Install• ruby -v -> recommended version 2.0+

• gem install calabash-cucumber

Install• ruby -v -> recommended version 2.0+

• gem install calabash-cucumber

• cd path-to-my-ios-project

Install• ruby -v -> recommended version 2.0+

• gem install calabash-cucumber

• cd path-to-my-ios-project

• calabash-ios setup

Install• ruby -v -> recommended version 2.0+

• gem install calabash-cucumber

• cd path-to-my-ios-project

• calabash-ios setup

• calabash-ios gen

Install• ruby -v -> recommended version 2.0+

• gem install calabash-cucumber

• cd path-to-my-ios-project

• calabash-ios setup

• calabash-ios gen

• in Xcode, build -cal scheme

Install• ruby -v -> recommended version 2.0+

• gem install calabash-cucumber

• cd path-to-my-ios-project

• calabash-ios setup

• calabash-ios gen

• in Xcode, build -cal scheme

• cucumber

If all goes well:

• LPSimpleExample[11298:13703] HTTPServer: Started HTTP server on port 37265

Build Directory• export APP_BUNDLE_PATH=/Users/rolandgr/

<…>/Build/Products/Debug-iphonesimulator/<…>-cal.app

Gherkin

• https://github.com/calabash/calabash-ios/wiki/01-Getting-started-guide

• Given …

• When …

• Then …

• And …

Useful Stuff

Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"

Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"

• cucumber NO_STOP=1

Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"

• cucumber NO_STOP=1

• cucumber -t @current

Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"

• cucumber NO_STOP=1

• cucumber -t @current

• cucumber -t @~excludedtag

Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"

• cucumber NO_STOP=1

• cucumber -t @current

• cucumber -t @~excludedtag

• calabash-ios console

Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"

• cucumber NO_STOP=1

• cucumber -t @current

• cucumber -t @~excludedtag

• calabash-ios console

• > start_test_server_in_background

Query Syntax• query(“button")

=> [

{"class"=>"UIRoundedRectButton", "frame"=>{"y"=>287, "width"=>72, "x"=>100, "height"=>37}, "UIType"=>"UIControl", "description"=>"<UIRoundedRectButton: 0x7d463d0; frame = (100 287; 72 37); opaque = NO; autoresize = RM+BM; layer = <CALayer: 0x7d46ae0>>"},

{"class"=>"UIRoundedRectButton", "frame"=>{"y"=>215, "width"=>73, "x"=>109, "height"=>37}, "UIType"=>"UIControl", "description"=>"<UIRoundedRectButton: 0x7d3a760; frame = (109 215; 73 37); opaque = NO; autoresize = RM+BM; layer = <CALayer: 0x7d3a8a0>>”}

]

• query("button index:0 label”)

"find the first button, and then inside of that find all labels”.

• query("button index:0 label", :text)

• touch("tabBarButton index:1")

accessibilityLabel accessibilityIdentifier

Custom Steps - Ruby API

Custom Steps - Ruby API• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-

iOS-Ruby-API

Custom Steps - Ruby API• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-

iOS-Ruby-API

• query, check for existence of elements, wait for elements to exist

Custom Steps - Ruby API• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-

iOS-Ruby-API

• query, check for existence of elements, wait for elements to exist

• keyboard

Custom Steps - Ruby API• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-

iOS-Ruby-API

• query, check for existence of elements, wait for elements to exist

• keyboard

• scroll

Custom Steps - Ruby API• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-

iOS-Ruby-API

• query, check for existence of elements, wait for elements to exist

• keyboard

• scroll

• rotate

Custom Steps - Ruby API• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-

iOS-Ruby-API

• query, check for existence of elements, wait for elements to exist

• keyboard

• scroll

• rotate

• backdoor -> call method in iOS app (implement in app delegate)

Custom Steps - Ruby API• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-

iOS-Ruby-API

• query, check for existence of elements, wait for elements to exist

• keyboard

• scroll

• rotate

• backdoor -> call method in iOS app (implement in app delegate)

• macro -> call other step definitions

http://www.slideshare.net/mobiletestsummit/ios-automated-testing-with-calabash-tips-and-tricks?qid=0afecec1-fe64-4168-bb69-a548bd01b832&v=default&b=&from_search=8

Tips and Tricks

Tips and Tricks• Resetting app data between scenarios

Tips and Tricks• Resetting app data between scenarios

• Avoid sleep - use wait instead:

Tips and Tricks• Resetting app data between scenarios

• Avoid sleep - use wait instead:

And I wait to see "send button"

Tips and Tricks• Resetting app data between scenarios

• Avoid sleep - use wait instead:

And I wait to see "send button"

• Wait before you interact with any element

Tips and Tricks• Resetting app data between scenarios

• Avoid sleep - use wait instead:

And I wait to see "send button"

• Wait before you interact with any element

• Wait after every action to finish

Tips and Tricks• Resetting app data between scenarios

• Avoid sleep - use wait instead:

And I wait to see "send button"

• Wait before you interact with any element

• Wait after every action to finish

• Escape quotes…

Tips and Tricks• Resetting app data between scenarios

• Avoid sleep - use wait instead:

And I wait to see "send button"

• Wait before you interact with any element

• Wait after every action to finish

• Escape quotes…

• https://groups.google.com/forum/#!forum/calabash-ios

Optional: Compare Images

• gem install imagemagick-binaries

• check you have compare: $ which compare

Optional: Compare Images

• gem install imagemagick-binaries

• check you have compare: $ which compare

Continuous Integration

Code Coverage

Other Info• UIKonf talk ‘Unit Testing on iOS’ by Tim

Brückmann https://speakerdeck.com/tibr/unit-testing-on-ios

• http://calaba.sh

• https://github.com/calabash/calabash-ios

• http://developer.xamarin.com/guides/testcloud/calabash/introduction-to-calabash/

Summary

• Automated Testing - investment that pays off

• Together with unit testing etc: make changes with confidence (still, manual QA required)

• Cross-platform mature toolset

• Continuous integration / code coverage

Acceptance Testing with Calabash

@roland9 roland@intercom.io

Xcake Dublin, 9th Dec 2014