cross mobile testautomation mit xamarin & specflow

COPYRIGHT, TECHTALK - WWW.TECHTALK.AT

ANDREAS WILLICH (@SABOTAGEANDI)

CHRISTIAN HASSA (@CHRISHASSA)

Mobile Quality Crew Meetup, Vienna, April 23rd 2015

Cross Mobile Testautomation withXamarin & SpecFlow

2

Topics

• Xamarin• Test automation in agile projects• Specification-By-Example• SpecFlow•UI automation on mobile devices•Mobile app automation with SpecFlow

3

Xamarin

4

Native development for each platform

5

Abstracting the platform

6

Xamarin approach

7

Compilation and run time

Android APK

Android/Linux kernelIL code for platform

specific profileVS solution with

projects

iOS APP

Android APK

Shared Code(C#,VB.NET)

Mono CLR for iOS (Monotouch)

Mono CLR for Android

(Monodroid)

Multi-Platformprofile assembly

Dalvik

Android.*Java.*

IL+JIT

AOT ARM Binary

iOS App (C#)

Android App(C#)

Xamarin.iOSprofile assembly

Xamarin.Androidprofile assembly

iOS/Cocoa Touch

iOS APP

ARMbinary

8

Native mobile apps with C#/Xamarin

1. Reuse existing .NET/C# Know-How

2. Powerful platform and tools(Linq, Xml, Events/Delegates, Parallel Programming, Visual Studio/TFS, …)

3. Shared code for multi-platform apps

9

Required Know-How

C#

10

Xamarin.iOSC#

11

Xamarin.Android

twitterListView.setOnItemClickListener (new OnItemClickListener () {public void onItemClick (AdapterView<?> parent, View view, int position, long id) {

Intent tweetDetails = new Intent (TwitterScreen.this, TweetDetailsScreen.class);tweetDetails.putExtra ("TweetID", TwitterFeed[position].ID);startActivity (tweetDetails);

}};

twitterListView.ItemClick += (s, e) => {var tweetDetails = new Intent (this, typeof (TweetDetailsScreen));tweetDetails.PutExtra ("TweetID", TwitterFeed[e.Position].ID);StartActivity (tweetDetails);

};

C#

12

Development environment

OS Mac OSX, Windows Windows

UI Designer Android, iOS

Build Android, iOS (via Xamarin Build Host on Mac OSX)

Debug Android (Device and ARM/x86 emulator)iOS (Device and emulator on Mac)

13

.NET libaries availabile on Xamarin

Assembly feature setmscorlib.dll SLOpenTK.dll OpenGL/OpenALSystem.dll SL+System.Core.dll SLSystem.Data.dll NET35-System.Data.Service.Client.dll vollständiger OData ClientSystem.Json.dll SLSystem.Runtime.Serialization.dll SLSystem.ServiceModel.dll SL (WCF)System.ServiceModel.Web.dll SL+System.Transactions.dll NET35System.Web.Services NET35-System.Xml.dll NET35System.Xml.Linq.dll NET35

Mono.CompilerServices.SymbolWriter.dll Für Compiler

Mono.Data.Sqlite.dll ADO.NET fürSQLite

Mono.Data.Tds.dll TDS ProtokollSupport

Mono.Security.dll Cryptographic API

monotouch.dll iOS BindingsMono.Android.dll Android Bindings

14

Xamarin.Forms

One UI definition for 3 platforms (iOS, Android, WP)in XAML (data binding)translates to native UI controls on each platformcan be mixed with custom UI

15

Test Automationin

agile projects

16

Test automation becomes expensive when …

• automating manual tests

• automation makes tests

unreadable

• automating only after

implementation

structure

readability

right moment

17

StructureManual tests

Asserts Multiple combined features

Structure ACT-ASSERT-ACT-ASSERT-ACT-ASSERT-…

Dependent featuresLong test path with high chance to breakCause and impact of error hard to trace

Automated Check

Single aspect of a single feature

ARRANGE –ACT –ASSERT

Independent featuresShort test path with lower chance to breakCause and impact of error easy to relate

18

What to automate

Userjourneys

Acceptance-criteria

Units

exploratory testing

Test automation pyramidSource: Mike Cohn many

few hard

easy

Automatability

19

// Go to web page 'http://localhost:40001/' using new browser instanceBrowserWindow localhostBrowser = BrowserWindow.Launch(

new System.Uri(this.RecordedMethod1Params.Url));

// Click 'Register found item' linkMouse.Click(uIFundstückerfassenHyperlink, new Point(56, 9));

// Click 'Save' buttonMouse.Click(uISpeichernButton, new Point(44, 14));

int fundNr1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9));

// Click 'Register found item' linkMouse.Click(uIFundstückerfassenHyperlink, new Point(63, 7));

// Click 'Save' buttonMouse.Click(uISpeichernButton, new Point(34, 11));

int fundNr2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9));

Assert.IsTrue(fundNr1 + 1 == fundNr2);

// Click 'Close' buttonMouse.Click(uICloseButton, new Point(26, 11));

Readability

20

A readable test case

Scenario: New found items should receive a consecutive number for the current year

Given the previous found item of thecurrent year had the number 145

When I register a new found item

Then the last found item of thecurrent year should have the number 146

21

When to test (right moment)

Acceptance criteria checks(ATDD, BDD)

Unit Tests(TDD)

business view

technical view

Exploratory tests User Journey tests

Performance, Scalability, Usability,Security, …

pro

du

ct d

efin

itio

n pro

du

ct critiqu

e

New dimension: defining the productSynergy: Specification of requirements and tests

Agile Testing Quadrants: Brian Marick

22

Automated checks don’t replace manual testing

Userjourneys

Acceptance-criteria

Units

exploratory testing

manual checkin definition of done for stories

Main success paths

Undiscovered acceptance criteria

No/(fewer) manualregression checks

Fewer pathsare enough

More timefor exploration

23

Specification-By-Example

24

Agile “requirements”

Specification-

By-ExampleImage from: 50 Quick ideas to improve your user stories: https://leanpub.com/50quickideas

Why?Outcome

How?Specification

25

Agile specifications

define upfront define just-in-time

Code

Reminderfor aconversation

Goals

Impacts

Epics

User Stories

Acceptance criteria

Why?Outcome

How?Specification

26

public void CalculateDiscount(Order order)

{

if (order.Customer.IsNew)

order.FinalAmount =

Math.Round(order.Total * 9/10);

}

Register as “bart_bookworm”Go to “/catalog/search”Enter “ISBN-0955683610”Click “Search”Click “Add to Cart”Click “View Cart”Verify “Subtotal” is “$33.75”

We’d like to encourage new customers to buy in our shop.Therefore we want offer 10% discount for their first order.

Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com

27

Acceptance criteria

Agile specifications

define upfront define just-in-time

Code

Goals

Impacts

Epics

User Stories

Bugs

Why?Outcome

How?Specification

28

Specification with examples

Examples …make abstract descriptions better understandable

are usually not formally documented or exchanged

Examples Tests

Requirements

describe validatefulfillment of

consist of

29

Register as “bart_bookworm”Go to “/catalog/search”Enter “ISBN-0955683610”Click “Search”Click “Add to Cart”Click “View Cart”Verify “Subtotal” is “$33.75”

public void CalculateDiscount(Order order)

{

if (order.Customer.IsNew)

order.FinalAmount =

Math.Round(order.Total * 9/10);

}

We’d like to encourage new customers to buy in our shop.Therefore we want offer 10% discount for their first order.


30Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com

New customers should receive 10% discount:Given the customer hasn‘t ordered yet

When the customer adds a book for EUR 37.50 to the shopping cart

Then the shopping cart shows a total price of EUR 33.75.

31

Actually, this is not quite right:Books on sale should be excluded!


32

“HappyPath”

Technical feasibility

Exceptions, border cases

3 Amigos


33

Collecting Acceptance Criteria

“I would try to put a book into the shopping cart …”

“I would try to remove a book from the shopping cart…”

“I’d check whether the shopping cart is empty, when I enter the shop …”

Books can be added to shopping cart.

Books can be removed from shopping cart.

Shopping cart should be empty when entering the shop.

... ? …

As a potential customerI want to collect books in a shopping cartSo that I can order several books at once.

“Imagine this story is already implemented:

How would you verify it?”

“I would try to add 1000 books to the shopping cart …”

34

Exploration through examples

Books in catalogue:

Title Author

Specification-By-Example Gojko Adzic

Impact Mapping Gojko Adzic

Explore It! Elisabeth Hendrickson

Competitive Engineering Tom Gilb

… I want to find books in the catalogue by author and title

Search for … Books found …

Spec Specification-By-Example

Hend Explore It!

et Explore It!, Competitive Engineering

Context

ActionAssertion

35

Key examples: Breaking the model

Books in catalogue:

Title Author

Specification-By-Example Gojko Adzic

Impact Mapping Gojko Adzic

Explore It! Elisabeth Hendrickson

Competitive Engineering Tom Gilb

… I want to find books in the catalogue by author and title

Search for … Books found …

Spec Specification-By-Example

Hend Explore It!

et Explore It!, Competitive Engineering

What happens, if I search for “Explore Specification”?

Can I search for single characters, e.g. “e”?

Is the number of search results limited, or paged?

Is the search also performed in the sub-title of a book?

36

Purpose of the examples

• Shared understanding:acceptance criteria

•Documentation:specification details

• Regression-tests:violated specifications

37

SpecFlow

38

SpecFlow – BDD for .NET

#62 in Visual Studio Gallery based on popularity (Apr 23 2015)

#303 most downloaded package on NuGetin the last 6 weeks (Apr 23 2015)

~1‘400 unique daily visitors on specflow.org

> 30’000 active users> 40 contributors

39

Feature: Description of feature or user story

Szenariogrundriss: Beschreibung des Akzeptanzkriteriums

Szenario: Beschreibung des Akzeptanzkriteriums

Gherkin Feature Files

Background: context for all scenarios in the feature file

Scenario: Description of acceptance criterion

Angenommen/Wenn/Dann: Automatisierte Szenario SchritteGiven/When/Then: automated scenario steps

Scenario Outline: Description of acceptance criterion

Angenommen/Wenn/Dann: Automatisierte Szenario SchritteGiven/When/Then: automated scenario steps with <place holder>

Examples: table with examples for <place holders>Examples: table with examples for <place holders>

Given: automated scenario steps

@tagname

@tagname

@tagname

@tagname

40

Abstract acceptance criteria

As a shop visitorI want to collect books in my shopping basketso that I can purchase multiple books at once.

Books can be added to the shopping basket

Books can be removed from the shopping basket

Shopping basket is initially empty

The same book can be added multiple times to the shopping basket

41

Scenarios: Examples in Gherkin



Given my shopping basket is empty

When I add the book “Harry Potter” to my shopping basket

Then my shopping basket should contain 1 copy of “Harry Potter”

42



Scenarios: Examples in Gherkin

Given my shopping basket contains 1 copy of “Harry Potter”


Then my shopping basket should contain 2 copies of “Harry Potter”


43


Structure of Scenarios




Title: Describes intention/abstract acceptance criterion

Arrange: Context, describes state of the system

Act: Execution of the feature

Assert: Assertion of observable behaviour

And I should see the warning: “Book already existed in basket”

Triple-Aconstraint“Checks”

Chainingup steps

44

Automated continuous validation




System

„Step Definitions“ are binding individual steps to an automatable interface of the application.

Automatableinterface

UIAutomation

Automation does not necessarily have to bind to the UI.

Automatability of system is supported/evolving with development.

45

UI automationon mobile devices

46

Available tools/libraries

• Android• Android 2.3+: Instrumentation• Android 4.2+: UiAutomator• selendroid• Appium• Robotium• Calaba.sh• Xamarin.UITest

• iOS• UIAutomation• Appium• ios-driver• Calaba.sh• Xamarin.UITest• Frank• KIF (Keep it functional)

47

Selenium: cross platform, existing know how

•Android• selendroid• Appium

• iOS• Appium• ios-driver

48

Appium (www.appium.io)

• Provides WebDriver API (JSON Wire Protocol)•Uses native UI Automation library• Can test native & hybrid Apps, mobile web apps• Clients for C#, Java, Ruby, Python, PHP, JavaScript•Works with devices and emulators

49

Examples

50

Automation with SpecFlow and Appium

51

SpecFlow & Appium

•Write acceptance criteria independent of the platform•Write bindings against a Selenium- like- API• Run it on a devices or emulator

52

Demo

53

SpecFlow automationon the

controller layer

54

Different levels of automation

Controller

Business Layer

Data Layer

Model

View

Deviceautomation

Trigger behaviourthrough controller

Assert behaviouron model, db, ..

Setup pre-conditionsthrough serviceinterfaces

Out-of-process

In-process

55

How?

• Reuse the same feature files•Write new bindings against the

Controller/ViewModel/…

56

Demo

57

Faster feedback from tests

UI Automation Test Controller Test

58

Why is it faster?

•No overhead for deploying and starting the app•No need to wait for UI animations to finish•No delays for searching UI controls

59

Advantages Controller Testing

• Faster

• Better testable App architecture (Xamarin.Forms & MVVM)

60

Controller tests cannot fully replace UI tests

•UI tests are still needed• Diversity of mobile devices• Changes of manufacturer• Different Hardware Screen sizes Densities …

•Automated UI tests are still faster than manual testing

61

Conclusions

62

Excuses for avoiding mobile test automation

• “This is just a small app”• “Only complex end-end tests would make sense”• “Maintaining automated tests is too costly”

•Diversity of devices• Small apps also run into regressions• Bad feedback on app store cannot be undone

63

Automated testing in mobile App development

• Business readable automated checks• specification of acceptance criteria• test first development

• Testability is an ongoing architectural concern• every new feature needs to be testable• not every acceptance criterion needs to be tested through UI

• Automated tests don’t replace manual testing entirely• exploratory testing• assertions that are hard to automate• …

64

Questions and further information

Andreas Willich ([email protected])Christian Hassa ([email protected])

cross mobile testautomation mit xamarin & specflow

Technology