December 2, 2013

Experiences Of Test Automation

@ Spotify Kristian Karl

Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds, by Henrik Kniberg & Anders Ivarsson

Why automation?

• We want to move faster

• Quick dev feed-back

• Continous Delivery

• Continously monitor the

state of the product

• Bug hunting From: ”Intelligent Test Automation” by Harry Robinson

What to automate?

Graphical user interface testing Usability testing Software performance testing System testing Functional testing Load testing Volume testing Stress testing Security testing Scalability testing Sanity testing

Unit testing Smoke testing Component testing API testing Regression testing Installation testing Maintenance testing Recovery and failover testing. Accessibility testing Monkey testing Integration testing

What to automate?

Graphical user interface testing

Our goals

• Create automated end-user regression tests on 4 major

platforms 1) Desktop – Windows and OSX

2) iOS – iPhone and iPad

3) Android

4) Webplayer

• Help testers, not replacing them

• Deliver automated regression tests for a feature as a part of

definition of done

• Deliver short feedback loops to teams using Dashboards

Our challenges

• Hard-to-test SUT (Experiences of test automation: Case Study 1, An Agile Team’s Test

Automation Journey: The First Year), Dorothy Graham & Mark Fewster)

• Maintenance of automation

• Expectations

• Flaky tests... or is it flaky SUT?

• Testability

• Test data, test environments

• Supporting services

Model-based testing

• Models are the abstraction layer

• Testers design the automation using

models

• Developers implements the automation

using the models as drivers

Login view displayed

Main view displayed

Valid credentials

App not running

Start app

Log out

Close

Exit app

Invalid credentials Toggle

’Remember Me’

Start app

public interface SimpleLogin {

public void e_Close();

public void e_Exit();

public void e_Init();

public void e_InvalidCredentials();

public void e_Logout();

public void e_StartClient();

public void e_ToggleRememberMe();

public void e_ValidPremiumCredentials();

public void v_ClientNotRunning();

public void v_LoginPrompted();

public void v_WhatsNew();

}

De

mo

http://graphwalker.org/demo

Test automator

• Professional Java developer.

• Test experience is not mandatory.

• Embedded in the squad (team).

• Test automators form their own Guild

Developers and developers

• Why not use developers for TA?

• Why use developers for TA?

• Test API’s

- Defined by TA

- Implemented by developers

Section name 18

Test interface + wrappers

Client

Before

Test model Test model Test model

QA/TA

Devs

Strings

Java test interface

Client

Test model Test model Test model

ObjC test interface

JSON

After

Before

String cmd = "android.view.View seekBarView = solo.getView(com.spotify.mobile.android.ui.view.CancellableSeekBar.class, 0);"; cmd += "int[] xy = new int[2];"; cmd += "seekBarView.getLocationOnScreen(xy);"; cmd += "solo.clickOnScreen(xy[0] + 9 + (seekBarView.getWidth() - 18) * " + position + "f, xy[1] + seekBarView.getHeight() / 2.0f)"; BeanRemoteClient.sendToServer(cmd);

After

PlayerAuto player = Pages.remote(PlayerAuto.class);

player.seekTrack(position);

Supporting services

• Continuous testing

• QA Lab

• TDS – Test Data Service

• TRS – Test Result Service

• Virtualization

Open source (and freeware) tools that we use

• yEd – editing FSM models [www.yworks.com]

• GraphWalker – generate test sequences from models [graphwalker.org github.com/spotify/python-graphwalker]

• Sikuli – image recognition tool [www.sikuli.org]

• NuRemote - communicating with iOS client [github.com/nevyn/NuRemoting]

• TestNG – group tests into suites and and run them [testng.org]

• Java – code implementation

• Jenkins – [test] job scheduler 24/7 [jenkins-ci.org]

De

mo

Q & A