Alexander Andelkovic, Spotify

STARWEST 2012 alex@spotify.com

Introduction

Background

Rapid Test Automation

Example

Summary

Fully Automated Testing

Questions

1/21

Lives in Stockholm, Sweden

15+ years testing experience

Test lead at Spotify

Passionate about advanced test techniques

2/21

Our reality:

- Tougher business competition

- Rapid releases needed to stay competitive

- Shorter release cycles (Year > Month > Week > Day)

Problem:

Difficult to keep-up with test automation of new

functionality as stories accumulates after each sprint

Solution:

Use Rapid Test Automation for faster test feedback loops

3/21

Test automation(TA) done by test developer and manual tester

Activity Traditional TA Agile TA Rapid TA

Modeling Test developer Manual tester Test developer

Manual tester Test developer

Coding Test developer Test developer Manual tester Test developer

Testing feedback Month(s) Week(s) Day(s)

4/21

Test automation(TA) done by test developer and manual tester

Activity Traditional TA Agile TA Rapid TA

Modeling Test developer Manual tester Test developer

Manual tester Test developer

Coding Test developer Test developer Manual tester Test developer

Testing feedback Month(s) Week(s) Day(s)

4/21

Test automation(TA) done by test developer and manual tester

Activity Traditional TA Agile TA Rapid TA

Modeling Test developer Manual tester Test developer

Manual tester Test developer

Coding Test developer Test developer Manual tester Test developer

Testing feedback Month(s) Week(s) Day(s)

4/21

Test automation(TA) done by test developer and manual tester

Activity Traditional TA Agile TA Rapid TA

Modeling Test developer Manual tester Test developer

Manual tester Test developer

Coding Test developer Test developer Manual tester Test developer

Testing feedback Month(s) Week(s) Day(s)

4/21

Agile TA Rapid TA

Finite State Machine(FSM) Extended Finite State Machine(EFSM)

Model-Based Testing used for auto generation of test procedures/vectors using models of system requirements

y-Ed used to create models

Graphwalker (Model Based testing tool) used to traverse models 5/21

- Simple scripts(Python) developed in Sikuli test framework to

build test automation

- Manual tester writes own test automation code

- No need to make test automation stable only keep usable code

6/21

- Exploratory testing done while modeling and coding

- Stress testing can be done immediately or overnight

- Manual tester give early/instant feedback to test developer of test

cases suited for robust regression testing

7/21

Features of Spotify desktop client rewritten into HTML 5

End user shall have same experience of the new rewritten HTML 5

features(What’s New, Top List, Profile) in the Spotify desktop client

8/21

Browse What’s New view

Main view

9/21

Play Top track

10/21

Search Track, Artist, Album

11/21

Write Message

11/21

Send track and message

11/21

12/21

Init

Loop

End

13/21

14/21

- Manual tester not dependant on test developer

- Test automation can be developed immediately when needed

- Valuable test feedback can be given within hour(s)

- Instant regression, performance and stress testing can be done

- Developers shows greather interest in testing

15/21

Fully Test Automation the next TEST frontier?

16/21

- GUITAR is a model-based system for automated GUI testing

- Developed by Atif Memon, associate professor in the

Dept. of Computer Science at the University of Maryland

- Automated GUI-based test case generation

- Automated test case execution

- Automated visualization of test case execution

Atif Memon 17/21

- Automated event/screen ripping with crawling of an

entire application

- Overlaying of test cases onto screenshots(node and

edge drawing)

- Recording of manual test cases through Selenium

IDE and execution with GUITAR Replayer

- Support for Web, iPhone, Android, Java apps

18/21

- WebTestingExplorer is an Automated test case generator for

web front-ends

- Developed by Scott McMaster former student of Atif Memon

- Generates and replay test cases, looking for differences in

state

- Detects unexpected HTTP status codes and Javascript errors

during exploration

Scott McMaster

19/21

E-mail: alex@spotify.com

Graphwalker: http://graphwalker.org/

yED: http://www.yworks.com/en/products_yed_about.html

Sikuli: http://sikuli.org/

Web Guitar:

http://sourceforge.net/apps/mediawiki/guitar/index.php?title=WebGuitar

Web Testing Explorer:

20/21

21/21