Alexander Andelkovic, Spotify

STARWEST 2011

Introduction

Background

Challenges

Success factors

Pitfalls

Tool

Examples

Summary

Questions

Lives in Stockholm, Sweden

15 years testing experience

Test engineer at Spotify

Passionate about advanced test techniques

Our reality:

- Testing in an agile developing cycle

- Commited to feature testing (Stories)

Problem:

Less time for story testing as stories accumulates

Solution:

Automate regression test, manual testers can focus on story testing

and rely on automation testing take the bulk of regression tests

Model-Based Testing (MBT) is the automatic generation of test procedures/vectors using models

of system requirements

Specific activities of the practice are:

1. Design models

2. Code test automation

3. Run tests

+ Easier to maintain test automation

+ Find bugs

+ Hire developers to work with testing

- Learning curve to get started

- Cost to setup test infrastructure

- MBT needs a different mindset

- Introducing test automation as a criteria for definition as done

- Test design (modelling) is done by testers

- Programmers do the automation

- Integrate test automation with nightly builds

- Tool chain

- Start with too complex modelling

GraphWalker is an Open Source Model-Based Testing Tool written in Java for

generating offline and online test sequences from Finite State Machines and

Extended Finite State Machines

The main features of GraphWalker are:

No UML: Uses GraphML, easier to get started with than UML

No exit/stop points:

- Long, unpredictable test sequences

- Does not want to walk the same path every time we execute a test

- Variation spiced with randomness. better 'test coverage'

- Stop generating test sequences are done by Stop Criterias

Online: Path walked through the model is decided at runtime (Web service)

Event-driven: Possibility to switch model caused by an event

ModelHandler: Re-use of models

yEd is a powerful diagram editor that can be used

to generate high-quality drawings of diagrams

yEd is free and runs on all major platforms:

Windows, Unix/Linux, and Mac OS

Supports .graphml format readable by Graphwalker

GraphML is a XML-based file format for graphs

Sikuli is a visual technology to automate and test graphical

user interfaces (GUI) using images (screenshots)

Sikuli includes Sikuli Script and Sikuli IDE, for writing visual scripts with

screenshots easily

Sikuli Script automates anything you see on the screen

without internal API's support. Uses OCR and image recognition

You can programmatically control a web page, a desktop application, or

even a smartphone application running in a simulator or via VNC

Access through Java API supported

Graphical editor for classification trees

Top features of CTE XL:

- Systematic and common classification tree method

- Automated test case generation

- Methodical linking with test automation tools

Problem: Many combinations

Generate:

Testsequence file (embedded code)

Execute:

CTEXL testcase model

Finite State Machine: Extended Finite State Machine:

Path walked through the model is decided runtime

An example of a script in QTP, executing a test using graphwalker

1. $> java -jar graphwalker.jar soap -f xml/login.xml

Now running as a SOAP server. For the WSDL file, see: http://myComputerName:9090/graphwalker-services?WSDL

2. WebService("SoapServicesService").SetTOProperty "WSDL", http://myComputerName:9090/graphwalker-services?WSDL

3. Do until not WebService("SoapServicesService").HasNextStep()

4. action = WebService("SoapServicesService").GetNextStep()

5. Loop

6. Reporter.ReportEvent micDone, "MBT Statistics", WebService("SoapServicesService").GetStatistics()

Mother model Sub model

Merged model (uses same data domain)

Purpose to reuse code and models (uses separate data domains)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18 SendSongs

WhatsNew

CheckInbox CleanInbox FreeUser PremiumUser

Switch model caused by an event

- Mindset

- Verifies functionality and finds bugs

- Test automation as a criteria of definition of done

- Testers to model

- Developers for test automation

- Integrate with nightly builds

E-mail: alex@spotify.com

Graphwalker: http://graphwalker.org/

yED: http://www.yworks.com/en/products_yed_about.html

Sikuli: http://sikuli.org/

CTE XL: http://www.berner-mattner.com/en/

berner-mattner-home/products/cte-xl/index.html