October 3, 2014

!

!

!

Andrii Dzynia Road Manager to Continuous Delivery

@adzynia

Quality Built In

2

Spotify brings you the right music for

every moment!

Started in 2006 (in Sweden) Now 1500+ employees, 500+ engineers 20+ million tracks available 4 data centres across the globe

What is Quality?

3

4

Not just software defects, but stakeholders expectations as well

when expectations matches the reality.

Quality is a “state” ….

Why Quality is Important?

5

Cost of Bug/Change

6

What is a way to build high quality product?

7

8Do not release until it’s perfect

9

No one is perfect.

Implementing internal back office system

Example

11

Step 1 - Prototyping phrase

Technical spike investigation Hacking different Back-End solutions

Ad-hoc design discussions Stub implementation of Front End

Getting early user’s feedback on Front End

12

13

Step 2 - Setting delivery pipeline

Add more unit tests Add more integration tests Add more functional tests Automate deployment configuration Add dashboards and system monitoring

Clarifying End User Needs

14

http://blog.crisp.se/2013/02/05/yassalsundman/continuous-delivery-vs-continuous-deployment

15

Step 3 - Connecting the dots

Connect back-end and from-end parts

into End-2-End system

Get user’s feedback and iterate

Keep releasing small, usable, incremental pieces16

17

What is under the hood ?

18

19

Test Automation Pyramid

20

How to avoid Ice-Cream effect?

21

One More Testing Pyramid

Testable micro-services architecture 22

Unit Tests

Component Tests

Functional Tests

System/End-2-End Tests

Integration Tests

Unit Tests

Component Tests

Functional Tests

JS Unit Tests

JS Component Tests

UI Functional

Tests

23

Unit Tests

Check logic of minimal code snippet Does not have any dependencies

or Mock all object dependencies

24

https://github.com/jsevellec/cassandra-unitEmbeddedCassandra

Very much like unit tests but emulating external components

e.g. cassandra database

Component Tests

25

Client App

Fake Back-End Server

Spotify Back End

Mocked Back-End

26

Integration Tests

Check an integration between components

You do not need whole system to check the integration

27

@docker containers

Create Test Environment ‘on the fly’

https://github.com/spotify/helios/blob/master/docs/testing_framework.md

28

Check services API behaviour or

End user use cases

Functional Tests

29

http://graphwalker.org

Model Based Testing

31

Code Review &

Human Testing on

Pull Request

Delivery Pipeline32

33

Was not covered during this talk …

Load Testing Gradual Rollouts A/B Testing Feature Toggles Monitoring/Alerting Information Radiators

… but stay tuned

34

Technical Test Engineer Test Engineer

! = Manual Tester ! = Test Automator

~ Software Engineer in Test

Test Engineering Roles and Responsibilities

~ Context Driven Tester

35

Technical Test Engineer

Work as a software developer Advocate testability of the product

Argue on software design with software engineers Help with building new/injecting existed development tools

36

https://github.com/jsevellec/cassandra-unit

Cassandra Unithttps://github.com/mikaellanger/job-dsl-plugin

Jenkins job-dsl-plugin

https://github.com/spotify/heliosDocker orchestration

Spoticloude.g. cli control over amazon cloud http://dashing.io

Dashboards

Development Productivity Tools

37

Knows business domain Focused on exploring the product

Free to use any programming language to test specific use case Free to use any programming language to automate his work

Test Engineer

38

Mind Map as a Tool

https://www.mindmup.com

Product tree Scenarios Playbooks Checklists

Session notes

Quality Engineers39

http://www.satisfice.com/blog/archives/1372

TEST JUMPERS: ONE VISION OF AGILE TESTING

http://www.satisfice.com/blog/archives/1364“RESPONSIBLE TESTER”

http://www.satisfice.com/articles/omega_tester.pdfOMEGA TESTER

@jamesmarcusbach calls us Test Jumpers !!!!

40

Any other ways to improve

quality?

41

!

test ideas during healthy discussions

test requirements via end users collaboration

discuss system design and conner cases earlier in the planing/design meetings

define definition of done

Break Uncertainty

42

But have responsible person to figure out what consensus means in each case

Use Google Docs collaboration for finding group consensus

Social Programming

43

write “checks” during implementation shape your thoughts via pair discussions peer review before merging to master

44

User Stories Planning meeting Standup meeting Open workspace TDD Refactoring etc……

Not mentioned Engineering Practices

http://www.extremeprogramming.org/rules.html

Employee Early Builds Testing

45

46

One week to do whatever you want

!

Do it alone or with a team !

Crazy ideas comes out

Hackathons

47

QA engineer at Spotify

http://continuousdelivery.com/2014/02/visualizations-of-continuous-delivery/

How to check you are on a right path ?

48

That is easy to spot 49

50

Take awaysMake Quality explicit Find Quality promoters Define your way of Quality

improvements Set right Quality constraints Share results as early as possible

Keep looking further quality improvements Probably you will never end :)

51

@adzynia adzynia@spotify.com

Spotify Engineering Culture part 1 part 2

and … !

Stop saying QA when you mean Testing

Check out http://www.spotify.com/jobs or @Spotifyjobs for more information

Want to join the band?