testing in a continuous delivery world - londoncd meetup - may 2014

TESTING IN A CONTINUOUS DELIVERY WORLD

Amy Phillips

Amy Phillips

! @itjustbroke !! testingthemind.wordpress.com

2

Songkick

Testing:

4

A search for information

Quality:

5

Value to some person ! - Gerald Weinberg

6

Test that you rolled a one

7

Test that you rolled a one

8

Test that you rolled a one

9

Test that you rolled a one

10

Test that you rolled a one

11

Test that you rolled a one

What were we doing?

! Checking the result !!!

! Testing the result

12

Explore It!: Reduce Risk and Increase Confidence with Exploratory Testing Elisabeth Hendrickson

13

! You can automate 100% of your system testing

!! Automation will guarantee your system

keeps working - after all you are testing literally everything

14

Some Myths about Automation

! You can never test everything to prove it is working

!! It takes just one test to prove that a

system is broken

15

Does it work?

! Not working according to requirements ! Doesn’t meet the user’s expectations ! Difficult to use ! Inconsistent ! Doesn’t scale ! Can’t be maintained ! Difficult to test ! …

16

What is a bug?

17

The (old) Songkick Way

Automated Test and build

Manual Test Feature & ReleaseQueued

for commit

Commit Deploy

18

Don’t Aim for Perfection

Why do we have this process?

Release

ProblemsMake everything

safe

Scheduled Continuous Releases Integration

!

!

!

20

What were our options?

or

What are our values?

!! Ship new features as soon as possible !! Developers responsible for production

quality

22

The (old) Songkick Way

Automated Test and build

Manual Test Feature & ReleaseQueued

for commit

Commit Deploy

1 Hour 2 Hours

Define a single test strategy

Developers Testers

Unit Tests

Integration Tests

Manual Checks

Manual Tests

Regression Tests

Automated Checks

Define a single test strategy

Developers Testers

Unit Tests

Integration Tests Manual ChecksManual Tests

Regression Tests

Automated Checks

25

!

Testing

100 %Team Effort

CheckingTesting

Agree on how to split test effort

26

!

100 %Team Effort

Testing Machine Human

Agree on how to split test effort

27

!!

! Dev teams are responsible for their own testing

A New Value

28

!! Test at the right level ! Everyone understands what is being

tested and why ! Everyone has a chance to influence

what should be tested ! Fast feedback

Shared ownership of Automated tests

29

!!

! What are we hoping to achieve with this change?

! Identify risks ! Agree on how to mitigate risks

Risk Assess Everything

30

The Songkick Way - low risk

Automated Test and build

Commit DeployAutomated Regression

31

The Songkick Way - medium risk

Automated Test and build

Manual Test on Dev env

Commit DeployAutomated Regression

32

The Songkick Way - high risk

Automated Test and build

Manual Test on Dev env

Commit DeployAutomated Regression

Manual Test on Prod env

Releasing doesn’t have to mean Launching

Trust the Test Results

! Never release if the build is red ! Green builds must mean the tests have

passed

!!You can break anything once ! - Songkick Maxim

35

Learn from your mistakes

Executing the right tests at the right time

Automated build and deploy process

Continuous Deployment

37

Tools

38

Use your brain

39

Keep on talking

Releases

0

28

56

84

112

140

Mar 2011 Jul 2011 Nov 2011 Mar 2012 Jul 2012

But this ended up being about more than just releases…

! Everyone cares about testing ! Fast and maintained automated tests ! Bug fixes often take just minutes

In Summary

! You must automate ! But don’t try to automate everything ! The automation you have doesn’t

necessarily make you safe ! Use problems to drive positive change ! Don’t neglect the human aspect

43

Songkick is hiring

44

Questions?! @itjustbroke !! testingthemind.wordpress.com !

! http://devblog.songkick.com/