testing in devops: ukstar conference

Test value first before correctness

27-28 February 2017LONDON

Laurent Py

@py_laurentlaurent.py@hiptest.nethttps://hiptest.net

PLAN

2

The match: quality VS speed

Step 1: Testing the idea

Step 2: Testing first increment in production

Step 3: General availability of the feature

Benefits of a progressive approach to quality

27-28 February 2017LONDON

QUALITY VS SPEED

WE HAVE SPEED

4

Between 5 to 10 deployments/day

Our DevOps

teamDevs/testers

Ops

Product manager

Users

Marketing

Support

4

WHAT ABOUT THE USER?

5

Speed of development feedback and learning

Value VS Quality

5

27-28 February 2017LONDON

STEP 1: TESTING THE IDEA

8

• Use personae • Capture the expected behavior with BDD • Define business assumptions

Benefits

Support

Devs/testeurs

Marketing

Ops

Product manager

WHAT VALUE DO WE WANT TO DELIVER?

8

9

• Create a shared understanding with examples• Based on a common business language• Definition of stop• When tests are automated → living

documentation

BDD AT A GLANCE

9

1111

27-28 February 2017LONDON

STEP 2: TESTING FIRST INCREMENT IN PRODUCTION

13

1. After step 1, we have our definition of success

2. We develop a first increment, do ET

3. We deploy it in production

FEATURE IS DEVELOPED

13

14

• Based on usage- NB users, clicks- NB requests…

• Impact on- Sign-up- Revenue- Retention…

LET’S MEASURE THE IMPACT

14

15

EXAMPLES OF METRICS

15

Notification feature• Usage

- NB clicks NB of notifications- NB of users

• Business: none

Test refactoring• Usage

- NB of refactoring actions- NB of impacts

• Business: Retention

16

LIVE CHAT TO GET REAL TIME FEEDBACK

16

1717

27-28 February 2017LONDON

STEP 3: FEATURE FULLY AVAILABLE

19

1. The impact has been validated

2. Investment on performance and non regression (checks are fully automated)

3. Feature made available to all users with a switch

FEATURE BECOMES FULLY AVAILABLE

19

20

Appsignal, Pingdom, Scalingo et Logmatic

MONITORING PERFORMANCE AND BUSINESS

20

2121

27-28 February 2017LONDON

CONCLUSION

23

•Test the value first• Invest progressively on quality•Create intimacy with the users•Create a suitable environment for experimentations

KEY TAKEAWAYS

23

Thank you!

27-28 February 2017LONDON

https://hiptest.net