CONTINUOUS

DELIVERY WHILE

MINIMISING

PERFORMANCE

RISKS

INTRODUCTION

OBJECTIVE

Put working software into production as quickly as possible,

whilst minimising risk of load-related problems:

› Bad response times

› Too small capacity

› Availability too low

› Excessive system resource use

RISK PREVENTION IS A BIG SUBJECT

Photo by chillihead: www.flickr.com/photos/chillihead/1778980935

PREVENTING RISK IS A BIG SUBJECT,

WHAT FOLLOWS IS TAKEN FROM OUR EXPERIENCE

CONTINUOUS DELIVERY LITERATURE PROVIDES METHODS

THAT HELP REDUCE RISK

› Blue-green deployments

› Dark launching

› Feature toggles

› Canary releasing

› Production immune systems

Jez Humble, http://continuousdelivery.com

BLUE-GREEN DEPLOYMENTS

Version n+1

Version n

Amazon

Route 53

Elastic

Load

Balancer

Elastic

Load

Balancer

Instances

Instances

DARK LAUNCHING

Web page DB

DARK LAUNCHING

Web page DB Weather SP

DARK LAUNCHING

Web page DB Weather SP

FEATURE TOGGLES

0%

CANARY RELEASING

100%

PRODUCTION IMMUNE SYSTEMS

USE CONTROLLED LOAD TESTING TO HELP CAPACITY

PLANNING

Instance

Amazon

Route 53

Elastic

Load

Balancer

RDS DB

Instance

RDS DB Instance

Read Replica

Instance

Instance

WORK WITH FAILURE

› Optimise for MTTD and MTTR, not MTBF

› Game day exercises

› Chaos monkey

› Go / NoGo meetings

› Retrospectives

BUT LEGACY SYSTEMS OFTEN LACK THE REQUIRED

RESILIENCE

WHILE WE WORK ON OUR RESILIENCE, WE USE LOAD TESTS

TO HELP IDENTIFY THE BIGGEST RISKS

PRE-PROD LOAD TESTING IS NOT FREE

› Extra code to maintain

› Usually test runs last several hours

› A production-like environment is expensive

› Realistic testing is hard

› Not all developers like writing (performance) tests

USE IT WISELY, WHERE PRODUCTION TESTING IS STILL

INAPPROPRIATE

› It provides no guarantee

› Use it to find any showstoppers you can

› Essentially, an optional service that teams can use

Photo by vastateparksstaff: www.flickr.com/photos/vastateparksstaff/5330257235

USE IT AS A PLAYGROUND TO TRY RISKY CHANGES

Functional integration

tests

Load tests

Build, unit test, etc.

Very often Less often At least once a day

(at night)

Functional integration

tests

Load tests

Build, unit test, etc.

Very often Less often At least once a day

(at night)

Load test script check

THE AIM IS NOT PERFECTION, GO FOR “AS REALISTIC AS

NEEDED”

SET UP TEST DATA IN THE WEEKEND, TO MINIMIZE

DISRUPTION

WHEN IS A PROBLEM REALLY A PROBLEM?

FIND AN OBJECTIVE WAY TO JUDGE YOUR FINDINGS

ESTABLISH REQUIREMENTS TO MAKE CLEAR WHAT IS

ACCEPTABLE

› Seen from the main stakeholders’ perspective

– Response time: users

– System resources: ops

– Capacity: business

› Specific

› Measurable

› Achievable

› Relevant

Concurrent users

Scale: Maximum load in a day, while

response times are still according to

spec.

Meter: Session table row count.

Intention: The website should at least be

able to manage our typical daily load, but

we would like some margin for growth

and marketing campaigns.

Stakeholder: Business

Fail: < 100k

Target: 200k

Now: 150k

SO USE A REAL BROWSER TO TEST

A REAL USER’S EXPERIENCE

Response time Fail [Today] Target

Homepage.FV > 6 sec 3.9 sec 2 sec

Homepage.RV > 5 sec 2.8 sec 1 sec

Checkout.FV > 8 sec 6.5 sec 2 sec

Details.FV > 6 sec 1.9 sec 2 sec

Details.RV > 5 sec 1.7 sec 1 sec

Search.FV > 6 sec 4.8 sec 2 sec

Search.RV > 5 sec 3.7 sec 1 sec

Cart.FV > 6 sec 4.4 sec 2 sec

Cart.RV > 5 sec 3.4 sec 1 sec

LoginForm.FV > 6 sec 3.5 sec 2 sec

LoginForm.RV > 5 sec 2.5 sec 1 sec

TO MAKE COMPARING SENSIBLE, MAKE YOUR TESTS

DETERMINISTIC

Stub systems that you have no control over

LOAD TESTING SHOULD BE OPTIONAL, THE ONLY THING

THAT COUNTS IS PRODUCTION!

› Your definition of done should reflect that

› The aim is to get early feedback from a safe environment

ANYTHING YOU FIND IS AN OPPORTUNITY TO FIX MORE

THAN ONE PROBLEM

SO WHAT MONITORING IS TYPICALLY NEEDED?

› Be able to localise where latency is coming from!

– For every system, all incoming and outgoing calls (count and

time spent stats)

› Finite resources (pools, CPU, I/O, etc.)

› Number of active users

› Response size, where possible

› Add whatever you need

It should be identical on all environments!

CONCLUSION

In order to put code live without pre-prod load testing, at least the following need to be in place: › Culture › State-of-the-art monitoring › Resilience

Without these, support your continuous delivery process with optional load tests and strong specs. Use the load tests to identify some pain points, so you can modify the code and add monitoring, making it safer to do (incremental) dark releases and canary testing in production.

QUESTIONS?

athomas@xebia.com

@a32an

www.xebia.com

blog.xebia.com

(we’re hiring)