smoothing the continuous delivery path – a tale of two teams - lyndsay prewer

@lyndspsimple software solutions to big business problems.

Making Software. Better.

Smoothing the continuous delivery pathA tale of two teams

@lyndsp

What’s this all about?

Continuous

DeliveryDone Well

@lyndsp

What’s this (Continuous Delivery) all about?

“... producing valuable

software in short cycles and ensure that the software

can be reliably released

at any time”

- https://en.wikipedia.org/wiki/Continuous_delivery

@lyndsp

Continuous doughnuts

Fast feedback

Risk Reduction

T: 0:00:000 T: 2:00:000

@lyndsp

There’s no (CD) silver bullet

1 (php) monolith50 deploys / day1.5 billion page views / monthhttp://goo.gl/XIB29i

>600 java microservices>100 deploys / day~2 billion requests per day

http://goo.gl/ZJBiK6

@lyndsp

Prefer Continuous Improvement over Continuous Delivery

▪ .Net monolith▪ 26 -> 24 -> 34 releases

per year▪ https://goo.gl/KF9xXw

@lyndsp 7

People first, practices/process second, products last

@lyndsp

.Net monolith team - how they roll

▪ Private sector▪ Significant legacy code▪ High daily traffic (~100 million

visits)▪ ~8 product teams▪ Weekly deployments:

– Pick release candidate– Regression test– Dog food– Deploy

@lyndsp

Scala microservices team - how they roll

▪ .Gov - HMRC Digital▪ 2 years old (greenfield)▪ Calendar spikes (Visits peak is ~4

million)▪ >30 product teams▪ Multiple deployments per day

– Teams “own” their own services– “You build it, you run it” -

https://goo.gl/DyfeOf

@lyndsp

Continuous

DeliveryDone Well

Patterns for successful practice

@lyndsp

Best practice #1 - Healthy CI - mind & body

@lyndsp

Best practice #1 - Healthy CI - mind & body

Almostgreen!

@lyndsp

Best practice #2 - Testing as an activity, not a phase

Product Owner

Define

Web Ops

Deploy

@lyndsp

Best practice #2 - Testing as an activity, not a phase

Product Owner

Define

Web Ops

Deploy

@lyndsp

Best practice #3 - Healthy automated tests

Unit Tests

@lyndsp

Best practice #3 - (Un)healthy automated tests

Unit Tests

The “Hour Glass”

@lyndsp

Best practice #3 - (Un)healthy automated tests

The “Ice Cream Cone”

@lyndsp

Best practice #3 - Healthy automated tests

Unit Tests

The “Tear Drop”

@lyndsp

Best practice #4 - Low (manual) cost deployment (and rollback)

@lyndsp

Best practice #5 - Metrics, monitoring and alerting

http://seluxkanaur.deviantart.com/art/Sauron-s-Eye-6-420535318

“Big data”Social mediaUser behaviourBusiness metricsService performanceService errorsDisk and Network

Monitor all things!

@lyndsp

Metrics, monitoring and alerting - getting it right

DetectAlert Respon

DisplayAnalyse

Time & Pain increasing!

@lyndsp

Metrics, monitoring and alerting - getting it wrong

DetectRespon

DisplayAnalyse

Time & Pain increasing!Alert

@lyndsp

Metrics, monitoring and alerting - where and how?

DetectAlert Respon

Display Analyse

@lyndsp

Metrics, monitoring and alerting - stories from the trenches

DetectAlert Respon

Display Analyse

@lyndsp

Best practice summary

1. Healthy CI (mind and body)

2. Testing as an activty, not a phase

3. Tear drop shape automated tests (focus on behaviour, with loose

coupling)

4. Low (manual) cost deployment and rollback

5. Metrics, monitoring and reporting

Product Owner

Define

Web Ops

Deploy

Unit Tests

Detect

Alert Respond

Display Analyse

@lyndsp

Shared Pain

Continuous

DeliveryDone Well

@lyndsp

Shared pain #1 - Zero down-time releases

https://en.wikipedia.org/wiki/VO2_max

@lyndsp

Shared pain #2 - Manual deployment steps

https://en.wikipedia.org/wiki/Tailor https://en.wikipedia.org/wiki/Zara_(retailer)

@lyndsp

Shared pain summary

1. Zero down time releases

2. Manual deployment steps

@lyndsp

Team specific accelerators

Continuous

DeliveryDone Well

@lyndsp

Accelerators - 5 why’s

● Create a timeline.● Gather those involved.● Walk through the timeline.● Choose an entry point event:

○ Event○ Why?○ Process improvement

● Repeat x5 (ish)

@lyndsp

Accelerators - 5 why’s - example

Event Why? Improvement

Error threshold alert triggered.

Calls to service X were timing out.

Implement failover / circuit breaker.

Calls to service X were timing out.

Queries run by Service X were taking too long.

Alert on long running queries.

Queries run by Service X were taking too long.

Index was missing for a new field.

... ... ...

@lyndsp

Accelerators - automated release notes - the problem #1

Much confusion and

@lyndsp

Accelerators - automated release notes - the problem #2

Agile Project Management

Source control system

Features

Stories

Defects

LoginReporting

PaymentsAPI’s

@lyndsp

Accelerators - automated release notes - the solution

Features

Stories

Defects

1. Identify source changes2. Identify stories/defects3. Identify areas4. Group by story/defect5. For each, summarise

areas and authors6. Group by:

a. What’s doneb. What’s in progressc. Untracked work

Reporting

PaymentsAPI’s

@lyndsp

Team specific accelerators

Continuous

DeliveryDone Well

@lyndsp

Accelerators - teams “own” their own microservices

▪ “You build it, you run it” - https://goo.gl/DyfeOf

▪ Prefer service over library dependencies

@lyndsp

Accelerators - backward compatible API’s (principle of least surprise)

https://en.wikipedia.org/wiki/VO2_max

ProductionStaging

Everyone shares(and cares)

@lyndsp

Team specific accelerators summary

Monolith:1. 5 -why’s2. Automated release noteMicroservices:3. Microservice dependencies, not library

dependencies4. Backward compatible API’s

Features

Stories

Defects

LoginReporting

PaymentsAPI’s

@lyndsp

References and resources

▪ Build Quality In (promo code - LP2015 -

https://goo.gl/ZRuQKw)

▪ Continuous Delivery

▪ Implementing Lean Software

Development

▪ Release It

@lyndspsimple software solutions to big business problems.

Making Software. Better.

Thank You

Twitter@EqualExperts@lyndsp

LinkedInlinkedin.com/company/equal-expertsuk.linkedin.com/in/lyndsp

UNITED KINGDOM

+44 203 603 7830helloUK@equalexperts.comEqual Experts UK Ltd30 Brock StreetLondon NW1 3FG

+91 20 6607 7763helloIndia@equalexperts.comEqual Experts India Private LtdOffice No. 4-CCerebrum IT Park No. B3Kumar City, Kalyani NagarPune, 411006

Webwww.equalexperts.comwww.lyndsayp.com

CANADA

+1 403 775 4861helloCanada@equalexperts.comEqual Experts Devices Inc205 - 279 Midpark way S.E.T2X 1M2Calgary, Alberta

PORTUGAL

+ helloPortugal@equalexperts.comEqual Experts PortugalRua Tomás da Fonseca - Torres de LisboaTorre G, 5º Andar1600-209 Lisboa

Thank You

smoothing the continuous delivery path – a tale of two teams - lyndsay prewer

Software

presentation to northumbria university of lyndsay williams'...

exponential smoothing methods.ppt -...

exponential smoothing

gaussian smoothing

bayesian filtering and smoothing - cambridge university...

mgt 267 project · simple exponential smoothing, holt’s...

optimal interest-rate smoothing - columbia...

ngrams smoothing

tmrs board meeting asset smoothing · agenda what is asset...

pc soundcard and digital audio history by lyndsay williams...

how to create a buzz - buzz education€¦ · how to create...

dividend smoothing

document resume author dugas, tim; green, lyndsay; … ·...

rita haake mary turner lyndsay hughes illinois ptacs

etc5410: nonparametric smoothing methods · etc5410:...

lyndsay vanhoy rhodes, phd - amazon web services...lyndsay...

lecture 8: nelson aalen estimator and smoothing kernel...

lyndsay fletcher, university of glasgow

gaussian kernel smoothing - semantic scholar · gaussian...

smoothing methods