Cloud native

Continuous Delivery

Christian Deger, christian@deger.eu, @cdeger

DevOpsCon Munich, 21.11.2017

Cloud native Continuous Delivery

• Cloud Native Computing Foundation

• Container packaged

• Dynamically managed

• Microservices oriented

• My opinion

• Serverless, FaaS

• Managed services

Cloud native Continuous Delivery

• Bring changes into production

• Fast

• Reliable

• Repeatable

• Traceable

• In order to

• Get fast feedback

• Lower risk

Microservices

Microservices

Speed

Microservices

SpeedScale the organization

Microservices

Speed

Fast local decisions

Scale the organization

Microservices

Speed

Fast local decisionsAutonomous teams

Scale the organization

Microservices

Speed

Fast local decisionsAutonomous teams

Scale the organization

Loosely coupled

Microservices

Speed

Fast local decisionsAutonomous teams

Strong boundaries

Scale the organization

Loosely coupled

Microservices

Speed

Fast local decisionsAutonomous teams

Strong boundaries

Scale the organization

Independent deployable

Loosely coupled

Microservices

Speed

Fast local decisionsAutonomous teams

Strong boundaries

Technology diversity

Scale the organization

Independent deployable

Loosely coupled

Development

“Change”

Dev

Development

“Change”

Operations

”Stability”

Ops

Development

“Change”

Operations

”Stability”

Dev and Ops silos

Development

“Change”

Operations

”Stability”

Dev and Ops silos

Cross-functional teams

Science and Continuous Delivery

Forsgren, Nicole and Humble, Jez, The Role of Continuous Delivery in IT and Organizational Performance (October 27, 2015). Forsgren, N., J. Humble

(2016). "The Role of Continuous Delivery in IT and Organizational Performance." In the Proceedings of the Western Decision Sciences Institute (WDSI)

2016, Las Vegas, NV. . Available at SSRN: http://ssrn.com/abstract=2681909 or http://dx.doi.org/10.2139/ssrn.2681909

Deployment Pipeline

Simplified deployment pipeline

Application code in

one repository per

service.

Simplified deployment pipeline

Application code in

one repository per

service.

CI

Deployment package

as artifact.

Simplified deployment pipeline

Application code in

one repository per

service.

CI

Deployment package

as artifact.

CD

Deliver package to

production

Code hosting is commodity

Application code in

one repository per

service.

CI

Deployment package

as artifact.

CD

Deliver package to

production

• GitHub, GitLab, Bitbucket, …

• Increased developer productivity

• Ecosystem: Apps and integrations

• Security

• Account Management, SSO, MFA

• git-secrets

• Backup using clone/fetch

Code hosting: Managed over self-hosted

One tool?

CI

CI/CD tool

with support for

deployment pipelines

CD

• Simpler

• Better overview

Two tools?

CI

CI tool

CD

CD tool

Artifact as

trigger/ handover

• Best tool for the job

• More complex

• Travis CI, CircleCI, GitLab CI, …

• Deploy agent needs access to production

• Use separate tools for CI and CD

• AWS Code*

• Definitely for OSS

• Not an option for AS24

Managed deployment pipelines?

• CD infrastructure should be the first task in a new project

• CD should not become a snowflake itself

• For disaster recovery you will need your CD infrastructure

• Aim for “CD as a service”

Automate CD infrastructure

• Containerized

• Isolated builds – bring your own agent

• Elastic agents

• Container as artifact

• Pipeline as code

• Declarative in service repository

• Fast and simple bootstrapping of new pipelines

• Avoid single, shared CI instance

New CI practices

• Everything that used to be good practices

• No CI theatre

• Embrace deployment pipelines

• No smarts in the CI tool

Old CI practices – Recap

Pets?

Cattle, not pets

Burgers, not cattle

Cloud native deployment pipeline

Application code and

infrastructure

specification in one

repository per service.

Cloud native deployment pipeline

Application code and

infrastructure

specification in one

repository per service.

CI

Deployment package

and infrastructure

declaration as

artifact.

Cloud native deployment pipeline

Application code and

infrastructure

specification in one

repository per service.

CI

Deployment package

and infrastructure

declaration as

artifact.

CD

1. Create or update

service infrastructure.

Cloud native deployment pipeline

Application code and

infrastructure

specification in one

repository per service.

CI

Deployment package

and infrastructure

declaration as

artifact.

CD

1. Create or update

service infrastructure.

2. New instances pull

down package and

start application.

No infrastructure monolith

• Follow microservices boundaries

• At least one stack per microservice

Decompose into Micro-Infrastructures

• Macro stack(s)

• Outputs parameters exported

• Keep it small, only things that don’t

change often

• No services

Macro-Infrastructure

• Network

• Security

• Bastion Host

• Services share macro stack

• Service stacks import parameters

• Service teams own service stack

• All services are in service stacks

Shared stack and service stacks

• Services have dependencies

• CD infrastructure

• Macro stack

• Base images (AMI, container)

• …

• But avoid explicit pipeline dependencies

• Try to reference pinned dependencies

Isolate deployment pipelines

Deployment

You build it,

you run it

How many environments?

V2V3

V6 V5

V4

V7

V5

V8

Enginee

r

CI Dev Staging

V1

V4

Prod

• Integrate in production

• Consumer contracts or CDCs

• Reduce impact of failures

• MTTR over MTBF

• Monitoring

• Canary releases

• Rollbacks

• Semantic monitoring

No staging environment

• Separate code deployment from feature release

• Trunk-based development

• No long lived feature branches

Feature toggles

Feature toggles – release and experiment

• Product is in charge of releasing a feature

• Canary releases

• A/B testing

Immutable deployment patterns

Function as a Service - FaaS

Done

Function as a Service - FaaS

Lifecycle of immutable servers/containers

Created

V3

Lifecycle of immutable servers/containers

Created

V3

Healthcheck

ok

V3

Lifecycle of immutable servers/containers

Created

V3

Healthcheck

ok

V3

Traffic from

load balancer

V3

Lifecycle of immutable servers/containers

V3

Created

V3

Healthcheck

ok

V3

Traffic from

load balancer

V3

Connections

drained

Lifecycle of immutable servers/containers

V3

Created

V3

Healthcheck

ok

V3

Traffic from

load balancer

V3

Terminated

V3

Connections

drained

Lifecycle of immutable servers/containers

V3

Created

V3

Healthcheck

ok

V3

Traffic from

load balancer

V3

Terminated

V3

Connections

drained

• No need for configuration management tools: Chef, Puppet, Ansible

• Patches/Security? Alert on base image age

• Simpler with stateless services

Rolling update

V3 V3 V3

Rolling update

V3 V3 V3 V4

Rolling update

V3 V3 V3 V4

Rolling update

V3 V3 V3 V4

Rolling update

V3 V3 V4V3

Rolling update

V3 V3 V4V3

Rolling update

V3 V3 V4

Rolling update

V3 V3 V4 V4

Rolling update

V3 V4 V4

Rolling update

V3 V4 V4 V4

Rolling update

V4 V4 V4

Rolling update

V4 V4 V4

• Only few additional resources required during deployment

• Takes some time

Blue/green

V3 V3 V3

Blue/green

V3 V3 V3 V4 V4 V4

Blue/green

V3 V3 V3 V4 V4 V4

Blue/green

V4 V4 V4V3 V3 V3

Blue/green

V4 V4 V4V3V3V3

Blue/green

V4 V4 V4V3V3V3

• Can keep drained instances for faster rollback

Blue/green

V4 V4 V4V3V3V3

Blue/green

V4 V4 V4

• Double the resources required during deployment

• Faster deployment

Canary analysis

V3 V3 V3

Canary analysis

V3 V3 V3 V4

• Make explicit, automated canary analysis

• Error rate

• Latency

• Load

• Alternative: Feature toggle based canaries

• Existing service in production

Dark launches

Service

Client

• New service to be launched

Dark launches

Old New

Client

• Fork real traffic to new service and

discard response

• Monitor new service under real load

• Compare responses

• Fork on server or client side

Dark launches

Old New

Client

Wrapping it up

• Build isolation

• Independent pipelines

• Elasticity

• Everything as code

• Pipelines owned by teams

Recommendations for deployment pipeline

• Time from commit to production – cycle time

• Time to bootstrap a new service including the deployment pipeline

Metrics

• “You build it, you deploy it, you run it”

• Embrace immutability

• Infrastructure follows microservices architecture

• Failures happen

• Reduce impact

• Fast detection

• Fast recovery

Important

Thank You!

• Regent's Rowing 8 by Jmf3333 [CC BY 3.0]

https://en.wikipedia.org/wiki/File:Regents_rowing.JPG

• Aquapark Aquacolors by Pantharei.2017 (Own work) [CC BY-SA 4.0]

https://commons.wikimedia.org/wiki/File:Aquapark_Aquacolors.jpg

• The Key of a Chamberlain by Niklitov [CC BY-SA 4.0]

https://commons.wikimedia.org/wiki/File:The_Key_of_a_Chamberlain_at_Kingdom_of_P

russia_Kalinigrad_Blindage_museum.JPG

• Beziers Fonseranes by Dedounet [CC BY-SA 1.0]

https://commons.wikimedia.org/wiki/File:Beziers_Fonseranes.jpg

Image attribution