devops workshop (section 3) · infrastructure as code desired state configuration convergence...

Devops Workshop (Section 3)

John Willis @botchagalupe

Section 3 - The First Way (Accelerate Flow)

Continuous Delivery Patterns and Practices

The First Way - Accelerate Flow

3

“There is nothing so useless as doing efficiently that which should not be done at all”

- Peter Drucker

▪ The First Way (a.k.a Continuous Delivery)

▪ Continuous delivery patterns and practices ▪ The deployment pipeline ▪ Creating consistency in the pipeline ▪ Automated Testing ▪ Deployment Strategies (Zero Downtime Release)

3

CD Patterns and Practices


▪ Continuous delivery patterns and practices ▪ The deployment pipeline ▪ Creating consistency in the pipeline ▪ Automated Testing ▪ Deployment Strategies

3


▪ First Way Principles

▪ Create automated and repeatable environments at each stage of the pipeline.

▪ Apply automated testing at every stage of the pipeline. ▪ Increase flow and shorten lead times. ▪ Global optimization over local optimization.

3


▪ Five Steps to Increase Value and Flow

1. Define value precisely from the perspective of the end customer in terms of a specific product with specific capabilities offered at a specific price and time.

2. Identify the entire value stream for each product or product family and eliminate waste.

3. Make the remaining value-creating steps flow. 4. Design and provide what the customer wants only when the

customer wants it. 5. Pursue perfection.

3

Source: Womack


▪ Continuous Integration ▪ The process of integrating components of a

feature, application or service ▪ Continuous Delivery

▪ Uses continuous integration to create installable artifacts (packages) that can be deployed

▪ Continuous Deployment ▪ The process of deploying a feature, application or

service to production

3


▪ Continuous Integration

• The build phase • Typically triggered by a code commit • Typically builds are done from trunk • The process happens every time someone commits code • Code is compiled and libraries are built • Builds trigger package, artifact and image creation

3

Source: Humble, Farley - Continuous Delivery


▪ Eight Principles of Continuous Delivery

1. Create a repeatable, reliable process for releasing software 2. Automate almost everything 3. Keep everything in version control 4. If it hurts, do it more frequently, and bring the pain forward 5. Build quality in 6. Done means released 7. Everybody is responsible for the delivery process 8. Continuous improvement

3

Source: Humble, Farley - Continuous Delivery


▪ Continuous Delivery Anti-Patterns

▪ Incongruent testing and production environments ▪ Testing takes too long ▪ Manual regression and acceptance tests ▪ Long lead times ▪ High technical debt ▪ Services are slow and hard to change

3


▪ Practices of Continuous Delivery

▪ Build binaries once ▪ Same process for deployment at every stage ▪ Smoke test your deployments ▪ If anything fails stop the line ▪ Peer reviews (pull requests) ▪ Code is committed to trunk

3


▪ Continuous Deployment

▪ Deploying changes to production ▪ Release versus deploy ▪ System, performance and load testing ▪ Faster deploys helps learn faster ▪ Installation, configuration and orchestration

3




3

The Deployment Pipeline

▪ The Deployment Pipeline

▪ Visibility ▪ All stages of the pipeline are visible to everyone responsible

for the delivery of the service. ▪ Feedback

▪ Each stage in the pipeline has designed “gates” created to eliminate downstream defects.

▪ Continually deploy ▪ The design of the pipeline is such that any patch, update or

new feature can be automated for delivery, deploy and release.

3


▪ About Terminology

▪ Binaries ▪ Artifacts ▪ Components ▪ Packages ▪ Artifact Repository ▪ Package Repository

3


3Source: DOES16 US - Edwards, Willis


3Source: Damon Edwards - DTO Solutions


▪ The Deployment Pipeline (Service Delivery Platform Design Patterns)

▪ Build Phase ▪ Source Repository ▪ Build Console

▪ Package Repository ▪ Built artifacts ▪ Store and retrieve

▪ Deployment Phase ▪ Pre-production (stage) ▪ Production

3

Resource: Edwards and Limoncelli


▪ What is a Service Delivery Platform? ▪ https://vimeo.com/46125904

3


https://vimeo.com/46125904

▪ Build Phase Example

▪ Update your code from source control (if existing) ▪ Service is coded (if new) ▪ Run a local build in your dev environment ▪ Service code is committed to the source repository ▪ Service is built ▪ Artifacts are tagged and packaged ▪ Packages are registered in a repository

3


▪ Commit

▪ Files are uploaded to the source repository ▪ All source should be working code ▪ Code should have full coverage unit tests ▪ Committed source should not break the build ▪ Pre-submit checks (bugs, lint, code styles) ▪ Automated trigger of the build stage

3


▪ Build

▪ Compiling code ▪ Invokes build tools (ANT, Maven, Mercury, IVY ,Make) ▪ Build time dependancies ▪ Creating and/or converting images ▪ Building container images ▪ Running functional/unit tests ▪ Automated trigger of integration and acceptance testing

3


▪ Packages/Artifacts

▪ Compiled executables ▪ Components ▪ Libraries ▪ Container Images ▪ TAR’d and/or compressed binaries

3


▪ Package/Artifact Repository

▪ Manages the distribution of packages and artifacts ▪ All packages/artifacts must be reproducible from source ▪ Cryptographically hashed or digitally signed ▪ Manages security access management ▪ Invokes vulnerability scanning ▪ Provides usage reporting

3


▪ Packages and Artifacts Repository Examples

▪ JAR, WAR, and EAR packages (Java) ▪ Gems (Ruby) ▪ Python packages ▪ Perl modules ▪ DLLs (Windows) ▪ ZIP or TAR files ▪ Container images ▪ RPM or DEB packages (Linux) ▪ Metadata or Reports ▪ Documentation

3


▪ Deploy Phase

▪ Promotion ▪ Provisioning ▪ Installation ▪ Configuration ▪ Orchestration

3


▪ Promotion

▪ Candidate releases are selected ▪ Versions are selected and marked ▪ Tagging strategies ▪ Multiple repositories strategies

3


▪ Promotion (Tagging strategies)

▪ Marked “Production” ▪ Marked “Latest” ▪ Pinning

3


▪ Promotion (Multiple Repository Strategies)

▪ Development ▪ Testing ▪ Production

3


▪ Provisioning

▪ Bare metal provisioning (PXE) ▪ Virtual image provisioning ▪ Cloud provisioning ▪ Container provisioning

3


▪ Installation

▪ Internally written installers ▪ Before and after scripts ▪ System level packages (RPM/YUM DEB/APT)

3


▪ Configuration Management (Non Immutable)

▪ Installs the service ▪ Infrastructure as Code ▪ Desired State Configuration ▪ Convergence ▪ CFEngine, Chef, Puppet, Ansible

3


3

“A butterfly flaps its wings in the Amazonian jungle, and subsequently a storm ravages half of Europe.”

- Neil Gaiman and Terry Pratchett

Creating Consistency in the Pipeline



3


▪ Consistent Pipeline Challenges

▪ Snowflakes ▪ Sprawl ▪ Drift ▪ Documentation

3


▪ Pets vs Cattle

▪ Servers as pets ▪ You name them and when they get sick, you

nurse them back to health

▪ Servers as cattle ▪ You number them and when they get sick, you

shoot them

3


Source: Randy Bias

▪ Consistent Environments in all Stages

▪ Goal is to create consistent environments ▪ All elements of the pipeline should be disposable and reproducible ▪ All environments should look like production ▪ Decrease variability between elements in the pipeline ▪ Repeatability increases speed rebuilding environments ▪ Reduced errors related to inconsistencies ▪ Increases security related to inconsistencies

3


▪ Version Control Everything

▪ Keeps a history of all changes ▪ Can easy check differences between versions ▪ Can restore and rebuild all elements ▪ Everything can be versioned and tagged ▪ All changes are visible and audited for everyone ▪ Changes can be automated

3


▪ What should be in version control (everything)

▪ All application code ▪ All configuration scripts ▪ All configuration management DSL code ▪ All image build scripts (VM’s, Containers) ▪ All meta definitions (JSON,YAML,TOML) ▪ All automated tests, test scripts and test DSL code) ▪ All documentation, procedures, release notes ▪ All templates (Cloudformation Teraform, Heat) ▪ All database schema abstractions, DNS, and Firewall definitions ▪ Network definitions (Switch configurations) ▪ Basically everything

3


▪ Google’s SysOps Death Squads

▪ Internal IT policy of supporting only 2 Linux versions at any given time.

▪ When a new version was introduced the oldest had to be rolled off.

▪ The “Death Squad” team would identify deprecated server versions and put them in a shared spreadsheet.

▪ The “Death Squad” team would help owners depreciate and/or pressure until marked done.

3


Source: Tom Limoncelli - The practice of System and Network Administration V1

3

“The least-cost way to ensure that the behavior of any two hosts will remain completely identical is always to implement the same changes in the same

order on both hosts.”


3

• Divergence

• Convergence

• Congruence

Source: Steve Traugott


▪ Why Order Matters

▪ Circular Dependancies ▪ Right Command Wrong Order ▪ Right Package Wrong Order

3


▪ The Difference a Byte Makes

▪ Lean Enterprise story ▪ A dependent library with only a couple of byte

difference created a bug that could not be recreated in test.

▪ Large Financial institution ▪ Applied 5 second desired state configuration

monitoring and saw 1B unplanned changes per day.

3


▪ Provisioning - Containers

▪ Type 1 Hypervisors ▪ Type 2 Hypervisors ▪ OS Level Virtualization

3


3


Source: IBM - http://www.slideshare.net/BodenRussell/realizing-linux-containerslxc

http://www.slideshare.net/BodenRussell/realizing-linux-containerslxc

▪ Infrastructure as Code

▪ Infrastructure primitives defined as DSL ▪ Highly parameterized ▪ Desired stated based

3


▪ Infrastructure as Code (Principles)

▪ Modularity ▪ Composability ▪ Extensibility . ▪ Flexibility ▪ Repeatability ▪ Declaration ▪ Abstraction ▪ Idempotence ▪ Convergence

3


▪ Infrastructure as Code (Solutions)

▪ CFEngine ▪ Puppet ▪ Chef ▪ Ansible ▪ Terraform ▪ Cloud Formation

3


▪ Immutable Infrastructure

3


▪ Configuration Management (Immutable)

▪ Deploy the immutable image from an artifact repository.

▪ Uses a meta layering tool to converge a service or cluster.

3


▪ Configuration Management - Netflix Example

▪ Used Java, Scala and Groovy ▪ Created packages from artifacts ▪ Turned packages into immutable AMI’s ▪ Used a meta layer for convergence

3


▪ Immutable Delivery

3


▪ Configuration Management - Container Model

▪ Immutable Delivery model ▪ Use orchestration for provisioning ▪ Add meta layer similar to the Netflix model

(however, much faster)

3


ContainerEcosystem

Decoupling the Container Ecosystem Terminology

• Container Runtimes

• Container Engines

• Container Orchestration

Container RuntimeRunc

Runlxc

Rkt

Runv

Railcar

Kata Containers

Container Engines• Container Runtime Interface (CRI)

• ContainerD

• Docker

• GKE

• CRI-O

• Openshift

Docker (engine)

• Moby

• Docker Engine Enterprise

• Docker Engine Community

As a Service Cloud Engine’s

• ECS (Amazon)

• ACS (Azure)

• GKE (Google *)

NOTE: ACS and GKE are now offered on-prem

Container Orchestration

๏ Kubernetes

๏ Swarm

๏ Mesos

Container Orchestration

๏ Kubernetes

๏ Swarm

๏ Mesos

๏ Nomad (Hashicorp)

CNCF Certified Kubernetes

• Alibaba Cloud, Alibaba Cloud Container Service• Apprenda, Kismatic Enterprise Toolkit (KET)• Appscode, Pharmer• Caicloud, Compass• Canonical, Canonical Distribution of Kubernetes• Cisco Systems, Cisco Container Platform• Cloud Foundry Foundation, Cloud Foundry Container Runtime• CoreOS, Tectonic• DaoCloud, DaoCloud Enterprise• Docker, Docker Enterprise Edition• Google, Google Kubernetes Engine• Heptio, AWS-Quickstart• Huawei, Huawei Cloud Container Engine• IBM, IBM Cloud Container Service and IBM Cloud Private• Loodse, Kubermatic Container Engine• Mesosphere, Kubernetes on DC/OS

• Microsoft, Azure ACS-Engine• Mirantis, Mirantis Cloud Platform• Netease, Netease Container Service Dedicated• Oracle, Oracle Container Engine, Oracle Linux• Pivotal Software, Pivotal Container Service (PKS)• Poseidon, Typhoon• Rancher, Inc., Rancher Kubernetes• Red Hat, OpenShift• SAP, Cloud Platform – Gardener• SUSE, SUSE CaaS (Container as a Service) Platform• Samsung SDS, Kraken• StackPointCloud, Stackpoint.io• Tencent Cloud, Tencent Cloud Container Service• VMware, Pivotal Container Service (PKS)• Weaveworks, kubeadm• Wise2C Technology, WiseCloud

Cloud (orchestration)

• EKS (Amazon)

• AKS (Azure)

• GKE (Google)

ServiceMesh

Service Mesh Capabilities

• Observability

• Traffic Control

• Service Discovery

• Resilience

• Deployment Strategies

• Security

ISTIO Architecture• Control Plane

• Manages and configures proxies to route traffic, enforce policies, and collect telemetry.

• Made up of three services: Pilot, Mixer, and Auth.

• Data Plane• Intercepts all inbound (ingress) and outbound

(egress) network traffic.

• Service mesh sidecar model (proxy) is an implementation of the data plane that runs as a container in a Kubernetes Pod.

https://github.com/meta-magic/kubernetes_workshop

https://github.com/meta-magic/kubernetes_workshop

▪ Immutable Delivery (Pro’s)

▪ Least variation pattern ▪ Fastest provision model ▪ Fits well with Microservices architectures ▪ Really no need for Infrastructure as code ▪ Binary consistency from desktop to production ▪ Built into the CI process

3


▪ Immutable Delivery (Con’s)

▪ DSL abstraction not as mature as Infrastructure as code

▪ Debugging is harder ▪ Need a good model for image management ▪ Not all delivery models fit well

3


▪ Consistency Summary

▪ Infrastructure as Code is in general better than scripted environment builds.

▪ Hybrid environments where immutability doesn’t make sense still need Infrastructure as Code.

▪ In environments where immutability makes sense Immutable Infrastructure and/or Immutability delivery is best.

3


3

“A production line that never stopped was either extremely good or extremely bad”

- Taiichi Ohno

Automated Testing



3



▪ Build quality in as early as possible ▪ Find bugs as early as possible ▪ Everything gets tested ▪ If it hurts, bring the pain forward ▪ Create fast feedback ▪ Version control tests ▪ Continuous testing ensures continuous improvement

3

Automated Testing

▪ The Nine Principles

1.Change the adversarial mindset to a collaboration mindset. 2.Change the dev then test mindset to test and dev mindset. 3.Instead of having a test team you have an everyone tests team. 4.Test early, test often and shorten the feedback loop. 5.Tests should have reasonable expectations. 6.Fix bugs when you find them. 7.Reduce test documentation, automate the expectation. 8.Done means released (and tested) 9.Test implement vs Implement Test

3

Automated Testing


▪ Test Driven Development (TDD) ▪ Acceptance Test Driven Development (ATDD) ▪ Behavior Driven Development (BDD) ▪ Automated Tests (Build and Deploy) ▪ Testing Tools

3

Automated Testing

▪ Test Driven Development (TDD)

▪ Red, Green, Refactor

3

Automated Testing

RED GREEN REFACTOR

Write a test based on the requirements.

Make it fail

Write a small piece of code to make it

pass

Refactor the code by improving it without changing the behavior and

Repeat

▪ Test Driven Development (TDD) - Benefits

▪ Prevents scope creep ▪ Catches design issues early ▪ Creates cleaner code ▪ Builds trust with other service owners ▪ Creates a consistent rhythm

3

Automated Testing

▪ Acceptance Test Driven Development (ATDD)

▪ Discuss ▪ With the business stake holders

▪ Distill ▪ Collaborate with the business stake holders - define Done

▪ Develop ▪ TDD

▪ Demonstrate ▪ Show the new feature to the business stakeholders, get feedback

3

Automated Testing

▪ Behavior Driven Development (BDD)

▪ Tests whether the software fulfills the business need

▪ Instead of how it works it tests how it behaves ▪ Based on Domain Driven Design ▪ Typically DSL based ▪ Tests are typically conversational

3

Automated Testing

▪ Behavior Driven Development (BDD) - Example

3

Automated Testing

Feature: Hello World

As a service manager I want our customers to be greeted when they visit our site So that they have a better experience

Scenario: Customers sees the welcome message When I go to the homepage Then I should see the welcome message

3Source: Wikipedia - Continuous Delivery

Automated Testing

3

Automated Testing

Source: Damon Edwards - DTO Solutions

▪ Build Phase

▪ Unit tests ▪ Acceptance Tests ▪ Integration Tests ▪ User Acceptance Tests

3

Automated Testing

▪ Always Be Automating … the tests

▪ Automated tests are always running ▪ Every commit ▪ Nightly functional testing ▪ Smoke testing ▪ Stability testing ▪ Performance testing ▪ Zero configuration

3

Automated Testing

▪ Unit Tests

▪ Testing the source code by class or function. ▪ Typically done at commit or before commit. ▪ Typically “mocks” are created for interface calls. ▪ Typically created by the developer. ▪ Develop intention checking. ▪ Should be run on every change. ▪ Unit tests should be fast. ▪ Clear results ▪ TDD based

3

Automated Testing

▪ Acceptance Tests

▪ Testing multiple classes or functions. ▪ Should verify a user story. ▪ Should be written before code. ▪ Checks for regressions. ▪ Tests what the customer expects. ▪ Specification is written in a domain specific language. ▪ Run after the build and all unit tests have passed.

3

Automated Testing

▪ Integration Tests

▪ Testing integration with other applications and services. ▪ All code and modules are tested as a group. ▪ Tests are a system wide view of the service. ▪ Tests real interfaces not “mocks”. ▪ Integration tests are typically more expensive. ▪ Run after the build, all unit tests and acceptance have

passed.

3

Automated Testing

▪ User Acceptance Tests (Manual)

▪ Testing done by customers. ▪ Ensure the applications or service meets the requirements. ▪ Catch defects from a customer’s perspective.

3

Automated Testing

3

Automated Testing

Source: Damon Edwards - DTO Solutions

▪ Deploy Phase

▪ System Testing ▪ Performance Testing ▪ Load Testing ▪ Security Testing ▪ Configuration Management Testing

3

Automated Testing

▪ System Testing

▪ Precondition for release being turned on in production. ▪ Testing of external interfaces (make a purchase, checking

stock quote) ▪ Synthetic testing, emulation of user interactions. ▪ Tests can be emulated from API’s, consoles, web,

devices, mobile and even from laser printers.

3

Automated Testing

▪ Performance Testing

▪ Tests the speed and latency of a service. ▪ Can test against customer SLA’s ▪ Performance regression tests (previous results)

3

Automated Testing

▪ Load Testing

▪ Tests how much load a service can handle. ▪ Tests involve increasing traffic to find max load. ▪ Performance regression load tests (previous results)

3

Automated Testing

▪ Security Testing

▪ SQL Injection ▪ Cross Site Scripting ▪ Unprotected Redirects ▪ Unsafe File Access ▪ Version-Specific Issues (CVEs) ▪ Symbol Dos ▪ Remote Code Execution

3

Automated Testing

3

Automated Testing

Source: - https://www.ruggedsoftware.org/

▪ Security Testing

▪ Create security related stories.

▪ Add security unit tests. ▪ Create a library of reusable

test cases patterns. ▪ Add vulnerability scans into

the pipeline.

Security and the Goldilocks Zone

• The fallacious nature of cyber security relates to the standard legacy security model specifically on the idea of perimeter security.

• This concept involves the implementation of a state-full firewall at a routed point within the network that very rarely gets looked at unless an operational change is required.

• The problem with having only premier security is that applications have changed significantly in the last ten years and the infrastructure they run upon is playing by the same old rules.

10

You Build It, You Secure It

DevSecOps as Supply Chain?

Source: Wikipedia - Continuous Delivery@botchagalupe

Software Supply Chain

Delivery Team

Version Control Build Test Release

DevOps Example

Stage

Prod

@botchagalupe

Security in the Software Supply Chain

107

Delivery Team


DevOps Example

Delivery Team


DevOps and Security

Stage

Prod

@botchagalupe

Security in the Software Supply Chain

108

Delivery Team


DevOps Example

Delivery Team


DevSecOps Example

Stage

Prod

@botchagalupe

109

Delivery Team


DevSecOps Supply Chain

Stage

Prod

The New Goldilocks Zone (DevSecOps)

Security Training Security Requirements Threat Modeling Architecture Review OWASP Top 10 IDE Plugins Code Examples

Fail the Build SAST/DAST/IAST Configuration Analysis Application Module Scanning Threat Modeling as Unit Test

Automated Pen Testing Static Code Analysis Security Policy Testing Configuration Analysis Security MonitoringConfiguration Monitoring

Best Practices for DevSecOps• Train development teams to develop secure code • Track security issues the same as software issues • Security as code, Security Built In. • Integrate security controls in the software

pipeline • Automate security test in the build process • Detect known vulnerabilities during the pipeline • Monitor security in production for known states • Inject failure to ensure security is hardened

Gene Kim, Jez Humble, Patrick Dubois, and John Willis. The DevOps Handbook; It Revolution Press, LLC.;2016.@botchagalupe

DevSecOps Operational Tips

• Work with and educate your auditor • Ruthlessly eliminate false positives to Developers • Explain the vulnerabilities in business impact terms • DevOps the vulnerability (JIRA, backlog, Kanban) • Open the code base to everyone in the organization • Educate on how to fix the Vulnerability

Audit the Software Supply Chain

DevSecOps Auditor Manifesto

• Inclusion over Exclusion • Transparency over Need-to-Know • Ownership of business risk over Delegation • Open Collaboration regarding Risk over Silo’s

▪ Tools - Unit Testing

▪ JUnit ▪ JTest ▪ CUnit ▪ RSpec ▪ Google Test ▪ Parasoft ▪ Visual Studio

3

Automated Testing

▪ Tools - Load and Performance Testing

▪ Apache Bench ▪ Apache JMeter ▪ HP LoadRunner ▪ Rational Performance Tester ▪ Selenium ▪ Neoload ▪ SOASTA

3

Automated Testing

▪ SOASTA - Myspace Load Test

▪ 1 million concurrent virtual users ▪ Test cases split between searching for and watching

music videos, rating videos, adding videos to favorites and viewing artists’ channel pages

▪ Transfer rate of 16 gigabits per second ▪ 6 terabytes of data transferred per hour ▪ Over 77,000 hits per second, in addition to live,

production traffic3

Automated Testing

▪ Tools - BDD Testing

▪ Cucumber ▪ JBehave ▪ SpecFlow

3

Automated Testing

▪ Tools - Security Testing

▪ Brakeman ▪ Gauntlet ▪ Nmap ▪ OWASP

3

Automated Testing

▪ Tools - Configuration Management Testing

▪ Chef ▪ ChefSpec (Unit) ▪ Test Kitchen (Integration) ▪ Cucumber-chef (BDD) ▪ Food Critic (Lint)

3

Automated Testing

▪ Tools - Configuration Management Testing

▪ Puppet ▪ rspec-puppet (Unit) ▪ Test Kitchen (Integration) ▪ Beaker (Integration) ▪ Puppet-lint (Lint)

3

Automated Testing

Devops Automated Deployment Pipeline

@botchagalupe

3

“Pushing code into production can be tricky because we are modifying a system while it is running. This is

like changing the tires of a car while it is speeding down the highway at 90 km/h: you can do it, but it

requires a lot of care and planning. ”

- Thomas A. Limoncelli

Deployment Strategies



3


▪ General Patterns and Constraints

▪ Changing data in the release can be more difficult. ▪ Orchestrated release rollbacks can be difficult. ▪ Best time to reply is during production (not at 3am) ▪ Log deployment activities ▪ Don’t delete old files ▪ Use warmup methodologies

3


▪ Upgrading Live Services

▪ Rolling Upgrades ▪ Canary ▪ Phased Roll-Outs ▪ Proportional Shedding ▪ Blue-Green Deployment ▪ Toggling Features

3


THE PRACTICE OF CLOUD SYSTEM ADMINISTRATION - Chapter 11 Upgrading Live Services.

▪ Rolling Upgrades

▪ Servers are removed from service and upgraded one at a time.

▪ The process is complete when when all the servers are updated.

▪ During the upgrade there is a temporary reduction in capacity.

▪ Server is first drained by removing from the load balancer.

3


▪ Canary

▪ Similar to rolling upgrade but to a subset of servers. ▪ Apply smoke, performance and load test. ▪ If no problems arise add more subsets. ▪ Server is first drained by removing from the load

balancer. ▪ Can also be done by percentage of servers. ▪ Canary”ing can be combined with other rollout strategies ▪ Designed to find a defective release.

3


▪ Phased Roll-Outs

▪ Roll-outs partitioned by users or groups. ▪ Groups are categorized by risk tolerance. ▪ If no problems arise add more subsets. ▪ Sometimes internal employees can be first roll-out

group. ▪ Power user or meta communities roll-outs

3


▪ Proportional Shedding

▪ A new service is built on new machines. ▪ The load balancers shed small percentages of traffic to

the new service. ▪ The percentage is increased if no issue are found. ▪ The old service is turn off when all traffic is moved to

the new service with no errors. ▪ With bare metal this can be expensive. Cloud makes

this more economical. 3


▪ Blue-Green Deploy

▪ Similar to proportional shedding accept you duplicate the application not the servers.

▪ Green is the live service. ▪ Blue is dormant and requires limited resources. ▪ When the release goes live the two are swapped. ▪ Rolling back is easier to do. ▪ Very popular technique.

3


▪ Toggling Features

▪ Also known as feature flags. ▪ Flags are set in the code that can turn on or off a feature. ▪ Allows developers to continuously deploy new code

decoupled from a release. ▪ Distrusted key value stores like Zookeeper, Etcd and

Consul can be used for feature toggling. ▪ Features can be gradually introduced or be point and

time released.3


▪ Case Studies - Cloud Deploy’s at Amazon

▪ What if you didn’t have any restrictions on the number of serves you can host.

▪ Make an entire copy of your running fleet (10’s of thousands of servers).

▪ Just flip the load balancers to the new fleet. ▪ Deploys that flip in seconds. ▪ Rollback in seconds.

3


▪ Case Studies - Cloud Deploy’s at Amazon (results)

▪ 75% reduction in outages triggered by software deploys.

▪ 90% reduction in outage minutes triggered by software deploys.

▪ ~0.001% of software deployments cause an outage. ▪ Instantaneous automated rollbacks. ▪ Reception in complexity.

3


▪ Case Studies - Dark Launch at Facebook

The secret for going from zero to seventy million users overnight is to avoid doing it all in one fell swoop. We chose to simulate the impact of many real users hitting many machines by means of a "dark launch" period in which Facebook pages would make connections to the chat servers, query for presence information and simulate message sends without a single UI element drawn on the page. With the "dark launch"

bugs fixed, we hope that you enjoy Facebook Chat now that the UI lights have been turned on.

3


devops workshop (section 3) · infrastructure as code desired state configuration convergence...

Documents