(arc402) deployment automation: from developers' keyboards to end users' screens | aws...

November 13, 2014 | Las Vegas NV

Chris Munns, AWS Solutions Architect

@chrismunns

• [email protected]

https://secure.flickr.com/photos/psd/4389135567/

Historically, there was no automation for developers:

Historically, development needed to be tightly controlled:

QA

Staging

Dev

Prod

QA

Staging

Dev

Prod

Unit

TestsIntegration

Tests

Usability

TestsPerformance

TestsAcceptance

Tests

System

Tests

Regression

Tests

Monitoring

A/B Tests

QA

Staging

Dev

Prod

QA

Staging

Dev

Prod

Unit

TestsIntegration

Tests

Usability

TestsPerformance

TestsAcceptance

Tests

System

Tests

Regression

Tests

Monitoring

A/B Tests

Where do we begin?

https://secure.flickr.com/photos/stevendepolo/5749192025/

If you are part of an Ops / “DevOps” /

“DevTools” / “CoreInfra” / whatever team that

has developers as an internal customer, it’s

your job to help them:

DEV QAPROD

Barker

Your environments should be as

similar to each other as possible!

Make the process so easy

a Caveman could do it*

*provided they have the appropriate access to!

Complexity of the process isn’t

necessarily bad, so long as not

everyone in the organization HAS

to know how the sausage is made

https://secure.flickr.com/photos/erix/2657100921

https://secure.flickr.com/photos/jasoneppink/499531891

Make the results of change

visible to everyone who

causes or deals with change!

Aim to reduce the “works on my machine” failures inherent with

developing on one OS and running production on another:

Docker is really changing how applications are

being built and deployed!

NEW!

“A Better Dev/Test Experience: Docker and AWS” on Medium!

https://medium.com/aws-activate-startup-blog/a-better-dev-test-experience-

docker-and-aws-291da5ab1238

http://bit.ly/1saojKw

https://medium.com/aws-activate-startup-blog/a-better-dev-test-experience-docker-and-aws-291da5ab1238

http://bit.ly/1saojKw

Dramatically lowers the complexity in running developer environments.

Let’s set it up:

[munns@maclaptop ~]$ vagrant init chef/centos-6.5

[munns@maclaptop ~]$ vagrant up

[munns@maclaptop ~]$ vagrant ssh

Last login: Fri Mar 7 16:57:20 2014 from 10.0.2.2

[vagrant@zekaih ~]$ uname -a

Linux zekaih.munnsdev.com 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64

x86_64 x86_64 GNU/Linux

• Continuous integration

• Continuous deployment

• Continuous integration

Continuous deployment always

continuous integration.

Continuous integration doesn’t mean

that the code gets deployed to

production every commit!

https://secure.flickr.com/photos/isherwoodchris/6917253693

CI tools:

CI bits

Green is

good!

Deploying code

https://secure.flickr.com/photos/simononly/15386966677

Convenience Control

Higher-level services Do it yourself

AWS

Elastic Beanstalk

AWS

OpsWorks

AWS

CloudFormation

AWS

CodeDeploy

AWS application/infrastructure management tools

AWS application/infrastructure management tools

Convenience Control

Higher-level services Do it yourself

AWS

Elastic Beanstalk

AWS

OpsWorks

AWS

CloudFormation

AWS

CodeDeploy

NEW!!

You’re not still configuring

your servers by hand,

right?

Options: Deciding factors:

How are you getting the bits from your code

repository to your destination environments?

Simplest of all methods. Use a deployment tool to either do a repo

sync, or copy the raw files from one environment to another:

Pros:– Easy to get started with

– No need for midprocess packaging steps

Cons:– Rollbacks could become a challenge

– Harder to do at large scale

AWS services can make it really easy to deploy from a repository:

– AWS CodeDeploy

– AWS Elastic Beanstalk

– AWS OpsWorks

Example with OpsWorks:

[root@saarbrucken infrahelper]# ll /srv/www/infrahelper/

total 8

lrwxrwxrwx 1 deploy apache 44 Oct 21 20:43 current ->

/srv/www/infrahelper/releases/20141021204316

drwxr-xr-x 7 deploy apache 4096 Oct 21 20:43 releases

drwxrwx--- 9 deploy apache 4096 Oct 21 20:43 shared

<----------------------DEPLOY HAPPENS--------------------->

[root@saarbrucken infrahelper]# ll /srv/www/infrahelper/

total 8

lrwxrwxrwx 1 deploy apache 44 Nov 7 21:44 current ->

/srv/www/infrahelper/releases/20141107214310

Bundle up your code, deploy the bundle:

Pros:– Very atomic solution to deploying code

– Easy to track versions and changes across environments

Cons:– Potentially very large deployable assets

Package will be largely unique to the OS/language you are using:

Example using FPM (Effing Package Management)

[munns@somehost ~]$ gem install fpm

…..

[munns@somehost ~]$ git clone https://github.com/teknogeek0/ReInvent2014-InfraHelper.git

…...

[munns@somehost ~]$ fpm -s dir -t rpm -n "InfraHelper" -v 1.0 --epoch 1 ReInvent2014-InfraHelper/=/opt/InfraHelper

Created package {:path=>"InfraHelper-1.0-1.x86_64.rpm"}

[munns@somehost ~]$ rpm -ivh InfraHelper-1.0-1.x86_64.rpm

…..

[munns@somehost ~]$ rpm -qa InfraHelper

InfraHelper-1.0-1.x86_64

[munns@somehost ~]$ ls -l /opt/InfraHelper/

total 32

drwxr-xr-x 2 root root 4096 Nov 4 23:07 flow

-rw-r--r-- 1 root root 156 Nov 4 23:06 Gemfile

-rwxr-xr-x 1 root root 661 Nov 4 23:06 IHQueueWatcher_control.rb

-rw-r--r-- 1 root root 2765 Nov 4 23:06 infrahelper_utils.rb

…..

Generate a new AMI with your updated code on it. Launch it:

Pros:– The most atomic way possible

• Won’t affect any currently running instances

– Can pretty easily run two versions side by side

Cons:– Bit more work involved

– Really have to think about data persistence

– Have to think about how rollbacks would happen

Bunch of tools to help you build AMIs quick and easy:

# ./packer validate webimage.json

Template validated successfully.

# ./packer build webimage.json

amazon-ebs output will be in this color.

==> amazon-ebs: Inspecting the source AMI...

==> amazon-ebs: Creating temporary keypair:

packer 5459736e-26a7-5983-db5a-df145dafa7e7

….

Build 'amazon-ebs' finished.

==> Builds finished. The artifacts of successful

builds are:

--> amazon-ebs: AMIs were created:

us-west-2: ami-5f9bd36f

# cat webimage.json

{

"variables": {

"aws_access_key": "",

"aws_secret_key": ""

},

"builders": [{

"type": "amazon-ebs",

"access_key": "{{user `aws_access_key`}}",

"secret_key": "{{user `aws_secret_key`}}",

"region": "us-west-2",

"source_ami": "ami-b5a7ea85",

"instance_type": ”m3.medium",

"ssh_username": "ec2-user",

"ami_name": "webserver {{timestamp}}",

}]

}

HOST

METRICS

SERVICE

METRICS

LOG

ANALYSISBUILD

METRICS

AWS Marketplace and Partners

• You can find, research, and buy

software

• Simple pricing, aligns with Amazon

EC2 usage model

• Launch in minutes

• AWS Marketplace billing integrated

into your AWS account

• Can also find SaaS offerings from

partners!

• 1900+ products across 25

categories

Learn more at: aws.amazon.com/marketplace

You’ve picked a deployment

method; now how are you

going to go about acting

upon it?

https://secure.flickr.com/photos/wscullin/3770015991

How do we go about rolling out our code?

What gotchas are there?

Replace code on all of the

instances without changing

them or taking removing traffic:Elastic Load Balancing (ELB)

Web/App instances

Amazon

DynamoDB

MySQL

Amazon RDS

Instance

Amazon

ElastiCache

Cache Node

users

v1v2

Amazon

Route 53

• Go through existing tier updating

the application in batches:ELB

Web/App instances

DynamoDB MySQL RDS

Instance

ElastiCache

Cache

Node

usersAmazon

Route 53

v1



Web/App instances

DynamoDB MySQL RDS

Instance

ElastiCache

Cache

Node

usersAmazon

Route 53

v2 v1



Web/App instances

DynamoDB MySQL RDS

Instance

ElastiCache

Cache

Node

usersAmazon

Route 53

v2

”WebAutoScalingGroup" : {

"Type" : "AWS::AutoScaling::AutoScalingGroup",

"Properties" : {

"AvailabilityZones" : { "Fn::GetAZs" : { "Ref" : "AWS::Region" } },

"LaunchConfigurationName" : { "Ref" : ”WebAutoScalingLaunchConfig" },

"MaxSize" : ”20",

"MinSize" : ”10"

},

"UpdatePolicy" : {

"AutoScalingRollingUpdate" : {

"MinInstancesInService" : “6",

"MaxBatchSize" : “2",

"PauseTime" : "PT5M"

}

}

}

CloudFormation— Auto Scaling with rolling updates:

”WebAutoScalingGroup" : {

"Type" : "AWS::AutoScaling::AutoScalingGroup",

"Properties" : {

"AvailabilityZones" : { "Fn::GetAZs" : { "Ref" : "AWS::Region" } },

"LaunchConfigurationName" : { "Ref" : ”WebAutoScalingLaunchConfig" },

"MaxSize" : ”20",

"MinSize" : ”10"

},

"UpdatePolicy" : {

"AutoScalingRollingUpdate" : {

"MinInstancesInService" : “6",

"MaxBatchSize" : “2",

"PauseTime" : "PT5M"

}

}

}

Replace 2 at a time, pause

for 5 minutes before doing

the next batch

CloudFormation— Auto Scaling with rolling updates:

Elastic Beanstalk—batch deployments and rolling configuration updates:

CodeDeploy—rolling deployments:

We stand up a duplicate part of our

infrastructure and slowly cut traffic

over to it:

As we shift more traffic over, let

Auto Scaling grow/shrink our

instances of the new or old

application:

ELB

Web/App instances

DynamoDB MySQL RDS

Instance

ElastiCache

Cache

Node

users

v1

Amazon

Route 53



over to it:




application:

ELB

Web/App instances

DynamoDB MySQL RDS

Instance

ElastiCache

Cache

Node

users

Amazon

Route 53

v1

ELB

v2



over to it:




application:

ELB

Web/App instances

DynamoDB MySQL RDS

Instance

ElastiCache

Cache

Node

users

v1

ELB

v2

Amazon

Route 53



over to it:




application:

ELB

Web/App instances

DynamoDB MySQL RDS

Instance

ElastiCache

Cache

Node

users

v2

Amazon

Route 53

Some

words of

caution!

https://secure.flickr.com/photos/e-coli/3888542890

Schema changes tied to deployments are a blocker to moving fast:

Unlink this from code deploys:

Be prepared for things

to go wrong!

https://secure.flickr.com/photos/akyamada/4071735996

"Resources" : {

"myMainSite" : {

"Type" : "AWS::Route53::RecordSetGroup",

"Properties" : {

"HostedZoneName" : {"Ref" : "HostedZoneName"},

"Comment" : "Main site with fail whale.",

"RecordSets" : [

{

"Name" : {"Ref" : "DNSRecordName"},

"Type" : "A",

"SetIdentifier": "Primary",

"Failover": "PRIMARY",

"AliasTarget" : {

"HostedZoneId" : {"Ref" : "ELBHostedZoneID"},

"DNSName" : {"Ref" : "ELBDnsName"},

"EvaluateTargetHealth" : "True"

}

},

{

"Name": {"Ref" : "DNSRecordName"},

"Type": "A",

"SetIdentifier" : "Secondary",

"Failover": "SECONDARY",

"AliasTarget": {

"HostedZoneId": { "Fn::FindInMap" : [

"S3RegionWebEndpoints", { "Ref" : "S3BucketRegion" },

"HostedZoneId" ]},

"EvaluateTargetHealth": "False",

"DNSName": { "Fn::FindInMap" : [


"DNSEndpoint" ]}

}

}

]

}

}

}

}

"Resources" : {

"myMainSite" : {

"Type" : "AWS::Route53::RecordSetGroup",

"Properties" : {

"HostedZoneName" : {"Ref" : "HostedZoneName"},

"Comment" : "Main site with fail whale.",

"RecordSets" : [

{

"Name" : {"Ref" : "DNSRecordName"},

"Type" : "A",

"SetIdentifier": "Primary",

"Failover": "PRIMARY",

"AliasTarget" : {

"HostedZoneId" : {"Ref" : "ELBHostedZoneID"},

"DNSName" : {"Ref" : "ELBDnsName"},

"EvaluateTargetHealth" : "True"

}

},

{

"Name": {"Ref" : "DNSRecordName"},

"Type": "A",

"SetIdentifier" : "Secondary",

"Failover": "SECONDARY",

"AliasTarget": {

"HostedZoneId": { "Fn::FindInMap" : [


"HostedZoneId" ]},

"EvaluateTargetHealth": "False",

"DNSName": { "Fn::FindInMap" : [


"DNSEndpoint" ]}

}

}

]

}

}

}

}

Example template:

http://bit.ly/113cNaa

HOST

METRICS

SERVICE

METRICS

LOG

ANALYSIS

EXTERNAL

SITE

METRICS

When things break during a deploy you’ll need to decide how to

react:

How do you decide? Deployment pattern and method will be deciding

factors:

Have a real-time communication method for the entire company:

Share knowledge:

QA

Staging

Dev

Prod

We need robots

https://secure.flickr.com/photos/spenceyc/7481166880

Good news, we’ve got

lots of “robots”!

https://secure.flickr.com/photos/peyri/10207629

(NEW!)

(NEW!)

(SOON!)

(NEW!)

https://secure.flickr.com/photos/jeffedoe/506027963

Code

Repository

Code

Repository

CI

Infra

CI

SaaS

Code

Repository

CI

Infra

CI

SaaSCode Bundler

Code

Repository

CI

Infra

CI

SaaSCode Bundler

Deploy

Object

Amazon S3

Bucket

Code

Repository

CI

Infra

CI

SaaSCode Bundler

Deploy

Object

Amazon S3

Bucket

AWS

OpsWorks

Code

Repository

CI

Infra

CI

SaaSCode Bundler

Deploy

Object

Amazon S3

Bucket

AWS

OpsWorks

Dev

Web/App

Servers

Code

Repository

CI

Infra

CI

SaaSCode Bundler

Deploy

Object

Amazon S3

Bucket

AWS

OpsWorks

Dev

Web/App

Servers

Dev/

QA

Users

Deploy

Object

Amazon S3

Bucket

AWS

OpsWorks

Prod

Web/App

Servers

Users

Deployment

Interface

Automation good!

https://secure.flickr.com/photos/macwagen/94975613

?https://secure.flickr.com/photos/dullhunk/202872717/

http://bit.ly/awsevals

http://bit.ly/awsevals

(arc402) deployment automation: from developers' keyboards to end users' screens | aws...

Technology

continuous integration

vagrant upmunns

aws codedeploy aws elastic

aws solutions architect

deployment tool

scaleaws services

code repository

vagrant init chefcentos