aws summit tel aviv - startup track - continuous deployment

AWS Summit 2013 Tel Aviv Oct 16 – Tel Aviv, Israel

Tiago Henriques

Solutions Architect, Amazon Web Services

Continuous Deployment

CONTINUOUS

INTEGRATION

DEVELOPER

SOURCE CODE

REPOSITORY

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

PICK

TASKS

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

SUBMIT

CODE

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

SCHEDULE

BUILD

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

RECURRENT

BUILDS

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

CODE

FETCH

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

CODE QUALITY

TESTS

TEST

RESULTS

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

BUILD OUTPUT

SOURCE CODE

REPOSITORY

PROJECT MANAGEMENT

SERVER

CONTINUOUS

INTEGRATION SERVER

DOCS

BINARIES

& PACKAGES

SOURCE CODE

REPOSITORY

DNS

CONTINUOUS

INTEGRATION SERVER

PROJECT

MANAGEMENT SERVER

BUILDS

PAIN POINTS • UNIT TESTS INCOMPLETE

• MOCK TESTS MAINTENANCE

• EXPENSIVE TEST ENVIRONMENT

• TEST ENVIRONMENT ≠ PRODUCTION

• DEPLOYMENT CYCLES

ON-DEMAND

PAY AS YOU GO

ELASTIC

=

PROGRAMMABLE

PLATFORM

IF YOU CAN PROGRAM IT

YOU CAN AUTOMATE IT

AWS

CLOUDFORMATION STACK-BASED DEPLOYMENT

SERVICE

CLOUDFORMATION

TEMPLATE

{ "Description" : "Create RDS with username and password", "Resources" : { "MyDB" : { "Type" : "AWS::RDS::DBInstance", "Properties" : { "AllocatedStorage" : "500", "DBInstanceClass" : "db.m1.small", "Engine" : "MySQL", "EngineVersion" : "5.5", "MasterUsername" : "MyName", "MasterUserPassword" : "MyPassword" } } } }

"AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "mysql" : [], "mysql-server" : [], "httpd" : [], "php" : [], "php-mysql" : [] } }, "sources" : { "/var/www/html" : "https://s3.amazonaws.com/my-builds/build-v4.zip" } }

{ "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" } }, }

CLOUDFORMATION

TEMPLATE

PROCEDURAL

DEFINITION Create it programmatically

KNOWN

CONFIGURATION Store stack configuration in

source control

PARAMETER

DRIVEN Dynamic and user-driven

templates

COLLABORATION Share templates with ease

as just files

APPLICATION

VERSIONS

+ INFRASTRUCTURE

VERSIONS

CLOUDFORMATION

TEMPLATE

HTTP://SORCERY.SMUGMUG.COM/

TEST ENVIRONMENTS

30,000 REQUESTS / SECOND

1 TB TRAFFIC / DAY

“…AWS seemed to be the best solution available to allow a small, independent company to rapidly develop and test a completely new infrastructure, and host it. We also loved the flexibility that AWS allowed us, when spinning up smaller test environments, for beta trials, QA, localization, and during development. The low initial cost was also crucial.” Alex Evans, CTO

LOAD TESTING

USING AMAZON EC2 TO SIMULATE

2.4 MILLION PLAYERS

CONTINUOUS

DEPLOYMENT SMALL, FREQUENT CHANGES

CONSTANTLY INTEGRATING INTO

PRODUCTION

KEY = ITERATION

ITERATION

= MODIFY THE SYSTEM TO BETTER

MEET THE EXPECTATIONS OF

YOUR USERS

11.6s

Mean time

between

deployments

(weekday)

1,079

Max number of

deployments in a

single hour

10,000

Mean number of

hosts

simultaneously

receiving a

deployment

30,000

Max number of

hosts

simultaneously

receiving a

deployment

DEPLOYMENTS AT

AMAZON.COM

SOFTWARE DEPLOY

≠ PRODUCT LAUNCH

1.5 BILLION PAGE VIEWS

OCTOBER 2012

$83 MILLION IN TRANSACTIONS

4.2 MILLION ITEMS SOLD

30 DEPLOYS PER DAY 1 DEPLOY EVERY 20 MINUTES

"Production is truly the only place you

can validate your code."

AWS OPSWORKS INTEGRATED APPLICATION

MANAGEMENT

14 BILLION REQUESTS/MONTH

50 000 DATABASE UPDATES / SEC

NO CACHE

A / B TESTING

DATA-DRIVEN

ARCHITECTURES

METRICS @ETSY

COST-ORIENTED

ARCHITECTURES

PHP+APACHE+VARNISH

NGINX+NODEJS

CONTINUOUS

INTEGRATION

CONTINUOUS

DEPLOYMENT

CONTINUOUS

DEPLOYMENT

=

CONTINUOUS

EXPERIMENTATION

CONTINUOUS

DEPLOYMENT

=

CONTINUOUS

IMPROVEMENT

INNOVATE

« Want to increase innovation?

Lower the cost of failure »

Joi Ito

SPEED AND AGILITY

Experiment

Often

Fail quickly at

a low cost

More

Innovation

Experiment

Infrequently

Failure is

expensive

Less

Innovation

“ON-PREMISE”

CloudOn Dev / Test Flows

AWS IL Conference

Oct 16 2013

82 | Proprietary and Confidential

What is CloudOn?

CloudOn is a versatile productivity

solution that allows you to connect with

cloud storage providers to create,

review, edit and share your Micosoft

Office files across different devices.

83 | Proprietary and Confidential

The CloudOn Production Environment

API RDS

Provisioning

Social

Storage

File Sharing

Sessions DB

84 | Proprietary and Confidential

Production Deployment Flow

Bootsrapping

Launch Instances

install 3rd party infra (Java, Tomcat, etc)

Configure roles

Deploy CloudOn Java services

(Not using OpsWorks yet)

85 | Proprietary and Confidential

Dev / Test Deployment Needs

Developer Needs

Push-button deploy by developers

Deploy any branch

Several deployments in parallel

Automated Regression Needs

Deploy specific release candidate combos

Run parallel regressions

Low Cost

86 | Proprietary and Confidential

The CloudOn Dev/Test Deployment Model

All In One

87 | Proprietary and Confidential

All In One (AIO) - Deploy

A single instance hosting all CloudOn services

Launches via Jenkins Job

Allows to override defaults: Branch and build

Instance type

Regions

Settings

Tagged by the developer name

Each CloudOn service runs on a different port, no collision

Allows re-deploy on existing AIO

All In One

88 | Proprietary and Confidential

All In One – External Services

What to do with the external, AWS services?

RDS

DynamoDB

ElastiCache

RDS

SQS

SES

89 | Proprietary and Confidential

All In One – SQS, DynamoDB, SES

Using a shared dev account - SQS, DynamoDB, SES

New Queue, Tables created for each AIO

Keyed by the instance private IP

Old Queues and Tables (from obsolete AIOs) deleted upon creation of the newest AIO

90 | Proprietary and Confidential

Using a local mySQL, memcached on the machine

Allows each instance to have its own data

Allows exploring private schema changes

(No cost)

All In One – RDS, Memcached

91 | Proprietary and Confidential

All In One – Cost

Medium instance (default, can be changed)

Most AIOs are automatically destroyed at midnight

(unless requested not to at creation time)

Weekly report on long-running AIOs

(still find rogue AIOs running for too long…)

92 | Proprietary and Confidential

All In One – Additional Benefits

• Push-button deploy by developers

• On demand

• Easy end-to-end integration test on AWS

• Any branch and version

• Allows operational testing

• Allows sandboxing for QA, partners

93 | Proprietary and Confidential

Thanks!