A complete step-by-step walk-through implementing

CI/CD for a microservice with Jenkins, CloudFormation

and Lambda

& aoepeople!

E-Commerce: Magento

CMS: TYPO3

Portals: ZF, Symfony,…

Mobile Searchperience: ElasticSearch

250+ people world-wide

(in 8 locations)

Global Enterprise Projects

Infrastructure: AWS

CloudFormation Lambda Jenkins

Pipeline plugin?

everything sounds better with “continuous”

is a software development practice

where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.”

- Martin Fowler, ThoughtWorks Chief Scientist

is the natural extension of Continuous

Integration: an approach in which teams ensure that every change to the system is releasable, and that we can release any version at the push of a button. Continuous Delivery aims to make releases boring, so we can deliver frequently and get fast feedback on what users care about.” https://www.thoughtworks.com/continuous-delivery

is a software development practice

where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.”

- Martin Fowler, ThoughtWorks Chief Scientist

is the natural extension of Continuous

Integration: an approach in which teams ensure that every change to the system is releasable, and that we can release any version at the push of a button. Continuous Delivery aims to make releases boring, so we can deliver frequently and get fast feedback on what users care about.” https://www.thoughtworks.com/continuous-delivery

Prod

Prod type “YESIKNOWWHATIMDOING” + click

Build Stage Prod

Deploy to Stage

Deploy to Prod

Build Test

…

Test

…

Commit

Trying to get as far as possible to the right

✔

✘ ✘

Test Deploy to Stage

Test Deploy to Prod

Build

… …

Commit

( )*

Selenium

(Half-baked)

AMI

Build

Test Deploy to Stage

Test Deploy to Prod

Build

… …

Commit

Static Code

Analysis Build

Unit Tests

Deploy to “tst”

Integration Tests

Performance Tests

Infrastructure Tests

Selenium Tests

Deploy to “prod”

Pet Cattle

not disposable disposable

nano

GET / Get current counter value

PUT / Increase counter and return increased value

DELETE / Reset counter to 0

(yes, that would be a perfect use-case

for Lambda + API Gateway

+ ElastiCache/DynamoDB…)

Load

Balancer S3 Bucket

www.example.com

Auto Scaling group

…

MySQL

Database

api.example.com

static html

Load

Balancer

Auto Scaling group

S3 Bucket

www.example.com

…

MySQL

Database

api.example.com

disposable/ immutable

Load

Balancer

Auto Scaling group

S3 Bucket

www.example.com

…

MySQL

Database

api.example.com

Auto Scaling group

…

1. Count Instances

2. Launch new ASG and wait until all

instances are healthy

Load

Balancer

Auto Scaling group

S3 Bucket

www.example.com

…

MySQL

Database

api.example.com

Auto Scaling group

…

3. Attach to ELB and wait until all

instances are “In Service”

4. Detach old ASG from ELB

Load

Balancer

Auto Scaling group

S3 Bucket

www.example.com

…

MySQL

Database

api.example.com

Auto Scaling group

…

5. Delete old ASG (by deleting it’s CFN stack)

… … …

Backup

Restore

deploy

trigger job

Internet

gateway

NAT

Gateway

Route

Table

pu

blic

su

bn

ets

private

su

bn

ets

Bastion

Webserver

Database

ELB

0.0.0.0/0:80+443 <my_ip>:22

<elb_sg>:80 <bastion_sg>:22

<websrv_sg>:3306

public

su

bn

ets

private

su

bn

ets

CloudFormation Lambda StackFormation

pre-process

Userdata

Lambda JS “Template”

superset of CFN template (JSON)

Parameters lookup from other stacks, env vars,…

Template

Parameter

Values

create/ update

Stack policies, Tags,…

Stack magento-stage-build-5

Stack magento-stage-build-6

Stack magento-prod-build-6

Stack magento-prod-build-5

CloudFormation Template

merge & pre-process

“CloudFormation+X” Template(s)

+ Dynamic Parameters

+ Stack Policies

+ Behavior

+ Tags …

Blueprint magento-{env:ENVIRONMENT}-build-{env:BUILD}

Blueprint magento-{env:ENVIRONMENT}-setup

Stack magento-stage-setup

Stack magento-prod-setup

blueprints: - stackname: 'magento-{env:BUILD}' template: 'magento.template' stackPolicy: 'policy.json' OnFailure: 'DO_NOTHING' parameters: Build: '{env:BUILD}' KeyPair: '{var:KeyPair}' VPC: '{resource:setupstack:VPC}' Subnet: '{resource:setupstack:Subnet}' InstanceSg: '{resource:setupstack:InstanceSg}' InstanceProfile: '{output:setupstack:InstanceProfile}' BootAmi: 'ami-06116566' tags: Environment: 'prod' Build: '{env:BUILD}'

enforce “immutability” by denying updates!

"UserData": { "Fn::Base64": { "Fn::Join" : [ "", [ "#!/usr/bin/env bash\n", "yum -y update\n", "echo \"Hello World\"\n" ]]}}

"UserData": { "Fn::Base64": { "Fn::FileContent": “userdata.sh” }}

"LoadBalancerName": { "Fn::Join" : [ "-", [ "elb", { "Ref": "EnvironmentName" }, "magento"] ]}

"LoadBalancerName": “elb-{Ref: EnvironmentName}-magento”

"SecurityGroupIngress": [{ "IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "CidrIp": “1.2.3.4/32" }]

"SecurityGroupIngress": [{ "IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "CidrIp": "1.2.3.4/32" /* Fabrizio’s home office IP */ }]

> stackformation ec2:ssh -t Environment:prod –c Type –c Build

filter by tag

Please select an instance [0] i-1033ed9b (Type: Frontend; Environment: prod; Build: 477) [1] i-4ff36ec8 (Type: Backend; Environment: prod ; Build: 477) [2] i-5ab4322b (Type: Worker; Environment: prod; Build: 477) [3] i-705ad42f (Type: Worker; Environment: prod; Build: 476) >

• will take jump hosts into account (ProxyCommand)

• auto-detects your local

(encrypted) private keys

• multiplexed ssh connections

• run commands directly

Build X

Build

Build X+1

Environment A (e.g. prod) Environment B (e.g. stage)

VPC

Static Resources

Build Build

Build X Build X+1

S3 Buckets Artifacts, Jenkins Backup, Static Website

Manual Setup ACM, HostedZone, KeyPair

IAM Setup Instance Roles,…

VPC VPC, Subnets, Bastion,…

Static Resources ELB, DB, Security Groups, DNS

Build ASG, Alarms, Scale Policies,…

manually

from devbox (via Stack- Formation)

via Jenkins Deployment Pipeline (via StackFormation)

these are the immutable ones!

- Instance Counter - Password Generator - GreenBlue Switcher - Stack Deleter

"CountInstances": { "Type": "Custom::InstanceCounter", "Properties": { "ServiceToken": {"Ref": "InstanceCounter"}, "AutoScalingGroupTags": [ {"Key": "Environment", "Value": "prod"}, {"Key": "Type", "Value": "Frontend"} ], "Min": 1, "Max": 10, "Factor": "1.5" } },

"FrontendAsg": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { [...] "DesiredCapacity": {"Fn::GetAtt": ["CountInstances", "Count"]}, "Tags": [ {"Key": "Environment", "Value": "prod", "PropagateAtLaunch": true}, {"Key": "Type", "Value": “Frontend", "PropagateAtLaunch": true} ] } },

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

env-{env:Environment}-setup

vpc

vpc-subnets

vpc-bastion

env-{env:Environment}-website

jenkins cfn-lambdahelper

s3-jenkinsbackup

s3-artifacts

iam

env-{env:Environment}-deploy{env:DEPLOY_ID}

env-{env:Environment}-setup

vpc

vpc-subnets

vpc-bastion

env-{env:Environment}-website

jenkins cfn-lambdahelper

s3-jenkinsbackup

s3-artifacts

iam

env-{env:Environment}-deploy{env:DEPLOY_ID}

api-tst.aoeplay.net

www-tst.aoeplay.net

Instance Profile

backup/restore

upload artifacts

download artifacts

Password Generator

StackDeleter InstanceCounter GreenBlueSwicher

store functions

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

Follow me on twitter!

My blog