scaling your app with docker swarm using terraform, packer on openstack
TRANSCRIPT
Scaling With Docker Swarm using Packer, Terraform & OpenStack
Bobby DeVeaux - March 9th 2017
https://joind.in/talk/55008
https://joind.in/talk/55008 @bobbyjason
• Created my first website at 9 Years old in 1995 😮
• Started coding PHP in 2001 - 16 years ago
• Developer, Team Leader, CTO, Director & Consultant
• Been using AWS for over 5 years
• Web Development, Message Queues, Automation, CI&CD
• Previously worked at SkyBet & DVSA
• Now a DevOps Consultant with UKCloud, Evangelising OpenStack
• Contributor to Terraform
• I ♥️ Docker, Terraform & Golang (or anything Hashicorp)
• #twitter: @bobbyjason
About Me ☁️
2
https://joind.in/talk/55008 @bobbyjason
• I’m here to spread the awareness of UKCloud & OpenStack
• I want you to use Docker Swarm
• I want you to love Terraform
• I want to show you how to scale an app using all the above
Why Am I Here?
3
https://joind.in/talk/55008 @bobbyjason
UKCloud at-a-glance
4
https://joind.in/talk/55008 @bobbyjason
• Docker for AWS
• Docker for Azure
• UKCloud offer Cloud Native Infrastructure using Openstack
Why Openstack?
5
https://joind.in/talk/55008 @bobbyjason
• Who’s using Docker yet?
• Who’s using Docker Swarm?
• Who’s using Terraform?
• Who’s using Packer?
• Who’s not played with any of them, and would love to?
Hands Up
6
https://joind.in/talk/55008 @bobbyjason
What Is Docker?
7
“Docker containers wrap a piece of software in a complete
filesystem that contains everything needed to run: code,
runtime, system tools, system libraries – anything that can be
installed on a server. This guarantees that the software will
always run the same, regardless of its environment.”- docker.com
https://joind.in/talk/55008 @bobbyjason
• Define multi-container setup
• docker-compose.yml
• beats running multiple docker run commands
• can specify multiple Dockerfiles
• docker-compose up .. similar to vagrant up
Docker Compose
8
https://joind.in/talk/55008 @bobbyjason
docker-compose.yml
9
version: "2"
services:
php-fpm:
image: php:7.0.8-fpm-alpine
ports:
- "9000:9000"
nginx:
image: nginx:alpine
ports:
- "80:80"
https://joind.in/talk/55008 @bobbyjason
• Allows tagging - useful for versioning:
• docker tag myapp:latest myapp:2.0.5
• Resides locally
Docker Tag
10
https://joind.in/talk/55008 @bobbyjason
• Image Repository - default for Docker
• Alternatives include:
• Amazon ECR
• Openstack Glance
• quay.io
Docker Hub
11
https://joind.in/talk/55008 @bobbyjason
• Login to selected image repository
• Docker HUB: $ docker login -u username -p password
• Amazon: $ eval `aws ecr get-login`
Docker Login
12
https://joind.in/talk/55008 @bobbyjason
• Push images & tags up to image repository
• docker push myapp:2.0.5
• docker push latest
Docker Push
13
https://joind.in/talk/55008 @bobbyjason
What Is Docker Swarm?
14
“Docker Swarm provides native clustering capabilities to turn a
group of Docker engines into a single, virtual Docker Engine.
With these pooled resources, you can scale out your
application as if it were running on a single, huge computer.”
- docker.com
https://joind.in/talk/55008 @bobbyjason 15
https://joind.in/talk/55008 @bobbyjason
• Docker 1.12 & now 1.13 has Docker Swarm natively, “Swarm Mode”
• Cluster management integrated with Docker Engine
• Decentralized design
• Declarative service model
• Scaling
• Desired state reconciliation
• Multi-host networking
• Service discovery
• Load balancing
• Secure by default
• Rolling updates
Swarm Mode?
16
https://joind.in/talk/55008 @bobbyjason
• docker swarm init —advertise-addr
• Initialises the node for Swarm mode
Docker Swarm Init
17
https://joind.in/talk/55008 @bobbyjason
• docker service create --env APPLICATION_ENV=dev --update-delay 10s --replicas 1 --network mynet --name php-fpm bobbydvo/ukc_php-fpm:latest
• docker service create --update-delay 10s --replicas 1 -p 80:80 --network mynet --name web bobbydvo/ukc_nginx:latest
• docker service update --image bobbydvo/ukc_nginx:1.81 web
Docker Service Create
18
https://joind.in/talk/55008 @bobbyjason
• docker service scale web=5
Docker Service Scale
19
https://joind.in/talk/55008 @bobbyjason
• docker stack deploy —compose-file docker-compose.yml mystack
Docker Stack Deploy (1.13 only)
20
version: "3"
services:
web:
tty: true
depends_on:
- php-fpm
image: bobbydvo/ukc_nginx:latest
ports:
- "80:80"
deploy:
mode: replicated
replicas: 2
update_config:
parallelism: 1
delay: 10s
failure_action: continue
monitor: 60s
max_failure_ratio: 0.3
php-fpm:
tty: true
build: ./
image: bobbydvo/dummyapp_php-fpm:latest
ports:
- "9000:9000"
environment:
- APPLICATION_ENV=prod
deploy:
mode: replicated
replicas: 1
update_config:
parallelism: 1
delay: 10s
failure_action: continue
monitor: 60s
max_failure_ratio: 0.3
https://joind.in/talk/55008 @bobbyjason
• We’re about to embark onto the interesting stuff
• Any questions?
…and Pause.
21
https://joind.in/talk/55008 @bobbyjason
• Packer
• Terraform
• Docker Swarm
• Jenkins For Release
How Do We Scale on Real Infrastructure
22
https://joind.in/talk/55008 @bobbyjason
• Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.
• Infrastructure as Code: Infrastructure is described using a high-level configuration syntax. This allows a blueprint of your datacenter to be versioned and treated as you would any other code. Additionally, infrastructure can be shared and re-used.
• Execution Plans: Terraform has a "planning" step where it generates an execution plan. The execution plan shows what Terraform will do when you call apply. This lets you avoid any surprises when Terraform manipulates infrastructure
What Is Terraform?
23
https://joind.in/talk/55008 @bobbyjason
• Terraform plan
• Will show you exactly what it’s going to do and how many resources are going to be affected
• No surprises.
Terraform Plan
24
https://joind.in/talk/55008 @bobbyjason
• terraform apply
• Applies the changes as shown in terraform plan using it’s resource graph. It know’s which resources have dependant resources and which ones don’t
• Parallelised building - using the resource graph it will make all the changes as quickly/efficiently as possible
Terraform Apply
25
https://joind.in/talk/55008 @bobbyjason
• Using the .tfstate file, it is fully aware of the resources.
• Terraform destroy will literally remove all the infrastructure it built for you. WARNING: There is no ctrl+z!
Terraform Destroy
26
https://joind.in/talk/55008 @bobbyjason
Terraform Demo
27
provider "openstack" {
}
resource "openstack_networking_network_v2" "example_network2" {
name = "example_network_2"
admin_state_up = "true"
}
resource "openstack_networking_subnet_v2" "example_subnet2" {
name = "example_subnet_2"
network_id = "${openstack_networking_network_v2.example_network2.id}"
cidr = "10.10.1.0/24"
ip_version = 4
dns_nameservers = ["8.8.8.8", "8.8.4.4"]
}
https://joind.in/talk/55008 @bobbyjason
• How long do your builds & deployments in travis / Jenkins take?
• What’s acceptable?
• ‘Quick’ is relative, and depends on your requirements.
• When I say quick deployments, I’m referring to efficient deployments using Foundation Images.
Who Likes Quick Deployments?
28
https://joind.in/talk/55008 @bobbyjason
• Ansible / Puppet / Chef means that lots of projects now build from the base box image, i.e. CentOS6 or Ubuntu 14.04 etc.
• Do you want to be building this each build? Some of you are clever, and don’t. Some of you are clever, but didn’t consider an alternative, or didn’t know how. Maybe some of you don’t even use automated builds…
• Using Packer and your provisioner of choice, you can export the artefact and store it as a Docker Container or Image in your cloud provider (Amazon AMI, OpenStack Glance, etc).
Foundation Images
29
https://joind.in/talk/55008 @bobbyjason
• Tool for creating identical machine images
• Supports multiple platforms
• Supports many provisioners (Ansible, Chef, Puppet, Bash.. etc.)
• Can export image in multiple formats AMIs for EC2, VMDK/VMX files for VMware, OVF exports for VirtualBox, etc.
What Is Packer?
30
https://joind.in/talk/55008 @bobbyjason
Packer template.json
31
{
"variables": {
"aws_access_key": "",
"aws_secret_key": ""
},
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "us-east-1",
"source_ami": "ami-fce3c696",
"instance_type": "t2.micro",
"ssh_username": "ubuntu",
"ami_name": "packer-example {{timestamp}}"
}]
}
https://joind.in/talk/55008 @bobbyjason
Ansible Provisioning
32
"provisioners": [
{
"type": "shell",
"inline": [
"rpm -iUvh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-7.noarch.rpm",
"yum -y update",
"yum -y install ansible",
"ansible --version"
]
},{
"type": "ansible-local",
"playbook_file": "./ansible/playbook.yml",
"role_paths": [
"./ansible/roles/init",
"./ansible/roles/server",
"./ansible/roles/mongodb",
"./ansible/roles/php7",
"./ansible/roles/nginx",
"./ansible/roles/supervisord",
"./ansible/roles/redis"
],
"group_vars": "./ansible/common/group_vars"
}
The cool bit;Putting it all together!
https://joind.in/talk/55008 @bobbyjason
Use Packer To Build CoreOS Image
34
{
"_comment": "Template file pulling from CoreOS",
"builders": [
{
"type": "openstack",
"image_name": "CoreOS-Docker-Alpha-1-13",
"source_image": "8e892f81-2197-464a-9b6b-1a5045735f5d",
"flavor": "c46be6d1-979d-4489-8ffe-e421a3c83fdd",
"ssh_keypair_name": "ukcloudos",
"ssh_private_key_file": "/Users/bobby/.ssh/ukcloudos",
"use_floating_ip": true,
"floating_ip_pool": "internet",
"ssh_username": "core",
"ssh_pty" : true
}
],
https://joind.in/talk/55008 @bobbyjason
Use Packer To Build CoreOS Image
35
"provisioners": [
{
"type": "shell",
"inline": [
"sudo sh -c 'echo GROUP=alpha > /etc/coreos/update.conf'",
"sudo systemctl restart update-engine",
"sudo update_engine_client -update",
"sudo sh -c 'mkdir /opt/'",
"sudo sh -c 'mkdir /opt/bin'",
"sudo sh -c 'curl -L https://github.com/docker/compose/releases/download/1.9.0/docker-
"sudo sh -c 'chmod +x /opt/bin/docker-compose'"
]
},{
"type": "file",
"source": "/Users/bobby/.ssh/ukcloudos",
"destination": "/home/core/.ssh/key.pem"
}
]
}
https://joind.in/talk/55008 @bobbyjason
Build Basic Docker Images - Nginx
36
FROM nginx:alpine
RUN apk update && apk upgrade && apk add --update curl wget bash tree
RUN mkdir -p /srv/web
ADD ./conf/default.conf /etc/nginx/conf.d/default.conf
ADD ./conf/fastcgi_params /etc/nginx/fastcgi_params
ADD ./index.php /srv/web
CMD ["nginx", "-g", "daemon off;"]
https://joind.in/talk/55008 @bobbyjason
Build Basic Docker Images - Nginx
37
server {
listen *:80;
server_name _ localhost;
root /srv/web;
index index.php index.html index.htm;
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location ~* \.(jpg|jpeg|png|gif|ico|css|js|woff)$ {
expires 365d;
}
location ~* \.php$ {
fastcgi_pass php-fpm:9000;
fastcgi_index index.php;
fastcgi_read_timeout 180;
include fastcgi_params;
}
}
https://joind.in/talk/55008 @bobbyjason
Build Basic Docker Images - Nginx
38
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param REQUEST_SCHEME $scheme;
fastcgi_param HTTPS $https if_not_empty;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param REDIRECT_STATUS 200;
https://joind.in/talk/55008 @bobbyjason
Build Basic Docker Images - php-fpm
39
FROM php:7.0.8-fpm-alpine
RUN apk update && apk upgrade && \
apk add --update curl wget bash tree autoconf gcc g++ make libffi-dev openssl-dev
RUN apk add supervisor
RUN pecl install xdebug \
&& docker-php-ext-enable xdebug \
&& docker-php-ext-install opcache
RUN docker-php-ext-install pdo_mysql
RUN mkdir -p /var/log/supervisor
COPY ./conf/php-dev.ini /usr/local/etc/php/
COPY ./conf/php-prod.ini /usr/local/etc/php/
COPY ./conf/php-prod.ini /usr/local/etc/php/php.ini
COPY ./conf/envars-development.conf /usr/local/etc/php-fpm.d
COPY ./conf/supervisord.conf /etc
WORKDIR /srv
COPY ./index.php /srv/web/
WORKDIR /srv
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/bin --filename=composer
CMD ["/usr/bin/supervisord","-n","-c","/etc/supervisord.conf"]
https://joind.in/talk/55008 @bobbyjason
Build Basic Docker Images - php-fpm
40
<?php
echo 'Welcome to your php-fpm Docker container.
You should copy your application into the /srv folder and overwrite this file.';
var_dump($_SERVER);
phpinfo();
[program:php-fpm]
command=php-fpm --nodaemonize -c /usr/local/etc/php/php-%(ENV_APPLICATION_ENV)s.ini
https://joind.in/talk/55008 @bobbyjason
Build Basic Docker Images - docker-compose.yml
41
version: "2"
services:
php-fpm:
build: ./php-fpm
image: bobbydvo/ukc_php-fpm:latest
ports:
- "9000:9000"
environment:
- APPLICATION_ENV=dev
web:
build: ./nginx
image: bobbydvo/ukc_nginx:latest
ports:
- "80:80"
$ docker-compose up —build
https://joind.in/talk/55008 @bobbyjason
Build Docker Images Regularly
42
#!/bin/bash
VERSION=1
CONTAINER=$1
BUILD_NUMBER=$2
if [[ $CONTAINER == 'all' ]];
then
for CONTAINER in php-fpm nginx dynamodb consul;
do
docker build ./$CONTAINER -t bobbydvo/ukc_$CONTAINER:latest
docker tag bobbydvo/ukc_$CONTAINER:latest bobbydvo/ukc_$CONTAINER:$VERSION.$BUILD_NUMBER
done
exit
fi
docker build ./$CONTAINER -t bobbydvo/ukc_$CONTAINER:latest
docker tag bobbydvo/ukc_$CONTAINER:latest bobbydvo/ukc_$CONTAINER:$VERSION.$BUILD_NUMBER
docker push bobbydvo/ukc_$CONTAINER:latest
docker push bobbydvo/ukc_$CONTAINER:$VERSION.$BUILD_NUMBER
https://joind.in/talk/55008 @bobbyjason
How NOT to Build Your Jenkins Job
43
echo $BUILD_NUMBER
docker -v
whoami
sudo docker login -u bobbydvo -p Lr6n9hrGBLNxBm
sudo ./build.sh $CONTAINER $BUILD_NUMBER
https://joind.in/talk/55008 @bobbyjason
How to Build Your Jenkins Job
44
docker login -u bobbdvo -p Lr6n9hrGBLNxBm
.docker/config.json:
{
"auths": {
"https://index.docker.io/v1/": {
"auth": “************************************”
}
}
https://joind.in/talk/55008 @bobbyjason
How to Build Your Jenkins Job
45
echo $BUILD_NUMBER
docker -v
whoami
#sudo docker login -u bobbydvo -p Lr6n9hrGBLNxBm
sudo ./build.sh $CONTAINER $BUILD_NUMBER
https://joind.in/talk/55008 @bobbyjason
How to Build Your Jenkins Job
46
https://joind.in/talk/55008 @bobbyjason
Building DummyPHP Docker Image
47
FROM bobbydvo/ukc_php-fpm:latest
WORKDIR /srv
COPY . /srv/
WORKDIR /srv
RUN composer install
CMD ["/usr/bin/supervisord","-n","-c","/etc/supervisord.conf"]
https://joind.in/talk/55008 @bobbyjason
On Merge Jenkins Hook
48
set -e
DUMMY_VERSION=$BUILD_VERSION
NGINX_VERSION='latest'
sudo docker-compose build
sudo docker run -i bobbydvo/dummyapp_php-fpm /srv/vendor/bin/phpunit -c /srv/app/phpunit.xml
# tag & push only if all the above succeeded (set -e)
sudo docker tag bobbydvo/dummyapp_php-fpm:latest bobbydvo/dummyapp_php-fpm:$DUMMY_VERSION
sudo docker push bobbydvo/dummyapp_php-fpm:$DUMMY_VERSION
sudo docker push bobbydvo/dummyapp_php-fpm:latest
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null [email protected] "dock
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null [email protected] "dock
https://joind.in/talk/55008 @bobbyjason
Infrastructure Next
49
https://joind.in/talk/55008 @bobbyjason
Terraform Build Infrastructure - Manager1
50
resource "openstack_compute_instance_v2" "swarm_manager" {
name = "swarm_manager_0"
count = 1
#coreos-docker-alpha
image_id = "0fe61d2f-0f9b-4dc8-8706-b45771f86d10"
flavor_id = "7d73f524-f9a1-4e80-bedf-57216aae8038"
key_pair = "${openstack_compute_keypair_v2.test-keypair.name}"
security_groups = ["${openstack_compute_secgroup_v2.example_secgroup_1.name}"]
user_data = "${data.template_file.cloudinit.rendered}"
network {
name = "${openstack_networking_network_v2.example_network1.name}"
floating_ip = "${openstack_networking_floatingip_v2.example_floatip_manager.address}"
}
}
https://joind.in/talk/55008 @bobbyjason
Terraform - Cloudinit.sh
51
data "template_file" "cloudinit" {
template = "${file("cloudinit.sh")}"
vars {
application_env = "dev"
}
}
https://joind.in/talk/55008 @bobbyjason
Terraform - Cloudinit.sh
52
docker swarm init
docker swarm join-token --quiet worker > /home/core/worker-token
docker swarm join-token --quiet manager > /home/core/manager-token
docker stack deploy --compose-file /home/core/docker-compose.yml mystack > /dev/null
https://joind.in/talk/55008 @bobbyjason
Terraform - Master X
53
resource "openstack_compute_instance_v2" "swarm_managerx" {
name = "swarm_manager_${count.index+1}"
count = 2
#coreos-docker-beta
image_id = "0fe61d2f-0f9b-4dc8-8706-b45771f86d10"
flavor_id = "7d73f524-f9a1-4e80-bedf-57216aae8038"
key_pair = "${openstack_compute_keypair_v2.test-keypair.name}"
security_groups = ["${openstack_compute_secgroup_v2.example_secgroup_1.name}"]
user_data = "${data.template_file.managerinit.rendered}"
network {
name = "${openstack_networking_network_v2.example_network1.name}"
}
}
https://joind.in/talk/55008 @bobbyjason
Terraform - Managerinit.sh
54
data "template_file" "managerinit" {
template = "${file("managerinit.sh")}"
vars {
swarm_manager = "${openstack_compute_instance_v2.swarm_manager.access_ip_v4}"
}
}
https://joind.in/talk/55008 @bobbyjason
Terraform - ManagerInit.sh
55
# Copy Tokens from master1 => masterX
sudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null
# Copy docker-compose.yml file
sudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null
sudo docker swarm join --token $(cat /home/core/manager-token) ${swarm_manager}
https://joind.in/talk/55008 @bobbyjason
Manager Cluster Demo
56
https://joind.in/talk/55008 @bobbyjason
Adding Workers
57
resource "openstack_compute_instance_v2" "swarm_slave" {
name = "swarm_slave_${count.index}"
count = "${var.swarm_node_count}"
#coreos-docker-beta
image_id = "0fe61d2f-0f9b-4dc8-8706-b45771f86d10"
flavor_id = "c46be6d1-979d-4489-8ffe-e421a3c83fdd"
key_pair = "${openstack_compute_keypair_v2.test-keypair.name}"
security_groups = ["${openstack_compute_secgroup_v2.example_secgroup_1.name}"]
user_data = "${data.template_file.slaveinit.rendered}"
network {
name = "${openstack_networking_network_v2.example_network1.name}"
}
}
https://joind.in/talk/55008 @bobbyjason
Adding Workers - slaveinit.sh
58
data "template_file" "slaveinit" {
template = "${file("slaveinit.sh")}"
vars {
swarm_manager = "${openstack_compute_instance_v2.swarm_manager.access_ip_v4}"
node_count = "${var.swarm_node_count + 3}"
}
}
https://joind.in/talk/55008 @bobbyjason
Adding Workers - slaveinit.sh
59
sudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -
sudo docker swarm join --token $(cat /home/core/worker-token) ${swarm_manager}
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /home/core/.ssh/key.
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /home/core/.ssh/key.
## Forces redistribution across all nodes
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /home/core/.ssh/key.
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /home/core/.ssh/key.
https://joind.in/talk/55008 @bobbyjason
Adding Workers - How Many?
60
variable "swarm_node_count" {
default = 1
}
variable "swarm_node_count" {
default = 5
}
https://joind.in/talk/55008 @bobbyjason
• Docker will bring up another container. Lets try.
What If A Container Dies?
61
https://joind.in/talk/55008 @bobbyjason
Final Demo + Questions
62
https://joind.in/talk/55008 @bobbyjason
• Provisioned Docker Containers
• Infrastructure as Code
• Automated Deployments for CI / CD
• Scalable Architecture
• Openstack + UKCloud
There you have it!
63
Thank you :-)
Bobby DeVeaux
@bobbyjason
https://joind.in/talk/55008