building a production-ready, fully-scalable docker swarm using terraform & packer on openstack
TRANSCRIPT
Scaling With Docker Swarm using Packer, Terraform & OpenStack
Bobby DeVeaux - March 28th 2017https://joind.in/talk/a76ea
https://joind.in/talk/a76ea@bobbyjason#doxlon
• Created my first website at 9 Years old in 1995 😮• Started coding PHP in 2001 - 16 years ago • Developer, Team Leader, CTO, Director & Consultant• Been using AWS for over 5 years • Web Development, Message Queues, Automation, CI&CD • Previously worked at SkyBet & DVSA• Now a DevOps Consultant with UKCloud, Evangelising OpenStack• Contributor to Terraform• I ♥ Docker, Terraform & Golang (or anything Hashicorp)• #twitter: @bobbyjason
About Me ☁
https://joind.in/talk/a76ea@bobbyjason#doxlon
• I’m here to spread the awareness of UKCloud & OpenStack• I want you to use Docker Swarm• I want you to love Terraform• I want to show you how to scale an app using all the above
Why Am I Here?
https://joind.in/talk/a76ea@bobbyjason#doxlon
UKCloud at-a-glance
https://joind.in/talk/a76ea@bobbyjason#doxlon
• Docker for AWS• Docker for Azure• UKCloud offer Cloud Native Infrastructure using Openstack
Why Openstack?
https://joind.in/talk/a76ea@bobbyjason#doxlon
‘Final’ Demo
https://joind.in/talk/a76ea@bobbyjason#doxlon
• Who’s using Docker yet?• Who’s using Docker Swarm?• Who’s using Terraform?• Who’s using Packer?• Who’s not played with any of them, and would love to?
Hands Up
https://joind.in/talk/a76ea@bobbyjason#doxlon
https://joind.in/talk/a76ea@bobbyjason#doxlon
• Docker 1.12 & now 1.13 has Docker Swarm natively, “Swarm Mode”• Cluster management integrated with Docker Engine• Decentralized design• Declarative service model• Scaling• Desired state reconciliation• Multi-host networking• Service discovery• Load balancing• Secure by default• Rolling updates
Swarm Mode?
https://joind.in/talk/a76ea@bobbyjason#doxlon
• docker swarm init —advertise-addr• Initialises the node for Swarm mode
Docker Swarm Init
https://joind.in/talk/a76ea@bobbyjason#doxlon
• docker service create --env APPLICATION_ENV=dev --update-delay 10s --replicas 1 --network mynet --name php-fpm bobbydvo/ukc_php-fpm:latest• docker service create --update-delay 10s --replicas 1 -p
80:80 --network mynet --name web bobbydvo/ukc_nginx:latest• docker service update --image bobbydvo/ukc_nginx:1.81
web
Docker Service Create
https://joind.in/talk/a76ea@bobbyjason#doxlon
• docker service scale web=5
Docker Service Scale
https://joind.in/talk/a76ea@bobbyjason#doxlon
• docker stack deploy —compose-file docker-compose.yml mystack
Docker Stack Deploy (1.13 only)
version: "3"services: web: tty: true depends_on: - php-fpm image: bobbydvo/ukc_nginx:latest ports: - "80:80" deploy: mode: replicated replicas: 2 update_config: parallelism: 1 delay: 10s failure_action: continue monitor: 60s max_failure_ratio: 0.3
php-fpm: tty: true build: ./ image: bobbydvo/dummyapp_php-fpm:latest ports: - "9000:9000" environment: - APPLICATION_ENV=prod deploy: mode: replicated replicas: 1 update_config: parallelism: 1 delay: 10s failure_action: continue monitor: 60s max_failure_ratio: 0.3
https://joind.in/talk/a76ea@bobbyjason#doxlon
• We’re about to embark onto the interesting stuff• Any questions?
…and Pause.
https://joind.in/talk/a76ea@bobbyjason#doxlon
• Packer• Terraform• Docker Swarm• Jenkins For Release
How Do We Scale on Real Infrastructure
https://joind.in/talk/a76ea@bobbyjason#doxlon
• Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.• Infrastructure as Code: Infrastructure is described using a
high-level configuration syntax. This allows a blueprint of your datacenter to be versioned and treated as you would any other code. Additionally, infrastructure can be shared and re-used.• Execution Plans: Terraform has a "planning" step where it
generates an execution plan. The execution plan shows what Terraform will do when you call apply. This lets you avoid any surprises when Terraform manipulates infrastructure
What Is Terraform?
https://joind.in/talk/a76ea@bobbyjason#doxlon
• How long do your builds & deployments in travis / Jenkins take?• What’s acceptable? • ‘Quick’ is relative, and depends on your requirements. • When I say quick deployments, I’m referring to efficient
deployments using Foundation Images.
Who Likes Quick Deployments?
https://joind.in/talk/a76ea@bobbyjason#doxlon
• Ansible / Puppet / Chef means that lots of projects now build from the base box image, i.e. CentOS6 or Ubuntu 14.04 etc.• Do you want to be building this each build? Some of you
are clever, and don’t. Some of you are clever, but didn’t consider an alternative, or didn’t know how. Maybe some of you don’t even use automated builds…• Using Packer and your provisioner of choice, you can export
the artefact and store it as a Docker Container or Image in your cloud provider (Amazon AMI, OpenStack Glance, etc).
Foundation Images
https://joind.in/talk/a76ea@bobbyjason#doxlon
• Tool for creating identical machine images• Supports multiple platforms• Supports many provisioners (Ansible, Chef, Puppet, Bash..
etc.)• Can export image in multiple formats AMIs for EC2,
VMDK/VMX files for VMware, OVF exports for VirtualBox, etc.
What Is Packer?
https://joind.in/talk/a76ea@bobbyjason#doxlon
Packer template.json{ "variables": { "aws_access_key": "", "aws_secret_key": "" }, "builders": [{ "type": "amazon-ebs", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-east-1", "source_ami": "ami-fce3c696", "instance_type": "t2.micro", "ssh_username": "ubuntu", "ami_name": "packer-example {{timestamp}}" }]}
https://joind.in/talk/a76ea@bobbyjason#doxlon
Ansible Provisioning"provisioners": [ { "type": "shell", "inline": [ "rpm -iUvh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-7.noarch.rpm", "yum -y update", "yum -y install ansible", "ansible --version" ] },{ "type": "ansible-local", "playbook_file": "./ansible/playbook.yml", "role_paths": [ "./ansible/roles/init", "./ansible/roles/server", "./ansible/roles/mongodb", "./ansible/roles/php7", "./ansible/roles/nginx", "./ansible/roles/supervisord", "./ansible/roles/redis" ], "group_vars": "./ansible/common/group_vars" }
The cool bit;Putting it all together!
https://joind.in/talk/a76ea@bobbyjason#doxlon
Build Docker Images Regularly#!/bin/bash
VERSION=1CONTAINER=$1BUILD_NUMBER=$2
if [[ $CONTAINER == 'all' ]];then
for CONTAINER in php-fpm nginx dynamodb consul;do
docker build ./$CONTAINER -t bobbydvo/ukc_$CONTAINER:latestdocker tag bobbydvo/ukc_$CONTAINER:latest bobbydvo/ukc_$CONTAINER:$VERSION.$BUILD_NUMBER
doneexit
fi
docker build ./$CONTAINER -t bobbydvo/ukc_$CONTAINER:latestdocker tag bobbydvo/ukc_$CONTAINER:latest bobbydvo/ukc_$CONTAINER:$VERSION.$BUILD_NUMBERdocker push bobbydvo/ukc_$CONTAINER:latestdocker push bobbydvo/ukc_$CONTAINER:$VERSION.$BUILD_NUMBER
https://joind.in/talk/a76ea@bobbyjason#doxlon
How NOT to Build Your Jenkins Job
echo $BUILD_NUMBERdocker -vwhoamisudo docker login -u bobbydvo -p Lr6n9hrGBLNxBmsudo ./build.sh $CONTAINER $BUILD_NUMBER
https://joind.in/talk/a76ea@bobbyjason#doxlon
How to Build Your Jenkins Job
docker login -u bobbdvo -p Lr6n9hrGBLNxBm
.docker/config.json:{"auths": {
"https://index.docker.io/v1/": {"auth": “************************************”
}}
https://joind.in/talk/a76ea@bobbyjason#doxlon
How to Build Your Jenkins Job
echo $BUILD_NUMBERdocker -vwhoami#sudo docker login -u bobbydvo -p Lr6n9hrGBLNxBmsudo ./build.sh $CONTAINER $BUILD_NUMBER
https://joind.in/talk/a76ea@bobbyjason#doxlon
How to Build Your Jenkins Job
https://joind.in/talk/a76ea@bobbyjason#doxlon
Building DummyPHP Docker Image
FROM bobbydvo/ukc_php-fpm:latest
WORKDIR /srvCOPY . /srv/WORKDIR /srv
RUN composer install
CMD ["/usr/bin/supervisord","-n","-c","/etc/supervisord.conf"]
https://joind.in/talk/a76ea@bobbyjason#doxlon
On Merge Jenkins Hookset -e
DUMMY_VERSION=$BUILD_VERSIONNGINX_VERSION='latest'
sudo docker-compose build
sudo docker run -i bobbydvo/dummyapp_php-fpm /srv/vendor/bin/phpunit -c /srv/app/phpunit.xml
# tag & push only if all the above succeeded (set -e)sudo docker tag bobbydvo/dummyapp_php-fpm:latest bobbydvo/dummyapp_php-fpm:$DUMMY_VERSIONsudo docker push bobbydvo/dummyapp_php-fpm:$DUMMY_VERSIONsudo docker push bobbydvo/dummyapp_php-fpm:latest
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null [email protected] "docker service update --image bobbydvo/dummyapp_php-fpm:$DUMMY_VERSION mystack_php-fpm"
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null [email protected] "docker service update --image bobbydvo/ukc_nginx:$NGINX_VERSION mystack_web"
https://joind.in/talk/a76ea@bobbyjason#doxlon
Infrastructure Next
https://joind.in/talk/a76ea@bobbyjason#doxlon
Terraform Build Infrastructure - Manager1resource "openstack_compute_instance_v2" "swarm_manager" {
name = "swarm_manager_0" count = 1
#coreos-docker-alpha image_id = "0fe61d2f-0f9b-4dc8-8706-b45771f86d10" flavor_id = "7d73f524-f9a1-4e80-bedf-57216aae8038" key_pair = "${openstack_compute_keypair_v2.test-keypair.name}" security_groups = ["${openstack_compute_secgroup_v2.example_secgroup_1.name}"]
user_data = "${data.template_file.cloudinit.rendered}"
network { name = "${openstack_networking_network_v2.example_network1.name}" floating_ip = "${openstack_networking_floatingip_v2.example_floatip_manager.address}" }
}
https://joind.in/talk/a76ea@bobbyjason#doxlon
Terraform - Cloudinit.sh
data "template_file" "cloudinit" { template = "${file("cloudinit.sh")}" vars { application_env = "dev" }}
https://joind.in/talk/a76ea@bobbyjason#doxlon
Terraform - Cloudinit.sh
docker swarm initdocker swarm join-token --quiet worker > /home/core/worker-tokendocker swarm join-token --quiet manager > /home/core/manager-token
docker stack deploy --compose-file /home/core/docker-compose.yml mystack > /dev/null
https://joind.in/talk/a76ea@bobbyjason#doxlon
Terraform - Master X resource "openstack_compute_instance_v2" "swarm_managerx" { name = "swarm_manager_${count.index+1}" count = 2
#coreos-docker-beta image_id = "0fe61d2f-0f9b-4dc8-8706-b45771f86d10" flavor_id = "7d73f524-f9a1-4e80-bedf-57216aae8038" key_pair = "${openstack_compute_keypair_v2.test-keypair.name}" security_groups = ["${openstack_compute_secgroup_v2.example_secgroup_1.name}"]
user_data = "${data.template_file.managerinit.rendered}"
network { name = "${openstack_networking_network_v2.example_network1.name}" }}
https://joind.in/talk/a76ea@bobbyjason#doxlon
Terraform - Managerinit.sh
data "template_file" "managerinit" { template = "${file("managerinit.sh")}" vars { swarm_manager = "${openstack_compute_instance_v2.swarm_manager.access_ip_v4}" }}
https://joind.in/talk/a76ea@bobbyjason#doxlon
Terraform - ManagerInit.sh
# Copy Tokens from master1 => masterXsudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /home/core/.ssh/key.pem core@${swarm_manager}:/home/core/manager-token /home/core/manager-token
# Copy docker-compose.yml filesudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /home/core/.ssh/key.pem core@${swarm_manager}:/home/core/docker-compose.yml /home/core/docker-compose.yml
sudo docker swarm join --token $(cat /home/core/manager-token) ${swarm_manager}
https://joind.in/talk/a76ea@bobbyjason#doxlon
Adding Workersresource "openstack_compute_instance_v2" "swarm_slave" { name = "swarm_slave_${count.index}" count = "${var.swarm_node_count}"
#coreos-docker-beta image_id = "0fe61d2f-0f9b-4dc8-8706-b45771f86d10" flavor_id = "c46be6d1-979d-4489-8ffe-e421a3c83fdd" key_pair = "${openstack_compute_keypair_v2.test-keypair.name}" security_groups = ["${openstack_compute_secgroup_v2.example_secgroup_1.name}"]
user_data = "${data.template_file.slaveinit.rendered}"
network { name = "${openstack_networking_network_v2.example_network1.name}" }}
https://joind.in/talk/a76ea@bobbyjason#doxlon
Adding Workers - slaveinit.sh
data "template_file" "slaveinit" { template = "${file("slaveinit.sh")}" vars { swarm_manager = "${openstack_compute_instance_v2.swarm_manager.access_ip_v4}" node_count = "${var.swarm_node_count + 3}" }}
https://joind.in/talk/a76ea@bobbyjason#doxlon
Adding Workers - slaveinit.shsudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /home/core/.ssh/key.pem core@${swarm_manager}:/home/core/worker-token /home/core/worker-tokensudo docker swarm join --token $(cat /home/core/worker-token) ${swarm_manager}
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /home/core/.ssh/key.pem core@${swarm_manager} "docker service scale mystack_php-fpm=${node_count}"
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /home/core/.ssh/key.pem core@${swarm_manager} "docker service scale mystack_web=${node_count}"
## Forces redistribution across all nodesssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /home/core/.ssh/key.pem core@${swarm_manager} "docker service update --force mystack_php-fpm"
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /home/core/.ssh/key.pem core@${swarm_manager} "docker service update --force mystack_web"
https://joind.in/talk/a76ea@bobbyjason#doxlon
Adding Workers - How Many?variable "swarm_node_count" {
default = 1}
variable "swarm_node_count" {default = 5
}
https://joind.in/talk/a76ea@bobbyjason#doxlon
• Docker will bring up another container. Lets try.
What If A Container Dies?
https://joind.in/talk/a76ea@bobbyjason#doxlon
• Provisioned Docker Containers• Infrastructure as Code• Automated Deployments for CI / CD• Scalable Architecture• Openstack + UKCloud
There you have it!
https://joind.in/talk/a76ea@bobbyjason#doxlon
‘Final’ Demo - A Release
Thank you :-)Bobby DeVeaux@bobbyjason
https://joind.in/talk/a76ea