Dockerize a small business

HOW DOCKER TRANSFORM OUR DEVELOPMENT & DEPLOYMENT PROCESS

Contents• The Dark age (Why Docker and Container matter)

• The Dockerization progress (How Docker fits in)

• Practice time

• Some useful recipes

• Example 1: Setup your own CI environments

• Example 2: Fantasy Football App

The Dark age

Modern “app” requirementsNot so thin front end app on various clients: browser, mobile, tablet

Assemble many different services from inside/outside sources

Running on any available set of physical resources (public/private/virtualized)

Modern “app” requirementsHow to setup development environment fast and reliable

How to ensure services interact consistently, avoids dependencies hell

How to avoids NxN different configs

How to migrate and scale quickly, ensure compatibility across different deployment environments

NxN compatibility nightmareMULTIPLICITY OF STACKS

nginx + modsecurity + openssl

postgresql + postgis

hadoop + hive + thrift + OpenJDK

Ruby + Rails + sass + Unicorn

Redis + redis-sentinel

Python 3 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs

Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client

MULTIPLICITY OF DEPLOYMENT ENVIRONMENTS

Development VM Public Cloud Contributor’s laptop

Production Servers Production Cluster Customer Data Center

Centos Ubuntu Debian

5 years running without package updateOutdate kernel Weird lib path

Custom repositories that no one maintained

isolation = k * repeatabilityWhere k is a constant that is inversely proportional to the probability of you saying "It works on my machine".

Multiple tools for every stack to solve the isolation problem◦ Compile your Python + virtualenv + pip + buildout + supervisord

◦ Rbenv + bundle + foreman + Capistrano + foreman

“This is like having sex with two condoms and anticonception pills.”

Still there will be inconsistency ◦ Incomplete version locks file

◦ Download cache in local machine

◦ Lack of / different version of C/C++ devel packages

Also, It takes hours to setup the environments

My code’s compiling

Virtualization saves the dayA virtual server for each app

Consistent environment

Easier for management (backup/migrate/increase resource …)

Vagrant automate dev environment setup

…for a price!Heavy weight / expensive / slow

Different VMs for different hypervisors

Image migration between different infrastructure/PaaS providers was/is quite painful

Resource allocation is not good enough

Still takes time to setup

People need something even more lightweight/atomic

Container is the futureAnd Docker popularized the technology

◦ Can encapsulate almost everything and its dependencies

◦ Run consistently on any hardware without modification

◦ Resource, network and content isolation

◦ Almost no resource overhead

◦ Good set of operation: run, start, stop, commit, pull, search… Perfect for CI, CD, auto scaling, hybrid clouds…

◦ Separation of duty: Dev worries about code, Ops worries about infrastructure

Why dev care: Build once, run anywhere A clean, safe, hygienic and portable runtime environment for your app.

Run each app in its own isolated container, so you can run various versions of libraries and other dependencies for each app without worrying

Reduce/eliminate concerns about compatibility on different platforms

Cheap, zero-penalty containers to deploy services

Instant replay and reset of image snapshots

Why ops care: Configure once, run anythingMake the entire lifecycle more efficient, consistent, and repeatable

Eliminate inconsistencies between development, test, production, and customer environments

Significantly improves the speed and reliability of continuous deployment and continuous integration systems

Because the containers are so lightweight, address significant performance, costs, deployment, and portability issues normally associated with VMs

The Dockerizationprogress

Use Docker as new building blockThinks Service and Volume

Design apps as multiple services talk with each others via API

Each service run in 1 or multiple, replicated containers

No containers run multiple services

Design persistent bits of app as multiple volumes

A service can either rw or ro a volume

Dev describes services and dependencies via compose file

New dev use compose file to setup dev environment

Ops look at the compose file and translate/create platform specific service configuration files.

Docker-powered processDev

Github

Local, privatereposistory

CI service

Build imagewith src

Run testsin image

Dev

Bootstrap project withDockerfile and

docker-compose.yml

Clone project$ docker-compose up

Docker-powered processOps

Github

Dockerreposistory

CI service

Public/privatecontainer service

image

Build imagewith src

Run testsin image Services

configuration

Run services

Bootstrap a new projectDev create new project with Dockerfile and docker-compose.yml

Dev declare dependencies and external services through docker-compose.yml◦ Services start with build: . will have a .: /app volume

Dev add test/CI related files and push to github

The image is built and pushed to private docker registry

Join an ongoing projectDev clone the repository

Dev download necessary volumes from other dev/staging server (db, configuration etc)

$ docker-compose up

Deploy and scale servicesOps look at the docker-compose file and create necessary services on the production dockerenvironment

Ops configure the volume storage options (nfs, ebs …)

Ops configure the forward proxy (nginx, certbot, load balancing…)

Ops run multiple containers base on the built image

Update and migrate New app image is built and pushed into registry.

New containers are sequentially created and replace the old ones (rolling update).

If there is data need to be migrated (scheduled downtime):◦ Stop the gateway (nginx)

◦ Hot backup the data volumes

◦ Run the migration scripts

◦ Restart the gateway

◦ Fall back is easy and instant since both old data volumes and images are there

PracticeSOME RECIPES TO BOOTSTRAP YOUR OWN DOCKERIZATION.

Docker cluster setup#Install docker on both master and workersyum update -ycurl -fsSL https://get.docker.com/ | sh

#Start & Enable docker servicesystemctl start docker; systemctl enable docker

#On the master, Init the swarm, we don't want outsider join our swarm.docker swarm init --listen-addr $PRIVATE_IP:2377

#On the workers, join the swarm, command is outputted from last commanddocker swarm join --secret $SECRET --ca-hash $HASH $MASTER_IP:2377

#Create a services, scale the servicesdocker service create -p 80:8000 --name whoami jwilder/whoamidocker service scale whoami=2#Wait containers being created and test it with curl

Nginx and Certbotmkdir /opt/nginx

docker run -d --name nginx nginxdocker cp nginx:/etc/nginx /opt/nginx/etcdocker cp nginx:/usr/share/nginx/html /opt/nginx/wwwdocker stop nginx; docker rm nginx

docker run –d --name nginx --restart=always \-p 80:80 –p 443:443 \-v /opt/nginx/etc:/etc/nginx \-v /opt/nginx/www:/usr/share/nginx/html nginx

docker run -it --rm \-v /opt/nginx/etc/certs:/etc/letsencrypt \-v /opt/nginx/www:/www \gzm55/certbot certonly --webroot -w /www –d example.com –d www.example.com

crontab –e30 2 * * 1 docker run -it --rm -v /opt/nginx/certs:/etc/letsencrypt -v /opt/nginx/www:/www gzm55/certbot renew >> /var/log/certbot.log35 2 * * 1 docker exec nginx nginx -s reload

Rails quickstartDOCKERFILE

FROM ruby:2.2.0

RUN apt-get update -qq && apt-get install -y build-essential libpq-dev nodejs

RUN mkdir /myapp

WORKDIR /myapp

ADD Gemfile /myapp/Gemfile

ADD Gemfile.lock /myapp/Gemfile.lock

RUN bundle install

ADD . /myapp

DOCKER-COMPOSE

version: '2’

services:db:image: postgres

web:build: .command: bundle exec rails s -p 3000

-b '0.0.0.0‘volumes:- .:/myapp

ports:- "3000:3000“

depends_on:- db

Python quickstartDOCKERFILE

FROM python:2.7

RUN mkdir /myapp

WORKDIR /myapp

ADD requirements.txt /myapp/

RUN pip install -r requirements.txt

ADD . /myapp/

DOCKER-COMPOSE

version: '2'

services:db:

image: postgres

web:build: .command: python app.pyvolumes:

- .:/myappports:

- "5000:5000"depends_on:

- db

Your own CI environment

Fantasy football apps