higher order infrastructure: from docker basics to cluster management - nicola paolucci - codemotion...

NICOLA PAOLUCCI • ATLASSIAN • @DURDN

Higher Order InfrastructureMicro-services on the Docker Swarm

Docker brought to the average software developer

Clear interfaces i.e. which ports to open

A standard format To package cloud applications

A caching layer To re-use building blocks

Central registry To store images

What’s Orchestration?

3

Services YOUR APPLICATION

Orchestration FRAMEWORKS

Data Center PHYSICAL INFRA

The advent of the Docker orchestration ecosystem

Easier upgrade and roll-out Because of immutable infrastructure, container registries even in heavily polyglot applications

Easier to reason about We can use high level domain specific

languages (like Docker compose’s YAML) to describe the relationships of our application

Baked in scalability and HA Orchestration frameworks provide a solid base to make your application easily scale and be

highly available

Is transforming the way we think about and deploy to the cloud

So let’s deploy a Polyglot Application Stack

Excitement

Even more excitement

Demo Disclaimer In order to convey concepts, the Demo will show a simplified scenario. Unless you, fine audience are all DevOps gods, in which case: Are you here just to mock me?!

Sample Polyglot Application: a voting platform

Reverse Proxy

Voting App

Results App

User from the Internet

Sample Polyglot Application: a voting platform

SQL database

key/value store

Worker

Python

NodeJS

Java

Components of the orchestration puzzleIn Docker’s own ecosystem

Provision machines Configure and prepare machines

to run Docker on a number of IaaS providers. Optionally

configuring them to be swarm cluster ready

Define services Define and link services together at a high level, without specifying

low level infrastructure information

Manage the nodes Allocate services to the cluster

nodes, restart policies, where to deploy workloads depending on

requirements

Networks & volumes Automatic overlay networks and cross-cluster volumes are critical

to complete the puzzle

Docker machine Docker compose Docker swarm Docker network et al.

Other supporting components are…

Discovery service we’ll use Consul

Reverse Proxy Otherwise called load balancer

Volume managers Out of scope for this talk

Other concerns Cloud infra is hard!

And more of them You’re not supposed to read this

Distributed Logging Kibanas of the world

swarm-master

Discovery service

Shared Swarm VXLAN Overlay network

Entry point fixed IP address

User from the Internet

Simplified architecture to support our app: machines

Swarm master

consul

database

db

Reverse proxy

reverse-proxy

Services

services

A word about Docker Swarm

15

CLUSTER MANAGER

Docker swarm

Deploy images and run containers on a full clusters as if you’re handling a single machine

Docker swarm

$ docker run -e \ constraint:instance==database \ --name db

swarm master

swarm node swarm node swarm node

container

container

container

container

container

container

discovery servicescheduler

Docker Swarm: Architecture

17

HELPER TOOL

Discovery Service

For our Swarm to know which nodes are added to the infrastructure and store information about them we need to use a key-value discovery service, like Consul.

Consul from HashiCorp

We need to setup the physical infra, we’ll use Docker machine

19

FIRST STEP

Docker machine

Simple command line tool to provision local and remote hosts with Docker installed. Fantastic to get up and running fast. It has drivers for many Internet service providers and IaaS.

Docker machine

$ docker-machine create -d virtualbox dev

INFO[0000] Downloading boot2docker.iso from... INFO[0001] Creating SSH key... INFO[0001] Creating VirtualBox VM... INFO[0006] Starting VirtualBox VM... INFO[0007] Waiting for VM to start... INFO[0041] "dev" has been created and is now active

Choose a provider

How to provision a box with docker-machine

Choose requirements Name it and label itDocker machine has drivers to provision hosts on a wide variety of IaaS platforms

Base image, memory, geographical area

Give it a name and choose labels to assign to the machine

docker-machine create -d digitalocean \ --digitalocean-access-token=$DO_TOKEN \ --digitalocean-region "ams3" \ consul

$

Specify the discovery service

Creating a machine part of the Swarm

Specify it’s part of the swarm Name itDocker machine has drivers to provision hosts on a wide variety of IaaS platforms

Base image, memory, geographical area

Give it a name

docker-machine create -d digitalocean \ [...] --digitalocean-image "debian-8-x64" \ --digitalocean-region "ams3" \ --swarm --swarm-master \ --swarm-discovery=consul://$(docker-machine ip consul):8500 \ --engine-opt="cluster-store=consul://$(docker-machine ip consul):8500" \ --engine-opt="cluster-advertise=eth0:2376" \ cluster

$

After all the provisioning we have

$ docker-machine ls

cluster digitalocean Running tcp://178.62.222.186:2376 cluster (master) v1.11.0 consul digitalocean Running tcp://178.62.242.131:2376 v1.11.0 db digitalocean Running tcp://128.199.39.208:2376 cluster v1.11.0 rproxy digitalocean Running tcp://128.199.60.17:2376 cluster v1.11.0 services digitalocean Running tcp://128.199.62.119:2376 cluster v1.11.0

In Digital Ocean UI

swarm-master

Reverse proxy

Discovery service

Shared Swarm VXLAN Overlay network

Entry point fixed IP address

User from the Internet

Simplified architecture to support our app: machines

rproxy

Swarm master

consul

database

db

Services

services

• Strategies• Spread• Binpack• Random

• Filters• Constraint• Affinity• Port• Dependency• Health

Swarm comes with strategies and filters

$ docker run -e \ constraint:instance==database --name db

WorkerJava

Voting AppPython

Results AppNodeJS

swarm-master

Reverse proxy

Discovery service

Shared Swarm VXLAN Overlay network

Entry point fixed IP address

User from the Internet

What Orchestration should do for us…

reverse-proxy

Swarm master

consul

Database

db

Services

services

We need to link our components across the cluster

29

TOOL NR.3

Docker compose

Docker compose

Describe the relation of your components in a simple YAML file called docker-compose.yml and docker-compose takes care of starting them and linking them in order.

1 bitbucket: 2 image: atlassian/bitbucket-server 3 ports: 4 - "7990:7990" 5 - "7999:7999" 6 links: 7 - db 8 volumes_from: 9 - license 10 user: root 11 privileged: true 12 db: 13 image: postgres 14 ports: 15 - "5432:5432" 16 environment: 17 18 license: 19 build: .

Dive into Compose configuration

Where is the image

docker-compose.yml a declarative way to define services

Ports and dependencies

Filters and affinities

Specify where is the image or at which folder the sources reside

Define which ports the application exposes and which other containers it depends upon

Specify filters, affinities and environment variables to tell the Swarm master where to deploy this specific service

version: “2”

services:

voting-app:

build: ./voting-app/.

image: docker.atlassian.io/npaolucci/voting-app

ports:

- "80"

depends_on:

- redis

environment:

- "constraint:instance==service"

- "VIRTUAL_HOST=vote.cluster.local"

result-app:

#build: ./result-app/.

Constraints are powerful

docker-compose.yml a declarative way to define services

Load environment file

We can deploy containers based on labels, node names, affinity rules or hardware characteristics

To pass environment variables to docker-compose you can load up an external environment variables file

version: “2”

services:

result-app:

build: ./result-app/.

image: docker.atlassian.io/npaolucci/result-app

ports:

- "80"

depends_on:

- db

environment:

- "constraint:instance==service"

- "VIRTUAL_HOST=results.cluster.local"

worker:

#build: ./worker

docker-compose.yml a declarative way to define services

services:

worker:

build: ./worker

image: docker.atlassian.io/npaolucci/worker

depends_on:

- redis

- db

environment:

- "constraint:instance==service"

redis:

image: redis

ports:

docker-compose.yml a declarative way to define services

services:

redis:

image: redis

ports:

- "6379:6379"

environment:

- "constraint:node==db"

services:

db:

image: postgres:9.4

volumes:

- "db-data:/var/lib/postgresql/data"

environment:

- "constraint:node==db"

volumes:

db-data:

Worker

Voting App

Results App

swarm-master

Reverse proxy

Discovery service

Shared Swarm VXLAN Overlay network

Entry point fixed IP address

User from the Internet

What Orchestration should do for us…

reverse-proxy

Swarm master

consul

Database

db

Services

services

Load environment fileYou can’t execute commands in the YAML. To pass environment variables to docker-compose you can use an file

Setting up the Reverse Proxy

services:

proxy:

image: jwilder/nginx-proxy

ports:

- "80:80"

env_file:

- proxy.env

volumes:

- "/tmp/docker-certs:/tmp/docker-certs"

cat proxy.env

DOCKER_HOST=tcp://178.62.222.186:3376

DOCKER_TLS_VERIFY=1

DOCKER_CERT_PATH=/tmp/docker-certs

constraint:node==rproxy

nginx-proxyTakes dynamically listens for containers exposing the right port and defining a VIRTUAL_HOST variable and refreshes nginx upstreams

Where is the DEMO Lebowski?

What about scaling?

Docker orchestration ecosystem has greatly matured

Compose improvements Like support for custom overlay networks, first class volumes and better support for Swarm

Swarm rescheduling on failure While the “--restart” flag has been there for a

while the most recent Swarm release has rescheduling on node failure. A welcome

feature that was missing.

Docker rewritten to use runC And the first deliverables of the Open Container

Initiative

In the past few releases

@durdn on Twitter

Thank you!