tiad 2016 : migrating 100% of your production services to containers

@slallema

Migrating 100% of your production services to containers

Simon LallemandSystem engineer

Over 30 million members and growing

1.5 million+joining each month

We are in 22 countries

150+ bare-metal servers200+ services3000+ running containers

?

How did we get there ?

(Pre)HistoryThe platform

evolution

It all started 10 years ago...

Web hosting

Dedicated server

1 rack

3 racks

5 racks

Industrialization eraStep 1

Step 1 : Installation and configuration of the servers

Chef

Foreman

Collins

Industrialization eraStep 2

Step 2 : Hardware uniformization

Limit the models of servers

Buy one rack at a time

Engineers don’t go to the DC

Industrialization eraStep 3

Step 3 : Abstract the hardware

Different workloads on one server

VM ? no.

=> Containers !

Containers : from the choice to the prod

Containersas container runtime

Dock... no, rkt !

Experimenting with rkt in early 2015

Taskforce to create the workflow

Workshops to validate

A lot of iterations

No existing tools: we made our own !

80% of the prod migrated in november 2015

The new infrastructure

On 100% of our new servers

CoreOS : easier to install and maintain

Perfect OS to run containers

CoreOSstable

rktas container runtime

we rkt !

Everything must run in containers !

(One of?) The biggest rkt deployment in production

We like the simplicity, stability and the modularity

?

How to build a container image ?

App containerspecification

A way of packaging

We like the appc approach because - it’s an open specification - it supports PODs - it’s easier to build tools around this spec than with docker images

But we also watch closely the OCI...

dgrContainer build and runtime tool

Harmonize the way we build

Quick build

One way of doing things

Easy to understand for newcomers

As little code replication as possible

Templating at container start

A good integration with rkt

github.com/blablacar/dgr

dgr : build directory of an ACI

├── aci-manifest.yml├── attributes│ └── redis.yml├── runlevels│ └── build│ └── install.sh└── templates └── etc └── redis └── redis.conf.tmpl

A standardized structureEase maintenance and teamwork

Inspired by config managementSeparation of templates, attributes and scripts.

dgr : ACI manifest

name: redis:0.1aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf

Simpler manifest formatFill only what is important

YAML <3

One process per ACIComposition is done only with POD

dgr : ACI manifest

name: elasticsearch:2.3.3aci: app: exec: - /bin/elasticsearch dependencies: - debian - java

Use of dependenciesCompositionLighter images

dgr : runlevel build

├── aci-manifest.yml├── attributes│ └── redis.yml├── runlevels│ └── build│ └── install.sh└── templates └── etc └── redis └── redis.conf.tmpl

Scripts executed in the containerBuild from inside the container with all the dependencies

dgr : runlevel build

#!/bin/bash

apt-get install -y redis-server

Install packages and stuffFor debian based containers you can install packages using apt-get for instance.

dgr : runlevel builder

name: aci-go-synapsebuilder: dependencies:

- example.org/aci-go- example.org/aci-git

aci: app: exec: - /bin/go-synapse

Build from outside of the containerDedicated builder image with go & git

dgr : runlevel builder

#!/dgr/bin/busybox sh

git clone https://github.com/blablacar/go-synapse

cd go-synapse

go build

cp dist/go-synapse ${ROOTFS}/bin

Builder scriptClone, build and copy binary to target ACI

Only /bin/go-synapse in final ACI

dgr : runlevel builder

name: example.org/gentoo-lighttpdbuilder: dependencies: - example.org/gentoo-stage4aci: dependencies: - example.org/base app: exec: [ /usr/sbin/lighttpd, -D ]

Other exampleBuilder with gentoo’s stage4

dgr : runlevel builder

#!/dgr/bin/busybox sh

emerge -v lighttpd

Install packages from outsideUse emerge to install your app and its dependencies.

Lighttpd with dependencies only in the final ACI

dgr : templates & attributes

├── aci-manifest.yml├── attributes│ └── redis.yml├── runlevels│ └── build│ └── install.sh└── templates └── etc └── redis └── redis.conf.tmpl

Render configuration filesTemplates stored in the aci

Default attributes stored in the aci

Overridable when used as dependencies

Overridable by environment var

dgr : templates & attributes

# templates/etc/redis.conf.tmpldaemonize noport {{ .redis.port }}…

# attributes/redis.ymldefault: redis: port: 6379

dgr : runlevel prestart

├── aci-manifest.yml├── attributes│ └── redis.yml├── runlevels│ └── prestart-late│ └── 10-init-db.sh└── templates └── etc └── redis └── redis.conf.tmpl

Initialize containerprestart-early and prestart-late scripts

Before and after templating

Initialize environment before exec

dgr : testing

├── aci-manifest.yml├── attributes├── runlevels├── templates└── tests └── wait.sh └── my_cool_tests.bats

TestingBats as default tester

wait.shWait for service to be ready

dgr : testing

#!/dgr/bin/bats -x

@test "Nginx should be running" { run bash -c "ps -aux | grep nginx" [ "$status" -eq 0 ] echo ${lines[0]} [[ "${lines[0]}" =~ "/usr/sbin/nginx" ]]}

@test "Nginx should listen on port: 80" { run bash -c "netstat -peanut | grep nginx" [ "$status" -eq 0 ] [[ "${lines[0]}" =~ ":80" ]] [[ "${lines[0]}" =~ "nginx" ]]}

dgr : subcommands

~ # dgr init

~ # dgr try

~ # dgr build

~ # dgr test

~ # dgr install

~ # dgr push

?

What about PODs ?

dgr : POD manifest

name: example.org/pod-myapp:1pod: apps: - dependencies: - example.org/nginx:1.10

- dependencies: - example.org/php:7.0 - example.org/myapp:42 app: exec: [/usr/sbin/php5-fpm, -F]

Grouping ACIACIs as a single unitSame execution context

Use dependencies (again)Create dedicated ACIs for the pod

Pin versionCan override attributes/templates for pod context

dgr : run a POD

$ rkt run \ example.org/pod-myapp_aci-nginx:1 \ example.org/pod-myapp_aci-php:1

?

How to contextualize a POD ?

dgr : run a POD

# attributes/redis.ymloverride: redis: maxmemory: 4GB

TEMPLATER_OVERRIDE={“redis”:{“maxmemory”: “4GB”}…}

Override attributesInject run context attributes as environment var

dgr : run a POD with systemd

[Service]ExecStart=/usr/bin/rkt run \ --set-env=TEMPLATER_OVERRIDE='{{.jsonAttrs}}' {{.acis}}

ggnManage services in fleet clusters

Pods Services and Environments

Different services that use the same pods

Environments (1 per DC for prod + dev…)

Abstraction of fleet commands

github.com/blablacar/ggn

INSERTLOGOHERE

ggn

├── prod-dc1│ └── attributes│ └── services│ └── redis-quota│ │ └── attributes│ │ └── service-manifest.yml│ │ └── unit.tmpl│ └── redis-rating├── preprod│ ...

Single directory treeDescribes the servicesThe environmentsVersioned in git

ggn : service manifest

containers: - aci.blbl.cr/pod-redis:3nodes: - hostname: redis-quota1 - hostname: redis-quota2 - hostname: redis-quota3

Define the servicePod image and versionNodes composing the service

ggn : run a POD

$ ggn prod-dc1 redis-quota update

$ ggn prod-dc1 redis-quota journal -f

github.com/blablacar/ggn

How do pods talk to each other ?

Service discovery

with nerve & synapseAirBnB’s smartstack

Dynamic topology of services

Scalability requires service discovery

We used smartstack of AirBnB

Since then we improved it and rewrote it in go

github.com/blablacar/go-nervegithub.com/blablacar/go-synapse

Service discovery/database/node1

XX

/database

?

What’s next ?

rktnetesOrchestrate all the things !

+

Questions?

@BlaBlaCarTechBlaBlaTech.com

@slallema

Thanks!@BlaBlaCarTech

BlaBlaTech.com