continuous delivery with jenkins, docker and mesos/marathon - jbcnconf

CONTINUOUS DELIVERY WITH JENKINS, DOCKER

AND MARATHON/MESOSJulia Mateo

@juliamateodc@duchessswiss http://jduchess.ch/

Monday, June 29, 15

GOAL

Monday, June 29, 15

CONTINUOUS DEPLOYMENT VS CONTINUOUS DELIVERY

• Continuous delivery : series of practices designed to ensure that code can be rapidly and safely deployed to production

• Continuous deployment is the next step of continuous delivery: Every change that passes the automated tests is deployed to production automatically

Monday, June 29, 15

CONTINUOUS DELIVERY

Source : http://xebia.github.io/cd-with-docker/#/

Monday, June 29, 15

From https://www.docker.com/whatisdocker/

Using Docker for Continuous Delivery

• Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications.

Monday, June 29, 15

DOCKER VS VMS

Server

Host OS

Hypervisor

GuestOS

Guest OS

Guest OS

Guest OS

Libs/Bins

Libs/Bins

Libs/Bins

App 1 App 2 App 3 App 4

Libs/Bins

Server

Host OS

Docker Engine

Libs/Bins Libs/Bins

App 1 App 2 App 3 App 4

c1 c2 c3 c4

VM1 VM2 VM3 VM4

Monday, June 29, 15

CONTAINERS AND IMAGES

DockerImages Docker

Images

libcontainer, Union FilesystemCentos Ubuntu

Jettyadd App.warContainer 1

OracleDBContainer 2

Read only

WritableWritable

Read only

Monday, June 29, 15

TO SUM UP :

• Simple : container as deployment unit

• Fast : Only need to build/upload the ∆ image

• Robust : less scope of error than custom deployment scripts

• Other advantages : docker facilitates fast rollback

Monday, June 29, 15

FIRST STEP : DEPLOY TO ONE NODE

Monday, June 29, 15

THE WEB APP TO DEPLOY

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

docker publish plugin

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

docker publish plugin

ssh jenkins plugin

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

docker publish plugin

ssh jenkins plugin

Monday, June 29, 15

DOCKER PUBLISH PLUGIN

Monday, June 29, 15

USING DOCKER INTO DOCKER

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

docker publish plugin

ssh jenkins plugin

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

docker publish plugin

ssh jenkins plugin

Monday, June 29, 15

DOCKER PRIVATE REPOSITORY

• Add the registry’s location to the repository name : my.registry.address:port/repositoryname

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

docker publish plugin

ssh jenkins plugin

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

docker publish plugin

ssh jenkins plugin

Monday, June 29, 15

JENKINS SSH PLUGIN

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

docker publish plugin

ssh jenkins plugin

Node 1

Node 2

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

docker publish plugin

ssh jenkins plugin

Node 1

Node 2

Container 1

Container 2

Container 3

Container 4

Container 5

Monday, June 29, 15

PROBLEMS

• Scalability

• Fault tolerance

Monday, June 29, 15

NEXT STEP : DEPLOY TO A MESOS

CLUSTER

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

docker publish pluginnode1Container 1

Container 3

Container 4Container 5

Container 2

Http request plugin

Master

Executor

Marathon

Executor

Monday, June 29, 15

• Abstraction of cluster resources

• Share resources across multiple frameworks (versions of the same fwk)

• Resource fair sharing : alternative to static partitioning

•Data locality

What is Mesos ?

Monday, June 29, 15

MESOS

ZKMaster Master

Executor Executor Executor

Executor ExecutorExecutor

Executor Executor Executor

Slaves

Monday, June 29, 15

HOW DOES IT WORK ?

http://mesos.apache.org/documentation/latest/mesos-architecture/

Monday, June 29, 15

USING MARATHON WITH MESOS

•Marathon is a Mesos framework written in Scala

• Provides easy deployment of Docker containers

•Manages of long running apps

• Rest API for developers

Monday, June 29, 15

MARATHON

ZKMaster Master

Executor Executor Executor

Executor ExecutorExecutor

Executor Executor Executor

Slaves

Monday, June 29, 15

MARATHON

ZKMaster Master

Executor Executor Executor

Executor ExecutorExecutor

Executor Executor Executor

Slaves

Marathon

Monday, June 29, 15

DEPLOY WITH DOCKER : TEST ENVIRONMENT

Localhost

docker publish plugin

Http request plugin

Master

Executor

Marathon

Executor

Monday, June 29, 15

JENKINS : HTTP REQUEST PLUGIN

Monday, June 29, 15

DEMO

Monday, June 29, 15

USING HAPROXY FOR LOAD BALANCING

Monday, June 29, 15

DEPLOYING TO PRODUCTION

• Canary releasing

• Blue Green deployment

Monday, June 29, 15

FEEDBACK FROM DEVELOPER PERSPECTIVE

•Many technologies in a short time : ansible, docker, mesos, marathon..

• All of them are quite recent technologies

Monday, June 29, 15

FEEDBACK FROM DEVELOPER PERSPECTIVE

•Docker, Mesos/Marathon : developer friendly

• Big and growing community

Monday, June 29, 15

NEXT IMPROVEMENTS

• Fine grained proxy configuration, service discovery...

Monday, June 29, 15

THANK YOU !

QUESTIONS

?Monday, June 29, 15