containerised continuous delivery at low cost and web scale!
TRANSCRIPT
![Page 1: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/1.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 1
CONTAINERISED CONTINUOUS DELIVERY AT LOW COST AND WEB SCALE!
Sunil Shah
![Page 2: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/2.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 2
At the end of this talk, you should be able to:1. Understand how Apache Mesos & DC/OS, works2. Deploy and configure a continuous deployment pipeline onto
DC/OS
OBJECTIVESIntroduction
![Page 3: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/3.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 3
Why is continuous delivery interesting now?
WHY BOTHER?Introduction
Google Trends for continuous delivery (blue) and continuous deployment (red)
![Page 4: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/4.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 4
1. It’s much easier to get compute resources nowadays!● Doesn’t cost much (sometimes it’s even free - just ask a graduate student)● EVERY platform and their dog has an API
WHY BOTHER?Introduction
![Page 5: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/5.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 5
1. It’s much easier to get compute resources nowadays!2. It turns out getting sleep is good for you!
● The National Sleep Foundation recommends 7-9 hours of sleep per night● Container orchestrators take the manual pain out of waking up and rebooting
an application (to varying degrees of success)● Let your devs dev and ops sleep!
WHY BOTHER?Introduction
![Page 6: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/6.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 6
1. It’s much easier to get compute resources nowadays!
2. It turns out getting sleep is good for you!3. Containers mean you can!
● No need for humans to ssh in and `apt-get package install python-mylibrary123`.
● Is this the beginning of the robot uprising?
WHY BOTHER?Introduction
![Page 7: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/7.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 7
MESOSPHERE DATACENTER OPERATING SYSTEM (DC/OS)Introduction
● Datacenter-wide services to power your apps
● Turnkey installation and lifecycle management
● Today’s most popular services (and those yet to come)
DC/OS Universe
Mesosphere DC/OS
Any Infrastructure
● Container operations & big data operations
● Fault tolerance & high availability
● Open source & production proven at scale
● Requires only a modern Linux distro (Windows coming soon)
![Page 8: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/8.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
DC/OS - THE DEFINITIVE PLATFORM FOR MODERN APPS
Tupperware& Bistro
Borg & Omega
Apache Mesos Project
Mesosphere
DC/OS OSS Project
Proprietary
DCOS Launched
2016201520132000 2010
![Page 9: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/9.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 9
APACHE MESOS: THE STORY OF
![Page 10: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/10.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 10
THE BIRTH OF MESOSApache Mesos: The Story Of
TWITTER TECH TALK
The grad students working on Mesos give a tech talk at Twitter.
March 2010
APACHE INCUBATION
Mesos enters the Apache Incubator.
Spring 2009
CS262B
Ben Hindman, Andy Konwinski and Matei Zaharia create “Nexus” as their
CS262B class project.
MESOS PUBLISHED
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center is
published as a technical report.
September 2010
December 2010
![Page 11: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/11.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 11
What does an operating system provide?
● Resource management● Programming abstractions● Security● Monitoring, debugging, logging
GRAD STUDENTS LEARNED HOW TO SHAREApache Mesos: The Story Of
Sharing resources between batch processing frameworks:
● Hadoop● MPI● Spark
![Page 12: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/12.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 12
CLUSTERING YOUR RESOURCES FOR YOUApache Mesos: The Story Of
Apache Mesos is a cluster resource manager.
It handles:
● Aggregating resources and offering them to schedulers● Launching tasks (i.e. processes) on those resources● Communicating the state of those tasks back to schedulers● Tasks can be:
● Long running services● Ephemeral / batch jobs
![Page 13: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/13.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 13
A SIMPLE MESOS CLUSTER ON MONDAY MORNINGApache Mesos: The Story Of
master
agent
scheduler
“Sir, I have some spare resources: 4 CPUs, 8 GB of memory and 1 TB of disk.”
![Page 14: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/14.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 14
A SIMPLE MESOS CLUSTER ON MONDAY MORNINGApache Mesos: The Story Of
master
agent
scheduler
“Hey, scheduler, would you like some compute resources?”
![Page 15: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/15.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 15
A SIMPLE MESOS CLUSTER ON MONDAY MORNINGApache Mesos: The Story Of
master
agent
scheduler
“Not right now, but thanks!”
![Page 16: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/16.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 16
A SIMPLE MESOS CLUSTER ON MONDAY MORNINGApache Mesos: The Story Of
master
agent
scheduler
“Happy Monday! Here’s a bunch of work.”
![Page 17: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/17.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 17
A SIMPLE MESOS CLUSTER ON MONDAY MORNINGApache Mesos: The Story Of
master
agent
scheduler
“Changed your mind?”
![Page 18: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/18.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 18
A SIMPLE MESOS CLUSTER ON MONDAY MORNINGApache Mesos: The Story Of
master
agent
scheduler
“How’d you know? Mind running this for me please?”
![Page 19: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/19.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 19
A SIMPLE MESOS CLUSTER ON MONDAY MORNINGApache Mesos: The Story Of
master
agent
scheduler
“Still got those spare resources? This task wants to run on them. Let me know how it goes.”
![Page 20: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/20.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 20
A SIMPLE MESOS CLUSTER ON MONDAY MORNINGApache Mesos: The Story Of
master
agent
scheduler
“Sure thing, it’s running.”
![Page 21: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/21.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 21
A SIMPLE MESOS CLUSTER ON MONDAY MORNINGApache Mesos: The Story Of
master
agent
scheduler
“Your task is running! I’ll let you know if that changes.”
![Page 22: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/22.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 22
FAIRNESS FOR ALL SCHEDULERSApache Mesos: The Story Of
master
agent
scheduler C scheduler Dscheduler B scheduler Escheduler A
agentagent agent agent
“Looks like you’re not using your fair share of resources - want some more?”
![Page 23: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/23.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 23
HELPING YOUR OPERATOR SLEEP WELLApache Mesos: The Story Of
master 2
agent
scheduler C scheduler Dscheduler B scheduler Escheduler A
agentagent agent agent
master 3master 1
ZooKeeper
“I just started! Who’s in charge around here?”
![Page 24: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/24.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 24
HELPING YOUR OPERATOR SLEEP WELLApache Mesos: The Story Of
master 2
agent
scheduler C scheduler Dscheduler B scheduler Escheduler A
agentagent agent agent
master 3master 1
ZooKeeper
“Check in with master 1!”
![Page 25: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/25.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 25
MESOS CLUSTERS CAN BE REALLY, REALLY LARGEApache Mesos: The Story Of
m 2
scheduler C scheduler Dscheduler B scheduler Escheduler A
m 1
ZooKeeper
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
aa
m 3 m 4 m 5 m 6 m 7 m 8 m 9
scheduler C scheduler Dscheduler B scheduler Escheduler Ascheduler C scheduler Dscheduler B scheduler Escheduler A
scheduler C scheduler Dscheduler B scheduler Escheduler Ascheduler C scheduler Dscheduler B scheduler Escheduler A
![Page 26: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/26.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 26
NOT USING MESOS IS EXPENSIVE!Apache Mesos: The Story Of
time
![Page 27: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/27.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 27
MARATHON(OR, HOW TO RUN MICROSERVICES ON MESOS)
![Page 28: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/28.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 28
Mesos can’t run applications on its own (!)
That’s where schedulers like Apache Aurora and Marathon come in.
Marathon keeps your application running! A bit like a distributed “init.d”.
MARATHON TALKS TO MESOSMarathon
![Page 29: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/29.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 29
A SELF SERVE INTERFACE TO YOUR CLUSTER
Marathon
`
![Page 30: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/30.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved.
30
MARATHON MANAGES THE APPLICATION LIFECYCLEMarathon
master
Marathon
agentagentagent agent agent
apachecon-demo:1.0
{ "container": { "docker": { "image": "ssk2/apachecon-demo:1.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "apachecon-demo-app", "instances": 1, "mem": 128}
Just launched my product!
![Page 31: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/31.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 31
MARATHON MANAGES THE APPLICATION LIFECYCLEMarathon
master
Marathon
agentagentagent agent agent
apachecon-demo:1.0
{ "container": { "docker": { "image": "ssk2/apachecon-demo:1.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "apachecon-demo-app", "instances": 5, "mem": 128}
Featured on Hacker News! SCALE! SCALE!
apachecon-demo:1.0 apachecon-demo:1.0 apachecon-demo:1.0 apachecon-demo:1.0
![Page 32: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/32.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 32
MARATHON MANAGES THE APPLICATION LIFECYCLEMarathon
master
Marathon
agentagentagent agent agent
apachecon-demo:2.0
{ "container": { "docker": { "image": "ssk2/apachecon-demo:2.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "apachecon-demo-app", "instances": 5, "mem": 128}
Time for v2!
apachecon-demo:1.0 apachecon-demo:1.0 apachecon-demo:1.0 apachecon-demo:1.0
apachecon-demo:2.0 apachecon-demo:2.0 apachecon-demo:2.0 apachecon-demo:2.0
apachecon-demo:1.0
![Page 33: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/33.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 33
MARATHON MANAGES THE APPLICATION LIFECYCLEMarathon
master
Marathon
agentagentagent agent agent
apachecon-demo:2.0
{ "container": { "docker": { "image": "ssk2/apachecon-demo:2.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "apachecon-demo-app", "instances": 5, "mem": 128}
Time for v2!
apachecon-demo:1.0 apachecon-demo:1.0 apachecon-demo:1.0 apachecon-demo:1.0
apachecon-demo:2.0 apachecon-demo:2.0 apachecon-demo:2.0 apachecon-demo:2.0
![Page 34: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/34.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 34
MARATHON MANAGES THE APPLICATION LIFECYCLEMarathon
master
Marathon
agentagentagent agent agent
apachecon-demo:2.0
{ "container": { "docker": { "image": "ssk2/apachecon-demo:2.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "apachecon-demo-app", "instances": 5, "mem": 128}
Time for v2!
apachecon-demo:1.0 apachecon-demo:1.0apachecon-demo:2.0 apachecon-demo:2.0
apachecon-demo:2.0 apachecon-demo:2.0
![Page 35: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/35.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 35
MARATHON MANAGES THE APPLICATION LIFECYCLEMarathon
master
Marathon
agentagentagent agent agent
apachecon-demo:2.0
{ "container": { "docker": { "image": "ssk2/apachecon-demo:2.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "apachecon-demo-app", "instances": 5, "mem": 128}
Time for v2!
apachecon-demo:2.0 apachecon-demo:2.0 apachecon-demo:2.0 apachecon-demo:2.0
![Page 36: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/36.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 36
JENKINS ON DC/OS (AND WHY YOU SHOULD PROBABLY BE RUNNING IT LIKE THIS)
![Page 37: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/37.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 37
WHEN IT BEGAN
Jenkins on DC/OS
Jenkins master
Continuous Integration is soooo
futuristic and this interface is beautiful.
![Page 38: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/38.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 38
THE OLD WORLD
Jenkins on DC/OS
Jenkins master
Life is good.
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins master
Oh, is that Jenkins cluster still up? I do all my builds on a Raspberry Pi now.
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins agent
Jenkins master
My builds take hours to run!
Jenkins agent
![Page 39: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/39.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 39
JUST USE WHAT YOU NEED, WHEN YOU NEED IT AND SHARE THE LOVE RESOURCESJenkins on DC/OS
Mesos master
Mesos agentMesos agentMesos agent Mesos agent Mesos agent
Jenkins agent (master 1)
Jenkins agent (master 1)
Jenkins agent (master 3)
Jenkins agent (master 3)
Jenkins agent (master 3)
Jenkins master 1
Life is good.
Jenkins master 2 Jenkins master 3
Life is good.
Life is good.
![Page 40: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/40.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 40
JUST USE WHAT YOU NEED, WHEN YOU NEED IT AND SHARE THE LOVE RESOURCESJenkins on DC/OS
Mesos master
Mesos agentMesos agentMesos agent Mesos agent Mesos agent
Jenkins agent (master 3)
Jenkins agent (master 3)
Jenkins agent (master 3)
Jenkins agent (master 3)
Jenkins agent (master 3)
Jenkins master 1
Team dinner! Let’s bounce.
Jenkins master 2 Jenkins master 3
Life is good.
Life is good.
![Page 41: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/41.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 41
BUILDING DOCKER IN DOCKER: ONE WEIRD TRICK
Jenkins on DC/OS
This brave new world of containers running in containers has a bit of a nesting problem.
We run everything inside a container to make it easy to bundle dependencies and to isolate it from other processes.
But when the thing that’s running wants to build a container, what do you do?
![Page 42: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/42.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 42
BUILDING DOCKER IN DOCKER: ONE WEIRD TRICK
Jenkins on DC/OS
One recommended approach is to bind mount in the host system’s Docker daemon.
Mesos agent
`docker run ssk2/apachecon-demo:f4ccd67``docker run mesosphere/jenkins:0.2.2`
Docker daemon
![Page 43: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/43.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 43
BUILDING DOCKER IN DOCKER: ONE WEIRD TRICK
Jenkins on DC/OS
This doesn’t work for Mesos though! It doesn’t track containers that it doesn’t launch.
The sibling container becomes orphaned and runs forever.
Mesos agent
`docker run ssk2/apachecon-demo:f4ccd67`
Docker daemon
![Page 44: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/44.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 44
BUILDING DOCKER IN DOCKER: ONE WEIRD TRICK
Jenkins on DC/OS
Our solution is to use a customised Docker-in-Docker container.
This is a little slower but Mesos takes care of the resources!
Mesos agent
`docker run mesosphere/jenkins:0.2.2`
Docker daemon
`docker run ssk2/apachecon-demo:f4ccd67`
![Page 45: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/45.jpg)
© 2016 Mesosphere, Inc. All Rights Reserved. 45
CONTINUOUS DEPLOYMENT
![Page 46: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/46.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 46
PIPELINE: A FIRST PASSContinuous Deployment
![Page 47: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/47.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 47
1. Configuration2. Build3. Deploy
![Page 48: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/48.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 48
Building a CD pipeline requires configuration in a couple of places:
1. Docker and Marathon files in your repo
2. Build configuration in Jenkins**in the future, you’ll be able to check in your build configuration alongside your repository too!
1. CONFIGURATIONContinuous Deployment
![Page 49: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/49.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 49
DEPENDENCY MANAGEMENT Docker is becoming the de-facto container format for
packaging applications:
● Encapsulates dependencies● Runs on your laptop● Runs on your cluster
DC/OS has native support for Docker.
Just stick a Dockerfile (or two) in the root of your repository!
1. CONFIGURATION
![Page 50: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/50.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 50
DEPENDENCY MANAGEMENT
FROM jekyll/jekyllADD site /srv/jekyll
1. CONFIGURATION
![Page 51: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/51.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Marathon application definitions are JSON files that describe:
● resources required● how many instances to run● what command to run● how to check your application is healthy
marathon.json should live in the root of your project repository.
51
APPLICATION CONFIGURATION
1. CONFIGURATION
![Page 52: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/52.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
{
"id": "apachecon-demo",
"container": {
"type": "DOCKER",
"docker": {
"image": "ssk2/cd-demo:latest",
"network": "BRIDGE",
"portMappings": [{
"containerPort": 80,
"protocol": "tcp"
}]
}
},
"labels": {
"HAPROXY_0_VHOST": "sunil-889-publicsl-781ifozhqg3z-1399492298.us-west-2.elb.amazonaws.com",
"HAPROXY_GROUP": "external"
},
"instances": 1,
"cpus": 0.1,
"mem": 128
}
52
APPLICATION CONFIGURATION
1. CONFIGURATION
![Page 53: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/53.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Continuous Deployment
2. BUILDING
53
It’s trivial to install Jenkins on DCOS:
1. Create a JSON file:{"jenkins": {"framework-name": "my-jenkins" }}
2. Install:
$ dcos package install --options=my-jenkins-config.json jenkins:
3. ???4. Profit!
![Page 54: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/54.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Continuous Deployment
2. BUILDING
54
Now, set up Jenkins:
1. Save your Docker Hub credentials
2. Set up triggered build to build and push Docker image
docker build . -t ssk2/whereisbot:$(GIT_BRANCH)
docker push ssk2/whereisbot:$(GIT_BRANCH)
3. Set up triggered build to update marathon.json using jq and PUT to Marathon
http PUT https://dcos/service/my-marathon/v2/app/ssk2/whereisbot < marathon.json
![Page 55: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/55.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Continuous Deployment
2. BUILDING
55
Next, let’s create a build:
1. Set up a build that polls GitHub periodically to build and push Docker imagedocker login -u ${DOCKER_HUB_USERNAME} -p ${DOCKER_HUB_PASSWORD} -e [email protected]
docker build -t ssk2/apachecon-demo:$(GIT_COMMIT) .
docker push ssk2/apachecon-demo:$(GIT_COMMIT)
2. Add a Marathon post deploy step pointing to the DC/OS Marathon:● Set any variables you’d like to override.
![Page 56: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/56.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 56
When you PUT to Marathon’s API, you trigger a deployment.
http PUT https://dcos/service/my-marathon/v2/app/ssk2/whereisbot < marathon.json
Marathon attempts to scale application to desired state by:
● Launching new instances● By default try to launch 100% of instances requested at once
● Killing old instances when new instances are healthy
3. DEPLOYINGContinuous Deployment
![Page 57: Containerised Continuous Delivery at Low Cost and Web Scale!](https://reader034.vdocuments.us/reader034/viewer/2022051522/58a1aa8f1a28abcf5a8ba452/html5/thumbnails/57.jpg)
THANK YOU!
57
Come and talk to us!● Email me at [email protected]● Slides will be up at http://mesosphere.github.io/presentations● Check out https://dcos.io