microservices on dc/os and container orchestration on...
TRANSCRIPT
![Page 1: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/1.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 1
Microservices on DC/OS and Container Orchestration on Mesos
Gilbert Song(宋子豪)
![Page 2: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/2.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 2
● Apache Mesos PMC, Committer● Mesosphere Distributed Systems
Engineer● M.S. of Computer Engineering from
University of California, Santa Barbara● Focus on Mesos Containerization● Passionate about Cloud Computing and
Distributed Systems
E-mail: [email protected]
WeChat: songzihao888358
Who am I
![Page 3: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/3.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 3
● From monolithic to microservices● Microservices on Mesosphere DC/OS● The architecture of DC/OS● Apache Mesos overview and fundamentals● Container standards/specifications supported by Mesos● Container Orchestration on Apache Mesos● Why should I pick Mesos● Latest features
Outline
![Page 4: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/4.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 4
From monolithic to microservices
4
Release Plan
Developer
Developer
Developer
Developer
Developer
QA Release Integration OPs Replace Old
with New Release
Bugs
Bugs
![Page 5: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/5.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved.
From monolithic to microservices
55
Release Plan
Developer
Developer
Developer
Developer
Developer
QA Release Integration
QA Release Integration
QA Release Integration
OPs Replace Old with New Release
OPs Replace Old with New Release
OPs Replace Old with New Release
Bugs
![Page 6: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/6.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 66
THE HISTORY OF INFRASTRUCTURE
VIRTUALMAINFRAME PHYSICAL (x86)UNIFIED
HYPERSCALE
![Page 7: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/7.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 77
TRADITIONAL IT APPROACH
PaaS 1
ContainerApp
1Big Data
Analytics 1Big Data
Analytics 2
PaaS 2
ContainerApp
2 Stateful Service
1
Stateful Service
2
![Page 8: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/8.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 88
INFRASTRUCTURE EVOLUTION
![Page 9: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/9.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 99
THE NEXT WAVE OF COMPUTING
VIRTUAL UNIFIED HYPERSCALEMAINFRAME PHYSICAL (x86)
![Page 10: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/10.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 1010
THE MESOSPHERE DC/OS APPROACH
Mesosphere DCOS
ContainerApps(All) Big Data Analytics
(All)
PaaS (All)
Stateful Service(All)
![Page 11: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/11.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 11
Datacenter Operating System (DC/OS)
Distributed Systems Kernel (Mesos)
DC/OS ENABLES MODERN DISTRIBUTED APPS
Big Data + Analytics EnginesMicroservices (in containers)
Streaming
Batch
Machine Learning
Analytics
Functions & Logic
Search
Time Series
SQL / NoSQL
Databases
Modern App Components
Distributed systems kernel to abstract resources
Ecosystem of frameworks & apps
Consistent architecture to run on top of kernel
User Interface (GUI & CLI)
Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage)
Any Infrastructure (Physical, Virtual, Cloud)11
![Page 12: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/12.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 1212
SILOED WORKLOADS
Typical Datacentersiloed, over-provisioned servers,
low utilization
Industry Average12-15% utilization
mySQL
microservice
Cassandra
Spark/Hadoop
Kafka
![Page 13: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/13.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 1313
DC/OS MULTIPLEXING
Typical Datacentersiloed, over-provisioned servers,
low utilization
DC/OS Datacenterautomated schedulers, workload multiplexing onto the
same machines
Industry Average12-15% utilization
DC/OS Multiplexing30-40% utilization, up to 96% at some customers
4X
mySQL
microservice
Cassandra
Spark/Hadoop
Kafka
![Page 14: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/14.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 1414
MESOSPHERE DC/OS Architecture
Overview
Virtual MachinesPhysical Servers Private Cloud Public Cloud
Security &Governance
Container Orchestration
Monitoring & Operations
User Interface & Command Line
HDFS Jenkins Marathon Cassandra Kubernetes
Spark Docker Rocket MongoDB +30 more...
Existing Infrastructure
Mesosphere DCOS
Services & Containers
![Page 15: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/15.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 1515
● What does a traditional OS kernel provide?○ Resource management Host cpu, memory, etc.○ Programming abstractions POSIX API: processes, threads, etc.○ Security and isolation Virtual memory, user, etc.
● Mesos: A kernel for data center applications○ Resource management Cluster cpu, memory, etc.○ Programming abstractions Mesos API: Task, Resource, etc.○ Security and isolation Containerization
Mesos: A kernel for data center applicationsMesos overview and fundamentals
![Page 16: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/16.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 1616
● Key concepts○ Framework○ Resource/Offer○ Task○ Executor
Programming abstractionsMesos overview and fundamentals
Master
Agent
Framework
Executor
Task Task
Executor
Task
Offer (Resources) Task/Executor
Resources Task/Executor
![Page 17: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/17.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 1717
Case study: Marathon Mesos overview and fundamentals
Master
Agent X
Marathon
OfferX: 8 cpus, 16G mem
Decline Offer
8 cpus, 16G mem
![Page 18: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/18.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 1818
Create a Marathon appMesos overview and fundamentals
Master
Agent X
Marathon
Executor
Task
OfferX: 8 cpus, 16G mem
Accept OfferLAUNCH(Task: 2 cpus, 2G mem)
POST /v2/apps
![Page 19: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/19.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 1919
Create a Marathon appMesos overview and fundamentals
Master
Agent X
Marathon
Executor
Task
TASK_RUNNING
TASK_RUNNING OfferX: 6 cpus, 14G mem
![Page 20: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/20.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 2020
A typical Mesos clusterMesos overview and fundamentals
Master
Agent
Marathon
Agent Agent Agent Agent Agent Agent Agent
Kafka Cassandra MarathonSpark
Master Master
Zookeeper
![Page 21: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/21.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 2121
Mesos helps improve cluster utilizationMesos overview and fundamentals
time
time
![Page 22: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/22.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 2222
● Production ready
● Proven scalability
● Highly customizable and extensible
Why should I pick Mesos?Why Mesos?
![Page 23: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/23.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 23
Production Ready
![Page 24: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/24.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 2424
PRODUCTION CUSTOMERS AND MESOS USERS
Proven reliable for large scale, mission-critical deployments
![Page 25: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/25.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 2525
POWERED BY MESOS
![Page 26: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/26.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 26
The history of MesosWhy Mesos?
CS262B
Ben Hindman, Andy Konwinski and Matei Zaharia create “Nexus” as their CS262B class project.
Spring 2009
TWITTER TECH TALK
The grad students working on Mesos give a tech talk at Twitter.
December 2010
September 2010
March 2010
MESOS PUBLISHED
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center is published as a technical report.
APACHE INCUBATION
Mesos enters the Apache Incubator.
![Page 27: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/27.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 27
The history of MesosWhy Mesos?
MESOSPHERE
Mesosphere is formed by engineers who have been using Mesos at Twitter and AirBnB.
April 2013
MESOS GRADUATES
Mesos graduates from the Apache Incubator to become a top level project.
August 2015
April 2015
June 2013
APPLE ANNOUNCES J.A.R.V.I.S.
Apple announces that the Siri infrastructure now runs on Mesos, atop “thousands” of nodes.
VERIZON SCALE DEMO
Verizon demonstrates launching 50,000 containers in less than 90 seconds using Mesos and Mesosphere’s Marathon scheduler.
![Page 28: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/28.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 28
Proven Scalability
![Page 29: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/29.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 2929
● Largest Mesos cluster○ > 30000 nodes○ > 250K containers
![Page 30: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/30.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 3030
Apple
● Siri is powered by Mesos!
![Page 31: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/31.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 3131
Verizon
● 50K containers in 50 seconds
![Page 32: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/32.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 3232
● Stateless master○ Inspired from the GFS design○ Agents hold truth about running tasks (distributed)○ Master state can be reconstructed when agents register
● Simple, only cares about○ Resource allocation and isolation○ Task management
● Implemented in C++○ Native performance○ No GC issue
Why Mesos is so scalable?Why Mesos?
![Page 33: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/33.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 3333
● Known that Mesos will scale to Twitter/Apple level○ Feature is easy to add, took time to make it scalable
● Quality assurance for free○ Imagine a test environment having 30k+ nodes with real workload
● Take backwards compatibility seriously○ We don’t want to break their production environment
What does it mean to you?Why Mesos?
![Page 34: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/34.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 34
Highly Customizable and Extensible
![Page 35: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/35.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 3535
● Every company’s environment is different○ Scheduling○ Service discovery○ Container image format○ Networking○ Storage○ Special hardware/accelerators (e.g., GPU, FPGA)
● No one-fits-all solution typically
Why this is important?Why Mesos?
![Page 36: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/36.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 3636
Pluggable schedulersWhy Mesos?
● For instance, you need separate schedulers for○ Long running stateless services○ Cron jobs○ Stateful services (e.g., database, DFS)○ Batch jobs (e.g., map-reduce)
● Monolithic scheduler?Monolithic schedulers do not make it easy to add new policies and specialized implementations, and may not scale up to the cluster sizes we are planning for.
--- From Google Omega Paper (EuroSys’13)
Mesos frameworks== pluggable schedulers
![Page 37: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/37.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 3737
Flexible service discoveryWhy Mesos?
● Mesos is not opinionated about service discovery○ DNS based○ ZK/Etcd/Chubby based (e.g., twitter, google, with client libraries)○ Your custom way, every company is different○ Mesos provides an endpoint to stream SD information
● DNS based solution does not scale wellLarger jobs create worse problems, and several jobs many be running at once. The variability in our DNS load had been a serious problem for Google before Chubby was introduced.
--- From Google Chubby paper (OSDI’06)
![Page 38: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/38.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 3838
● Container image format● Networking● Storage● Security● Custom isolation● Container lifecycle hooks
Pluggable and extensible containerizationWhy Mesos?
![Page 39: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/39.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 39
● Container Image● OCI (Open Container Initiative)● Docker● Appc
● Container Network● CNI
● Container Storage● Docker Volume (dvdi)● CSI (new Container Storage Interface)
Container standards/specifications supported by Mesos
![Page 40: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/40.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 4040
● Support Container Network Interface (CNI) from 1.0○ A spec for container networking○ Supported by most network vendors
● Implemented as an isolator○ --isolation=network/cni,...
Container network supportUnified Containerizer
![Page 41: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/41.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 4141
● Proposed by CoreOS : https://github.com/containernetworking/cni
● Simple contract between container runtime and CNI plugin defined in the form of a JSON schema○ CLI interface○ ADD: attach to network○ DEL: detach from network
Container Network Interface (CNI)Unified Containerizer
Mesos Agent
Containerizer
Container
Executor
T1 T2
CNI Plugin
IPAM
veth
Network
![Page 42: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/42.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 42
● Simpler and less dependencies than Docker CNM● Backed by Kubernetes community as well● Rich plugins from network vendors● Clear separation between container and network management● IPAM has its own pluggable interface
42
Why CNI?Unified Containerizer
![Page 43: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/43.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 4343
Existing CNI plugins● ipvlan● macvlan● bridge● flannel● calico● contiv● contrail● weave● …
CNI pluginsUnified Containerizer
You can write your own plugin, and Mesos supports it!
![Page 44: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/44.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 4444
● Support Docker volume plugins from 1.0○ Define the interface between container runtime and storage provider○ https://docs.docker.com/engine/extend/plugins_volume/
● A variety of Docker volume plugins○ Ceph○ Convoy○ Flocker○ Glusterfs○ Rexray
Container storage supportUnified Containerizer
![Page 45: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/45.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 45
CSIUnified Containerizer
![Page 46: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/46.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 46
● Design issues with docker volume plugins● Issues with other storage spec:
● Kubernetes Flex Volume● Libstorage
● Need a new container storage spec
Why CSI
![Page 47: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/47.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 4747
ContainerizerContainerization in Mesos
47
Containerizer
● Between agents and containers● Launch/update/destroy containers● Provide isolations between containers● Report container stats and status
Mesos Master Mesos Master Mesos Master
Zookeeper
Marathon Framework
Cassandra Framework
Mesos Agent
Containerizer
Container
Executor
T1 T2
Mesos Agent
Containerizer
Container
Executor
T1 T2
Mesos Agent
Containerizer
Container
Executor
T1 T2
![Page 48: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/48.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 4848
Docker containerizer● Delegate to Docker daemon
Mesos containerizer● Using standard OS features (e.g.,
cgroups, namespaces)● Pluggable architecture allowing
customization and extension
Currently supported containerizersContainerization in Mesos
Very stable. Used in large scale production clusters
![Page 49: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/49.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 4949
Docker containerizer● Delegate to Docker daemon
Mesos containerizer● Using standard OS features (e.g.,
cgroups, namespaces)● Pluggable architecture allowing
customization and extension● Support Docker, Appc, OCI (soon)
images natively w/o dependency
Currently supported containerizersContainerization in Mesos
Very stable. Used in large scale production clusters
![Page 50: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/50.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 5050
Docker containerizer● Delegate to Docker daemon
Unified containerizer● Using standard OS features (e.g.,
cgroups, namespaces)● Pluggable architecture allowing
customization and extension● Support Docker, Appc, OCI (soon)
images natively w/o dependency
Currently supported containerizersContainerization in Mesos
Very stable. Used in large scale production clusters
![Page 51: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/51.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 5151
Start from 0.28, you can run your Docker container on Mesos without a Docker daemon installed!
● One less dependency in your stack● Agent restart handled gracefully, task not affected● Compose well with all existing isolators● Easier to add extensions
Container image supportUnified Containerizer
![Page 52: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/52.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 5252
● Mesos supports multiple container image format○ Docker (without docker daemon)○ Appc (without rkt)○ OCI (ready soon)○ CVMFS (experimental)○ Host filesystem with tars/jars○ Your own image format!
Pluggable container image formatUnified Containerizer
Used in large scale production clusters
![Page 53: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/53.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 53
● Unified containerizer● GPU support● Nested container and task group (Pods)● Debug container● Multi role and hierarchy role
Latest Features
![Page 54: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/54.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 5454
● New in Mesos 1.1○ Building block for supporting Pod like feature
● Highlighted features○ Support arbitrary levels of nesting○ Re-use all existing isolators○ Allow dynamically creation of nested containers
Nested container supportNested container support
![Page 55: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/55.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 5555
Nested container supportNested container support
Mesos Master Mesos Master Mesos Master
Zookeeper
Marathon Framework
Cassandra Framework
Mesos Agent
Containerizer
Container
Executor
T1 T2
Mesos Agent
Containerizer
Container
Executor
T1 T2
Mesos Agent
Containerizer
Container
Executor
T1 T2
Container
Executor
T1 T2
Nested Container Nested Container
![Page 56: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/56.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 5656
New Agent API for Nested ContainersNested container support
message agent::Call {
enum Type {
// Calls for managing nested containers
// under an executor's container.
LAUNCH_NESTED_CONTAINER = 14;
WAIT_NESTED_CONTAINER = 15;
KILL_NESTED_CONTAINER = 16;
}
}
![Page 57: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/57.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 5757
Launch nested containerNested container support
Container
ExecutorMesos Agent
Containerizer
LAUNCH
Nginx
![Page 58: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/58.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 5858
Watch nested containerNested container support
Container
ExecutorMesos Agent
Containerizer
WAIT
NginxExit Status = 0
![Page 59: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/59.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 5959
Arbitrary levels of nestingNested container support
Container
Executor
Nginx
Mesos Agent
Containerizer
LAUNCHDebug
![Page 60: Microservices on DC/OS and Container Orchestration on Mesospic.huodongjia.com/ganhuodocs/2017-06-15/1497513595.14.pdfDC/OS MULTIPLEXING Typical Datacenter siloed, over-provisioned](https://reader030.vdocuments.us/reader030/viewer/2022041014/5ec54e9845156e34a17d00ba/html5/thumbnails/60.jpg)
© 2017 Mesosphere, Inc. All Rights Reserved. 60
Demo