introduction to mesos and the datacenter operating system

© 2016 Mesosphere, Inc. All Rights Reserved.

Introduction to Mesos and the Datacenter Operating System Artem Harutyunyan ([email protected])

© 2015 Mesosphere, Inc. All Rights Reserved. 2

$ whoamiINTRO

ARTEM HARUTYUNYAN● ALICE Offline (2004 - 2010)

● AliEn (porting to Windows, authentication, scheduling, etc)

● Fellow at PH/SFT (2010 - 2012)

● CernVM Co-Pilot (LHC@home 2.0)

● CernVM (File System) Infrastructure

● Architect/Engineering Director at Qualys (2012 - 2015)

● Engineering Director at Mesosphere (2015 - )


Apache Mesos is a general purpose cluster manager.

WHAT IS APACHE MESOS?MESOS


Wait!? Like a batch system?

IS IT A BATCH SYSTEM?MESOS


NO!

IS IT A BATCH SYSTEM?MESOS


Master

Agents

Framework FrameworkFramework

6

MESOS

2-LEVEL SCHEDULING


MESOS

LEVEL OF INDIRECTION

Mesos (agents)

Framework

Mesos (master)

Framework

responsible for allocation (and reallocation) of resources

Framework


MESOS

LEVEL OF INDIRECTION

Mesos (agents)

Mesos (master)

framework

responsible for allocation (and reallocation) of resources

framework


AUTOMATED OPERATIONS OF DISTRIBUTED SYSTEMSGOAL

Software should manage itself.

● Most distributed systems are difficult to manage but they don’t need to be.


BUILT ON MESOSMESOS

2009 2010 2013 2014 2015


PORTED TO MESOSMESOS

2011 2012 2013 2014 2015


BREAK OUT OF TRADITIONAL INFRASTRUCTURE SILOSRESULT

• Many silos.

• Management nightmare.

• Lengthy cycles to deploy code.

• Low utilization.

Apache Mesos and the DC/OS

TRADITIONAL APPROACH

PaaS 1

ContainerApp

1Big Data

Analytics 1Big Data

Analytics 2

PaaS 2

ContainerApp

2Stateful Service

1

Stateful Service

2

Complete workload portability

UNIFIED APPROACH

ContainerApps(All) Big Data Analytics

(All)

PaaS (All)

Stateful Service(All)

• High performance and resource isolation.

• Easy scalability and multi-tenancy.

• Fault tolerant and highly available.

• Highly efficient with highest utilization.

Deploys on-prem or in cloud


THE DATACENTER COMPUTER


1. TREAT MACHINES AS CATTLE NOT PETS

Keep the base operating system small and simple, run “containerized” applications.

2. AUTOMATE WITH SOFTWARE NOT HUMANS

Let software schedule software, i.e., handle failures, improve utilization, and manage maintenance.

PRINCIPLES


DATACENTER KERNELMESOS

provides common functionality every new distributed system re-implements:

• resource allocation• resource deallocation• resource reservations• resource isolation• resource monitoring• failure detection• package distribution• task starting, killing, cleanup• volume management• …

don’t reinvent the wheel!


INTERESTING FEATURESMESOS

● Quota

● Oversubscription

● Maintenance

● Persistence Primitives

● External Storage

● Security

● CVMFS Integration (presented at Mesoscon NA 2016)

● Optimistic Offers (upcoming)

Distributed Systems Kernel (Mesos)

DATACENTER OPERATING SYSTEM (DC/OS)

Distributed systems kernel to abstract resources

18




User Interface (GUI & CLI)

Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage)

19

Datacenter Operating System (DC/OS)




User Interface (GUI & CLI)

Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage)

20

Datacenter Operating System (DC/OS)

On Premise AWS Azure


● Mesos!

● API, CLI and GUI

● Service Discovery & Load Balancing

● Storage Volumes

● Package Manager

● Installer for on premise and cloud

WHAT’S IN THE BOX?

24

DC/OS ● 100% open source (ASL2.0)+ Kicked off with a big, diverse

community● An umbrella for ~30 OSS projects

+ Roadmap and designs+ The build tool chain+ Docs and tutorials

● Not limited in any way● Familiar, with a few new features

25

THEUNIVERSE

27

THE COMMUNITY


Check outdcos.io!

THANK YOU

introduction to mesos and the datacenter operating system

Documents