mesos gets pluggable - introducing mesos modules

Kapil Arya & Niklas Nielsen

Mesos Gets Pluggable Introducing Mesos Modules

© 2015 Mesosphere, Inc. 2

Niklas [email protected]

Kapil [email protected]


Mesos Modules & HooksModules & HooksArwwwww


How and why modules was introduced in Mesos

Our humble thoughts on how modules and extensibility in Mesos can evolve in the future

How Mesos Modules work and give you concrete examples of modules in action

© 2015 Mesosphere, Inc.

Modules

5


Different organizationsDifferent needs

6

● Hardly anyone run clusters the same way○ Different scales○ Different hardware○ Different workloads○ Different external tooling○ Different security needs

One cluster with turbo chargers please


• Mesos was built with this in mind!• The subsystems are lightweight insight

and control over HTTP

• Excellent for tooling around

• Different subsystems can be enabled and configured in a modular way

• Most notable: Isolation mechanisms

Good news!

7


New “extensions” to subsystems like isolators had to be upstreamed

But…

● Mesos can be made even more customizable and extendable

● Not all organizations can share their work

● Support proprietary and experimental integrations

● Not create bespoke forks of Mesos

However...

8


• Tie into and control task launch

• Dynamically setup execution environments

• Pass signatures through Mesos

• All of this, transparently to the framework and user

We needed it to support bespoke security subsystems

9


• Be able to extend and replace any component in Mesos• Allocator algorithms

• Authentication mechanisms

• Advanced scheduling features like oversubscription

• Anything!

The general thought of Modules was bigger

10

Imagine ifI could write my own?


Modules are old news

Many large software systems support libraries to

• Extend behavior• Isolate and abstract complexity• Make this a configuration rather than a

build exercise

For example

• Browsers (Firefox)• Server software (Apache Webserver)• Linux kernel

11

Wish I had modules already


What is a module anyway?

Module, plugin, extension, library …

Adds or replace a full component

For example:

• An isolator (works together with existing ones) in the agent

• The allocator and authenticators in the master

12


And how about hooks?

More often than not, you don’t want to replace a full component

Just want to tie into events and their context

For example:

• Launch task requests at the master

• Launch task requests at the agent

• Exit and cleanup events

13

Psst - I just launched a task


And who is using it?

Powering new exciting features and integrations!

• Oversubscription modules• Static (fixed) estimator

• Dynamic estimator and QoS Controller, project Serenity

• Networking integration with project Calico

14


Module Mechanics


A demo!

16

● A hook module that tags TaskStatus messages


Components

17

Isolator InterfaceIsolator Module

Hook Module H1

Hook Module H2

Hook Interface

Mesos Master/Agent

Module library

ModuleManager

Module spec JSON

Initialization

Initialize subsystems

use module objects

Module libraryinitialize modules

get module object

readspec

call hooks


Initialization

18

● First phase: ○ load module libraries○ compatibility checks, etc.○ libprocess not available

● Second phase○ initialize a specific module○ module-specific parameters○ libprocess available


class TestHook : public Hook{public: Result<Labels> slaveTaskStatusLabelDecorator( const FrameworkID& frameworkId, const TaskStatus& status) { Labels labels; if (status.state() == TASK_RUNNING) { Label* newLabel = labels.add_labels(); newLabel->set_key("whereami"); newLabel->set_value("mesoscon"); } return labels; }};

A Hook Module

19

// Create and return an object or TestHook type. static Hook* createHook(const Parameters& parameters){ // Any initialization checks go here.

return new TestHook();}

// Declares a Hook module named ‘org_apache_mesos_TestHook'mesos::modules::Module<Hook> org_apache_mesos_TestHook( MESOS_MODULE_API_VERSION, MESOS_VERSION, "Apache Mesos", "[email protected]", "Test Hook module.", NULL, createHook);


{ "libraries": [ { "file": "/path/to/libmodule.so", "modules": [

{ "name": "org_apache_mesos_TestHookModule", "parameters": [ { "key": "agent_addr", "value": "agent.host.domain" }, { "key": "...", "value": "..." } ] } ] } ]}

Specifying Modules to Master/Agent

20


● Build without building Mesos○ Just have Mesos installed

● Modules compile into a shared libraries○ Multiple modules per library

● Specify modules on command line:mesos-agent.sh <master-parameters> --modules=file:///path/to/modules.json --isolation=”my_isolator” --hooks=”my_hook”

Using Modules

21


● Add/replace a full component● Implement the interface● Asynchronous (actor model)

● Existing modularized interfaces:○ Allocator○ Authentication○ Authorizer○ Isolator○ QoSController○ ResourceEstimator

Replacement Modules

22


● Listen/Intercept interesting calls● Occasionally modify the behavior

○ Trigger initialization/cleanup● Allows us to “tag” certain tasks, statuses, etc.

● Two broad categories○ Task launch sequence○ Status updates

Hook Modules

23


● Co-exists with the parent process ○ separate thread of execution

● Create Master/Agent http “listen” endpoints● No callbacks

Anonymous Modules

24

One module to rule them all!


● Do not block○ Hooks are synchronous○ Use libprocess/pthreads

● Exit semantics○ Avoid assertions

Writing Modules

25


● Logs○ stdout/stderr

● Run debug module with non-debug Master/Agent○ gdb

Debugging

26

What crashed the Master?


● Dependency on other modules● Compatibility within set of modules● Upgrade path

○ rebuild modules when updating Mesos

Dependency and Compatibility

27


Future Work


● Safeguard against unsafe modules○ Limit data exposure○ Execute modules in a separate process

● Module certification● ACL’s● Runtime functionality checks

○ whitelist services○ can it add routes or not

Better Safety and Security

29


● More module interfaces● Load/Unload a module without rebooting Master/Agent● Upgrade path● Express dependability on other modules● Inter-module communication● Non-C++ modules

Future Work

30

Thanks for [email protected] [email protected]

❏ Documentation: http://mesos.apache.org/documentation/latest/modules/

❏ Modules repo: https://github.com/mesos/modules

❏ Mailing list: [email protected]

mailto:[email protected]





mesos gets pluggable - introducing mesos modules

Technology