mesoscon 2016: frameworks in harmony

© 2016 Mesosphere, Inc. All Rights Reserved. 1

FRAMEWORKS IN HARMONY

MesosCon 2016


BRENDEN MATTHEWSEngineer

Core Mesos committer, contributor to numerous Mesos frameworks, and various other projects in the Mesos

ecosystem. He spends his free time contemplating existentialism, cooking steak, and craigslist furniture hacking.

@brndnmtthws

VISHNU MOHANSolutions Engineer

Works closely with customers in the field on Performance and Automation. When he’s not analyzing sandbox logs or coding

against some API for fun, he may be found rock climbing, hiking, running or scuba diving.

@vm_mesos


AGENDAMesosCon 2016: Frameworks in Harmony


● Problem Definition, Motivations and Applications● Intro to Roles, Weights, Quotas, Reservations, Revocation & Oversubscription● Theory

● Dominant Resource Fairness (DRF)● Hierarchical Dominant Resource Fairness (H-DRF)

● Mesos Quotas● Scenarios and putting it all together● Demo on

AGENDAMesosCon 2016: Frameworks in Harmony


MOTIVATIONS/APPLICATIONS

MesosCon 2016: Frameworks in Harmony


MOTIVATIONS

Running multiple environments on one cluster

Guaranteeing resource availabilityAPPLICATIONS

Multitenancy

Isolation

Introducing new frameworks

MOTIVATIONS/APPLICATIONSMesosCon 2016: Frameworks in Harmony


DEFINITIONSMesosCon 2016: Frameworks in Harmony


ROLES

A method of identifying frameworks for use with reservations, auth, ACLs, and other Mesos features.

WEIGHTS

Control the relative share of resources that is offered to different roles

INVERSE OFFERS

Mechanism to reclaim resources for maintenance, reallocation and pre-emption

DEFINITIONSMesosCon 2016: Frameworks in Harmony

RESERVATIONS

Mechanism to reserve resources on specific agents (Static and Dynamic)

QUOTAS

Lay away resources for consumption via cluster-wide dynamic reservation

OVERSUBSCRIPTION

Temporarily take advantage of unused resources


DOMINANT RESOURCE FAIRNESS(DRF)



DRF

Core resource allocation algorithm for Mesos

Generalization of Max-Min Fairness to multiple resources

WHY MAX-MIN FAIRNESS?

Generality

Ability to provide performance isolationhttps://www.cs.berkeley.edu/~alig/papers/drf.pdf

WEIGHTED MAX-MIN FAIRNESS

Priority

Reservation

Deadline-Based Allocation

User is guaranteed to receive her fair share irrespective of the demands of others

Applicable to a wide variety of resources

DOMINANT RESOURCE FAIRNESSMesosCon 2016: Frameworks in Harmony

https://www.cs.berkeley.edu/~alig/papers/drf.pdf



DRF CORE CONCEPTS

In a multi-resource environment, the allocation of a user should be determined by the user’s “dominant share”, which is the maximum share that the user has been allocated of any resource

DRF then seeks to maximize the minimum dominant share across all users

DRF CORE PROPERTIES

Sharing Incentive

Strategy Proofness

Pareto Efficiency

Envy Freeness

NICE-TO-HAVE PROPERTIES

Single Resource Fairness

Bottleneck Fairness

Population Monotonicity

Resource Monotonicity

DOMINANT RESOURCE FAIRNESS contd.MesosCon 2016: Frameworks in Harmony


EXAMPLE

Total Capacity: 9 CPUs, 18 GB RAM

ROLE A

<1 CPU, 4 GB>, Dominant Resource: Mem

<x CPU, 4x GB>, Dominant Share: 2/9 (i.e., 4/18)

ROLE B

<3 CPUs, 1 GB>, Dominant Resource: CPU

<3y CPU, y GB>, Dominant Share: 1/3 (i.e., 3/9)

DRF IN ACTIONMesosCon 2016: Frameworks In Harmony

SOLUTION

x = 3, y = 2


DRF IN ACTION contd.MesosCon 2016: Frameworks In Harmony


DRF ALLOCATION SCHEDULE (Quantum 1)MesosCon 2016: Frameworks in Harmony

ScheduleRole A <1 CPU, 4 GB> Role B <3 CPU, 1 GB> CPU

Total Alloc.RAM

Total Alloc.res. shares dom. share res. shares dom. share

Role B <0,0> 0 <3/9, 1/18> 1/3 3/9 1/18




Total Alloc.RAM


Role B <0,0> 0 <3/9, 1/18> 1/3 3/9 1/18

Role A <1/9, 4/18> 2/9 <3/9, 1/18> 1/3 4/9 5/18




Total Alloc.RAM


Role B <0,0> 0 <3/9, 1/18> 1/3 3/9 1/18

Role A <1/9, 4/18> 2/9 <3/9, 1/18> 1/3 4/9 5/18

Role A <2/9, 8/18> 4/9 <3/9, 1/18> 1/3 5/9 9/18




Total Alloc.RAM


Role B <0,0> 0 <3/9, 1/18> 1/3 3/9 1/18

Role A <1/9, 4/18> 2/9 <3/9, 1/18> 1/3 4/9 5/18

Role A <2/9, 8/18> 4/9 <3/9, 1/18> 1/3 5/9 9/18

Role B <2/9, 8/18> 4/9 <6/9, 2/18> 2/3 8/9 10/18


DRF ALLOCATION SCHEDULE (Quantum 5) - CPU Maxed OutMesosCon 2016: Frameworks in Harmony


Total Alloc.RAM


Role B <0,0> 0 <3/9, 1/18> 1/3 3/9 1/18

Role A <1/9, 4/18> 2/9 <3/9, 1/18> 1/3 4/9 5/18

Role A <2/9, 8/18> 4/9 <3/9, 1/18> 1/3 5/9 9/18

Role B <2/9, 8/18> 4/9 <6/9, 2/18> 2/3 8/9 10/18

Role A <3/9, 12/18> 2/3 <6/9, 2/18> 2/3 1 14/18


DRF ALLOCATION AND RESOURCE CONSUMPTIONMesosCon 2016: Frameworks In Harmony


HIERARCHICALDOMINANT RESOURCE FAIRNESS(H-DRF)https://www.cs.berkeley.edu/~alig/papers/h-drf.pdf


https://www.cs.berkeley.edu/~alig/papers/h-drf.pdf

https://www.cs.berkeley.edu/~alig/papers/h-drf.pdf


MESOSROLES



USE CASES

● Ensuring that resources on a particular agent are only offered to a particular framework

● Partitioning a cluster between organizations: resources reserved for use by Organization A will only be offered to frameworks that have registered using Organization A’s role (also, Reservations)

● Expressing that one group of frameworks has “higher priority” (and offered more resources) than another group of frameworks (also, Weights)

● Setting a guaranteed resource allocation for one or more frameworks belonging to a role (also, Quotas)

● Ensuring that persistent volumes created by one framework are not offered to frameworks registered with a different role

MESOS ROLESMesosCon 2016: Frameworks in Harmony


MESOS QUOTAS



QUOTA

Quotas are a mechanism for dynamically reserving resources.

WHY QUOTA?

DRF may offer resources to frameworks beyond their fair share. Operators can use quotas to ensure that frameworks always receive enough resources.

MESOS QUOTASMesosCon 2016: Frameworks in Harmony

QUOTA CORE CONCEPTS

Quotas provide a mechanism for guaranteeing resources for roles. Frameworks may register with that role, and will be allocated at least the specified amount of resources for that role.

In other words, quotas are an implementation of cluster-wide dynamic resource reservations by role.

© 2016 Mesosphere, Inc. All Rights Reserved.

LIMITATIONS

Can’t specify granularity or constraints (e.g., can’t distribute resources evenly across fault domains).

Cannot update an existing quota (must be removed and re-added).

The current implementation is both the minimum and the limit. No more offers will be sent once all reserved resources are allocated for that role.

25

QUOTA FEATURES

Minimum Guarantee

Dynamically Allocated

Configured at Runtime

Applies Per Role

Cluster-wide

Easy Reasoning

Complementary to Dynamic Reservations

MESOS QUOTAS contd.MesosCon 2016: Frameworks in Harmony


QUOTA USAGEMesosCon 2016: Frameworks in Harmony

core@ip-10-0-6-151 ~ $ curl -d '{"role":"myRole","guarantee":[{"name":"cpus","type":"SCALAR","scalar":{"value":1}},{"name":"mem","type":"SCALAR","scalar":{"value":100}}]}' -X POST http://master.mesos:5050/quota

core@ip-10-0-6-151 ~ $ curl http://master.mesos:5050/quota{"infos":[{"guarantee":[{"name":"cpus","role":"*","scalar":{"value":1.0},"type":"SCALAR"},{"name":"mem","role":"*","scalar":{"value":100.0},"type":"SCALAR"}],"role":"myRole"}]}

core@ip-10-0-6-151 ~ $ curl -X DELETE http://master.mesos:5050/quota/myRole

core@ip-10-0-6-151 ~ $ curl http://master.mesos:5050/quota{}


DEMO!MesosCon 2016: Frameworks in Harmony


CONCLUSIONS & FUTURE WORK



Q&AMesosCon 2016: Frameworks in HarmonyThursday, 11:00am: Quota: Reliable

Guarantees - Jörg Schad & Joris Van Remoortere, Mesosphere - Grand Mesa A, http://sched.co/6jtn

Thursday, 12:00pm: Maintenance: Automating Collaboration Between Frameworks and Operators - Joseph Wu, Mesosphere - Chasm Creek (this room), http://sched.co/6jtt

Thursday, 2:00pm: Resource Allocation in Apache Mesos: Present and Future - Benjamin Mahler, Mesosphere - Grand Mesa A, http://sched.co/6jtm


VISIT OUR BOOTHLocated at D1, near the main entrance

Learn more by visiting dcos.io and mesosphere.com

Vishnu’s Office HoursThursday 10-12:50pm


Charlie


ReferencesMesosCon 2016: Frameworks in Harmony

DRF: https://www.cs.berkeley.edu/~alig/papers/drf.pdf

H-DRF: http://www.cs.berkeley.edu/~alig/papers/h-drf.pdf

https://www.usenix.org/conference/nsdi11/dominant-resource-fairness-fair-allocation-multiple-resource-types

Mesos Docs:

https://www.cs.berkeley.edu/~alig/papers/mesos.pdf

https://mesos.apache.org/documentation/latest/roles/

https://mesos.apache.org/documentation/latest/weights/

https://mesos.apache.org/documentation/latest/attributes-resources/

https://mesos.apache.org/documentation/latest/reservation/

https://mesos.apache.org/documentation/latest/quota/

https://mesos.apache.org/documentation/latest/oversubscription/

n-level Hierarchies: https://issues.apache.org/jira/browse/MESOS-1763


http://www.cs.berkeley.edu/~alig/papers/h-drf.pdf














mesoscon 2016: frameworks in harmony

Software