cluster management systems - github pages · computation model: frameworks l a framework (e.g.,...
TRANSCRIPT
ClusterManagementSystemsIonStoicaCS294
September26,2016
Motivationl Rapidinnovationincloudcomputing
l Todayl Nosingleframeworkoptimalforallapplicationsl Eachframeworkrunsonitsdedicatedclusterorcluster
partition
Dryad
Pregel
CassandraHypertable
2011slide
ComputationModel:Frameworksl Aframework (e.g.,Hadoop,MPI)managesoneormorejobs inacomputercluster
l Ajob consistsofoneormoretasksl Atask (e.g.,map,reduce)isimplementedbyoneormoreprocessesrunningonasinglemachine
3
cluster
FrameworkScheduler(e.g.,JobTracker)
Executor(e.g.,TaskTracker)
Executor(e.g.,TaskTraker)
Executor(e.g.,TaskTracker)
Executor(e.g.,TaskTracker)
task1task5
task3task7 task4
task2task6
Job1: tasks1,2,3,4Job2:tasks5,6,7
2011slide
OneFrameworkPerClusterChallengesl Inefficientresourceusage
l E.g.,HadoopcannotuseavailableresourcesfromPregel’scluster
l Noopportunityforstat.multiplexing
l Hardtosharedatal Copyoraccessremotely,expensive
l Hardtocooperatel E.g.,NoteasyforPregeltouse
graphsgeneratedbyHadoop
4
Hadoop
Pregel
0%#
25%#
50%#
0%#
25%#
50%#
Hadoop
Pregel
2011slideNeedtorunmultipleframeworksonsamecluster
Whatdowewant?
l Commonresourcesharinglayerl Abstracts(“virtualizes”)resourcestoframeworksl Enablediverseframeworkstoshareclusterl Makeiteasiertodevelopanddeploynewframeworks(e.g.,Spark)
5
MPIHadoopMPIHadoop
ResourceManagementSystem
Uniprograming Multiprograming
2011slide
FineGrainedResourceSharing
l Taskgranularitybothintime&spacel Multiplexnode/timebetweentasksbelongingtodifferent
jobs/frameworks
l Taskstypicallyshort;median~=10sec,minutes
l Whyfinegrained?l Improvedatalocalityl Easiertohandlenodefailures
62011slide
Goals
l Efficientutilizationofresources
l Supportdiverseframeworks (existing&future)
l Scalability to10,000’sofnodes
l Reliability infaceofnodefailures
2011slide
Approach:GlobalScheduler
8
GlobalScheduler
OrganizationpoliciesResourceavailability
• Responsetime• Throughput• Availability• …
Jobrequirements
2011slide
Approach:GlobalScheduler
9
GlobalScheduler
OrganizationpoliciesResourceavailability
• TaskDAG• Inputs/outputs
JobrequirementsJobexecutionplan
2011slide
Approach:GlobalScheduler
10
GlobalScheduler
OrganizationpoliciesResourceavailability
• Taskdurations• Inputsizes• Transfersizes
JobrequirementsJobexecutionplan
Estimates
2011slide
Approach:GlobalScheduler
l Advantages:canachieveoptimalschedulel Disadvantages:
l Complexityà hardtoscaleandensureresiliencel Hardtoanticipatefuture frameworks’ requirementsl Needtorefactorexistingframeworks
11
GlobalScheduler
OrganizationpoliciesResourceavailability
TaskscheduleJobrequirementsJobexecutionplan
Estimates
2011slide
Motivations,NowandThenl Recallmainmotivationsin2011:
l Efficientresourceusagel Enablerapidinnovationl Mainusecase:bigdataframeworks
l Whathaschangedsincethen?
12
WhathasChangedSinceThen?l Workloads:runlongrunningapplications(e.g.,fron-endservices)emergedasacommonworkloadl Noneedforfinegrainallocation
l Containershaveevolvedfromanisolationmechanismtoapackagingsolution(e.g.,Docker)l Containerorchestrationamajorusecaseàmanageentire
applifecycle:develop,test,deploy
l Cloudhasbecameprevalentl EasytoscaleupusingPAYG(statisticmultiplexingless
important) 13
Howtoclassifysystems?
l Whatisthegranularityofresourcesharing?l Tasksvsinstanceallocation
l Howdotheymakeresourcemanagementdecisions?l Centralizedvs.distributedallocationl Whomapsresourcestoapps,andwhomapsresourcesto
tasks?l Whatresourcemanagementpolicestheyenforce?
14
ThisLecture
l Mesos
l Yarn
l Omega(andBorg)
l Kubernetes
15