Mechanisms for Matchmaking and Parallel High Throughput Computing

in the Condor Distributed System

Rajesh Raman, raman@cs.wisc.edu

Todd Tannenbaum, tannenba@cs.wisc.edu

http://www.cs.wisc.edu/condorOct 27, 1997

Condor ProjectCondor Project

OverviewOverview• What is Condor ?What is Condor ?• Projects and CollaborationsProjects and Collaborations• High Throughput ComputingHigh Throughput Computing• ClassAds and MatchMakingClassAds and MatchMaking• Parallel Computing with CondorParallel Computing with Condor

What is Condor ?What is Condor ?

High Throughput ComputingHigh Throughput Computing Distributed ResourcesDistributed Resources

• Physically distributedPhysically distributed• Distributed ownershipDistributed ownership

Resource ManagementResource Management• Increase utilization of resourcesIncrease utilization of resources• Simple interface to execution environmentSimple interface to execution environment

– User level interfaceUser level interface– Application level interfaceApplication level interface

Important MechanismsImportant Mechanisms

MatchmakingMatchmaking Checkpointing (and migration)Checkpointing (and migration)

• Owner policies require resource reclamationOwner policies require resource reclamation• Need to save (resumable) state of application Need to save (resumable) state of application

Remote System CallsRemote System Calls• Preserves submission environment in Preserves submission environment in

execution environment.execution environment. SandboxingSandboxing

• Security concernsSecurity concerns

The Condor TeamThe Condor Team

Prof. Miron Livny, PIProf. Miron Livny, PI Research StaffResearch Staff

• Todd TannenbaumTodd Tannenbaum• Derek WrightDerek Wright• Adding 2 more...Adding 2 more...

Condor Team, cont.Condor Team, cont.

Graduate Students Graduate Students • Rajesh Raman (MatchMaking)Rajesh Raman (MatchMaking)• Jim Basney (Split Execution)Jim Basney (Split Execution)• Shrinivas Ashwin (Mr. Parallel)Shrinivas Ashwin (Mr. Parallel)• Adiel Yoaz (Accounting)Adiel Yoaz (Accounting)

Undergraduate StudentsUndergraduate Students• Tom StanisTom Stanis

Condor AlmuniCondor Almuni

• Mike LitzkowMike Litzkow• David DewittDavid Dewitt• Marvin SolomonMarvin Solomon• Many others… (Produced XXX Masters and Many others… (Produced XXX Masters and

XXX PhDs]XXX PhDs]

Current Collaborators and Current Collaborators and ProjectsProjects

NCSANCSA• PACIPACI• National GridNational Grid

UW-FlockUW-Flock• Intel Sponsorship: $4.2 Million Intel Sponsorship: $4.2 Million • Graduate School, EngineeringGraduate School, Engineering

metaNEOS: metacomputing metaNEOS: metacomputing environments for optimizationenvironments for optimization• with Prof. Michael Ferriswith Prof. Michael Ferris

Condor Pool InstallationsCondor Pool Installations

UniversitesUniversites• U of Wisconsin, U of Illinois, U of Michigan, U of Wisconsin, U of Illinois, U of Michigan,

Dartmouth, Duke, U of Washington, U of Dartmouth, Duke, U of Washington, U of Virginia, U of California-BerkeleyVirginia, U of California-Berkeley

GovernmentGovernment• NCSA, Nasa, US Navy, NSA, NIKHEF NCSA, Nasa, US Navy, NSA, NIKHEF

(Amsterdam), INFN (Italy)(Amsterdam), INFN (Italy) CommercialCommercial

• Hewlett-Packard Labs, J.P. Morgan, Mercedez-Hewlett-Packard Labs, J.P. Morgan, Mercedez-Benz, Dragon SystemsBenz, Dragon Systems

Power of Computing Power of Computing EnvironmentsEnvironments

Power = Work / TimePower = Work / Time High Performance ComputingHigh Performance Computing

• Fixed amount of work; how much time?Fixed amount of work; how much time?• Response time/latency orientedResponse time/latency oriented• Traditional Performance metrics: FLOPS, MIPSTraditional Performance metrics: FLOPS, MIPS

High Throughput ComputingHigh Throughput Computing• Fixed amount of time; how much work?Fixed amount of time; how much work?• Throughput orientedThroughput oriented• Application specific performance metricsApplication specific performance metrics

Distributed Ownership of Distributed Ownership of ResourcesResources

Commodity resourcesCommodity resources• Underutilized: 70% of a pool's cycles are not Underutilized: 70% of a pool's cycles are not

utilizedutilized• Fragmented: owned by different peopleFragmented: owned by different people

Can provide HTC with these cycles, BUTCan provide HTC with these cycles, BUT• Must not impact QOS to ownerMust not impact QOS to owner

Owners specify access policyOwners specify access policy• Expressed with control expressionsExpressed with control expressions

– The current state of the resource (e.g., load average)The current state of the resource (e.g., load average)– Characteristics of the request (e.g., who wants to use it?)Characteristics of the request (e.g., who wants to use it?)– Time of day, random numbers, etcTime of day, random numbers, etc

Condor ArchitectureCondor Architecture

Startds ( Represent owners of resources)Startds ( Represent owners of resources)• Implement owner's access control policyImplement owner's access control policy

Schedds( Represent customers of the Schedds( Represent customers of the system)system)• Maintain persistent queues of resource Maintain persistent queues of resource

requestsrequests ManagerManager

• Collector: Database of resourcesCollector: Database of resources• Negotiator: MatchmakerNegotiator: Matchmaker• Accountant: Priority maintenanceAccountant: Priority maintenance

Condor Architecture, cont.Condor Architecture, cont.

MatchmakingMatchmaking

Customers Customers • Require resources with certain characteristicsRequire resources with certain characteristics• Discriminating customersDiscriminating customers• Requests place constraints on resourcesRequests place constraints on resources

Distributed ownershipDistributed ownership• Resources service requests which match owner's Resources service requests which match owner's

policypolicy• Discriminating resourcesDiscriminating resources• Resource offers place constraints on customersResource offers place constraints on customers

Matchmaking is symmetricMatchmaking is symmetric

Matchmaking with Matchmaking with Classified AdvertisementsClassified Advertisements

Parties requiring matchmaking advertiseParties requiring matchmaking advertise• Characteristics and requirements (i.e., constraints)Characteristics and requirements (i.e., constraints)

Advertisements matched by a MatchmakerAdvertisements matched by a Matchmaker Matched parties contact each other to "claim”Matched parties contact each other to "claim”

• Communication, authentication, constraint Communication, authentication, constraint verification, negotiation of terms, etc.verification, negotiation of terms, etc.

• Claiming does not involve the MatchmakerClaiming does not involve the Matchmaker Method is symmetricMethod is symmetric

• No client/server relation imposedNo client/server relation imposed

Classified Advertisement Classified Advertisement Matchmaking FrameworkMatchmaking Framework

Expression and evaluation of Expression and evaluation of characteristicscharacteristics• ClassAd, Closure, EvaluationContextClassAd, Closure, EvaluationContext

Advertising ProtocolAdvertising Protocol• Contents of advertisementsContents of advertisements• Publication protocolPublication protocol

Matchmaking AlgorithmMatchmaking Algorithm• Relates ad contents to matching process Relates ad contents to matching process • Priority schemes, Ranking schemes, etc.Priority schemes, Ranking schemes, etc.

Classified Advertisement Classified Advertisement Matchmaking Framework Matchmaking Framework (contd.)(contd.) Matchmaking ProtocolMatchmaking Protocol

• How are relevant parties informed of a How are relevant parties informed of a successful match?successful match?

• What information are they given?What information are they given? Claiming ProtocolClaiming Protocol

• How do matched parties claim each other to How do matched parties claim each other to cooperate?cooperate?

ClassAd: Mechanism for ClassAd: Mechanism for expressing characteristicsexpressing characteristics

A ClassAd is a set of names, each of which is A ClassAd is a set of names, each of which is bound to an expression. e.g., bound to an expression. e.g., [ [ Name => "Joe Hacker" ; Height => 182 ; Sex => Name => "Joe Hacker" ; Height => 182 ; Sex => "Male" ; "Male" ; Disposition => (TimeOfDay() < 600) ? "Sour" : Disposition => (TimeOfDay() < 600) ? "Sour" : UNDEFINED ;UNDEFINED ; Requirements => (other.Height < Height) && Requirements => (other.Height < Height) && (other.Sex == (other.Sex == "Female")"Female") ] ]

ExpressionsExpressions• Constants, attribute references, function callsConstants, attribute references, function calls

ClassAd (contd.)ClassAd (contd.) Attribute references may refer to attributes Attribute references may refer to attributes

in other adsin other ads• Attribute references "trigger" expression Attribute references "trigger" expression

evaluationevaluation• Scope resolutionScope resolution• Evaluates to UNDEFINED if no such expression Evaluates to UNDEFINED if no such expression

existsexists Values Values

• String, integer, real, UNDEFINED and ERROR String, integer, real, UNDEFINED and ERROR typestypes

• Operators are total (i.e., defined over all values)Operators are total (i.e., defined over all values)

Closure: Evaluation Closure: Evaluation Environment for a ClassAdEnvironment for a ClassAd

Determines which Determines which ClassAd's attributes to ClassAd's attributes to lookuplookup

Closure isClosure is• ClassAd and an ordered ClassAd and an ordered

mapping of (scope-name, mapping of (scope-name, closure) pairsclosure) pairs

• No name may be repeatedNo name may be repeated

EvaluationContext: EvaluationContext: Evaluation Environment for Evaluation Environment for several ClassAdsseveral ClassAds A set of closures which is self-containedA set of closures which is self-contained

• No closure reference leaves the contextNo closure reference leaves the context• Condor's "Standard Context" is a bit more Condor's "Standard Context" is a bit more

complexcomplex– Includes closures for a matchmaker Includes closures for a matchmaker

"advertisement”"advertisement”

Matchmaking in CondorMatchmaking in Condor

Opportunistic Resource ExploitationOpportunistic Resource Exploitation• Resource availability is unpredictableResource availability is unpredictable

– Exploit resources as soon as they are availableExploit resources as soon as they are available– Return resources as soon as they are unavailableReturn resources as soon as they are unavailable

• Matchmaking performed continuouslyMatchmaking performed continuously Attractive for malleable parallel Attractive for malleable parallel

applicationsapplications• Request more resources after execution Request more resources after execution

commencescommences– Granted immediately if resources are available, orGranted immediately if resources are available, or– As soon as resources become availableAs soon as resources become available

Matchmaking in Condor Matchmaking in Condor (contd.)(contd.)

Advertising protocolAdvertising protocol• Startd's, Schedd's send classads to CollectorStartd's, Schedd's send classads to Collector• Must contain a "Requirements” expressionMust contain a "Requirements” expression

– Optionally contain a"Rank” and “CurrentRank” Optionally contain a"Rank” and “CurrentRank” expressionsexpressions

• Startds send a "private ad" containing a Startds send a "private ad" containing a capabilitycapability

Matchmaking protocolMatchmaking protocol• Give the matched Startd and Schedd the Give the matched Startd and Schedd the

capability from the startd's private adcapability from the startd's private ad

Matchmaking in Condor Matchmaking in Condor (contd.)(contd.)

Matchmaking AlgorithmMatchmaking Algorithm• Request ad A matched with offer ad B “iff”Request ad A matched with offer ad B “iff”

– A's "Requirements" expression evaluates to TRUE, andA's "Requirements" expression evaluates to TRUE, and– B's "Requirements" expression evaluates to TRUE, andB's "Requirements" expression evaluates to TRUE, and– B‘s"Rank" expression value is greater than B‘s"Rank" expression value is greater than

"CurrentRank", and"CurrentRank", and– A’s "Rank" expression value is its greatest when A’s "Rank" expression value is its greatest when

evaluated against Bevaluated against B

Claiming protocolClaiming protocol• Negotiate "heartbeat" frequency, checkpoint Negotiate "heartbeat" frequency, checkpoint

transfer, etc.transfer, etc.

Condor ParallelismCondor Parallelism

Job LevelJob Level• Condor clusters of processesCondor clusters of processes• DagManDagMan

Task LevelTask Level• Interfacing Condor and PVMInterfacing Condor and PVM

– PVM: Message PassingPVM: Message Passing– Condor: Resource ManagementCondor: Resource Management

• PVM Resource Manager InterfacePVM Resource Manager Interface– pvm_reg_rm()pvm_reg_rm()

Interfacing Condor and Interfacing Condor and PVM, cont.PVM, cont.

Interfacing Condor and Interfacing Condor and PVM, cont.PVM, cont.

CARMI -vs- PVM CARMI -vs- PVM • Resource RequestsResource Requests

– PVM: SynchronousPVM: Synchronous– CARMI: AsynchronousCARMI: Asynchronous

• Resource Request MechanismResource Request Mechanism– PVM: Hostname and Type StringPVM: Hostname and Type String– CARMI: ClassAdCARMI: ClassAd

– CARMI Resource ClassCARMI Resource Class

• Task ManagementTask Management– CARMI: Additional NotificationsCARMI: Additional Notifications– CARMI: Additional OperationsCARMI: Additional Operations

Master-Worker ModelMaster-Worker Model A good fit for an A good fit for an

opportunistic opportunistic environmentenvironment

MasterMaster• Runs on Submit MachineRuns on Submit Machine• Manages pool of tasksManages pool of tasks

WorkerWorker• Runs on remote Runs on remote

machinesmachines• Receives pieces of work Receives pieces of work

from the Master, returns from the Master, returns answeranswer

Shadow

Master

PVMd

Starter

PVMd

Worker

Starter

PVMd

Worker

Additional Condor/PVM Additional Condor/PVM FrameworksFrameworks

CoCheckCoCheck• Checkpoint a Worker or set of WorkersCheckpoint a Worker or set of Workers• Requirements for a consistent checkpointRequirements for a consistent checkpoint

– Synchronize all processesSynchronize all processes– Flush PVM messages in transitFlush PVM messages in transit– Perform Checkpoint (save image)Perform Checkpoint (save image)– Remap TIDsRemap TIDs

WoDiWoDi• A framework for Master-Worker applicationsA framework for Master-Worker applications• Performs optimizationsPerforms optimizations

Future WorkFuture Work

DebugDebug PortPort DebugDebug PortPort DebugDebug PortPort DebugDebug Port…..Port…..

Future Work Part IIFuture Work Part II

MatchmakingMatchmaking• Aggregate Resources/RequestsAggregate Resources/Requests

AccountingAccounting• AuthenticationAuthentication

FlockingFlocking Java UniverseJava Universe Split ExecutionSplit Execution

SummarySummary

Condor is an implementation of a High Condor is an implementation of a High Throughput Computing system in an Throughput Computing system in an opportunistic environment.opportunistic environment.

Major Mechanisms to achieve HTC:Major Mechanisms to achieve HTC:• MatchmakingMatchmaking• CheckpointingCheckpointing• Remote system callsRemote system calls• SandboxingSandboxing

Questions ? Questions ?