sphinx: a scheduling middleware for data intensive ... · data intensive applications on a grid...
TRANSCRIPT
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
1
Sphinx: A Scheduling Middleware for Data Intensive Applications on a Grid
Richard CavanaughUniversity of Florida
Collaborators:Janguk In, Sanjay Ranka, Paul Avery, Laukik Chitnis, Gregory Graham (FNAL), Pradeep Padala, Rajendra Vippagunta, Xing Yan
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
2
The Problem of Grid Scheduling
o Decentralised ownershipo No one controls the grid
o Heterogeneous compositiono Difficult to guarantee execution environments
o Dynamic availability of resourceso Ubiquitous monitoring infrastructure needed
o Complex policieso Issues of trusto Lack of accounting infrastructureo May change with time
o Information gathering and processing is critical!
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
3
A Real Life Exampleo Merge two grids into a single multi-VO
“inter-grid”
o How to ensure that o neither VO is harmed?o both VOs actually benefit?o there are answers to questions like:
o “With what probability will my job be scheduled and complete before my conference deadline?”
o Clear need for a scheduling middleware!
FNAL
Rice
UI
MIT
UCSD
UF
UW
Caltech
UM
UTA
ANL
IU
UC
LBL
SMU
OU
BU
BNL
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
4
Some Requirements for Effective Grid Scheduling
o Information requirementso Past & future dependencies of
the applicationo Persistent storage of
workflowso Resource usage estimationo Policies
o Expected to vary slowly over time
o Global views of job descriptions
o Request Tracking and Usage Statistics
o State information important
o Resource Properties and Statuso Expected to vary slowly with
timeo Grid weather
o Latency measurement important
o Replica managemento System requirements
o Distributed, fault-tolerant scheduling
o Customisabilityo Interoperability with other
scheduling systemso Quality of Service
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
5
Incorporate Requirementsinto a Framework
VDT Server
VDT Client
??
?
o Assume the GriPhyN Virtual Data Toolkit:o Client (request/job submission)
o Globus clientso Condor-G/DAGMano Chimera Virtual Data System
o Server (resource gatekeeper)o Globus serviceso RLS (Replica Location Service)o MonALISA Monitoring Serviceo etc
VDT Server
VDT Server
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
6
Incorporate Requirementsinto a Framework
o Assume the GriPhyN Virtual Data Toolkit:o Client (request/job submission)
o Globus clientso Condor-G/DAGMano Chimera Virtual Data System
o Server (resource gatekeeper)o Globus serviceso RLS (Replica Location Service)o MonALISA Monitoring Serviceo etc
VDT Server
VDT Client
o Framework design principles:o Information driveno Flexible client-server modelo General, but pragmatic and simple
o Implement now; learn; extend over time
o Avoid adding middleware requirements on grid resources
o Take what is offered!
?
Scheduler
VDT Server
VDT Server
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
7
The Sphinx Framework
Sphinx Server
VDT Client
VDT Server Site
MonALISA Monitoring Service
Globus Resource
Replica Location Service
Condor-G/DAGMan
Request Processing
Data Warehouse
Data Management
InformationGathering
Sphinx ClientChimera
Virtual Data System
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
8
Sphinx Scheduling Server
Sphinx Server
Control Process
Job Execution Planner
Graph Reducer
Graph Tracker
Job Predictor
Graph Data Planner
Job Admission Control
Message Interface
Graph Predictor
Graph Admission Control
Data Warehouse
Data Management
Information Gatherer
o Functions as the Nerve Centre
o Data Warehouseo Policies, Account Information,
Grid Weather, Resource Properties and Status, Request Tracking, Workflows, etc
o Control Processo Finite State Machine
o Different modules modify jobs, graphs, workflows, etc and change their state
o Flexibleo Extensible
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
9
Policy Constraints
o Defined by Resource Providerso Actual grid sites (resource centres)o VO management
o Applied to Request Submitterso VO, group, user, or even a proxy request (e.g. workflow)
o Valid over a Period of Timeo Can be dynamic (e.g. periodic) or constant
o Global accounting and book-keeping is necessary
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
10
Quality of Serviceo For grid computing to become economically viable, a
Quality of Service is neededo “Can the grid possibly handle my request within my required
time window?”o If not, why not? When might it be able to accommodate such
a request?o If yes, with what probability?
o But, grid computing today typically:o Relies on a “greedy” job placement strategies
o Works well in a resource rich (user poor) environmento Assumes no correlation between job placement choices
o Provides no QoS
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
11
Quality of Serviceo As a grid becomes resource limited,
o QoS becomes even more important!o “greedy” strategies may not be a good choice
o Strong correlation between job placement choices
o Sphinx is designed to provide QoS through time dependent, global views of o Requests (workflows, jobs, allocation, etc)o Policieso Resources
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
12
Resource Usage Estimation
o User Requirementso Upper limits on CPU, memory, storage, bandwidth usage
o Domain Specific Knowledgeo Applications are often known to depend logarithmically,
linearly, etc on certain input parameters, data size or type
o Historical Estimates o Record the performance of all applicationso Statistically estimate resource usage within some confidence
level
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
13
Data Management
o Smart Replication:o Graph based
o Examine and insert replication nodes to minimise overall completion time
o Distribute and collect required datao Particularly useful in data parallelism
o “Hot Spot” basedo Monitor current and historical data access
patterns and replicate to optimise future access
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
14
Data Management
o Smart Replication:o Graph based
o Examine and insert replication nodes to minimise overall completion time
o Distribute and collect required datao Particularly useful in data parallelism
o “Hot Spot” basedo Monitor current and historical data access
patterns and replicate to optimise future access
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
15
Early Sphinx Prototype Test Results
o Simple sanity checkso 120 canonical virtual data workflows
submitted to US-CMS Grido Round-robin strategy
o Equally distribute work to all siteso Upper-limit strategy
o Makes use of global information (site capacity)
o Throttle jobs using just-in-time planningo 40% better throughput (given grid
topology)
o Conclusion: Prototype is working!
D ist r ib ut io n o f jo b s across t he U S- C M S Grid T est b ed
60 60 60 6052
106
25
57
0
20
40
60
80
100
120
DGT IGT UCSD CALTECH
Sit es
Round Robin
Upper Limit
A verage t ime to co mplete a jo b
833 816 781723
3250
684 679659
0
500
1000
1500
2000
2500
3000
3500
DGT IGT UCSD CALTECH
Sites
Round Robin
Upper Limi t
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
16
Some Current and Future Activities
o Policy Based Schedulingo Quality of Serviceo Graph Partitioningo Data Parallelismo Prediction Moduleo Useful Views and Fusion of Monitoring Data
18.09.2003 Data Mining and Exploration Middleware for Distributed and Grid Computing – University of Minnesota
17
Conclusions
o Scheduling on a grid has unique requirementso Informationo System
o Decisions based on global views providing a Quality of Service are importanto Particularly in a resource limited environment
o Sphinx is an extensible, flexible grid middleware which o Already implements many required features for effective
global schedulingo Provides an excellent “workbench” for future activities!