what’s new in work queue
DESCRIPTION
What’s New in Work Queue. Michael Albrecht, University of Notre Dame CCL Workshop, June 2012. Overview. New “Batch Job” Systems Moab / Cluster MPI Queue Work Queue Enhancements Hierarchical Work Queue. Batch Job Systems. Abstraction Layer Library for generic task submission - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/1.jpg)
What’s New in Work What’s New in Work QueueQueue
Michael Albrecht, University of Notre Dame
CCL Workshop, June 2012
![Page 2: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/2.jpg)
2
Overview
New “Batch Job” SystemsMoab / ClusterMPI Queue
Work Queue EnhancementsHierarchical Work Queue
![Page 3: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/3.jpg)
3
Batch Job Systems
Abstraction Layer Library for generic task submission Used by Makeflow
![Page 4: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/4.jpg)
4
Moab & other Clusters
Similar to “SGE” Explicitly support Moab scheduler Support other, similar schedulers
Set name, submit and remove commands
moab
![Page 5: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/5.jpg)
5
MPI Queue
Workflow
W WW WW W W W W
W W W W W
![Page 6: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/6.jpg)
6
MPI Queue
Workflow
W W W W W
W W W W F
![Page 7: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/7.jpg)
7
MPI Queue
Enables arbitrary computation on “MPI-only” clusters
Ranks 1-N talk to Rank 0, which acts as a “foreman”
Assumes shared parallel filesystem for cluster
API very similar to Work QueueFully supported by Batch Job/Makeflow
![Page 8: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/8.jpg)
Work Queue
![Page 9: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/9.jpg)
9
Work Queue is Wonderful
Easily harness 100’s-1000’s of cores
Combine multiple resources for one project
Dynamically scale computational resources
PrivateCluster
CampusCondor
Pool
PublicCloud
Provider
SharedSGE
Cluster
Makefile
Makeflow
Local Files and Programs
sge_submit_workers
W
W
W
ssh
WW
WW
W
Wv
W
condor_submit_workers
W
W
W
![Page 10: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/10.jpg)
10
Work Queue has Limits
Bandwidth, file size, and computation length constrain potential number of workers
![Page 11: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/11.jpg)
11
Work Queue has Limits
Natural parallelism of a workflow leaves extra workers idle
T1 T2 T3 T4
T0
T5
???W5W1 W2 W3 W4
![Page 12: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/12.jpg)
12
Work Queue has Limits
Synchronous transfer leaves network resources idle, increases dispatch time
![Page 13: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/13.jpg)
Add more indirection!
“All problems in computer science can be solved by another layer of indirection”
-David Wheeler
“…except for the problem of too many layers of indirection”
-Kevlin Henney
![Page 14: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/14.jpg)
14
Hierarchical Work Queue
Master(Makeflow)
Worker Worker Worker Worker Worker Worker Worker Worker
![Page 15: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/15.jpg)
15
Hierarchical Work Queue
Master(Makeflow)
Worker Worker Worker Worker Worker Worker Worker Worker
Foreman Foreman
![Page 16: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/16.jpg)
16
Hierarchical Work Queue
Master(Makeflow)
Worker Worker Worker Worker Worker Worker Worker Worker
Foreman Foreman
Shared FS
![Page 17: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/17.jpg)
17
Hierarchical Work Queue
Master(Makeflow)
Worker Worker Worker Worker Worker Worker Worker Worker
Foreman ForemanShared FS Shared FS
![Page 18: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/18.jpg)
18
Shared FS
Hierarchical Work Queue
Master(Makeflow)
Worker Worker Worker Worker Worker Worker Worker Worker
Foreman ForemanShared FS Shared FS
Foreman Foreman
![Page 19: What’s New in Work Queue](https://reader030.vdocuments.us/reader030/viewer/2022033106/56812cf0550346895d91bbf7/html5/thumbnails/19.jpg)
19
Hierarchical Work Queue
19
http://www.nd.edu/~cclhttp://www.nd.edu/~ccl
Coming
Soon!