ducks – distributed user-mode chirp- knowledgeable server joe thompson jay doyle
TRANSCRIPT
![Page 1: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/1.jpg)
DUCKS – Distributed User-mode Chirp-Knowledgeable ServerJoe ThompsonJay Doyle
![Page 2: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/2.jpg)
Performance
Usa
bili
ty
Chirp &
Condor
2
3
1
DUCKS Motivation
![Page 3: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/3.jpg)
DUCKS Goals
Bring together functionality of Condor and CHIRP in an easy to use package.
Abstract Condor and CHIRP interfaces.
Intelligently distribute files over CHIRP servers.
Provide simple interface for the Chirp Active Storage Program-To-Data model.
Provide simple interface for the Condor Data-To-Program model.
![Page 4: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/4.jpg)
MySQL
GarbageCollector
ChirpTrack
er
TimeoutHandler
IncomingMessageQueue
IncomingMessageQueue
IncomingMessageQueue
ChirpServer
List
TransactionList
TransactionHandler
DUCKS
Client
![Page 5: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/5.jpg)
ducks_put <username> <local_file> <ducks_name>
Local Machine
DUCKS Server
Chirp Node
Store_Request<username> <ducks_name>
<filesize>
1) Verify the <username/ducks_name> pair is not already in the DB
2) Find a Chirp node with enough free space to store the file
3) Query database for path name to use on the Chirp node
Store_Response<Chirp_node> <Path_on_node>
4) Parse response to get the Chirp storage location for <local_file>
chirp_put<local_file> <chirp_node>
<path_on_node>
Store_success<username> <ducks_name>
<chirp_node> <path_on_node>
5) Update DB to reflect this file storage
File
![Page 6: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/6.jpg)
ducks_get <username> <ducks_name> <local_name>
Local Machine
DUCKS Server
Chirp Node
Get_Request<username> <ducks_name>
1) Query DB for Chirp location of <ducks_name>
Get_Response<Chirp_node> <Path_on_node>
2) Parse response to get the Chirp storage location for <ducks_name>
chirp_get<chirp_node> <path_on_node>
<local_file>
File
![Page 7: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/7.jpg)
ducks_delete <username> <ducks_name>
Local Machine
DUCKS Server
Garbage Collectordelete_Request
<username> <ducks_name>
1) Set the delete_flag of the <username/ducks_name> entry in the DB
2) Periodically query the DB for files with the delete_flag set
DUCKS DB
<chirp_node> <path_on_node>……
Result Set
3) Delete files in the list
Chirp Node
Dele
te
File
4) Remove files from DB
![Page 8: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/8.jpg)
ducks_ls <username> <search_string>
Local Machine
DUCKS Server
ls_Request<username> <search_string>
1) Query the DB for all files owned by the user that match “%<search_string>%”;
2) Start a background thread and pass it the result set containing the found filesResult
Set
3) Iterate through the set and send the file information to the client
File_info
ls Thread
![Page 9: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/9.jpg)
Distribution via Chirp Active Storage to nodes already storing input
Wrapper/
Submit Scripts
input01.txt
input02.txt
input03.txt
input04.txtexe/libs request
ducks_get
Chirp NodesClient
DUCKS Server
Job
Request
Response with locations of input files
![Page 10: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/10.jpg)
Client
Condor
exe/libs/input_file request
Chirp Nodes
DUCKS Server
Distribute tasks to any available node
Get input and exe/lib files from other nodes in the Chirp cluster with ducks_get
![Page 11: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/11.jpg)
![Page 12: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/12.jpg)
![Page 13: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/13.jpg)
![Page 14: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/14.jpg)
![Page 15: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/15.jpg)
![Page 16: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/16.jpg)
1 (x1000) 10 (x100) 100 (x10) 1000 (x1)0
5
10
15
20
25
30
Transfer Speed Comparison: DUCKS vs Standalone Chirp (avg. 3 runs)
ChirpDUCKS
File Size (MB) (x # Files)
Tra
nsfe
r S
peed
s (
MB
/s)
![Page 17: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/17.jpg)
Condor DUCKS (JtoD) DUCKS(DtoJ)0
50
100
150
200
250
300
Execution Time Comparison: DUCKS vs Standalone Condor (avg. 15 runs)
Job Execution Method
Execu
tio
n T
ime (
s)
![Page 18: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/18.jpg)
DUCKS Future Work
The basic framework is implemented.
Add a more robust file interface.
Implement DUCKS management of job status information. (Queued, Running, Complete)
![Page 19: DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle](https://reader035.vdocuments.us/reader035/viewer/2022062422/56649ea95503460f94bad9fe/html5/thumbnails/19.jpg)
?