ducks – distributed user-mode chirp-knowledgeable server
DESCRIPTION
DUCKS – Distributed User-mode Chirp-Knowledgeable Server. Joe Thompson Jay Doyle. DUCKS Motivation. 3. Usability. 2. Chirp & Condor. 1. Performance. DUCKS Goals. Bring together functionality of Condor and CHIRP in an easy to use package. Abstract Condor and CHIRP interfaces. - PowerPoint PPT PresentationTRANSCRIPT
DUCKS – Distributed User-mode Chirp-Knowledgeable ServerJoe ThompsonJay Doyle
Performance
Usability
Chirp &
Condor
2
3
1
DUCKS Motivation
DUCKS GoalsBring together functionality of Condor and CHIRP in an easy to use package. Abstract Condor and CHIRP interfaces.Intelligently distribute files over CHIRP servers.Provide simple interface for the Chirp Active Storage Program-To-Data model.Provide simple interface for the Condor Data-To-Program model.
MySQL
GarbageCollector
ChirpTrack
erTimeoutHandler
IncomingMessageQueue
IncomingMessageQueue
IncomingMessageQueue
ChirpServer
List
TransactionList
TransactionHandler
DUCKS
Client
ducks_put <username> <local_file> <ducks_name>
Local Machine
DUCKS Server
Chirp NodeStore_Request
<username> <ducks_name> <filesize>
1) Verify the <username/ducks_name> pair is not already in the DB
2) Find a Chirp node with enough free space to store the file
3) Query database for path name to use on the Chirp node
Store_Response<Chirp_node> <Path_on_node>
4) Parse response to get the Chirp storage location for <local_file>
chirp_put<local_file> <chirp_node>
<path_on_node>
Store_success<username> <ducks_name>
<chirp_node> <path_on_node>
5) Update DB to reflect this file storage
File
ducks_get <username> <ducks_name> <local_name>
Local Machine
DUCKS Server
Chirp Node
Get_Request<username> <ducks_name>
1) Query DB for Chirp location of <ducks_name>
Get_Response<Chirp_node> <Path_on_node>
2) Parse response to get the Chirp storage location for <ducks_name>
chirp_get<chirp_node> <path_on_node>
<local_file>
File
ducks_delete <username> <ducks_name>
Local Machine
DUCKS Server
Garbage Collectordelete_Request
<username> <ducks_name>
1) Set the delete_flag of the <username/ducks_name> entry in the DB
2) Periodically query the DB for files with the delete_flag set
DUCKS DB
<chirp_node> <path_on_node>……
Result Set3) Delete files in the list
Chirp Node
Delete File
4) Remove files from DB
ducks_ls <username> <search_string>
Local Machine
DUCKS Server
ls_Request<username> <search_string>
1) Query the DB for all files owned by the user that match “%<search_string>%”;
2) Start a background thread and pass it the result set containing the found filesResult
Set3) Iterate through the set and send the file information to the client
File_info
ls Thread
Distribution via Chirp Active Storage to nodes already storing input
Wrapper/
Submit Scripts
input01.txt
input02.txt
input03.txt
input04.txtexe/libs request
ducks_get
Chirp NodesClient
DUCKS Server
Job
Requ
est
Response with locations of input files
Client
Condor
exe/libs/input_file request
Chirp Nodes
DUCKS Server
Distribute tasks to any available node
Get input and exe/lib files from other nodes in the Chirp cluster with ducks_get
1 (x1000) 10 (x100) 100 (x10) 1000 (x1)0
5
10
15
20
25
30
Transfer Speed Comparison: DUCKS vs Standalone Chirp (avg. 3 runs)
ChirpDUCKS
File Size (MB) (x # Files)
Tran
sfer
Spe
eds
(MB/
s)
Condor DUCKS (JtoD) DUCKS(DtoJ)0
50
100
150
200
250
300
Execution Time Comparison: DUCKS vs Standalone Condor (avg. 15 runs)
Job Execution Method
Exec
utio
n Ti
me
(s)
DUCKS Future WorkThe basic framework is implemented.Add a more robust file interface.Implement DUCKS management of job status information. (Queued, Running, Complete)
?