efficiently sharing common data htcondor week 2015 zach miller (zmiller@cs.wisc.edu) center for high...

Post on 22-Dec-2015

219 Views

Category:

Documents

0 Downloads

Preview:

Click to see full reader

TRANSCRIPT

Efficiently SharingCommon Data

HTCondor Week 2015

Zach Miller (zmiller@cs.wisc.edu)Center for High Throughput Computing

Department of Computer SciencesUniversity of Wisconsin-Madison

› Input files are never reusedh From one job to the nexth Multiple slots on same machine

› Input files are transferred serially from the machine where the job was submitted

› This results in the submit machine often transferring multiple copies of the same file simultaneously (bad!), sometimes to the same machine (even worse!).

“Problems” with HTCondor

2

› Enter the HTCache!› Runs on the execute machine› Runs under the condor_master just like any

other daemon› One daemon serves all users of that

machine› Runs with same privilege as the startd

HTCache

3

› Cache is on disk

› Persists across restarts› Configurable size› Configurable cache replacement policy

HTCache

4

› The cache is shared

› All slots use same local cache › Even if user is different (data is data!)› Thus, the HTCache needs the ability to

write files into a job’s sandbox as the user that will run the job

HTCache

5

› Instead of fetching files from the shadow, the job instructs the HTCache to put specific files into the sandbox

› If the file is in the cache, the HTCache COPIES the file into the sandbox

› Each slot gets its own copy, in case the job decides to modify it. (As opposed to hard or soft-linking the file into the sandbox)

Preparing Job Sandbox

6

› If the file is not in the cache, the HTCache fetches the file directly into the sandbox and then possibly adds it to the cache

› Wait… possibly?

Preparing Job Sandbox

7

› Yes, possibly.

› Obvious case: File is larger than cache› Larger question: which files are the best to

keep?› Cache policy is one of those things where it

is rarely a “one solution works best in all cases”

Cache Policy

8

› There are 10 problems in Computer Science:h Cachingh Levels of Indirectionh Off-by-one errors

› Allow flexible caching by adding a level of indirection. Don’t use size, time, etc., but rather the “value” of a file.

Cache Policy

9

› How do we determine the value?› Another trick: punt to the admin!› The cache policy is implemented as a

plugin, using a dynamically loaded library:

double valuationFun (long size, long age, int stickiness,long uses, long bytes_seeded, long time_since_seed) {

return (stickiness – age) * size;

}

Cache Policy

10

› The plugin determines the “value” of a file using the input parameters:h File sizeh Time file entered cacheh Time last accessedh Number of hitsh “Stickiness” (This is a hint provided by the

submit node… more on that later)

Cache Policy

11

› When deciding whether or not to cache a file, the HTCache considers all files currently in the cache, plus the file under consideration

› Computes the “value” of each file› Finds the “maximum value cache” that fits

in the allocated size› May or may NOT include the file just

fetched

Cache Policy

12

› There is a submit-side component as well, although it has a slightly different roleh Does not have a dedicated disk cacheh Instead, serves all files requested by jobsh Periodically scans the queue, counts the

number of jobs that use each input file, and broadcasts this “stickiness” value to all HTCache daemons

Submit Node HTCache

13

› Suppose I have a cluster of 25 eight-core machines

› I have a 1GB input file common to all my jobs (a common scenario for say, BLAST)

› I submit 1000 jobs› Old way: Each time a job starts up it

transfers the 1GB file to the sandbox (1TB)

Example

14

› New way: Each of the 25 machines gets the file once, shares it among all 8 slots, and it persists across jobs

› Naïve calculation: 25GB transfer (as opposed to 1TB).

› Of course, this ignores competition for the cache.

Example

15

› This is where “stickiness” helps› If I submit a separate batch of 50 jobs using

a different 1GB input, the HTCache can look at the stickiness and decide not to evict the first 1GB file since 1000 jobs are scheduled to use it is opposed to 50

› It’s possible to write a cache policy tailored to your cluster’s particular workload

Example

16

› This already has huge advantages.› Even if cache does nothing useful and

makes all the wrong choices, it can do NO WORSE than the existing method of transferring file every time.

› A huge advantage: Multiple slots share same cache! (And this advantage grows as number of cores grows)

› Massively reduces network load on Schedd

Success!

17

HTCache Results

18

› Although the load is reduced, the Schedd is still the single source for all input files

However…

19

› What if there was a way to get the files from somewhere else?

› Maybe even bits of the files from multiple different sources?

› Peer-to-peer?› We already have an HTCache deployed on

all the execute nodes…

However…

20

BitTorrent

21

› The HTCache running on the submit node acts as a SeedServer

› It always has all pieces of files that may be read. If you recall, it is not managing a cache, only serving the already existing files in place.

› When a job is submitted, input files are then automatically added to the seed server

Submit Node w/ BitTorrent

22

› The HTCache uses BitTorrent to retreive the file directly into the sandbox first.

› Optionally adds the file to its own cache

› Thus, BitTorrent is used to transfer files even if they won't end up in the cache

Execute Node w/ BitTorrent

23

Putting It All Together

24

Putting It All Together

25

› “GradStudent-ware”h Was done as a class projecth Doesn’t yet meet the exceedingly high

standards for committing into our main code repository.

› BitTorrent traffic is completely independent from HTCondor. As such, doesn’t work with the shared_port daemon

Project Status

26

› Obvious statement of the year: Caching is good!

› Runner-up: Using peer-to-peer file transfer can be faster than one-to-many file transfer!

› However, the nature of scientific workloads and multi-core machines creates an environment where these are especially advantageous

Conclusion

27

› Thank you!

› Questions? Comments?

› Ask now, talk to me at lunch, or email me at zmiller@cs.wisc.edu

Conclusion

28

top related