![Page 1: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/1.jpg)
Douglas ThainComputer Sciences
DepartmentUniversity of Wisconsin-
Madison
(In Bologna for June 2000)
[email protected]://www.cs.wisc.edu/
condor
Remote I/O in Condor
![Page 2: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/2.jpg)
www.cs.wisc.edu/condor
Outline
› Introduction
› Using Remote I/O
› Implementation
› Build Your Own: Bypass
› Conclusion
![Page 3: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/3.jpg)
www.cs.wisc.edu/condor
Introduction
› Distributed systems provide you with access to a diverse array of machines. INFN Condor Pool (200) UW/CS Condor Pool (500) The Grid: (10000s?)
› Although you have permission to use these machines, they may be unfriendly to your application.
![Page 4: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/4.jpg)
www.cs.wisc.edu/condor
Introduction (Cont.)
› Remote I/O is an adapter which provides a friendly execution environment on an unfriendly machine.
› Condor uses remote I/O to homogenize the many machines in a Condor pool.
› Can we adapt this to the Grid?
![Page 5: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/5.jpg)
www.cs.wisc.edu/condor
What is Unfriendly?
› Programs can technically execute: Correct CPU and OS and enough memory
› But missing some critical items: No input files. No space for output files. No shared filesystem. No login - run as "nobody"?
![Page 6: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/6.jpg)
www.cs.wisc.edu/condor
Range of Unfriendliness
› Anonymous compute node on the Grid: Run as "nobody", with no access to disk.
› Machine at other institution: Can login, have some disk, but no file system.
› Machine down the hall: Can login, share one NFS mount, but not
another.
![Page 7: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/7.jpg)
www.cs.wisc.edu/condor
Why use an unfriendly machine?
› After all, homogeneous clusters are the norm: 10s or 100s of identical
machines. Centrally administrated. Shared filesystem
ClusteClusterr
FileServer
![Page 8: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/8.jpg)
www.cs.wisc.edu/condor
Because:You need more
machines!› Another hundred idle machines
could be found across the street or in the next department..
ClusteClusterr
FileServer
ClusteClusterr
FileServer
![Page 9: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/9.jpg)
www.cs.wisc.edu/condor
You need more machines! (Cont.)
› But, your application may not find the resources it needs.
ClusteClusterr
FileServer
ClusteClusterr
FileServer
HELP!
![Page 10: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/10.jpg)
www.cs.wisc.edu/condor
You need more machines! (Cont.)
› The problem is worse when we consider a global data Grid of many resources!
ClusterFile
Server
FileServer
FileServer
Cluster
FileServer
ClusterFile
Server
Cluster
Cluster
FileServer
HELP!HELP!
HELP!HELP!
HELP!
![Page 11: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/11.jpg)
www.cs.wisc.edu/condor
Solution: Remote I/O
› Condor remote I/O creates a friendly environment on an unfriendly machine.
ClusteClusterr
FileServer
ClusteClusterr
FileServer
Just like home!
![Page 12: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/12.jpg)
www.cs.wisc.edu/condor
Outline
› Introduction
› Using Remote I/O
› Implementation
› Build Your Own: Bypass
› Conclusion
![Page 13: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/13.jpg)
www.cs.wisc.edu/condor
Using Remote I/O
› Condor provides several "universes": Vanilla – Unmodified UNIX jobs Standard - UNIX jobs + remote I/O
Scheduler Globus (Not described here) PVM/MPI
![Page 14: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/14.jpg)
www.cs.wisc.edu/condor
Which Universe?
ClusteClusterr
FileServer
ClusteClusterr
FileServer
VANILLAVANILLASTANDARDSTANDARD
![Page 15: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/15.jpg)
www.cs.wisc.edu/condor
Vanilla Universe
› Submit any sort of UNIX program to the Condor system.
› Advantages: No relinking required. Any program at all, including
• Binaries• Shell scripts• Interpreted programs (java, perl)• Multiple processes
![Page 16: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/16.jpg)
www.cs.wisc.edu/condor
Vanilla Universe (Cont.)
› Disadvantages: No checkpointing. Very limited remote I/O services.
• Specify input files explicitly.• Specify output files explicitly.
Condor will refuse to start a vanilla job on a machine that is unfriendly.
• ClassAds: FilesystemDomain and UIDDomain
![Page 17: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/17.jpg)
www.cs.wisc.edu/condor
Standard Universe
› Submit a specially-linked UNIX application to the Condor system.
› Advantages: Checkpointing for fault tolerance. Remote I/O services:
• Friendly environment anywhere in the world.• Data buffering and staging.• I/O performance feedback.• User remapping of data sources.
![Page 18: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/18.jpg)
www.cs.wisc.edu/condor
Standard Universe (Cont.)
› Disadvantages: Must statically link with Condor
library. Limited class of applications:
• Single-process UNIX binaries.• Certain system calls prohibited.
![Page 19: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/19.jpg)
www.cs.wisc.edu/condor
System Call Limitations
› Standard universe does not allow: Multiple processes:
• fork(), exec(), system()
Inter-process communication:• semaphores, messages, shared memory
Complex I/O:• mmap(), select(), poll(), non-blocking I/O, …
Kernel-level threads• (User level threads are OK.)
![Page 20: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/20.jpg)
www.cs.wisc.edu/condor
System Call Limitations (Cont.)
› Too restrictive? Use the vanilla universe.
![Page 21: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/21.jpg)
www.cs.wisc.edu/condor
System Call Features
› The standard universe does allow: Signals
• But, Condor reserves SIGTSTP and SIGUSR1.
Sockets• Keep it brief - network connections, by
nature, cannot migrate or checkpoint.
![Page 22: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/22.jpg)
www.cs.wisc.edu/condor
System Call Features (Cont.)
› The standard universe does allow: Complex I/O on sockets
• select(), poll(), and non-blocking I/O can be used on sockets, but not other sorts of files.
User-level threads
![Page 23: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/23.jpg)
www.cs.wisc.edu/condor
Which Universe?
› Vanilla: Perfect for a Condor pool of identical machines.
› Standard: Needed for heterogeneous Condor pools,
flocked pools, and more generally, unfriendly machines on the Grid.
› The rest of this talk concerns the standard universe.
![Page 24: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/24.jpg)
www.cs.wisc.edu/condor
Using the Standard Universe
› Link with Condor library.
› Submit the job.
› Get brief I/O feedback while running.
› Get complete I/O feedback when done.
› If needed, remap files.
![Page 25: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/25.jpg)
www.cs.wisc.edu/condor
Link with Condor Library
› Simply use condor_compile in front of your normal link line.
› For example, gcc main.o utils.o -o program
› Becomes: condor_compile gcc main.o utils.o -o program
› Despite the name, only re-linking is required, not re-compiling.
![Page 26: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/26.jpg)
www.cs.wisc.edu/condor
Submit Job
› Create a submit file:
% vi program.submit
› Submit the job:
% condor_submit program.submit
Universe = standard
input = program.inoutput = program.out
executable = program
queue 3
![Page 27: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/27.jpg)
www.cs.wisc.edu/condor
Brief I/O Summary
% condor_q -io-- Schedd: c01.cs.wisc.edu : <128.105.146.101:2016>ID OWNER READ WRITE SEEK XPUT BUFSIZE BLKSIZE756.15 joe 244.9 KB 379.8 KB 71 1.3 KB/s 512.0 KB 32.0 KB758.24 joe 198.8 KB 219.5 KB 78 45.0 B /s 512.0 KB 32.0 KB758.26 joe 44.7 KB 22.1 KB 2727 13.0 B /s 512.0 KB 32.0 KB
3 jobs; 0 idle, 3 running, 0 held
![Page 28: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/28.jpg)
www.cs.wisc.edu/condor
Complete I/O Summaryin Email
Your condor job "/usr/joe/records.remote input output" exited with status 0.
Total I/O:104.2 KB/s effective throughput5 files opened104 reads totaling 411.0 KB316 writes totaling 1.2 MB102 seeks
I/O by File:
buffered file /usr/joe/outputopened 2 times4 reads totaling 12.4 KB4 writes totaling 12.4 KB
buffered file /usr/joe/inputopened 2 times100 reads totaling 398.6 KB311 write totaling 1.2 MB101 seeks
![Page 29: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/29.jpg)
www.cs.wisc.edu/condor
Complete I/O Summaryin Email
› The summary helps identify performance problems. Even advanced users don't know exactly how their programs and libraries operate.
![Page 30: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/30.jpg)
www.cs.wisc.edu/condor
Complete I/O Summary in Email (Cont.)
› Example: CMSSIM - physics analysis program. “Why is this job so slow?” Data summary:
• read 250 MB from 20 MB file. Very high SEEK total -> random access. Solution: Increase buffer to 20 MB.
![Page 31: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/31.jpg)
www.cs.wisc.edu/condor
Buffer Parameters
› By default: buffer_size = 524288 (512 KB) buffer_block_size = 32768 (32 KB)
› Change parameters in submit file: buffer_size = 20000000 buffer_block_size = 32768
![Page 32: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/32.jpg)
www.cs.wisc.edu/condor
If Needed, Remap Files
› Suppose the program is hard-coded to open datafile, but you want each instance to get a slightly different copy. In the submit file, add:
file_remaps = "datafile = /usr/joe.data.$(PROCESS)"
› Process one gets /usr/joe.data.1
› Process two gets /usr/joe.data.2
› And so on...
![Page 33: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/33.jpg)
www.cs.wisc.edu/condor
If Needed, Remap Files (Cont.)
› The same syntax will allows the user to direct the application to other third-party data sources such as web servers:
file_remaps = "datafile = http://www.cs.wisc.edu/usr/joe/data”
![Page 34: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/34.jpg)
www.cs.wisc.edu/condor
Outline
› Introduction
› Using Remote I/O
› Implementation
› Build Your Own: Bypass
› Conclusion
![Page 35: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/35.jpg)
www.cs.wisc.edu/condor
The Big Picture
![Page 36: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/36.jpg)
www.cs.wisc.edu/condor
The Machines
Has all of your files, or knows where to find them.
Accepts your identity and credentials
Allows you to run a process, but it might not:› have some of your files.› accept your identity.
![Page 37: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/37.jpg)
www.cs.wisc.edu/condor
General Strategy
› Trap all the application's I/O operations. open(), close(), read(), write(), seek(),
…
› Route them to the correct service.
› Cache both service decisions and actual data.
![Page 38: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/38.jpg)
www.cs.wisc.edu/condor
Application
› Plain UNIX program.
› Unaware that it is part of a distributed system.
› Statically linked against Condor library.
![Page 39: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/39.jpg)
www.cs.wisc.edu/condor
Condor Library
› Sends system calls to various services via RPC.
› Buffers and stages data.
› Asks shadow for policy decisions.
![Page 40: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/40.jpg)
www.cs.wisc.edu/condor
Shadow
› Makes policy decisions for application.
› Executes remote system calls for application.
![Page 41: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/41.jpg)
www.cs.wisc.edu/condor
Opening a File
Shadow CondorLibrary
Application
Open("datafile",O_RDONLY);
![Page 42: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/42.jpg)
www.cs.wisc.edu/condor
Opening a File
Shadow CondorLibrary
Application
Where is "datafile?"
Open("datafile",O_RDONLY);
![Page 43: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/43.jpg)
www.cs.wisc.edu/condor
Opening a File
Shadow CondorLibrary
Application
URL: local:/usr/joe/datafileBuffering: none.
Open("datafile",O_RDONLY);
Where is "datafile?"
![Page 44: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/44.jpg)
www.cs.wisc.edu/condor
Opening a File
Shadow CondorLibrary
ApplicationOpen("/usr/joe/datafile",O_RDONLY)
Foreign Machine
Open("datafile",O_RDONLY);
Where is "datafile?"
URL: local:/usr/joe/datafileBuffering: none.
![Page 45: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/45.jpg)
www.cs.wisc.edu/condor
Opening a File
Shadow CondorLibrary
Application
Foreign Machine
Open("datafile",O_RDONLY);
Where is "datafile?"
Open("/usr/joe/datafile",O_RDONLY) Success
URL: local:/usr/joe/datafileBuffering: none.
![Page 46: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/46.jpg)
www.cs.wisc.edu/condor
Opening a File
Shadow CondorLibrary
Application
Success
Foreign Machine
Open("datafile",O_RDONLY);
Where is "datafile?"
Success
URL: local:/usr/joe/datafileBuffering: none.
Open("/usr/joe/datafile",O_RDONLY)
![Page 47: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/47.jpg)
www.cs.wisc.edu/condor
Shadow Responses
› URL: remote: Use remote system calls. local: Use local system calls. special: Use local system calls,
disable checkpointing. http: Fetch from a web server. Others in development…
![Page 48: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/48.jpg)
www.cs.wisc.edu/condor
Shadow Responses (Cont.)
› Buffering: None. Buffer partial data. Stage whole file to local disk.
![Page 49: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/49.jpg)
www.cs.wisc.edu/condor
Some Fast, Some Slow
Shadow CondorLibrary
Application
Foreign Machine
RPC over network:Several milliseconds,or (much) worse!
Function call:Less than amicrosecond?
System call:10s or 100s ofmicroseconds
![Page 50: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/50.jpg)
www.cs.wisc.edu/condor
Reading data from a file
Shadow CondorLibrary
Application
Read 1024 bytes from "datafile"
Read 1024 bytes from "/usr/joe/datafile"
Success
Success
Foreign Machine
Library remembers where datafile is - no need to communicate with the shadow
Low latency, random-access data source: Read directly
![Page 51: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/51.jpg)
www.cs.wisc.edu/condor
Reading data from a file
Shadow
CondorLibrary
Application
Read 1024 bytes from "otherfile" up to 32 times
Read 32768 bytes from "otherfile"
High-latency, random-access data source: Buffer large chunks
Databuffer
![Page 52: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/52.jpg)
www.cs.wisc.edu/condor
Reading data from a file
Shadow CondorLibrary
Application
High-latency, sequential-access data source: Stage file to local disk.
FTPServer
Local copy of "otherfile"
Open("datafile",O_RDONLY);
Where do I open "datafile"?
URL: ftp://server/datafileBuffer: Stage to disk.
![Page 53: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/53.jpg)
www.cs.wisc.edu/condor
Reading data from a file
Shadow CondorLibrary
Application
Random access service can be provided from the local copy.
FTPServer
Local copy of "otherfile"
![Page 54: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/54.jpg)
www.cs.wisc.edu/condor
Guiding Principle
› Policy in shadow, mechanisms in library. Shadow makes policy decisions
because it knows the system configuration.
Library is closest to the application, so it routes system calls to the destination selected by the shadow.
![Page 55: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/55.jpg)
www.cs.wisc.edu/condor
Policy at Shadow
ShadowUser Override
"I know file x can be quickly loaded from ftp://ftp.cs.wisc.edu/y"
Condor Library
"There is plenty of space to stage files over here."
Scheduling System
"The foreign machine is not in your cluster"
![Page 56: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/56.jpg)
www.cs.wisc.edu/condor
Policy at Shadow
ShadowUser Override
"I know file x can be quickly loaded from ftp://ftp.cs.wisc.edu/y"
Condor Library
"There is plenty of space to stage files over here."
Scheduling System
"The foreign machine is not in your cluster"
"Direct all requests for x to ftp://ftp.cs.wisc.edu/y"
![Page 57: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/57.jpg)
www.cs.wisc.edu/condor
Policy Decisions
› May be different on each foreign machine In same building: "use foreign machine” In other country: "use home machine”
› May change as job migrates same building -> other country
› May change by user control "Let's see if NFS is faster than AFS”
![Page 58: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/58.jpg)
www.cs.wisc.edu/condor
Outline
› Introduction
› Using Remote I/O
› Implementation
› Build Your Own: Bypass
› Conclusion
![Page 59: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/59.jpg)
www.cs.wisc.edu/condor
Build Your Own: Bypass
› Generalize remote I/O -> split execution.
› Building split execution systems is hard.
› Bypass is a tool for building split execution systems.
![Page 60: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/60.jpg)
www.cs.wisc.edu/condor
Build Your Own: Bypass (Cont.)
› Unlike Condor, Bypass can be used on any UNIX program without re-linking.
› Example: GASS Agent
![Page 61: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/61.jpg)
www.cs.wisc.edu/condor
Generalized Split Execution
Shadow Agent
Application
Trap a subset of available system calls
Replace them with arbitrary code.
Allow RPCs to a shadow in the home environment.
Allow arbitrary code at the home machine.
![Page 62: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/62.jpg)
www.cs.wisc.edu/condor
Split Execution is Hard
› Trapping system calls involves a large body of knowledge of particular OS and version Library entry points:
• _read, __read, __libc_read
System call entries:• socket(), open("/dev/tcp")
Wacky header files:• #define stat(a,b) _xstat(VERSION,a,b)
![Page 63: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/63.jpg)
www.cs.wisc.edu/condor
Split Execution is Hard (Cont.)
› RPCs must be platform-neutral Byte sizes and ordering
• off_t is 8 bytes on Alpha, but 4 bytes on Intel
Structure contents and order• struct stat has different members on different
platforms
Symbolic values• O_CREAT is a source-level symbol, but its actual
value is different on every platform.
![Page 64: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/64.jpg)
www.cs.wisc.edu/condor
Split Execution is Hard (Cont.)
› The code replacing system calls must be able to execute the original system calls!
› Example: Sandboxing Trap open(). Check for unauthorized file names.
• Return failure for some.• Re-invoke the original open() for others.
![Page 65: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/65.jpg)
www.cs.wisc.edu/condor
Bypass Makes it Easy!SpecificationFile
KnowledgeFile
Bypass
We provide:ugly details of systemcall trapping.
You provide: How you want the
system to work.
YourAgent
YourShadow
![Page 66: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/66.jpg)
www.cs.wisc.edu/condor
Example: GASS Agent
› Let's create an Agent that changes all calls to UNIX open() and close() into their analogues in Globus GASS. This will instrument the application with remote file fetching and staging.
ApplicationAgent(THE GRID)
Open(“http://www.yahoo.com/index.html”,O_RDONLY);
Globus_gass_open(“http://www.yahoo.com/index.html”,O_RDONLY);
![Page 67: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/67.jpg)
www.cs.wisc.edu/condor
Example: GASS Agent (Cont.)
agent_prologue{{
@include "globus_common.h"@include "globus_gass_file.h"
}};
int open( const char *name, int flags, [int mode] )agent_action{{
globus_module_activate( GLOBUS_GASS_FILE_MODULE );return globus_gass_open( namame, flags, mode );
}};
int close( int fd )agent_action{{
return globus_gass_close( fd );}};
![Page 68: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/68.jpg)
www.cs.wisc.edu/condor
Example: GASS Agent (Cont.)
› Generate the source code. bypass -agent gass.bypass
› Compile into a shared library. g++ gass_agent.C (libraries) -shared -o
gass.so
› Insert the library into your environment. setenv LD_PRELOAD /path/to/gass.so
![Page 69: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/69.jpg)
www.cs.wisc.edu/condor
Example: GASS Agent (Cont.)
› Now, run any plain old UNIX program. The program may be given URLs in place of filenames. Globus GASS will stage and cache the needed files.% cp http://www.yahoo.com/index.html /tmp/yahoo.html
% grep address http://www.cs.wisc.edu/index.html
<LI> <A HREF="/academic.html">Academic information</A>
![Page 70: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/70.jpg)
www.cs.wisc.edu/condor
Bypass
› Uses ideas from Condor, but is a separate tool.
› User specifies design, Bypass provides details.
![Page 71: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/71.jpg)
www.cs.wisc.edu/condor
Bypass (Cont.)
› Can be applied to any unmodified, dynamically-linked UNIX program at run time. Works on Linux, Solaris, IRIX, OSF/1. Static linking only on HP-UX.
![Page 72: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/72.jpg)
www.cs.wisc.edu/condor
Bypass (Cont.)
› The "knowledge file" is certainly not complete! Our experience: Each new OS version
has new tricks in the standard library that must be foleded into the knowledge file.
![Page 73: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/73.jpg)
www.cs.wisc.edu/condor
Outline
› Introduction
› Using Remote I/O
› Under the Hood
› Build Your Own: Bypass
› Conclusion
![Page 74: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/74.jpg)
www.cs.wisc.edu/condor
Future Work
› Lots of new plumbing, but still adding faucets FTP, SRB, GASS, SAM …
› Find and use third-party staging grounds? Turn checkpoint server into general
staging ground.
![Page 75: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/75.jpg)
www.cs.wisc.edu/condor
Future Work (Cont.)
› Interaction with CPU scheduling: Release CPU while waiting for slow
tape? Stage data, then allocate CPU?
![Page 76: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/76.jpg)
www.cs.wisc.edu/condor
In Summary…
› Harnessing large numbers of CPUs requires that you use unfriendly machines.
› Remote I/O is an adapter which provides a friendly execution environment on an unfriendly machine.
![Page 77: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/77.jpg)
www.cs.wisc.edu/condor
In Summary… (Cont.)
› Condor uses remote I/O to homogenize the many machines in a Condor pool.
› Bypass allows the quick construction of split execution systems, allowing remote I/O techniques to be used outside of Condor.
![Page 78: Douglas Thain Computer Sciences Department University of Wisconsin-Madison (In Bologna for June 2000) thain@cs.wisc.edu Remote](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d3e5503460f94a16b4e/html5/thumbnails/78.jpg)
www.cs.wisc.edu/condor
Need More Info?
› Contact Douglas Thain [email protected]
› Condor Web Page: http://www.cs.wisc.edu/condor
› Bypass Web Page: http://www.cs.wisc.edu/condor/bypass
› Questions now?