A Managed Object Placement Service (MOPS) using NEST and
GridFTP Dr. Dan Fraser
John Bresnahan, Nick LeRoy, Mike Link, Miron Livny, Raj Kettimuthu
SCIDAC Center for Enabling Distributed Petascale Science (CEDpS)www.cedps.org
Overview
• Brief CEDPS overview
• Focus on data movement
• Managed Object Placement Service (MOPS)– Internal resource management (awareness)
• GFork capability
– External awareness & interaction• NEST (Network Storage Technology)
Petascale Data Challenge
• DOE facilities generatemany petabytes of data(2 petabytes = all U. S. academic research libraries!)
Massive data
U
U
U
U
U
DOEfacilities
• Remote users (at labs universities, industry) need data!
• Rapid, reliable accesskey to maximizingvalue of $B facilities
U
Remotedistributed users
U
U
• Reliable: recoverfrom many failures
• Predictable: data arrives when scheduled
• Secure: protect expensive resources & data• Scalable: deal with many
users & much data
Bridging the Divide (1):Move Data to Users
When & Where NeededC
B
A
• Fast: >10,000x faster thanusual Internet
“Deliver this 100 Terabytes to
locations A, B, C by 9am tomorrow”
• Flexible: easyintegration of functions
• Secure: protect expensive resources & data• Scalable: deal with many
users & much data
Bridging the Divide (2):Allow Users to Move
ComputationNear Data
A
• Science services:provide analysisfunctions neardata source
“Perform mycomputation F ondatasets X, Y, Z”
Y Z
XF
• Instrument: includemonitoring points inall system components
• Monitor: collect data inresponse to problems
• Diagnose: identify thesource of problems
Bridging the Divide (3):Troubleshoot
End-to-EndProblemsC
B
A
“Why did my datatransfer (or remoteoperation) fail?”
• Identify & diagnose failures & performanceproblems
What is GridFTP
• Widely used, open source, production quality data mover– Separate control and data channels– Parallel streams (~3-5x faster than TCP/IP)– Parallel stripes (multiple servers)– Partial file transfer– Multiple security options (GSI, SSH)– Third party control– Extensible for both file system & protocols
GridFTP Modularity
Data StorageInterfaces (DSI) -POSIX -SRB -HPSS -NEST
GridFTP Server-separate control, data-striping
XIO Drivers -TCP -UDT (UDP) -parallel streams -GSI -SSH
Client Interfaces -Globus-URL-Copy -C Library -RFT (3rd party)
I/OFileSystems
Clients
GridFTP Advanced Configurations
• GFork (Internal awareness)– Robust unix fork/setuid model– Allows server state to be maintained across
connections
• Dynamic backends– Stability in the event of backend failure– Growing resource pools for peak demands
• Storage/Access Allocation (External awareness)– NEST (Network Storage Technology)
Why is awareness important?
• Currently, GridFTP does everything it is asked• If asked, GridFTP in a worst case scenario could:
– Use all available memory & buffers on the server– Write until the file system is full– Slow down all the transfers when overloaded
(Worst case scenarios do not happen very often)
• Many tools designed to work around these limitations– SRM, DCache, …
Services should be able to protect both themselves and their environments
GFork (Internal Awareness)
Client
Server Host
GForkServer
GridFTPPlugin
GridFTP Server Instance
Fork
GridFTP Server Instance
GridFTP Server Instance
State Sharing Link
ClientClient Inherited Links
Control Channel Connections
External Awareness:Why storage allocations ?
• Users need both temporary storage, and long-term guaranteed storage.
• Administrators need a storage solution with configurable limits and policy.
• Administrators will benefit from NeST’s autonomous reclamations of expired storage allocations.
External Awareness: GridFTP + NeST
GridFTP Server
NeST Callout
Disk
Storage
NeST Server
NeST Client
Negotiator
globus-url-copy
(Lot operations, etc.)(File transfers)
(GSI-FTP)
Overview of NeST
• NeST: Network Storage Technology• Lightweight: Configuration and installation can be
performed in minutes.• Multi-protocol: Supports Chirp, GridFTP, NFS,
HTTP– Chirp is NeST’s internal protocol
• Secure: GSI authentication• Allocation: NeST negotiates “mini storage
contracts” between users and server.
Storage allocations in NeST
• Lot – abstraction for storage allocation with an associated handle– Handle is used for all subsequent
operations on this lot
• Client requests lot of a specified size and duration. Server accepts or rejects client request.
External Awareness Architecture
Client
GridFTP Server
ACL Plugin
DSI Plugin
Main Codebase
NEST
ACL Plugin• Authorize/Init
– Grant access Yes/No– Plugin establishes context (initializes state for future requests)
• Create/Modify/Read a file– Given pathname and size– Creates a transaction
• Update Transaction– Plug in may timeout waiting– Progessively commit bytes as ‘complete’– Finished flag
Granting Access
Client
GridFTP Server
ACL Plugin
DSI Plugin
Main Codebase
Client connects
GSI ID
Allow?Y
230 Enter
GSI HandshakeNow known ID sent to auth pluginDo whatever needed to determine if allowedNotify client of access
NEST
Recieving a File
Client
GridFTP Server
ACL Plugin
DSI Plugin
Main Codebase
Path/size
Allow?Y
150 Begin
RECV file
Reserve Space
Start transfer
01010101010101010101
Receive Bytes
Update Transaction
Transaction Complete
NEST
Notes
• Sending a file– Same interactions as receiving, only simpler
(no space reservation)
• ACLs can be chained together– Chaining semantics still being worked out
Using NeST• Init
– NeST can use the client username/GSI subject to initialize.
• Create/modify– Reserve space with a given timeout
• Pathname is key to transaction• If expires reservation and uncommitted data is lost
• Update– Commit bytes, reset timeout.
• Complete– Clean up state
Conclusion• Services Must be able to protect
themselves
• Awareness of environment (Internal & External) is key
• Managed Object Placement Service– Straight-forward technology advancements– Capability greater than sum of parts
• Invitation to work together…