Download - Space allocations in NeST
![Page 1: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/1.jpg)
Joseph StanleyComputer Sciences DepartmentUniversity of Wisconsin-Madison
[email protected]://www.cs.wisc.edu/condor
Space allocations in NeST
![Page 2: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/2.jpg)
www.cs.wisc.edu/condor
Outline
› Overview of NeST
› Space allocations
› Implementation
› Big picture
› Measurements and Future Work
![Page 3: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/3.jpg)
www.cs.wisc.edu/condor
Overview of NeST
› Network storage server designed for the Grid
› Flexibility Unprivileged, user-level software Multiple protocols
• Chirp, GridFTP, FTP, HTTP, NFS Multiple concurrency models
• Processes, threads, non-blocking Portable
![Page 4: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/4.jpg)
www.cs.wisc.edu/condor
Overview of NeST
› Functionality Exports Unix based file system Space allocation ( Lot )
› Manageability User management – Dynamic
users Security – GSI authentication
![Page 5: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/5.jpg)
www.cs.wisc.edu/condor
Dispatcher
Transfer MgrConcurrencies
Control flow
Data flow
Chirp FTP Grid FTP NFS
Common protocol layer
HTTP
Physical network layer
Storage Mgr
Lot Mgr
Physical storage layer
NeST structure
![Page 6: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/6.jpg)
www.cs.wisc.edu/condor
Why space allocations ?
› Data generation and storage Data intensive applications Need to allocate space for temporary files
› Data migration Many simultaneous data flows
• Competition for storage space• Possible that all flows end in partial transfers
Need to ensure at least one succeeds
![Page 7: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/7.jpg)
www.cs.wisc.edu/condor
Space allocations in NeST
› Lot – abstraction for space allocation
› Create lot for a specified size and duration
› User and group lots
› Guaranteed and best-effort lots
› Hierarchical lots
![Page 8: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/8.jpg)
www.cs.wisc.edu/condor
Motivation for hierarchical lot
› SRMs on top of NeSTs
› Client requests streamlined through SRMs
› SRMs create a large lot
› SRMs distribute sub-lots among clients carved from the large lot
![Page 9: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/9.jpg)
www.cs.wisc.edu/condor
Lot operations
› Create, Delete, Update› MoveFile
Moves files across lots
› AddUser, RemoveUser Lot level access control List of users allowed to request sub-lots
› Attach / Detach Associates a path to a lot
![Page 10: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/10.jpg)
www.cs.wisc.edu/condor
Implementation - two choices
› Rely on the underlying quota system Kernel-assisted lot
› Provide the space guarantee within NeST User-level lot
› A third choice? Modify kernel
![Page 11: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/11.jpg)
www.cs.wisc.edu/condor
Dispatcher
Transfer MgrConcurrencies
Control flow
Data flow
Chirp FTP Grid FTP NFS
Common protocol layer
HTTP
Physical network layer
Storage Mgr
Lot Mgr
Physical storage layer
Kernel-assisted lot
Quota system
![Page 12: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/12.jpg)
www.cs.wisc.edu/condor
Kernel-assisted lot› Quota system provides the guarantee› Chirp-like protocols pose no challenge
Aware of transfer size
› GridFTP-like protocols ? Transfer size may be unknown Don’t worry at NeST level Quota system watches disk writes
![Page 13: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/13.jpg)
www.cs.wisc.edu/condor
Dispatcher
Concurrencies
Control flow
Data flow
Chirp FTP Grid FTP NFS
Common protocol layer
HTTP
Physical network layer
Physical storage layer
Storage Mgr
Lot Mgr
Transfer Mgr
User-level lot
![Page 14: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/14.jpg)
www.cs.wisc.edu/condor
User-level lot
› NeST provides the guarantee
› Chirp-like protocols pose no challenge
› GridFTP-like protocols ? Iterative data transfer Transfer manager repeatedly
queries lot manager
![Page 15: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/15.jpg)
www.cs.wisc.edu/condor
So which one ?
› Kernel-assisted Pros
Less complexPOSIX access
ConsNeeds root privilegesDepends on quota system – Less portabilityCannot distinguish among different allocations for
the same userDoes not have a hierarchical organization
![Page 16: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/16.jpg)
www.cs.wisc.edu/condor
So which one ?
› User-level Pros
Flexible, PortableHierarchical organization is easyEasy to differentiate among lots for the
same user Cons
Difficult to provide POSIX accessComplexity in the transfer and lot
managers
![Page 17: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/17.jpg)
www.cs.wisc.edu/condor
Big picture
NeST
Job
DAGMan
SRM
NeST
NeST
Remote site
GridFTP/DiskRouter transfer
StorkRequest allocation
Chirp allocations
Activate 3rd party put
Chirp input
Chirp output
![Page 18: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/18.jpg)
www.cs.wisc.edu/condor
Where are the overheads ?
› Lot operations
› Puts
› Recovery
![Page 19: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/19.jpg)
www.cs.wisc.edu/condor
Lot operations
![Page 20: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/20.jpg)
www.cs.wisc.edu/condor
Puts using Chirp
›With increase in transfer size overhead is amortized
![Page 21: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/21.jpg)
www.cs.wisc.edu/condor
Puts using GridFTP
›With increase in transfer size, overheads become comparable
![Page 22: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/22.jpg)
www.cs.wisc.edu/condor
Recovery time
›Compaction reduces lot overhead to a manageable level
![Page 23: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/23.jpg)
www.cs.wisc.edu/condor
Future Work
› Advance reservations
› Combo - Space and disk bandwidth reservation
› Recovery management Transaction based lot operations
![Page 24: Space allocations in NeST](https://reader035.vdocuments.us/reader035/viewer/2022062723/56813f93550346895daa8486/html5/thumbnails/24.jpg)
www.cs.wisc.edu/condor
Questions ?
› Demo on Wednesday Room 3351, CS building, 9am – noon
› More information available at www.cs.wisc.edu/condor/nest
› People John Bent, Joseph Stanley, Nick LeRoy, Alain
Roy Miron Livny, Andrea Arpaci-Dusseau, Remzi
Arpaci-Dusseau, Arie Shoshani