the globus striped gridftp framework and server bill allcock 1 (presenting) john bresnahan 1 raj...
Post on 27-Mar-2015
215 Views
Preview:
TRANSCRIPT
The Globus Striped GridFTP Framework and Server
Bill Allcock1 (presenting) John Bresnahan1
Raj Kettimuthu1 Mike Link2 Catalin Dumitrescu2 Ioan Raicu2 Ian Foster1,2
1 Math & Computer Science Division, Argonne National Laboratory, Argonne, IL 60439, U.S.A.
2 Dept of Computer Science, University of Chicago, Chicago, IL 60615, U.S.A.
Introduction Problem we are addressing A brief discussion of the GridFTP Protocol Design / Architecture of our implementation Performance results
Technology Drivers Internet revolution: 100M+ hosts
Collaboration & sharing the norm Universal Moore’s law: x103/10 yrs
Sensors as well as computers Petascale data tsunami
Gating step is analysis & our old infrastructure?
114 genomes735 in progress
You are here
What issues are we addressing?
Striping storage systems are often clusters, and we need
to be able to utilize all of that parallelism Collective Operations
essentially, the striping should be invisible to the outside world
Uniform interface Ideally, any data source can be treated the
same way
What issues are we addressing? Network Protocol Independence
TCP has well known issues with high Bandwidth-Delay Product networks
Need to be able to take advantage of aggressive protocols on circuits.
Diverse Failure Modes Much happening under the covers, so must be
resilient to failures End-to-End Performance
We need to be able to manage performance for a wide range of resources
The GridFTP Protocol
What is GridFTP? A secure, robust, fast, efficient, standards based,
widely accepted data transfer protocol A Protocol
Multiple Independent implementation can interoperate This works. Fermi Lab has an implementation with their
DCache system and U. Virginia has a .Net implementation that work with ours.
Lots of people have developed clients independent of the Globus Project.
The Globus Toolkit supplies a reference implementation: Server Client tools (globus-url-copy) Development Libraries
GridFTP: The Protocol
Existing standards RFC 959: File Transfer Protocol RFC 2228: FTP Security Extensions RFC 2389: Feature Negotiation for the File
Transfer Protocol Draft: FTP Extensions GridFTP: Protocol Extensions to FTP for the Grid
Grid Forum Recommendation GFD.20 http://www.ggf.org/documents/GWD-R/GFD-R.020.pdf
What did the GridFTP protocol add? Extended Block Mode
data is sent in “packets” with a header containing a 64 bit offset and length
allows out-of-order reception of packets Restart and Performance Markers
allows for robust restart and perf monitoring SPAS/SPOR
striped PASV and striped PORT allows a list of IP/ports to be returned
What did the GridFTP Protocol add? Data Channel Authentication
Needed since in third party transfer, you don’t know who will connect to the listener.
ESTO/ERET allows for additional processing on the data prior
to storage/transmission We use this for partial file transfers
SBUF/ABUF manual and automatic TCP buffer tuning
Options to set parallelism/striping parameters
1Establish control connection
2Establish data connection
Client
Server Server Server
Client
1Establish control connection
2Establish control connection
3Establish data connection
Client Server Model 3rd Party Transfer Model
Client/Server vs 3rd Party
Parallelism vs Striping
Architecture / Design of our Implementation
Data Channel
Server PI
DTP
Description of transfer: completely server-internal communication. Protocol is unspecified and left up to the implementation.
Server PI
DTP
Internal IPC API Internal IPC API
Client PIInfo on transfer: restart markers, performance markers, etc. Server PI optionally processes these, then sends them to the client PI
ControlChannels
Overall Architecture
Possible Configurations
Control
Data
Typical Installation
Control
Data
Separate Processes
Striped Server
Control
Data
Control
Striped Server (future)
Data
Data Storage Interface
Data Processing Module
Data Channel Protocol Module
Datasourceor sink
Data channel
Data Transfer Processor
Data Storage Interface This is a very powerful abstraction Several can be available and loaded
dynamically via the ERET/ESTO commands Anything that can implement the interface
can be accessed via the GridFTP protocol We have implemented
POSIX file (used for performance testing) HPSS (tape system; IBM) Storage Resource Broker (SRB; SDSC) NeST (disk space reservation; UWis/Condor)
Extensible IO (XIO) system Provides a framework that implements a
Read/Write/Open/Close Abstraction Drivers are written that implement the functionality (file,
TCP, UDP, GSI, etc.) Different functionality is achieved by building protocol
stacks GridFTP drivers will allow 3rd party applications to easily
access files stored under a GridFTP server Other drivers could be written to allow access to other
data stores. Changing drivers requires minimal change to the
application code. Ported GridFTP to use UDT in less than a day
AFTER the UDT driver was written
Network ProtocolNetwork
Protocol
Globus XIO Approach
ApplicationDisk
Network Protocol
Special Device
Glo
bus
XIO
Driver
Driver
Driver
Globus XIO Framework
Moves the data from user to driver stack.
Manages the interactions between drivers.
Assist in the creation of drivers.
Asynchronous support. Close and EOF Barriers. Error checking Internal API for passing
operations down the stack.
User API
Fram
ework
Driver Stack
Transform
Transform
Transport
What issues are we addressing?
Striping storage systems are often clusters, and we need
to be able to utilize all of that parallelism Collective Operations
essentially, the striping should be invisible to the outside world
Uniform interface Ideally, any data source can be treated the
same way
What issues are we addressing? Network Protocol Independence
TCP has well known issues with high Bandwidth-Delay Product networks
Need to be able to take advantage of aggressive protocols on circuits.
Diverse Failure Modes Much happening under the covers, so must be
resilient to failures End-to-End Performance
We need to be able to manage performance for a wide range of resources
Performance Results
Comparison in Stream Mode
Throughput comparison on a LAN (0.2 ms RTT and 612 Mbps bandwidth)
0
20
40
60
80
100
120
140
160
1M 10M 100M 1000M
File size
Thro
ughp
ut in
mbp
s
NCFTP
ZEBRA
WUFTP
Throughput comparison on a WAN (60 ms RTT and 1 Gbps bandwidth)
0
20
40
60
80
100
120
140
160
180
1M 10M 100M 1000M
File size
Thro
ughp
ut in
mbp
s
NCFTP
ZEBRA
WUFTP
Parallel Stream PerformanceThroughput on a LAN (0.2 ms RTT and 612 Mbps bandwidth)
0
100
200
300
400
500
600
700
0 5 10 15 20 25 30 35
Number of Streams
Th
rou
gh
pu
t in
mb
ps
iperf
zebra mem
zebra disk
bonnie
Throughput on DOT cluster (2.2 ms RTT and 1 Gbps bandwidth)
0
100
200
300
400
500
600
700
800
900
1000
0 5 10 15 20 25 30 35
Number of Streams
Th
rou
gh
pu
t in
mb
ps
iperf
zebra mem
zebra disk
bonnie
0
500
1000
1500
2000
2500
3000
3500
4000
4500
0 5 10 15 20 25 30 35
Streams
See
k co
un
t
Zebra (disk-disk)
Bandwidth vs Streams
0
200
400
600
800
1000
1200
1400
0 5 10 15 20 25 30 35
Streams
Ba
nd
wid
th (
in M
bp
s)
( Iperf)memory-memory
( Zebra)memory-memory
( Zebra)disk-disk
Bonnie
Memory to MemoryStriping Performance
BANDWIDTH Vs STRIPING
0
5000
10000
15000
20000
25000
30000
0 10 20 30 40 50 60 70
Degree of Striping
Ban
dw
idth
(M
bp
s)
# Stream = 1 # Stream = 2 # Stream = 4 # Stream = 8 # Stream = 16 # Stream = 32
Disk to Disk Striping PerformanceBANDWIDTH Vs STRIPING
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
20000
0 10 20 30 40 50 60 70
Degree of Striping
Ban
dw
idth
(M
bp
s)
# Stream = 1 # Stream = 2 # Stream = 4 # Stream = 8 # Stream = 16 # Stream = 32
Storage Performance SDSC
0
200
400
600
800
1000
1200
1400
1600
1800
0 1 2 3 4 5 6 7 8 9
Number of nodes
Dis
k t
hro
ug
hp
ut
(in
Mb
ps)
Write
Read
Storage Performance NCSA
0
200
400
600
800
1000
1200
1400
1600
1800
0 1 2 3 4 5 6 7 8 9
Number of nodes
Dis
k th
rou
gh
pu
t (i
n M
bp
s)
Write
Read
Scalability ResultsGridFTP Server Performance
Upload 10MB file from 1800 clients to ned-6.isi.edu:/dev/null
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
1500
1600
1700
1800
1900
2000
0 500 1000 1500 2000 2500 3000 3500
Time (sec)
# o
f Co
ncu
ren
t Ma
chin
es
/ Re
spo
nse
Tim
e (
sec)
0
10
20
30
40
50
60
70
80
90
100
CP
U %
/ Th
rou
gh
pu
t (M
B/s
))
Throughput
Response Time
Load
CPU %
Memory Used
Summary
The GridFTP protocol provides a good set of features for data movement requirements in the Grid.
The Globus Striped Server (Zebra) implementation of this protocol provides a flexible design / architecture for integrating with different communities, storage systems, and protocols.
Our implementation is robust and performant over a range of environments.
Questions?
top related