profiling grid data transfer protocols and servers george kola, tevfik kosar and miron livny...
TRANSCRIPT
![Page 1: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/1.jpg)
Profiling Grid Data Profiling Grid Data Transfer Protocols Transfer Protocols
and Serversand ServersGeorge Kola, Tevfik Kosar and George Kola, Tevfik Kosar and
Miron LivnyMiron Livny
University of Wisconsin-MadisonUniversity of Wisconsin-Madison
USAUSA
![Page 2: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/2.jpg)
2/33
MotivationMotivation Scientific experiments Scientific experiments
are generating large are generating large amounts of dataamounts of data
Education research & Education research & commercial videos are commercial videos are not far behindnot far behind
Data may be Data may be generated and stored generated and stored at multiple sitesat multiple sites
How to efficiently How to efficiently store and process this store and process this data ?data ?
AppliApplicatiocatio
nn
First First DataData
Data Data VoluVolume me
(TB/y(TB/yr)r)
UsersUsers
SDSSSDSS 19991999 1010 100s100s
LIGOLIGO 20022002 250250 100s100s
ATLAATLAS/S/CMSCMS
20052005 5,0005,000 1000s1000sSource: GriPhyN Proposal, 2000
WCEWCER R
20042004 500+500+ 100s100s
![Page 3: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/3.jpg)
3/33
MotivationMotivation Grid enables large scale computationGrid enables large scale computation ProblemsProblems
Data intensive applications have Data intensive applications have suboptimal performancesuboptimal performance
Scaling up creates problemsScaling up creates problems Storage servers thrash and crashStorage servers thrash and crash
Users want to reduce failure rate and Users want to reduce failure rate and improve throughput improve throughput
![Page 4: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/4.jpg)
4/33
Profiling Protocols and Profiling Protocols and ServersServers
Profiling is a first step Profiling is a first step Enables us to understand how time is spentEnables us to understand how time is spent Gives valuable insightsGives valuable insights HelpsHelps
computer architects add processor featurescomputer architects add processor features OS designers add OS featuresOS designers add OS features middleware developers to optimize the middleware developers to optimize the
middlewaremiddleware application designers design adaptive application designers design adaptive
applicationsapplications
![Page 5: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/5.jpg)
5/33
ProfilingProfiling We (middleware designers) are aiming We (middleware designers) are aiming
for automated tuningfor automated tuning Tune protocol parameters, concurrency Tune protocol parameters, concurrency
levellevel Depends on dynamic state of network, Depends on dynamic state of network,
storage serverstorage server We are developing low overhead online We are developing low overhead online
analysisanalysis Detailed Offline + Online analysis would Detailed Offline + Online analysis would
enable automated tuningenable automated tuning
![Page 6: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/6.jpg)
6/33
ProfilingProfiling RequirementsRequirements
Should not alter system characteristicsShould not alter system characteristics Full system profileFull system profile Low overheadLow overhead
Used OProfileUsed OProfile Based on Digital Continuous Profiling Based on Digital Continuous Profiling
InfrastructureInfrastructure Kernel profilingKernel profiling No instrumentationNo instrumentation Low overhead/tunable overheadLow overhead/tunable overhead
![Page 7: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/7.jpg)
7/33
Profiling SetupProfiling Setup Two server machines
Moderate server: 1660 MHzAthlon XP CPU with 512 MB RAM
Powerful server: dual Pentium 4 Xeon 2.4 GHz CPU with 1 GB RAM.
Client Machines were more powerful – Client Machines were more powerful – dual Xeonsdual Xeons To isolate server performanceTo isolate server performance
100 Mbps network connectivity Linux kernel 2.4.20, GridFTP server , GridFTP server
2.4.3 , NeST prerelease2.4.3 , NeST prerelease
![Page 8: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/8.jpg)
8/33
GridFTP ProfileGridFTP Profile
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
Perc
en
tag
e o
f C
PU
Tim
e
Idle EthernetDriver
InterruptHandling
Libc Globus Oprofile IDE File I/O Rest ofKernel
Read From GridFTP Write To GridFTP
Read Rate = 6.45 MBPS, Write Rate = 7.83 MBPS
=>Writes to server faster than reads from it
![Page 9: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/9.jpg)
9/33
GridFTP ProfileGridFTP Profile
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
Pe
rce
nta
ge
of
CP
U T
ime
Idle EthernetDriver
InterruptHandling
Libc Globus Oprofile IDE File I/O Rest ofKernel
Read From GridFTP Write To GridFTP
Writes to the network more expensive than Writes to the network more expensive than readsreads
=> Interrupt coalescing=> Interrupt coalescing
![Page 10: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/10.jpg)
10/33
GridFTP ProfileGridFTP Profile
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
Pe
rce
nta
ge
of
CP
U T
ime
Idle EthernetDriver
InterruptHandling
Libc Globus Oprofile IDE File I/O Rest ofKernel
Read From GridFTP Write To GridFTP
IDE reads more expensive than writesIDE reads more expensive than writes
![Page 11: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/11.jpg)
11/33
GridFTP ProfileGridFTP Profile
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
Pe
rce
nta
ge
of
CP
U T
ime
Idle EthernetDriver
InterruptHandling
Libc Globus Oprofile IDE File I/O Rest ofKernel
Read From GridFTP Write To GridFTP
File system writes costlier than readsFile system writes costlier than reads=> Need to allocate disk blocks=> Need to allocate disk blocks
![Page 12: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/12.jpg)
12/33
GridFTP ProfileGridFTP Profile
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
Pe
rce
nta
ge
of
CP
U T
ime
Idle EthernetDriver
InterruptHandling
Libc Globus Oprofile IDE File I/O Rest ofKernel
Read From GridFTP Write To GridFTP
More overhead for writes because of higher More overhead for writes because of higher transfer ratetransfer rate
![Page 13: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/13.jpg)
13/33
GridFTP Profile SummaryGridFTP Profile Summary Writes to the network more expensive than Writes to the network more expensive than
readsreads Interrupt coalescingInterrupt coalescing DMA would helpDMA would help
IDE reads more expensive than writesIDE reads more expensive than writes Tuning the disk elevator algorithm would helpTuning the disk elevator algorithm would help
Writing to file system is costlier than Writing to file system is costlier than readingreading Need to allocate disk blocksNeed to allocate disk blocks Larger block size would helpLarger block size would help
![Page 14: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/14.jpg)
14/33
NeST ProfileNeST Profile
0.0
10.0
20.0
30.0
40.0
50.0
60.0
Pe
rce
nta
ge
of
CP
U T
ime
Idle EthernetDriver
InterruptHandling
Libc NeST Oprofile IDE File I/O Rest ofKernel
Read From NeST Write To NeST
Read Rate = 7.69 MBPS, Write Rate = 5.5 MBPS
![Page 15: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/15.jpg)
15/33
NeST ProfileNeST Profile
0.0
10.0
20.0
30.0
40.0
50.0
60.0
Pe
rce
nta
ge
of
CP
U T
ime
Idle EthernetDriver
InterruptHandling
Libc NeST Oprofile IDE File I/O Rest ofKernel
Read From NeST Write To NeST
Similar trend as GridFTP
![Page 16: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/16.jpg)
16/33
NeST ProfileNeST Profile
0.0
10.0
20.0
30.0
40.0
50.0
60.0
Perc
en
tag
e o
f C
PU
Tim
e
Idle EthernetDriver
InterruptHandling
Libc NeST Oprofile IDE File I/O Rest ofKernel
Read From NeST Write To NeST
More overhead for reads because of higher transfer rate
![Page 17: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/17.jpg)
17/33
NeST ProfileNeST Profile
0.0
10.0
20.0
30.0
40.0
50.0
60.0
Pe
rce
nta
ge
of
CP
U T
ime
Idle EthernetDriver
InterruptHandling
Libc NeST Oprofile IDE File I/O Rest ofKernel
Read From NeST Write To NeST
Meta data updates (space allocation) makes NeST writes
more expensive
![Page 18: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/18.jpg)
18/33
GridFTP versus NeSTGridFTP versus NeST GridFTP
Read Rate = 6.45 MBPS, write Rate = 7.83 MBPS
NeST Read Rate = 7.69 MBPS, write Rate = 5.5
MBPS GridFTP is 16% slower on reads
GridFTP I/O block size 1 MB (NeST 64 KB) Non-overlap of disk I/O & network I/O
NeST is 30% slower on writes Lots (space reservation/allocation)
![Page 19: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/19.jpg)
19/33
Effect of Protocol Effect of Protocol ParametersParameters
Different tunable parametersDifferent tunable parameters I/O block sizeI/O block size TCP buffer sizeTCP buffer size Number of parallel streamsNumber of parallel streams Number of concurrent transfersNumber of concurrent transfers
![Page 20: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/20.jpg)
20/33
Read Transfer RateRead Transfer Rate
![Page 21: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/21.jpg)
21/33
Server CPU Load on ReadServer CPU Load on Read
![Page 22: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/22.jpg)
22/33
Write Transfer RateWrite Transfer Rate
![Page 23: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/23.jpg)
23/33
Server CPU Load on WriteServer CPU Load on Write
![Page 24: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/24.jpg)
24/33
Transfer Rate and CPU Transfer Rate and CPU LoadLoad
![Page 25: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/25.jpg)
25/33
Server CPU Load and L2 Server CPU Load and L2 DTLB missesDTLB misses
![Page 26: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/26.jpg)
26/33
L2 DTLB MissesL2 DTLB Misses
Parallelism triggers the kernel to use larger page size
=> lower DTLB miss
![Page 27: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/27.jpg)
27/33
Profiles on powerful serverProfiles on powerful server Next set of graphs were obtained using Next set of graphs were obtained using
the powerful serverthe powerful server
![Page 28: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/28.jpg)
28/33
Parallel Streams versus Parallel Streams versus ConcurrencyConcurrency
![Page 29: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/29.jpg)
29/33
Effect of File Size (Local Effect of File Size (Local Area)Area)
![Page 30: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/30.jpg)
30/33
Transfer Rate versus Transfer Rate versus Parallelism in Short Latency Parallelism in Short Latency
(10 ms) Wide Area(10 ms) Wide Area
![Page 31: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/31.jpg)
31/33
Server CPU UtilizationServer CPU Utilization
![Page 32: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/32.jpg)
32/33
ConclusionConclusion Full system profile gives valuable insightsFull system profile gives valuable insights Larger I/O block size may lower transfer rateLarger I/O block size may lower transfer rate
Network, disk I/O not overlappedNetwork, disk I/O not overlapped Parallelism may reduce CPU loadParallelism may reduce CPU load
May cause kernel to use larger page sizeMay cause kernel to use larger page size Processor feature for variable sized pages would Processor feature for variable sized pages would
be usefulbe useful Operating system support for variable page size Operating system support for variable page size
would be usefulwould be useful Concurrency improves throughput at increased Concurrency improves throughput at increased
server loadserver load
![Page 33: Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA](https://reader035.vdocuments.us/reader035/viewer/2022070409/56649e765503460f94b777db/html5/thumbnails/33.jpg)
33/33
QuestionsQuestions ContactContact
[email protected]@cs.wisc.edu www.cs.wisc.edu/condor/publications.htmlwww.cs.wisc.edu/condor/publications.html