![Page 1: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/1.jpg)
Presented by:
Leveraging open source tools to gain insight into OpenStack SwiftMay 20, 2015
Michael Factor, IBM Fellow, Storage and Systems, IBM Research - Haifa
Dmitry Sotnikov, System and Storage Researcher, IBM Research - Haifa
Deep dive insights into Swift
The work was done with help of:
Yaron Weinsberg George Goldberg
For more information contact: [email protected]
![Page 2: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/2.jpg)
Swift Monitoring
• Monitoring Swift With StatsD• https://swiftstack.com/blog/2012/04/11/swift-monitoring-with-statsd/
• Unified Instrumentation and Metering of Swift• https://wiki.openstack.org/wiki/UnifiedInstrumentationMetering
• Administrator’s Guide• http://docs.openstack.org/developer/swift/admin_guide.html#cluster-telem
etry-and-monitoring
“Once you have all this great data, what do you do with it? Well, that’s going to require its own post.“
“Monitoring Swift With StatsD” by SwiftStack, Inc2
![Page 3: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/3.jpg)
Swift Monitoring Flow
• StatsD allows deep instrumentation of the Swift code and can report over 100 metrics.
• Collectd gathers statistics about the system• Graphite is an enterprise-scale monitoring tool that
stores and displays numeric time-series data• Logstash is a data pipeline that can normalize the
data to a common format.• Elasticsearch is a search server that allows
indexing large amounts of data.• Kibana is a browser based analytics and search
interface for Elasticsearch. • Spark is a fast and general engine for large-scale
data processing. • RequestStopper catches the request and returns
success, enabling isolating overheads in a non-production system.
Pro
xy S
erve
r
Co
nta
ine
r S
erve
r
Ob
ject
Se
rver
Acco
un
t S
erve
r
StatsD
CPU Statistics
RAM Statistics
Disk Statistics
Monitoring, Analytics and Visualization Node
Swift Node
RequestStopper
3
![Page 4: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/4.jpg)
Benchmark Tool
• COSBench, Intel’s Cloud Object Storage Bench-marking tool• https://github.com/intel-cloud/cosbench
4
![Page 5: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/5.jpg)
Where Our Journey Starts
• Swift 1.13• 1 container• Half a million small objects • 100 COSBench workers• What should be the cluster size to run more then 1000 PUTs a
second? (with reasonable response time)
5
![Page 6: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/6.jpg)
• 3 proxy node (Proxy servers only)• 7 storage nodes (Object, Container and Account servers)
• 20 HDD• 2 SSD• 256 GB RAM
• 3 clients machines connected to Proxy• All network connections are 10 Gbps
Our Hardware - Story #1
520 operations per second
6
![Page 7: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/7.jpg)
Swift Data Path Flow
• The Put object request arrives to one of the Proxies.• The Proxy sends the request to R (e.g., 3) storage
nodes, that will hold the R (e.g., 3) replicas of that object.
• Next, the container database is updated asynchronously to reflect the new object in it. (https://swiftstack.com/openstack-swift/architecture/)
• It is not fully asynchronous, but on timeout of 0.5 sec – first it tries to make a synchronous update.
• When at least two of the three writes to the object servers return successfully, the proxy server process will notify the client that the upload was successful.
Proxy Server
Object Server
Container Server
Client Put Request Response
7
![Page 8: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/8.jpg)
Swift Data Path Flow
Proxy Server
Object Server
Container Server
Client Put Request Response
Null Container Server
Null Object Server
Null Proxy Server
While nulling out a server is not useful for a production system, it is useful to diagnose performance bottlenecks 8
![Page 9: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/9.jpg)
RequestStopper
https://gist.github.com/gilv/7e70ba055f24bcc472b6 9
![Page 10: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/10.jpg)
Swift Data Path Flow – Put Request Response Time
Proxy Server
Object Server
Container Server
Client Put Request Response
RequestStopper at Container Server
RequestStopper at Object Server
RequestStopper at Proxy Server
192.47 ms
47.3 ms
32.89 ms
1.86 ms
10
![Page 11: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/11.jpg)
Object PUT Operations Average Response Time per Swift Component 100 Workers, 500K Objects
SWIFT 1.13 : 1 container SWIFT 1.13 : 100 containers0
50
100
150
200
250
Network RTT to Proxy Proxy ServerObject Server Container Server
Resp
onse
Tim
e (m
s)
X 4.7 faster
11
![Page 12: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/12.jpg)
SWIFT 1.13 : 1 container
SWIFT 2.2 : 1 container
SWIFT 1.13 : 100 containers
SWIFT 2.2 : 100 containers
0
50
100
150
200
250
Network RTT to Proxy Proxy Server Object Server Container Server
Resp
onse
Tim
e (m
s)
Object PUT Operations Average Response Time Comparison of Swift 1.13 vs Swift 2.2100 Workers, 500K Objects
X 3 faster
X 1.5 faster
12
![Page 13: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/13.jpg)
Mixed Workload: 1 Container, 100 Workers, 500K Objects
13
SWIFT 2.2 SWIFT 1.130
200
400
600
800
1000
1200
1400
1600
1800
Mixed Workload - 1 container
Read Write Delete
Ope
ratio
ns p
er S
econ
d
At Mixed Workload SWIFT 2.2 achieves 70% performance improvement
![Page 14: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/14.jpg)
SWIFT Scalability – Swift 2.2100 Containers, 100 Workers, 500K Objects
14
2 3 4 5 6 70
50010001500200025003000
Measured Operation Ratio
Number of Storage Servers
Put O
pera
tions
per
Sec
ond
2 3 4 5 6 70
100200300400500600
Response Time Distribution
60%-RT 80%-RT 90%-RT 95%-RT 99%-RT
Number of Storage Servers
Tim
e (m
s)
This is a maximal performance that can be achieved by 100 COSBench worker, for Swift 2.2, so adding a new node does not improves the performance.
![Page 15: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/15.jpg)
#Workers Operations per second Average Response Time (ms)
100 workers2854.31 38.98
200 workers3455.17 62.68
400 workers4323.52 101.96
Influence of number of COSBench Workers on Performance – Swift 2.27 Storage Nodes, 500K Objects, 100 Container
15
![Page 16: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/16.jpg)
Story #1 Conclusions: RequestStopper
• In some cases the limiting factor is not throughput but response time• Response time of the native Swift 1.13 with 1 container is 192 ms ~5.2 op/sec per COSBench worker 520 op/sec per 100 COSBench workers• Reducing the response time to 65 ms at Swift 2.2 helps to get ~1560 IOPS on
same cluster
16
![Page 17: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/17.jpg)
Story #1 Conclusions: Container Server
• The difference in the Container Server performance between Swift 2.2 and Swift 1.13 was due in large part to the container merge_items speedup patch (https://review.openstack.org/#/c/116992/)
• Container Sharding (https://review.openstack.org/#/c/139921/) still has a potential to improve the performance for this workload by a factor of 1.5
17
![Page 18: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/18.jpg)
System Size Influence on Performance
18
10 100 1000 100000
10
20
30
40
50
60
70
80
90
1 container 100 containers
Number of Objects per Container
Aver
age
Resp
onse
Tim
e (m
s)
5 kops 10 kops 50 kops 100 kops 500 kops0
50
100
150
200
250
300
350
1 container 100 containers
Number of Objects at System
Aver
age
Resp
onse
Tim
e (m
s)
SWIFT performance (response time) is influenced by the number of objects per container. In our environment we identified an optimum number of objects – need to evaluate what affects the optimal number of objects per container
![Page 19: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/19.jpg)
Where Our Journey Continues
• September 2014• Swift 1.13• 1 container• Half million small objects • What should be the cluster size to run more then 1000 PUTs in a
second? ( with reasonable response time )
19
![Page 20: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/20.jpg)
20
Kibana – Put Request Response Time Percentiles
Resp
onse
Tim
e (s
ec)
![Page 21: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/21.jpg)
Average Response Time – for 1 seconds intervals – SWIFT 2.2
2:23:532:27:432:31:332:35:232:39:132:43:032:46:532:50:432:54:332:58:233:02:153:06:053:09:553:13:453:17:353:21:253:25:153:29:050
50
100
150
200
250
300
350
400
450
Tim
e (m
s)
2:23:51 2:31:03 2:38:15 2:45:27 2:52:39 2:59:51 3:07:03 3:14:15 3:21:27 3:28:390
50100150200250300350400450
Tim
e (m
s)
21
https://bugs.launchpad.net/swift/+bug/1450656 ?
There is some peak each 30 sec
21
Resp
onse
Tim
e (s
ec)
Time
![Page 22: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/22.jpg)
22
Graphite – PUT Request Response Time
30 seconds30 seconds
Resp
onse
Tim
e (s
ec)
Time
![Page 23: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/23.jpg)
Zoom-in on Swift Response time outliers (> 0.5 sec) Request Granularity PUT workload, 500K objects, 100 Containers, 100 Workers, Swift 2.2
0:36:00 0:36:43 0:37:26 0:38:09 0:38:52 0:39:36 0:40:190
2
4
6
8
10
12
14
16
30 seconds
23Time
Resp
onse
Tim
e (s
ec)
![Page 24: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/24.jpg)
The effect of fs.xfs.xfssyncd_centisecs on PUT response timePUT workload, 500K objects, 100 Containers, 100 Workers, Swift 2.2
8:08:398:09:548:11:098:12:248:13:398:14:548:16:098:17:248:18:398:19:548:21:098:22:248:23:398:24:548:26:098:27:248:28:398:29:540
100
200
300
400
500
600
700
800
300 seconds 60 seconds
300 seconds
60 seconds
24
Resp
onse
Tim
e (m
s)
Time
![Page 25: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/25.jpg)
The effect of fs.xfs.xfssyncd_centisecs on PUT response timePUT workload, 500K objects, 100 Containers, 100 Workers, Swift 2.2
Seconds Avg-ResTime 60%-RT 80%-RT 90%-RT 95%-RT 99%-RT 100%-RT
10 83.26 30 50 350 520 700 1,62030 43.34 30 40 50 60 530 3,69060 38.81 30 40 50 70 270 5,900
300 31.89 30 40 50 70 220 9,530
Increasing of the fs.xfs.xfssyncd_centisecs improves the 99% percentile at the price of 100% percentile degradation
25
![Page 26: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/26.jpg)
Story #2
26
![Page 27: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/27.jpg)
• 2 proxy node (Proxy servers only)• 4 object nodes (Object servers)
• 15 HDD• 128 GB RAM
• 2 metadata nodes (Container and Account servers)• 2 SSD• 128 GB RAM
• 2 clients machines connected to Proxy• Internal network connections are 10 Gbps
Our Hardware - Story #2
27
![Page 28: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/28.jpg)
28
Object PUT Workload100 Workers, 100 Containers, 500K Objects
![Page 29: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/29.jpg)
29
Clients Transmitted Throughput
Thro
ughp
ut (M
B/se
c)
Time
![Page 30: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/30.jpg)
30
Clients Transmitted vs. Proxy Servers Received Throughput Comparison
Proxy Servers Received Throughput
Clients Transmitted Throughput =
Total Client Transmitted NetworkTotal Proxy Received Network
Thro
ughp
ut (M
B/se
c)
Time
![Page 31: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/31.jpg)
31
Proxy Servers Received vs. Proxy Servers Transmitted Throughput Comparison
Thro
ughp
ut (M
B/se
c)
Time Proxy Server Received ThroughputProxy Server Transmitted Throughput
X3
Proxy Servers Transmitted Throughput
Proxy Servers Received Throughput
![Page 32: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/32.jpg)
32
Proxy Servers Received and Transmitted vs. Object Servers Received Throughput Comparison
Thro
ughp
ut (M
B/se
c)
Time Proxy Server Transmitted ThroughputProxy Server Received ThroughputObject Server Received Throughput
X3
Proxy Servers Transmitted Throughput
Proxy Servers Received Throughput
Object Servers Received Throughput
![Page 33: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/33.jpg)
33
Network vs. Disks Throughput Comparison
Thro
ughp
ut (M
B/se
c)
Time Proxy Server Transmitted ThroughputProxy Server Received ThroughputObject Server Received ThroughputObject Servers Disks Write Throughput
Object Servers Disks Write Throughput
Proxy Servers Received Throughput
Object Servers Received Throughput
X12
![Page 34: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/34.jpg)
34
Total Disks Capacity
![Page 35: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/35.jpg)
Disks Capacity Utilization
35
Thro
ughp
ut (M
B/se
c)
Time
New object creation part
Rewrite workload
The expected disks capacity for all the workload
![Page 36: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/36.jpg)
Number of async_pending requests over the time
36
#asy
nc_p
endi
ng re
ques
ts p
er s
ec
Time
Object1Object2Object3Object4
![Page 37: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/37.jpg)
37
Variable object size workload
64 KB128 KB
32 KB15 KB
512 KB 1 MBTh
roug
hput
(MB/
sec)
Time Proxy Server Received ThroughputProxy Server Transmitted ThroughputObject Servers Disks Write Throughput
![Page 38: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/38.jpg)
38
Disk vs. Client Perceived Bandwidth
64 KB
128 KB
32 KB
15 KB
512 KB 1 MB
The overhead is not flat 3x, but instead is a function of object size
Ratio
![Page 39: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/39.jpg)
• 2 proxy node (Proxy servers only)• 5 object nodes (Object servers)
• 15 HDD• 128 GB RAM
• 3 metadata nodes (Container and Account servers)• 3 SSD• 128 GB RAM
• 2 clients machines connected to Proxy• Internal network connections are 10 Gbps
Our Hardware - Story #3 – without async_pendings
39
![Page 40: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/40.jpg)
40
Network vs. Disks Throughput Comparison
Thro
ughp
ut (M
B/se
c)
Time Proxy Server Received ThroughputProxy Server Transmitted ThroughputObject Servers Disks Write Throughput
![Page 41: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/41.jpg)
41
Disks Capacity UtilizationTh
roug
hput
(MB/
sec)
Time
![Page 42: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/42.jpg)
42
Number of async_pending requests over the time
#asy
nc_p
endi
ng re
ques
ts p
er s
ec
Time
![Page 43: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/43.jpg)
Back to Story #2
43
![Page 44: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/44.jpg)
44
PUT Request Average Response Time (Lower is better)
Object1Object2Object3Object4Proxy1Proxy2
PUT
Requ
est R
espo
nse
Tim
e (m
s)
Time
Object Server “Object1” has much higher response time
“Object2/3/4” have lower response times than proxies
Proxies
![Page 45: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/45.jpg)
Disks Read and Write Throughputs
45
Object1.Read
Object1.Write
Object2.Read
Object2.Write
Object3.Read
Object3.Write
Object4.Read
Object4.Write
![Page 46: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/46.jpg)
Processes statistics over object servers
46
#Pro
cess
Wall Clock Time
Blocked processes – processes that are waiting to IO response
Running Blocked
![Page 47: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/47.jpg)
The idle CPU comparison over object servers (Higher is better)
CPU0-CPU9 CPU10-CPU19 CPU20-CPU29 CPU30-CPU39 Stopped47
Perc
ent (
%)
Wall Clock Time
![Page 48: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/48.jpg)
• Micro benchmark results (Vdbench 8k random write workload):
• “Object1” shows an average response time of ~37 ms • “Object2”, “Object3”, “Object4” show an average response time of
~30 ms
• Our investigation revealed that “Object1” server consists of older hardware components even though all servers were “supposed” to be the same
48
![Page 49: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/49.jpg)
4 Object Servers
3 Object Servers – without Object1
~10% Throughput improvement, although 25% object servers reduction
49
![Page 50: Leveraging open source tools to gain insight into OpenStack Swift](https://reader036.vdocuments.us/reader036/viewer/2022062300/55cc5e96bb61eb95138b46bc/html5/thumbnails/50.jpg)
Object Size Avg-Res Time Avg-Proc Time Throughput Bandwidth15 KB 38.98 ms 38.94 ms 2854.31 op/s 42.81 MB/S 1 MB 105.37 ms 103.6 ms 967.04 op/s 967.04 MB/S
10 MB 852.06 ms 578.62 ms 117.43 op/s 1.17 GB/S
Effect of Object Size on Cluster Bandwidth
50
Back of the envelope calculation:User Bandwidth * Number of Replicas < Total Proxy Backend Bandwidth
At our case we have 3 proxy servers, and 10 Gbit network:User Bandwidth*3 < 3*10 Gbit User Bandwidth < 1.25 GB/sec