cloud iaas performance & price performance - vogel.de · cloud iaas performance &...
Post on 05-Jun-2018
241 Views
Preview:
TRANSCRIPT
Cloud IaaS Performance & Price-Performance
Comparing Linux Compute Performance of Amazon Web Services, T-Systems Dynamic Services for Infrastructure with vCloud, Microsoft Azure and ProfitBricks
Prepared for T-Systems Commercial Report
Published on 12/4/2016
Cloud Spectator Performance Report
1
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Contents
1. Introduction……………………………………………………… 2
Why Performance Matters 2
2. Executive Summary…………………………………………… 4
Findings 4
3. Methodology…………………………………………………… 6
Process 6
Tests Used 7
VM Configurations and Pricing 8
Understanding Performance Results 9
Key Considerations 10
4. Performance Findings………………………………………… 11
Processor Performance 11
Memory Bandwidth 16
Disk IOPS: Sequential and Random Operations 18
Internal Network 23
5. Price-Performance Findings………………………………… 25
Processor Price-Performance 25
Memory Bandwidth Price-Performance 26
Disk IOPS Price-Performance 27
Internal Network Price-Performance 29
6. Conclusion…………………………………………………… 30
7. About…………………………………………………………. 30
Cloud Spectator Performance Report
2
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Introduction
In an effort to simulate an end-user experience regarding performance of virtual machines across various cloud
computing providers, Cloud Spectator ran its iterative benchmark suite for 48 total hours on 2 different sets of
virtual machines (VMs) on each of the following providers: Amazon Web Services (AWS), Dynamic Services
for Infrastructure with vCloud (DSI vCloud), Microsoft Azure and ProfitBricks (PB).
Cloud Spectator measured performance on 2 vCPU, 4 vCPU, and 8 vCPU VMs for this study. For each VM
size, three types of VMs with different memory allocations were provisioned, each one tested for a 24-hour
period.
This study not only examined the performance of each vendor, but also tracked performance variability for each
of the 24-hour periods on 2vCPU, 4vCPU and 8vCPU machines. The methodology allowed Cloud Spectator to
capture performance variability over time on the same VM. Performance stability varied between resource types
and VM types/sizes throughout the course of the study.
Why Performance Matters
Performance and pricing are both key considerations in the public cloud industry, together having a substantial
impact on a company’s annual operating costs. Cloud users may need fewer resources on better-performing
services, which can lower costs. Since many users only consider price and not price-performance, they may pay
more because they require additional resources to achieve a desired level of performance. While some providers
try to differentiate offerings by cutting prices, others try to differentiate by focusing on improved performance
and user experience.
Differences in performance outputs of VMs
across IaaS providers can greatly affect quality of
service as well as annual operating costs. The
graph on the right illustrates an example of the
average processor performance from a sample of
six Cloud Service Providers (CSPs) as studied by
Cloud Spectator. CSP 1 has a processor
performance three times as high as CSP 6 (names
removed), which gives CSP 1 a notable
advantage in many processor-intensive
workloads. CSPs 2-5 exhibit a closer resemblance
in processor performance, but do not offer nearly
as much processing power as CSP 1 does.
Cloud Spectator Performance Report
3
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
The table below lists the 3 hardware components studied in this project, and each purpose as a function in the
server.
CPU & MEMORY PERFORMANCE
STORAGE PERFORMANCE
NETWORK PERFORMANCE
The performance of all applications is highly dependent on the CPU. The CPU is responsible for the processing and orchestration of all applications. The relationship between CPU performance and RAM is also observed by examining RAM bandwidth. While memory performance is not considered one of the key bottlenecks in performance for many applications, a subset of applications—particularly HPC and in-memory databases—is highly dependent on large sustained memory bandwidth.
Because most applications and all data reside on the disk, having fast disk performance is a key consideration for best application performance in many cases.
In a cloud environment, network performance is a critical piece. Scalability, in many cases, is dependent on the availability of additional VMs that must maintain a strong network backbone.
Cloud Spectator Performance Report
4
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Executive Summary
T-Systems commissioned Cloud Spectator to measure and compare the performance of VMs on four different
cloud providers: Amazon Web Services, T-System’s DSI vCloud, Microsoft Azure and ProfitBricks. Raw VM
performance was examined to evaluate the component attributes of processor, memory, disk, and internal
network on each provider’s VMs. The purpose of the study was to understand, from an end-user perspective,
performance among cloud providers with similarly sized VMs.
Findings
vCPU Performance Findings
For this study, Cloud Spectator evaluated vCPU
performance by benchmarking the VMs using
Geekbench 3, a suite of benchmark tests that
simulate tasks such as encoding and image
processing. Testing occurred over the course of a
48-hour testing period.
vCPU Performance Key Findings: DSI vCloud exhibited the highest processor performance in the
study.
DSI vCloud provided the highest processor price-performance
value of the 4vCPU and 8vCPU VMs in the study; Microsoft had
the highest value for 2vCPU VMs.
Memory Performance Findings
Cloud Spectator evaluated memory bandwidth by
benchmarking the VMs using Geekbench 3’s
memory tests, which leverages STREAM to
produce requests to memory. Testing occurred over
the course of a 48-hour testing period.
Memory Performance Key Findings: DSI vCloud achieved the highest memory bandwidth in the study.
DSI vCloud provided the highest memory price-performance value
in the study.
Cloud Spectator Performance Report
5
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Storage Performance Findings
Storage performance evaluation utilized the fio disk
benchmark, which tests the sequential read/write
and random read/write operations of storage. In this
study a 4 KB block size was used. Testing occurred
over a 48-hour test period. Persistent storage
(offered as “block storage” or “redundant storage”)
was used in all storage tests.
Storage Performance Key Findings: DSI vCloud VMs offered the highest median and maximum disk
performance.
Despite being one of the lower performers in disk IOPS, Microsoft
displayed the most stable disk performance throughout the study.
DSI vCloud provided the highest storage price-performance value
examined in the study.
Internal Network Performance Findings
Internal network performance is measured by the
throughput between VMs within the same data
center of the cloud provider (measured using Iperf2)
over the course of a 48-hour test period.
Internal Network Performance Key Findings: DSI vCloud and ProfitBricks VMs achieved the highest internal
network throughput of all the VMs examined in the study.
DSI vCloud and ProfitBricks provided the highest internal network
price-performance value of all the VMs examined in the study.
AWS exhibited stable internal network throughput with coefficient
of variation (CV) values under 1%.
Cloud Spectator Performance Report
6
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Methodology
Cloud Spectator strives to create a transparent and detailed methodology to allow readers to understand the
testing process and recreate any studies. If any information is unclear or if you have any questions, please email
the team at contact@cloudspectator.com or call +1 (617) 300-0711.
Process
1. One iteration of a 24-hour test cycle ran on each of the two sets of VMs per type on each provider for a total of 48 hours of testing per VM
type.
2. Each VM was provisioned with a Linux Ubuntu 14.04 operating system by default, available from all providers. For AWS, the HVM image
was used.
3. Before the test period, and after provisioning the VMs, system updates and upgrades were conducted via apt-get.
4. The following dependencies were installed for testing:
a. Git. Git was used to clone the test repository on the VM.
b. MySQL. For automation purposes, mysql-server was installed to automate data uploads.
c. Pip. Used to download the appropriate libraries for Python in order to run the testing. SQL Alchemy was downloaded to interact
with MySQL and upload data.
d. Libmysqlclient-dev. MySQL database’s development files, which are necessary for the SQL Alchemy and MySQL interaction.
5. Each test cycled through in the following sequence: Geekbench 3 (process & memory), fio sequential operations, fio random operations,
Iperf internal network throughput (for more information on testing, see Tests Used).
a. For fio testing (to measure disk IOPS), sequential operations ran first. Files from the sequential tests were deleted, and fio
recreated files before running random operations. Once random operations completed, the files were also deleted. Thus, before
each disk IOPS test, the files associated with the tests were deleted and recreated.
6. Internal network testing was conducted by creating a clone of the existing VM within the same region/availability zone in a client and server
architecture, connected through a private network. Iperf was installed on both the client and server. The server used “Iperf –s” to listen on
port 5001 (default port) for a TCP connection made by the client. Iperf ran bi-directionally for 60 seconds.
7. At the end of each test iteration, results were uploaded into Cloud Spectator’s database through use of SQLAlchemy (Python—see 4c in
Process).
Price-Performance Comparisons
Cloud Spectator’s price-performance calculation, the CloudSpecs Score, provides information on how much
performance is realized for each unit of cost. The CloudSpecs Score is an indexed, comparable score ranging
from 0-100 indicative of value based on a combination of cost and performance. The value is scaled; e.g., a
Cloud Service Provider (CSP) with a score of 100 gives 4x the value of a CSP with a score of 25. The
CloudSpecs Scores in this report can only be compared with equivalent configurations; e.g., a 1 vCPU VM on
Provider A can only be compared to a 1 vCPU VM on Provider B.
The calculation of the CloudSpecs Score: 1. provider_value = [Provider Performance Score] / [Provider Cost]
2. best_provider_value = max{provider_values}
3. CSP’s CloudSpecs Score = 100*provider_value / best_provider_value
Cloud Spectator Performance Report
7
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Tests Used
Processor & Memory Bandwidth: Geekbench 3
Geekbench 3, a licensable product created by Primate Labs, is a cross-platform processor benchmark that can
measure single-core and multi-core performance by simulating real-world workloads. The Geekbench 3 test
suite is comprised of 27 individual tasks/workloads: 13 integer workloads, 10 floating point workloads, and 4
memory-bandwidth tasks. While processor and memory bandwidth are both performance factors that contribute
to the final score provided by Geekbench 3, the test suite weighs processing performance much more heavily
than memory bandwidth. Also, memory bandwidth is not necessarily affected by the amount of memory
available for the VM, so VMs with larger amounts of memory may not exhibit larger bandwidth. For more
information on Geekbench 3 and to see its individual workloads, please see
http://www.primatelabs.com/geekbench/.
Geekbench 3 Tasks (Figure 3.1)
TEST TOOL TASK
Separate CPU tests that are all aggregated
into a final score. Subtests include: Integer Math, Floating Point
Math
DESCRIPTION
Integer Geekbench 3
AES, Twofish, SHA1, SHA2, BZip2 Compression,
BZip2 Decompression, JPEG Compression, JPEG
Decompression, PNG Compression, PNG
Decompression, Sobel, Lua, Dijkstra
Integer and Floating Point tasks together represent
vCPU performance. The performance of all
applications is highly dependent on the vCPU
since the vCPU is responsible for the processing
and orchestration of all applications. Floating
Point Geekbench 3
Black Scholes, Mandelbrot, Sharpen Filter, Blur Filter,
SGEMM, DGEMM, SFFT, DFFT, N-Body, Ray Trace
Memory Geekbench 3 STREAM Copy, STREAM Scale, STREAM Add,
STREAM Triad
While memory performance is not considered one
of the key bottlenecks in performance for many
common applications, a subset of applications—
particularly HPC and in-memory databases—is
highly dependent on large sustained memory
bandwidth.
Sequential and Random Disk IOPS: fio
Fio is an open source I/O generator that spawns a number of threads and processes to conduct a particular type
of I/O action specified. For the purpose of this study, fio was used to measure disk IOPS by tracking direct I/O
to the VM’s network storage. 5 x 200 MB files were created for sequential operations testing, and 5 x 200 MB
files were created for random operations testing. All operations were 50% read and 50% write. Each test
iteration used a 4 KB block size. Each test iteration lasted 60 seconds.
Internal Network Throughput: Iperf
Iperf is an open source tool used to measure TCP or UDP network bandwidth performance. Cloud Spectator
used Iperf to measure the network throughput between VMs residing in the same region/availability zone. A
screen session was created for Iperf as a server machine on each appropriate VM (see Process 6). Each Iperf test
iteration lasted 60 seconds, and data was transferred in one direction, from the test VM to the Iperf server VM.
Cloud Spectator Performance Report
8
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
VM Configurations
VM configurations were matched to standardize by virtual processors. 2 vCPUs, 4 vCPUs, and 8 vCPUs
machines from each provider were used in this study that most closely matched in size. While some providers
may offer the option for local storage, none was used and all disk testing was conducted on persistent SAN
storage. The storage columns in Figures 3.3 A through C are reflective of the SAN storage provisioned for each
VM.
2 vCPU Virtual Machines (Figure 3.3 A)
Provider Instance vCPU RAM (GB) Storage (GB)
Amazon c4.large 2 3.75 100 (EBS SSD)
Amazon m4.large 2 8 100 (EBS SSD)
Amazon r3.large 2 15 100 (EBS SSD)
Microsoft A2 Standard 2 3.5 100 (Page Blob)
Microsoft D2 v2 2 7 100 (Page Blob)
Microsoft D11 v2 2 14 100 (Page Blob)
DSI vCloud 2c4r 2 4 100 (default)
DSI vCloud 2c8r 2 8 100 (default)
DSI vCloud 2c8r 2 16 100 (default)
ProfitBricks 2c4r 2 4 100 (HDD)
ProfitBricks 2c8r 2 8 100 (HDD)
ProfitBricks 2c8r 2 16 100 (HDD)
4 vCPU Virtual Machines (Figure 3.3 B)
Provider Instance vCPU RAM (GB) Storage (GB)
Amazon c4.xlarge 4 7.5 500 (EBS SSD)
Amazon m4.xlarge 4 16 500 (EBS SSD)
Amazon r3.xlarge 4 30.5 500 (EBS SSD)
Microsoft A3 Standard 4 7 500 (Page Blob)
Microsoft D3 v2 4 14 500 (Page Blob)
Microsoft D12 v2 4 28 500 (Page Blob)
DSI vCloud 4c8r 4 8 500 (default)
DSI vCloud 4c16r 4 16 500 (default)
DSI vCloud 4c32r 4 32 500 (default)
ProfitBricks 4c8r 4 8 500 (HDD)
ProfitBricks 4c16r 4 16 500 (HDD)
ProfitBricks 4c32r 4 32 500 (HDD)
Cloud Spectator Performance Report
9
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
0
500
1000
1500
2000
2500
3000
3500
4000
Alpha Bravo Charlie Delta Echo
Mp
ixel
s/se
c
Sample Performance Graph
8 vCPU Virtual Machines (Figure 3.3 C)
Provider Instance vCPU RAM (GB) Storage (GB)
Amazon c4.2xlarge 8 15 1000 (EBS SSD)
Amazon m4.2xlarge 8 32 1000 (EBS SSD)
Amazon r3.2xlarge 8 61 1000 (EBS SSD)
Microsoft A4 Standard 8 14 1000 (Page Blob)
Microsoft D4 v2 8 28 1000 (Page Blob)
Microsoft D13 v2 8 56 1000 (Page Blob)
DSI vCloud 8c16r 8 16 1000 (default)
DSI vCloud 8c32r 8 32 1000 (default)
DSI vCloud 8c64r 8 64 1000 (default)
ProfitBricks 8c16r 8 16 1000 (HDD)
ProfitBricks 8c32r 8 32 1000 (HDD)
ProfitBricks 8c64r 8 64 1000 (HDD)
Data Center Locations (Figure 3.4)
Provider Amazon Microsoft DSI vCloud ProfitBricks
Region EU (Frankfurt) West Europe Munich Germany
Understanding Performance Results
A total of three sets of VMs were compared in this study: 2 vCPU, 4 vCPU, and 8 vCPU VMs. Each set was
compared by examining the minimum, 5TH percentile, median, 95TH percentile, and maximum scores achieved
in the total 48-hour period. The information is illustrated through box-and-whisker plots and value tables
designed to visualize the performance and performance variation captured over the course of the study. An
example of a performance percentile graph is displayed below:
Legend
Maximum: highest score achieved on this VM over the duration of
the testing.
95TH Percentile (High-Score Category): 95% of all scores on this
VM achieved this score or lower.
Median (Median-Score Category): The number separating the
higher half of the scores of that VM from the lower half. If the
median is closer to the 95th percentile, then more high scores were
observed than low scores; vice versa.
5TH Percentile (Low-Score Category): 5% of all scores on this
provider achieved this score or lower.
Minimum: lowest score achieved on this VM over the duration of
the testing.
Cloud Spectator Performance Report
10
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Key Considerations
Testing was conducted on specific VM types for each provider. Different VM configurations may yield different comparative results
between the providers. AWS and Microsoft offered fixed VM configurations. DSI vCloud and ProfitBricks offered independently
customizable VM configurations.
Users may experience different performance across different physical hosts. Factors such as user contention or malfunctions of the
physical hardware can cause suboptimal performance.
VMs selected were the base offerings across providers; greater performance may be obtained on certain providers by paying for additional
features/services.
Due to lack of availability, HDD was selected on ProfitBricks as SDD was not available within the selected data center.
Time of server setup and provisioning may vary across providers; this report only takes into consideration the performance of created
machines. Users seeking quick scalability of resources should evaluate the usability and provisioning time of multiple providers.
Cloud Spectator Performance Report
11
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Performance Findings
Processor Performance – Integer Operations
Below are results of integer-based processor testing on all three providers. The processor tests are categorized
into two categories: integer and floating point tasks. The tables on the right specify the scores achieved by each
provider’s VMs.
--Units Measured in Geekbench Performance Score--
--Units Measured in Geekbench Performance Score--
0
1000
2000
3000
4000
5000
6000
7000
8000
m4
.la
rge
c4.la
rge
r3.la
rge
A2
D2v2
D11v2
DS
I_2c4r
DS
I_2c8r
DS
I_2c16r
PB
_2c4
r
PB
_2c8
r
PB
_2c1
6r
AWS Azure DSI vCloud PB
PE
RF
OR
MA
NC
E S
CO
RE
Figure 4.1A: CPU (INTEGER) PERFORMANCE (2vCPU VMs)
0
2000
4000
6000
8000
10000
12000
14000
m4
.xla
rge
c4.x
larg
e
r3.x
larg
e
A3
D3v2
D12v2
DS
I_4c8r
DS
I_4c16r
DS
I_4c32r
PB
_4c8
r
PB
_4c1
6r
PB
_4c3
2r
AWS Azure DSI vCloud PB
PE
RF
OR
MA
NC
E S
CO
RE
Figure 4.1B: CPU (INTEGER) PERFORMANCE (4vCPU VMs)
Provider VM Min 5TH Median 95TH Max
2 v
CP
Us
AWS
m4.large 3463 3474 3506 3527 3530
c4.large 4055 4074 4122 4154 4164
r3.large 3357 3375 3389 3411 3418
A2 1839 2351 2458 2505 2552
Microsoft D2v2 5557 5738 5941 6090 6170
D11v2 5438 5567 5869 6271 6310
DSI vCloud
2c4r 2059 2311 6430 6598 6631
2c8r 1735 2091 6241 6555 6614
2c16r 2961 3606 5867 6646 6701
PB
2c4r 3434 3600 3881 4080 4208
2c8r 3356 3602 3834 4026 4220
2c16r 2926 3736 3791 3818 3823
Provider VM Min 5TH Median 95TH Max
4 v
CP
Us
AWS
m4.xlarge 6650 6758 6915 6954 6969
c4.xlarge 8033 8068 8122 8165 8209
r3.xlarge 6632 6670 6702 6734 6746
A3 4164 4607 4756 4858 4910
Microsoft D3v2 9820 10016 10457 10942 11010
D12v2 9894 10098 10328 10647 10797
DSI
vCloud
4c8r 10166 11791 12752 13099 13146
4c16r 9505 12207 12932 13072 13125
4c32r 9269 11795 12941 13057 13102
PB
4c8r 6043 6435 6757 7016 7124
4c16r 6235 6450 6746 6985 7053
4c32r 6289 6425 6745 7025 7215
Cloud Spectator Performance Report
12
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
--Units Measured in Geekbench Performance Score--
<This Space Intentionally Left Blank>
0
5000
10000
15000
20000
25000
30000m
4.2
xla
rge
c4.2
xla
rge
r3.2
xla
rge
A4
D4v2
D13v2
DS
I_8c16r
DS
I_8c32r
DS
I_8c64r
PB
_8c1
6r
PB
_8c3
2r
PB
_8c6
4r
AWS Azure DSI vCloud PB
PE
RF
OR
MA
NC
E S
CO
RE
Figure 4.1C: CPU (INTEGER) PERFORMANCE (8vCPU VMs)
Provider VM Min 5TH Median 95TH Max
8 v
CP
Us
AWS
m4.2xlarge 13531 13610 13658 13706 13741
c4.2xlarge 15321 15951 16026 16075 16130
r3.2xlarge 12294 13129 13250 13290 13304
A4 8412 8494 8687 8837 8976
Microsoft D4v2 16805 17524 18465 19082 19382
D13v2 11328 11530 14964 19574 19743
DSI
vCloud
8c16r 15373 21223 24553 24810 24888
8c32r 12102 22311 24299 24517 24636
8c64r 7327 21071 24099 24498 24666
8c16r 11250 11539 11942 12262 12499
PB 8c32r 11200 11540 11865 12191 12464
8c64r 11170 11489 11916 12262 12514
Cloud Spectator Performance Report
13
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Processor Performance – Floating Point Operations
Below are results of floating-point-based processor testing on all three providers. The processor tests are
categorized into two categories: integer and floating point tasks. The tables on the right specify the scores
achieved by each provider’s VMs.
--Units Measured in Geekbench Performance Score--
--Units Measured in Geekbench Performance Score--
0
1000
2000
3000
4000
5000
6000
7000
m4
.la
rge
c4.la
rge
r3.la
rge
A2
D2v2
D11v2
DS
I_2c4r
DS
I_2c8r
DS
I_2c16r
PB
_2c4
r
PB
_2c8
r
PB
_2c1
6r
AWS Azure DSI vCloud PB
PE
RF
OR
MA
NC
E S
CO
RE
Figure 4.2A: CPU (FLOATING POINT) PERFORMANCE
(2vCPU VMs)
0
2000
4000
6000
8000
10000
12000
14000
m4
.xla
rge
c4.x
larg
e
r3.x
larg
e
A3
D3v2
D12v2
DS
I_4c8r
DS
I_4c16r
DS
I_4c32r
PB
_4c8
r
PB
_4c1
6r
PB
_4c3
2r
AWS Azure DSI vCloud PB
PE
RF
OR
MA
NC
E S
CO
RE
Figure 4.2B: CPU (FLOATING POINT) PERFORMANCE
(4vCPU VMs)
Provider VM Min 5TH Median 95TH Max
2 v
CP
Us
AWS
m4.large 3371 3380 3404 3428 3485
c4.large 3979 3989 4010 4128 4136
r3.large 3344 3354 3367 3427 3446
A2 2268 2282 2378 2420 2458
Microsoft D2v2 5468 5545 5762 5883 5952
D11v2 5300 5360 5647 6095 6238
DSI
vCloud
2c4r 1650 1890 6248 6383 6502
2c8r 1449 1694 6148 6357 6493
2c16r 2732 3244 5842 6367 6510
2c4r 2937 3046 3229 3419 3564
PB 2c8r 2943 3040 3178 3356 3452
2c16r 2378 2996 3029 3079 3102
Provider VM Min 5TH Median 95TH Max
4 v
CP
Us
AWS
m4.xlarge 6613 6797 6877 6943 6953
c4.xlarge 8038 8078 8137 8212 8240
r3.xlarge 6795 6830 6857 6881 6916
A3 4615 4687 4825 4910 4956
Microsoft D3v2 10303 10402 10923 11303 11461
D12v2 10150 10411 10689 10952 11223
DSI vCloud
4c8r 7384 10798 12722 12899 12997
4c16r 9872 12131 12792 12919 12998
4c32r 9180 12148 12773 12896 13009
4c8r 5207 5559 5736 5922 6036
PB 4c16r 5554 5681 5828 5963 6060
4c32r 5556 5698 5848 6033 6125
Cloud Spectator Performance Report
14
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
--Units Measured in Geekbench Performance Score--
Observations: Processor Performance
The scores in Figures 4.1 A – C and 4.2 A – C are indexed through a combination of floating point and integer
performance on processors. Performance variability was determined by measuring each VM’s coefficient of
variation (CV), which is a percentage expressing the relationship between the average and standard deviation.
The variability for most VMs were relatively low with values of 5% or lower. Microsoft’s 8vCPU D13v2 had
elevated CV values ranging up to 26%, and DSI’s 2vCPU VMs ranged up 40%.
Additional Observations DSI vCloud achieved the highest performance across the 2vCPU, 4vCPU and 8vCPU processor tests in the study, followed by Microsoft,
then AWS and ProfitBricks.
The difference between Amazon’s compute-oriented offering and its general offering (c4 vs. m4, respectively) was relatively small at ~17%.
The difference between Microsoft’s stronger D-series v2 and A-series Standard was much greater at 125%.
DSI vCloud and ProfitBricks VMs do not offer different VM types, thus there is negligible difference between the VMs on each provider.
0
5000
10000
15000
20000
25000
30000
m4
.2xla
rge
c4.2
xla
rge
r3.2
xla
rge
A4
D4v2
D13v2
DS
I_8c16r
DS
I_8c32r
DS
I_8c64r
PB
_8c1
6r
PB
_8c3
2r
PB
_8c6
4r
AWS Azure DSI vCloud PB
PE
RF
OR
MA
NC
E S
CO
RE
Figure 4.2C: CPU (FLOATING POINT) PERFORMANCE
(8vCPU VMs)
Provider VM Min 5TH Median 95TH Max
8 v
CP
Us
AWS
m4.2xlarge 13434 13482 13552 13696 13798
c4.2xlarge 13246 15918 16008 16127 16212
r3.2xlarge 13351 13479 13547 13635 13788
A4 8834 8932 9116 9292 9370
Microsoft D4v2 18531 19209 20485 21225 21561
D13v2 12152 12443 16836 21878 22015
DSI
vCloud
8c16r 13282 19514 25190 25402 25538
8c32r 10558 22094 24849 25175 25319
8c64r 9255 20319 24439 25142 25336
8c16r 9344 10052 10386 10685 10911
PB 8c32r 9751 10038 10336 10662 10892
8c64r 9799 10067 10413 10767 11124
Cloud Spectator Performance Report
15
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Memory Bandwidth
Below are the results of the memory bandwidth testing on all three providers. Because memory bandwidth is
generally not affected by the amount of provisioned RAM, VMs with larger amounts of RAM do not
necessarily have higher scores. The tables on the right specify the scores achieved by each provider’s VMs.
--Units Measured in Geekbench Performance Score--
--Units Measured in Geekbench Performance Score--
0
1000
2000
3000
4000
5000
6000
m4
.la
rge
c4.la
rge
r3.la
rge
A2
D2v2
D11v2
DS
I_2c4r
DS
I_2c8r
DS
I_2c16r
PB
_2c4
r
PB
_2c8
r
PB
_2c1
6r
AWS Azure DSI vCloud PB
PE
RF
OR
MA
NC
E S
CO
RE
Figure 4.3A: MEMORY PERFORMANCE (2vCPU VMs)
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
m4
.xla
rge
c4.x
larg
e
r3.x
larg
e
A3
D3v2
D12v2
DS
I_4c8r
DS
I_4c16r
DS
I_4c32r
PB
_4c8
r
PB
_4c1
6r
PB
_4c3
2r
AWS Azure DSI vCloud PB
PE
RF
OR
MA
NC
E S
CO
RE
Figure 4.3B: MEMORY PERFORMANCE (4vCPU VMs)
Provider VM Min 5TH Median 95TH Max
2 v
CP
Us
AWS
m4.large 3162 3165 3198 3247 3251
c4.large 3407 3413 3439 3451 3456
r3.large 2225 2238 2259 2277 2283
A2 1713 1786 1965 2070 2107
Microsoft D2v2 2904 2941 3008 3258 3900
D11v2 2826 2891 2967 3060 3089
DSI vCloud
2c4r 1393 1700 4826 5442 5590
2c8r 990 1170 4285 5191 5280
2c16r 1881 2295 4249 5549 5602
2c4r 1511 2005 2249 2347 2400
PB 2c8r 1799 1898 2063 2100 2132
2c16r 1336 1453 1739 1785 1840
Provider VM Min 5TH Median 95TH Max
4 v
CP
Us
AWS
m4.xlarge 4594 5476 5555 5620 5636
c4.xlarge 5679 5798 5941 6029 6081
r3.xlarge 3765 4336 4393 4430 4448
A3 1853 1915 2011 2118 2165
Microsoft D3v2 2902 2974 3065 3401 3987
D12v2 2840 2854 3232 3380 3450
DSI
vCloud
4c8r 3375 5009 6728 7991 8054
4c16r 4637 6366 7474 7946 8019
4c32r 4415 6102 7537 7911 8029
4c8r 1867 2040 2427 2550 2598
PB 4c16r 1953 2024 2107 2169 2216
4c32r 2126 2133 2173 2239 2266
Cloud Spectator Performance Report
16
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
--Units Measured in Geekbench Performance Score--
Observations: Memory Bandwidth Performance
The scores in Figures 4.3 A – C are indexed STREAM test results on the memory bandwidth of the RAM.
DSI’s performance varied the most across the VMs averaging 23%, followed by Microsoft at 6%, ProfitBricks
at 3%, and AWS at 1%.
Additional Observations AWS’s memory bandwidth scales up from 2vCPU to 4vCPU for both VMs, but appears to cap at ~6000 (Geekbench Score). The compute
instance type (c4) does not reach the cap until the 8vCPU VM.
DSI vCloud’s memory bandwidth scales from an average median of 4453 to 7246 from 2vCPU to 4vCPU, but only scales to 7736 on the
8vCPU VMs.
Microsoft and PB memory bandwidth does not scale as the core count increases.
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
m4
.2xla
rge
c4.2
xla
rge
r3.2
xla
rge
A4
D4v2
D13v2
DS
I_8c16r
DS
I_8c32r
DS
I_8c64r
PB
_8c1
6r
PB
_8c3
2r
PB
_8c6
4r
AWS Azure DSI vCloud PB
PE
RF
OR
MA
NC
E S
CO
RE
Figure 4.3C: MEMORY PERFORMANCE (8vCPU VMs)
Provider VM Min 5TH Median 95TH Max
8 v
CP
Us
AWS
m4.2xlarge 5436 5474 5547 5598 5636
c4.2xlarge 4630 5874 6028 6070 6096
r3.2xlarge 4124 4222 4390 4415 4429
A4 1749 1787 1988 2126 2178
Microsoft D4v2 2857 2897 3021 3097 3148
D13v2 2170 2199 2651 3185 3210
DSI
vCloud
8c16r 3621 5042 8488 8599 8646
8c32r 1895 3474 8140 8475 8525
8c64r 1527 2681 6580 8311 8437
8c16r 1966 1973 1988 2015 2031
PB 8c32r 1969 1975 1992 2018 2035
8c64r 1975 1988 2006 2032 2049
Cloud Spectator Performance Report
17
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
SAN Disk IOPS: Sequential Operations
Below are results of disk IOPS testing on all providers, specifically for sequential operations (50% read, 50%
write). Disk IOPS was tested with direct I/O, so results are not reflective of cached performance, which may
sustain higher IOPS on each provider. The tables on the right specify the scores achieved by each provider’s
VMs.
--Units Measured in IOPS--
--Units Measured in IOPS--
0
2000
4000
6000
8000
10000
12000
m4
.la
rge
c4.la
rge
r3.la
rge
A2
D2v2
D11v2
DS
I_2c4r
DS
I_2c8r
DS
I_2c16r
PB
_2c4
r
PB
_2c8
r
PB
_2c1
6r
AWS Azure DSI vCloud PB
IOP
S
Figure 4.4A: STORAGE PERFORMANCE - SEQUENTIAL R/W
(100GB)
0
2000
4000
6000
8000
10000
12000
m4
.xla
rge
c4.x
larg
e
r3.x
larg
e
A3
D3v2
D12v2
DS
I_4c8r
DS
I_4c16r
DS
I_4c32r
PB
_4c8
r
PB
_4c1
6r
PB
_4c3
2r
AWS Azure DSI vCloud PB
IOP
S
Figure 4.4B: STORAGE PERFORMANCE - SEQUENTIAL R/W
(500GB)
Provider VM Min 5TH Median 95TH Max
2 v
CP
Us
AWS
m4.large 299 299 300 3003 3004
c4.large 299 299 300 3003 3004
r3.large 299 299 300 3004 3004
A2 496 502 503 504 505
Microsoft D2v2 220 501 503 504 504
D11v2 446 502 503 504 504
DSI
vCloud
2c4r 2105 3288 6067 9142 11056
2c8r 1838 3328 6025 8470 10724
2c16r 2757 4130 6304 8106 9930
2c4r 1500 2134 3411 3798 3850
PB 2c8r 1702 2083 3524 3809 3855
2c16r 1843 2111 3547 3834 3858
Provider VM Min 5TH Median 95TH Max
4 v
CP
Us
AWS
m4.xlarge 1814 1832 1874 2985 2986
c4.xlarge 1491 1493 1510 2984 2986
r3.xlarge 1627 2242 2286 2986 2986
A3 490 500 503 504 504
Microsoft D3v2 496 502 503 504 504
D12v2 405 502 503 504 504
DSI
vCloud
4c8r 1823 2705 4144 6208 8122
4c16r 4093 5492 7072 8393 10441
4c32r 2064 3068 6008 8276 8788
4c8r 1784 2414 3527 3873 3899
PB 4c16r 1548 2125 3455 3858 3895
4c32r 1888 2459 3465 3848 3884
Cloud Spectator Performance Report
18
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
--Units Measured in IOPS--
<This Space Intentionally Left Blank>
0
2000
4000
6000
8000
10000
12000
14000
m4
.2xla
rge
c4.2
xla
rge
r3.2
xla
rge
A4
D4v2
D13v2
DS
I_8c16r
DS
I_8c32r
DS
I_8c64r
PB
_8c1
6r
PB
_8c3
2r
PB
_8c6
4r
AWS Azure DSI vCloud PB
IOP
S
Figure 4.4C: STORAGE PERFORMANCE - SEQUENTIAL R/W
(1000GB)
Provider VM Min 5TH Median 95TH Max
8 v
CP
Us
AWS
m4.2xlarge 2962 2976 2984 2986 2987
c4.2xlarge 2954 2977 2983 2986 2987
r3.2xlarge 2820 2976 2983 2986 2987
A4 498 502 503 504 504
Microsoft D4v2 472 501 503 504 504
D13v2 218 500 503 504 504
DSI
vCloud
8c16r 4370 5615 7593 10283 11695
8c32r 2677 4450 7615 9827 10765
8c64r 2005 5088 7546 9897 10474
8c16r 1517 2279 3405 3863 3895
PB 8c32r 1663 2490 3592 3870 3906
8c64r 2923 3598 3829 3885 3917
Cloud Spectator Performance Report
19
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
SAN Disk IOPS: Random Operations
Below are results of disk IOPS testing on all providers, specifically for random operations (50% read, 50%
write). Disk IOPS was tested with direct I/O, so results are not reflective of cached performance, which may
sustain higher IOPS on each provider. The tables on the right specify the scores achieved by each provider’s
VMs.
--Units Measured in IOPS--
--Units Measured in IOPS--
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
m4
.la
rge
c4.la
rge
r3.la
rge
A2
D2v2
D11v2
DS
I_2c4r
DS
I_2c8r
DS
I_2c16r
PB
_2c4
r
PB
_2c8
r
PB
_2c1
6r
AWS Azure DSI vCloud PB
IOP
S
Figure 4.5A: STORAGE PERFORMANCE - RANDOM R/W
(100GB)
0
2000
4000
6000
8000
10000
12000
14000
m4
.xla
rge
c4.x
larg
e
r3.x
larg
e
A3
D3v2
D12v2
DS
I_4c8r
DS
I_4c16r
DS
I_4c32r
PB
_4c8
r
PB
_4c1
6r
PB
_4c3
2r
AWS Azure DSI vCloud PB
IOP
S
Figure 4.5B: STORAGE PERFORMANCE - RANDOM R/W
(500GB)
Provider VM Min 5TH Median 95TH Max
2 v
CP
Us
AWS
m4.large 299 299 300 3003 3004
c4.large 299 300 300 3004 3004
r3.large 299 299 300 3003 3004
A2 382 498 503 504 504
Microsoft D2v2 408 498 502 503 503
D11v2 390 497 502 503 503
DSI
vCloud
2c4r 363 1961 4184 7791 9469
2c8r 474 1453 4206 7280 9194
2c16r 407 1760 3968 6360 8620
2c4r 230 293 660 1363 1774
PB 2c8r 243 301 664 1383 1631
2c16r 240 302 697 1235 1384
Provider VM Min 5TH Median 95TH Max
4 v
CP
Us
AWS
m4.xlarge 2968 2979 2986 2987 2988
c4.xlarge 2980 2985 2986 2988 2988
r3.xlarge 2374 2985 2986 2988 2989
A3 482 499 502 503 503
Microsoft D3v2 484 499 502 503 503
D12v2 386 490 502 503 503
DSI
vCloud
4c8r 305 570 1203 2623 3837
4c16r 609 1182 2690 7234 11860
4c32r 349 650 1992 4593 6225
4c8r 214 289 607 1263 1618
PB 4c16r 179 272 520 1007 1150
4c32r 196 254 533 842 984
Cloud Spectator Performance Report
20
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
--Units Measured in IOPS--
Observations: Sequential and Random Disk IOPS Performance
Providers offer unique approaches to SAN disk, from hardware components (SSD vs. traditional magnetic
drives) to performance throttling. Even the similarities can be quite different; while Amazon and DSI both
utilize SSD, the performance difference between the two providers is highly apparent (see Figures 4.4-4.5).
Additional Observations DSI vCloud achieved the highest maximum IOPS observed throughout the study.
DSI vCloud’s storage performance variability is moderately high across the three volume sizes with an average CV of 23% for sequential
read/write and 46% for random read/write operations.
AWS’s variability drops as the storage volume increases; exhibiting a CV of 142% on the 2vCPU/100GB VMs and dropping down to an
average 24% on the 4vCPU/500GB VMs and under 1% on the 8vCPU/1000GB VMs (which is due to AWS’s burst functionality) when
looking at the sequential read/write results.
Microsoft’s storage performance was the most stable with the highest CV ranging up to 6%.
ProfitBricks’ variability was elevated for sequential read/write at 15%, but high on random read/write at 40%.
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
m4
.2xla
rge
c4.2
xla
rge
r3.2
xla
rge
A4
D4v2
D13v2
DS
I_8c16r
DS
I_8c32r
DS
I_8c64r
PB
_8c1
6r
PB
_8c3
2r
PB
_8c6
4r
AWS Azure DSI vCloud PB
IOP
S
Figure 4.5C: STORAGE PERFORMANCE - RANDOM R/W
(1000GB)
Provider VM Min 5TH Median 95TH Max
8 v
CP
Us
AWS
m4.2xlarge 2974 2984 2986 2987 2988
c4.2xlarge 2972 2985 2986 2988 2988
r3.2xlarge 2537 2984 2986 2987 2988
A4 421 499 502 503 503
Microsoft D4v2 277 496 502 503 503
D13v2 410 497 502 503 503
DSI
vCloud
8c16r 567 983 2459 6587 8069
8c32r 1268 1844 4330 7976 9246
8c64r 479 1737 4062 7492 8896
8c16r 225 313 613 970 1406
PB 8c32r 231 278 495 972 1417
8c64r 717 787 1040 1629 1695
Cloud Spectator Performance Report
21
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Internal Network Throughput
Below are the results of internal network testing on all providers. Internal network was examined by running
TCP connections between two servers within the same data center/region/zone. The tables on the right specify
the scores achieved by each provider’s VMs. T
--Units Measured in Mbit/s--
--Units Measured in Mbit/s--
0
1000
2000
3000
4000
5000
6000
7000
8000
m4
.la
rge
c4.la
rge
r3.la
rge
A2
D2v2
D11v2
DS
I_2c4r
DS
I_2c8r
DS
I_2c16r
PB
_2c4
r
PB
_2c8
r
PB
_2c1
6r
AWS Azure DSI vCloud PB
MB
IT/S
Figure 4.6: INTERNAL NETWORK PERFORMANCE
(2vCPU VMs)
0
1000
2000
3000
4000
5000
6000
7000
8000
m4
.xla
rge
c4.x
larg
e
r3.x
larg
e
A3
D3v2
D12v2
DS
I_4c8r
DS
I_4c16r
DS
I_4c32r
PB
_4c8
r
PB
_4c1
6r
PB
_4c3
2r
AWS Azure DSI vCloud PB
MB
IT/S
Figure 4.6B: INTERNAL NETWORK PERFORMANCE
(4vCPU VMs)
Provider VM Min 5TH Median 95TH Max
2 v
CP
Us
AWS
m4.large 445 445 445 445 445
c4.large 610 610 610 610 610
r3.large 492 492 492 492 492
A2 42 435 476 477 477
Microsoft D2v2 735 831 1275 1429 1430
D11v2 531 939 1282 1430 1431
DSI vCloud
2c4r 46 51 1808 4430 5381
2c8r 50 57 1603 3975 5008
2c16r 1981 2510 4033 6900 7132
2c4r 4271 4792 5346 5663 5891
PB 2c8r 4792 5226 6015 6355 6675
2c16r 4777 5170 5884 6138 6327
Provider VM Min 5TH Median 95TH Max
4 v
CP
Us
AWS
m4.xlarge 753 754 754 754 754
c4.xlarge 1227 1227 1227 1227 1228
r3.xlarge 706 706 707 707 707
A3 670 761 855 953 953
Microsoft D3v2 643 725 1795 2262 2621
D12v2 686 988 1501 2257 2568
DSI
vCloud
4c8r 67 779 3985 7345 7530
4c16r 41 49 2627 4305 5254
4c32r 48 56 3038 4432 5530
4c8r 3437 3780 4464 5107 5307
PB 4c16r 4865 5429 6041 6395 6669
4c32r 4862 5331 5817 6094 6263
Cloud Spectator Performance Report
22
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
--Units Measured in Mbit/s--
Observations: Internal Network Throughput Performance
With regards to internal network throughput, DSI exhibited high performance variability with CVs averaging
56% across all three VM sizes. Microsoft VMs’ performance variability was relatively lower with CVs
averaging 23% across all the VMs examined, followed by ProfitBrick averaging 7%. AWS’s throughput was
very stable, exhibiting CVs that were under 1%.
Additional Observations DSI vCloud VMs recorded the highest maximum values for internal network across all three VM sizes.
AWS’s VMs deliver a specific network throughput depending on the size and family of VM. As the size of the VMs increase, the network
throughput increases as well. The throughput starts at 445-610Mbit/s for the 2vCPU VMs and rises to 707-1227Mbit/s for the 4vCPU VMs
and capping at 962-2382Mbit/s for the 8vCPU VMs.
Microsoft’s VMs varied by instance type as well, with the A-series limited between 400-1,600Mbit/s, while the D-series fluctuated between
500-2600Mbit/s.
ProfitBricks exhibited the highest median values for 2vCPU and 4vCPU VMs. The median performance values primarily ranged between 4-
6Gbps.
0
1000
2000
3000
4000
5000
6000
7000
8000
m4
.2xla
rge
c4.2
xla
rge
r3.2
xla
rge
A4
D4v2
D13v2
DS
I_8c16r
DS
I_8c32r
DS
I_8c64r
PB
_8c1
6r
PB
_8c3
2r
PB
_8c6
4r
AWS Azure DSI vCloud PB
MB
IT/S
Figure 4.6C: INTERNAL NETWORK PERFORMANCE
(8vCPU VMs)
Provider VM Min 5TH Median 95TH Max
8 v
CP
Us
AWS
m4.2xlarge 962 962 962 962 962
c4.2xlarge 2378 2382 2382 2382 2382
r3.2xlarge 962 962 962 962 962
A4 697 776 1061 1433 1594
Microsoft D4v2 751 858 1355 2017 2573
D13v2 296 351 1253 1878 2634
DSI
vCloud
8c16r 4070 4278 6145 7174 7334
8c32r 3875 4279 6314 6511 6786
8c64r 53 69 1754 5865 6630
8c16r 2281 3473 4191 4901 5236
PB 8c32r 3503 4155 5026 5590 5625
8c64r 2191 3050 4103 4702 5326
Cloud Spectator Performance Report
23
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Price-Performance Analysis Results
The following section provides the price-performance comparisons of the VMs tested in the performance
analysis section above. The values used in the comparison are taken from the raw performance scores shown in
the preceding section and the pricing for the VMs being examined (see Methodology). Scores are out of 100,
with the highest value VM in each test scoring 100, while the remaining VM scores are measured against the
price-performance value of highest scoring VM.
Provider Performance Analysis: Provider Pricing
Below are the prices per tested virtual machine across all providers and all virtual machine sizes. Storage prices
are broken out from the virtual machine price as persistent storage was tested for every machine size.
2vCPU/100GB VM & Storage Pricing
Amazon Microsoft DSI vCloud ProfitBricks
VM Type c4.large m4.large r3.large A2 Std D2v2 D11v2 2c4r 2c8r 2c16r 2c4r 2c8r 2c16r
VM € € 86.96 € 92.80 € 129.79 € 73.88 € 98.48 € 131.11 € € € € 71.54 € 84.68 € 110.96
Storage € € 10.58 € 10.58 € 10.58 € 4.22 € 4.22 € 4.22 € € € € 4.00 € 4.00 € 4.00
Total € 97.54 € 103.38 € 140.37 € 78.10 € 102.70 € 135.33 € 116.76 € 114.89 € 128.22 € 75.54 € 88.68 € 114.96
4vCPU/500GB VM & Storage Pricing
Amazon Microsoft DSI vCloud ProfitBricks
VM Type c4.xlarge m4.xlarge r3.xlarge A3 Std D3v2 D12v2 4c8r 4c16r 4c32r 4c8r 4c16r 4c32r
VM € € 173.27 € 184.96 € 259.59 € 147.75 € 196.37 € 262.87 € € € € 143.08 € 169.36 € 221.92
Storage € € 52.90 € 52.90 € 52.90 € 21.10 € 21.10 € 21.10 € € € € 20.00 € 20.00 € 20.00
Total € 226.17 € 237.85 € 312.48 € 168.85 € 217.47 € 283.97 € 160.27 € 177.69 € 181.52 € 163.08 € 189.36 € 241.92
8vCPU/1000GB VM & Storage Pricing
Amazon Microsoft DSI vCloud ProfitBricks
VM Type c4.2xlarge m4.2xlarge r3.2xlarge A4 Std D4v2 D13v2 8c16r 8c32r 8c64r 8c16r 8c32r 8c64r
VM € € 346.55 € 369.91 € 519.18 € 295.50 € 392.74 € 524.51 € € € € 286.16 € 338.72 € 443.84
Storage € € 105.79 € 105.79 € 105.79 € 42.20 € 42.20 € 42.20 € € € € 40.00 € 40.00 € 40.00
Total € 452.34 € 475.70 € 624.97 € 337.70 € 434.94 € 566.71 € 193.83 € 342.15 € 319.77 € 326.16 € 378.72 € 483.84
*prices based on 730 hours per month*
Pricing Considerations: Regional pricing based off the data center locations listed in the Methodology section.
DSI vCloud
pricing is based on the resource pool utilization, not the individual VMs or storage volumes
Pricing is accurate as of September 16, 2016
USD to EUR conversion rate of .889 was used for AWS
Cloud Spectator Performance Report
24
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Cloud Spectator Performance Report
25
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
vCPU VALUE ANALYSIS RESULTS: INTEGER
vCPU Value – Integer Tasks (Multicore): 2vCPU
Provider VM CloudSpecs
Score
Amazon m4.large 59
Amazon c4.large 73
Amazon r3.large 42
Microsoft A2 54
Microsoft D2v2 100
Microsoft D11v2 75
DSI vCloud 2c4r 95
DSI vCloud 2c8r 94
DSI vCloud 2c16r 79
PB 2c4r 89
PB 2c8r 75
PB 2c16r 57
vCPU Value – Integer Tasks (Multicore): 4vCPU
Provider VM CloudSpecs
Score
Amazon m4.xlarge 37
Amazon c4.xlarge 45
Amazon r3.xlarge 27
Microsoft A3 35
Microsoft D3v2 60
Microsoft D12v2 46
DSI vCloud 4c8r 100
DSI vCloud 4c16r 91
DSI vCloud 4c32r 90
PB 4c8r 52
PB 4c16r 45
PB 4c32r 35
vCPU Value – Integer Tasks (Multicore): 8vCPU
Provider VM CloudSpecs
Score
Amazon m4.2xlarge 23
Amazon c4.2xlarge 28
Amazon r3.2xlarge 17
Microsoft A4 20
Microsoft D4v2 34
Microsoft D13v2 21
DSI vCloud 8c16r 100
DSI vCloud 8c32r 56
DSI vCloud 8c64r 59
PB 8c16r 29
PB 8c32r 25
PB 8c64r 19
59
73
42
54
100
75
95 94
79
89
75
57
0
10
20
30
40
50
60
70
80
90
100
m4.la
rge
c4
.larg
e
r3.la
rge
A2
D2
v2
D1
1v2
DS
I_2c4
r
DS
I_2c8
r
DS
I_2c1
6r
PB
_2
c4r
PB
_2
c8r
PB
_2
c16
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of vCPU (Integer) - 2vCPU VMs
3745
2735
60
46
10091 90
5245
35
0
10
20
30
40
50
60
70
80
90
100
m4.x
larg
e
c4
.xla
rge
r3.x
larg
e
A3
D3
v2
D1
2v2
DS
I_4c8
r
DS
I_4c1
6r
DS
I_4c3
2r
PB
_4
c8r
PB
_4
c16
r
PB
_4
c32
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of vCPU (Integer) - 4vCPU VMs
2328
1720
34
21
100
56 59
2925
19
0
10
20
30
40
50
60
70
80
90
100
m4.2
xla
rge
c4
.2xla
rge
r3.2
xla
rge
A4
D4
v2
D1
3v2
DS
I_8c1
6r
DS
I_8c3
2r
DS
I_8c6
4r
PB
_8
c16
r
PB
_8
c32
r
PB
_8
c64
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of vCPU (Integer) - 8vCPU VMs
Cloud Spectator Performance Report
26
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
vCPU VALUE ANALYSIS RESULTS: FLOATING POINT
vCPU Value – Floating Point Tasks (Multicore): 2vCPU
Provider VM CloudSpecs
Score
Amazon m4.large 59
Amazon c4.large 73
Amazon r3.large 43
Microsoft A2 54
Microsoft D2v2 100
Microsoft D11v2 74
DSI vCloud 2c4r 95
DSI vCloud 2c8r 95
DSI vCloud 2c16r 81
PB 2c4r 76
PB 2c8r 64
PB 2c16r 47
vCPU Value – Floating Point Tasks (Multicore): 4vCPU
Provider VM CloudSpecs
Score
Amazon m4.xlarge 36
Amazon c4.xlarge 45
Amazon r3.xlarge 28
Microsoft A3 36
Microsoft D3v2 63
Microsoft D12v2 47
DSI vCloud 4c8r 100
DSI vCloud 4c16r 91
DSI vCloud 4c32r 89
PB 4c8r 44
PB 4c16r 39
PB 4c32r 30
vCPU Value – Floating Point Tasks (Multicore): 8vCPU
Provider VM CloudSpecs
Score
Amazon m4.2xlarge 22
Amazon c4.2xlarge 27
Amazon r3.2xlarge 17
Microsoft A4 21
Microsoft D4v2 36
Microsoft D13v2 23
DSI vCloud 8c16r 100
DSI vCloud 8c32r 56
DSI vCloud 8c64r 59
PB 8c16r 25
PB 8c32r 21
PB 8c64r 17
59
73
43
54
100
74
95 95
8176
64
47
0
10
20
30
40
50
60
70
80
90
100
m4.la
rge
c4
.larg
e
r3.la
rge
A2
D2
v2
D1
1v2
DS
I_2c4
r
DS
I_2c8
r
DS
I_2c1
6r
PB
_2
c4r
PB
_2
c8r
PB
_2
c16
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of vCPU (Floating Point) -2vCPU VMs
3645
2836
63
47
10091 89
4439
30
0
10
20
30
40
50
60
70
80
90
100
m4.x
larg
e
c4
.xla
rge
r3.x
larg
e
A3
D3
v2
D1
2v2
DS
I_4c8
r
DS
I_4c1
6r
DS
I_4c3
2r
PB
_4
c8r
PB
_4
c16
r
PB
_4
c32
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of vCPU (Floating Point) -4vCPU VMs
2227
1721
36
23
100
56 59
25 2117
0
102030
4050
6070
8090
100
m4.2
xla
rge
c4
.2xla
rge
r3.2
xla
rge
A4
D4
v2
D1
3v2
DS
I_8c1
6r
DS
I_8c3
2r
DS
I_8c6
4r
PB
_8
c16
r
PB
_8
c32
r
PB
_8
c64
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of vCPU (Floating Point) -8vCPU VMs
Cloud Spectator Performance Report
27
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
MEMORY VALUE ANALYSIS RESULTS
Memory Value (Multicore): 2vCPU
Provider VM CloudSpecs
Score
Amazon m4.large 75
Amazon c4.large 85
Amazon r3.large 39
Microsoft A2 61
Microsoft D2v2 71
Microsoft D11v2 53
DSI vCloud 2c4r 100
DSI vCloud 2c8r 90
DSI vCloud 2c16r 80
PB 2c4r 72
PB 2c8r 56
PB 2c16r 37
Memory Value (Multicore): 4vCPU
Provider VM CloudSpecs
Score
Amazon m4.xlarge 56
Amazon c4.xlarge 62
Amazon r3.xlarge 33
Microsoft A3 28
Microsoft D3v2 34
Microsoft D12v2 27
DSI vCloud 4c8r 100
DSI vCloud 4c16r 100
DSI vCloud 4c32r 99
PB 4c8r 35
PB 4c16r 26
PB 4c32r 21
Memory Value (Multicore): 8vCPU
Provider VM CloudSpecs
Score
Amazon m4.2xlarge 27
Amazon c4.2xlarge 30
Amazon r3.2xlarge 16
Microsoft A4 13
Microsoft D4v2 16
Microsoft D13v2 11
DSI vCloud 8c16r 100
DSI vCloud 8c32r 54
DSI vCloud 8c64r 47
PB 8c16r 14
PB 8c32r 12
PB 8c64r 9
75
85
39
61
71
53
100
90
8072
56
37
0
10
20
30
40
50
60
70
80
90
100
m4.la
rge
c4
.larg
e
r3.la
rge
A2
D2
v2
D1
1v2
DS
I_2c4
r
DS
I_2c8
r
DS
I_2c1
6r
PB
_2
c4r
PB
_2
c8r
PB
_2
c16
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Memory - 2vCPU VMs
5662
3328
3427
100 100 99
35
2621
0
10
20
30
40
50
60
70
80
90
100
m4.x
larg
e
c4
.xla
rge
r3.x
larg
e
A3
D3
v2
D1
2v2
DS
I_4c8
r
DS
I_4c1
6r
DS
I_4c3
2r
PB
_4
c8r
PB
_4
c16
r
PB
_4
c32
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Memory - 4vCPU VMs
2730
16 13 1611
100
5447
14 12 9
0
10
20
30
40
50
60
70
80
90
100
m4.2
xla
rge
c4
.2xla
rge
r3.2
xla
rge
A4
D4
v2
D1
3v2
DS
I_8c1
6r
DS
I_8c3
2r
DS
I_8c6
4r
PB
_8
c16
r
PB
_8
c32
r
PB
_8
c64
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Memory - 8vCPU VMs
Cloud Spectator Performance Report
28
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
STORAGE VALUE ANALYSIS RESULTS: SEQUENTIAL OPERATIONS
Storage Value – Sequential Operations: 2vCPU
Provider VM CloudSpecs
Score
Amazon m4.large 6
Amazon c4.large 6
Amazon r3.large 4
Microsoft A2 12
Microsoft D2v2 9
Microsoft D11v2 7
DSI vCloud 2c4r 99
DSI vCloud 2c8r 100
DSI vCloud 2c16r 94
PB 2c4r 86
PB 2c8r 76
PB 2c16r 59
Storage Value – Sequential Operations: 4vCPU
Provider VM CloudSpecs
Score
Amazon m4.xlarge 20
Amazon c4.xlarge 17
Amazon r3.xlarge 18
Microsoft A3 7
Microsoft D3v2 6
Microsoft D12v2 4
DSI vCloud 4c8r 65
DSI vCloud 4c16r 100
DSI vCloud 4c32r 83
PB 4c8r 54
PB 4c16r 46
PB 4c32r 36
Storage Value – Sequential Operations: 8vCPU
Provider VM CloudSpecs
Score
Amazon m4.2xlarge 16
Amazon c4.2xlarge 17
Amazon r3.2xlarge 12
Microsoft A4 4
Microsoft D4v2 3
Microsoft D13v2 2
DSI vCloud 8c16r 100
DSI vCloud 8c32r 57
DSI vCloud 8c64r 60
PB 8c16r 27
PB 8c32r 24
PB 8c64r 20
6 6 412 9 7
99 10094
86
76
59
0
10
20
30
40
50
60
70
80
90
100
m4.la
rge
c4
.larg
e
r3.la
rge
A2
D2
v2
D1
1v2
DS
I_2c4
r
DS
I_2c8
r
DS
I_2c1
6r
PB
_2
c4r
PB
_2
c8r
PB
_2
c16
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Storage (Sequential R/W) -100GB
20 17 18
7 6 4
65
100
83
5446
36
0
10
20
30
40
50
60
70
80
90
100
m4.x
larg
e
c4
.xla
rge
r3.x
larg
e
A3
D3
v2
D1
2v2
DS
I_4c8
r
DS
I_4c1
6r
DS
I_4c3
2r
PB
_4
c8r
PB
_4
c16
r
PB
_4
c32
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Storage (Sequential R/W) -260GB
16 1712
4 3 2
100
57 60
27 24 20
0102030405060708090
100
m4.2
xla
rge
c4
.2xla
rge
r3.2
xla
rge
A4
D4
v2
D1
3v2
DS
I_8c1
6r
DS
I_8c3
2r
DS
I_8c6
4r
PB
_8
c16
r
PB
_8
c32
r
PB
_8
c64
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Storage (Sequential R/W) -500GB
Cloud Spectator Performance Report
29
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
STORAGE VALUE ANALYSIS RESULTS: RANDOM OPERATIONS
Storage Value – Random Operations: 2vCPU
Provider VM CloudSpecs
Score
Amazon m4.large 8
Amazon c4.large 8
Amazon r3.large 6
Microsoft A2 18
Microsoft D2v2 13
Microsoft D11v2 10
DSI vCloud 2c4r 98
DSI vCloud 2c8r 100
DSI vCloud 2c16r 85
PB 2c4r 24
PB 2c8r 20
PB 2c16r 17
Storage Value – Random Operations: 4vCPU
Provider VM CloudSpecs
Score
Amazon m4.xlarge 83
Amazon c4.xlarge 87
Amazon r3.xlarge 63
Microsoft A3 20
Microsoft D3v2 15
Microsoft D12v2 12
DSI vCloud 4c8r 50
DSI vCloud 4c16r 100
DSI vCloud 4c32r 72
PB 4c8r 25
PB 4c16r 18
PB 4c32r 15
Storage Value – Random Operations: 8vCPU
Provider VM CloudSpecs
Score
Amazon m4.2xlarge 49
Amazon c4.2xlarge 52
Amazon r3.2xlarge 38
Microsoft A4 12
Microsoft D4v2 9
Microsoft D13v2 7
DSI vCloud 8c16r 100
DSI vCloud 8c32r 100
DSI vCloud 8c64r 100
PB 8c16r 15
PB 8c32r 10
PB 8c64r 17
8 8 6
1813 10
98 100
85
24 2017
0
10
20
30
40
50
60
70
80
90
100
m4.la
rge
c4
.larg
e
r3.la
rge
A2
D2
v2
D1
1v2
DS
I_2c4
r
DS
I_2c8
r
DS
I_2c1
6r
PB
_2
c4r
PB
_2
c8r
PB
_2
c16
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Storage (Random R/W) -100GB
8387
63
2015 12
50
100
72
2518 15
0
10
20
30
40
50
60
70
80
90
100
m4.x
larg
e
c4
.xla
rge
r3.x
larg
e
A3
D3
v2
D1
2v2
DS
I_4c8
r
DS
I_4c1
6r
DS
I_4c3
2r
PB
_4
c8r
PB
_4
c16
r
PB
_4
c32
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Storage (Random R/W) -260GB
49 52
38
12 9 7
100 100 100
1510
17
0
10
20
30
40
50
60
70
80
90
100
m4.2
xla
rge
c4
.2xla
rge
r3.2
xla
rge
A4
D4
v2
D1
3v2
DS
I_8c1
6r
DS
I_8c3
2r
DS
I_8c6
4r
PB
_8
c16
r
PB
_8
c32
r
PB
_8
c64
r
AWS Azure DSI vCloud PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Storage (Random R/W) -500GB
Cloud Spectator Performance Report
30
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
INTERNAL NETWORK VALUE ANALYSIS RESULTS: THROUGHPUT
Internal Network Value – Throughput: 2vCPU
Provider VM CloudSpecs
Score
Amazon m4.large 6
Amazon c4.large 9
Amazon r3.large 5
Microsoft A2 9
Microsoft D2v2 18
Microsoft D11v2 13
DSI vCloud 2c4r 22
DSI vCloud 2c8r 20
DSI vCloud 2c16r 44
PB 2c4r 100
PB 2c8r 96
PB 2c16r 72
Internal Network Value – Throughput: 4vCPU
Provider VM CloudSpecs
Score
Amazon m4.xlarge 10
Amazon c4.xlarge 17
Amazon r3.xlarge 7
Microsoft A3 16
Microsoft D3v2 26
Microsoft D12v2 17
DSI vCloud 4c8r 78
DSI vCloud 4c16r 46
DSI vCloud 4c32r 52
PB 4c8r 86
PB 4c16r 100
PB 4c32r 75
Internal Network Value – Throughput: 8vCPU
Provider VM CloudSpecs
Score
Amazon m4.2xlarge 6
Amazon c4.2xlarge 17
Amazon r3.2xlarge 5
Microsoft A4 10
Microsoft D4v2 10
Microsoft D13v2 7
DSI vCloud 8c16r 100
DSI vCloud 8c32r 58
DSI vCloud 8c64r 17
PB 8c16r 41
PB 8c32r 42
PB 8c64r 27
6 9 5 918
1322 20
44
10096
72
0
10
20
30
40
50
60
70
80
90
100
m4.la
rge
c4
.larg
e
r3.la
rge
A2
D2
v2
D1
1v2
DS
I_2c4
r
DS
I_2c8
r
DS
I_2c1
6r
PB
_2
c4r
PB
_2
c8r
PB
_2
c16
r
AWS Azure DSI PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Internal Network -2vCPU VMs
1017
716
2617
78
4652
86
100
75
0102030405060708090
100
m4.x
larg
e
c4
.xla
rge
r3.x
larg
e
A3
D3
v2
D1
2v2
DS
I_4c8
r
DS
I_4c1
6r
DS
I_4c3
2r
PB
_4
c8r
PB
_4
c16
r
PB
_4
c32
r
AWS Azure DSI PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Internal Network -4vCPU VMs
6
17
510 10 7
100
58
17
41 42
27
01020
3040
5060
7080
90100
m4.2
xla
rge
c4
.2xla
rge
r3.2
xla
rge
A4
D4
v2
D1
3v2
DS
I_8c1
6r
DS
I_8c3
2r
DS
I_8c6
4r
PB
_8
c16
r
PB
_8
c32
r
PB
_8
c64
r
AWS Azure DSI PB
Clo
ud
Sp
ecs S
co
re
Price-Performance of Internal Network -8vCPU VMs
Cloud Spectator Performance Report
31
Copyright 2016 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Conclusion
Selecting the right provider and virtual machines requires a thorough and accurate performance
comparison. The study conducted for this report offers a general understanding of performance strengths and
weaknesses across each included vendor’s VMs, and should be considered a source of information to help guide
readers in their own testing and analyses. The processor, memory bandwidth, network storage, and internal
network were all examined and results clearly show that no single provider can offer everything to fit
everyone’s needs. Therefore, in order to accurately select the right provider for a business or an application,
performance analysis is crucial.
Results from this study show that T-Systems’ DSI vCloud VMs displayed strong overall performance
and offered high price-performance value. DSI VMs demonstrated high performance for processing, storage
and internal network. DSI performance was highest in the study for processing and storage. Price-performance
value of DSI was highest in the study for storage and memory bandwidth, while having the highest value for
processor price-performance for two out of three VM sizes.
Performance in the industry cannot be assumed to be equal or even similar, as illustrated in this report.
When examining processor and memory bandwidth performance, tiered providers such as AWS and Microsoft
offer varying performance depending on the family/series of the VM, despite having equivalent amounts of
vCPUs and similar amounts of memory.
While this study was conducted in the manner of understanding a typical end user experience, it should
not be assumed to be accurate for all use cases. Stress testing was conducted to better understand fluctuation
and theoretically sustained performance, and should be seen as a general indication of overall provider
performance.
About
About Cloud Spectator
Cloud Spectator is a cloud analyst agency focused on cloud Infrastructure-as-a-Service (IaaS) performance. The
company actively monitors several of the largest IaaS providers in the world, comparing VM performance (i.e.,
CPU, RAM, disk, internal network, and workloads) and pricing to achieve transparency in the cloud market.
The company helps cloud providers understand their market position and helps business make intelligent
decisions in selecting cloud providers and lowering total cost of ownership. The firm was founded in early 2011
and is located in Boston, MA.
For questions about this report, to request a custom report, or if you have general inquiries about our products
and services, please contact Cloud Spectator at +1 (617) 300-0711 or contact@cloudspectator.com.
For press/media related inquiries, please contact:
Ken Balazs
VP Sales & Marketing
kbalazs@cloudspectator.com
top related