© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

STG302: Maximizing EC2 and Elastic Block

Store Disk Performance

Miles Ward, Sr. Manager Solutions Architecture, Amazon Web Services

November 12, 2013

EBS =

So.. what storage workloads?

A few definitions…

A “Normal” Hard Drive

❶ Select a new type of Provisioned IOPS volume

❸ Specify the number of I/O operations per

second your application needs, up to 4000

IOPS per volume. The volume will deliver the

specified I/O operations per second.

❷ Specify the volume capacity

Review: Provisioned IOPS Volumes

Minimum ratio of capacity to IOPS = 1:30

$aws ec2 create-volume --availability-zone us-east-1a --size 134 --volume-type io1 --iops 4000

Amazon EBS Standard

Amazon Elastic

Block Storage

(EBS)

• IOPS: ~100 IOPS steady-state, with best-effort bursts to hundreds

• Throughput: variable by workload, best effort to 10s of MB/s.

• Latency: Varies, reads typically <20 ms writes typically <10 ms

• Capacity: As provisioned, up to 1 TB

EBS PIOPS

Amazon Elastic

Block Storage

(EBS)

• IOPS: Within 10% of up to 4000 IOPS,

99.9% of a given year, as provisioned.

• Throughput: 16 KB per I/O = up to 64 MB/s, as provisioned.

• Latency: low and consistent. Second/IOPS, at recommended QD

• Capacity: As provisioned, up to 1 TB

* *

USE PIOPS AND EBS OPTIMIZED

PLEASE

Why the ? *

EC2

An I/O

EBS

Why the ? * A bigger I/O

EBS

>16 KB? = more than one I/O

EC2

EC2

Why the ? * A smaller I/O

EBS

< 16 KB? = no “bonus” random I/Os

EC2

Why the ? * An I/O

EBS

Once it’s gone, it’s gone.

EC2

Why the ? * An I/O

EBS

Just because Amazon EC2 sends more

work doesn’t mean there’s enough

bandwidth to handle it!

EC2

Why the ? * An I/O

Without more bandwidth,

more EBS volumes or higher PIOPS won’t help!

EC2

Why the ? * an I/O

EBS

Just because EC2 sends more work

doesn’t mean there’s enough

IOPS to handle it!

Why the ? *

EC2

An I/O

EBS

Adding bandwidth alone

or increasing instance size won’t help!

EBS-Optimized

EBS-Optimized

Oh, YEAH!! *

EC2

A “boatload” of I/O

*

Architecting for Performance

• IOPS consistency requires EBS-

optimized instances

• Maximum throughput delivered by

Amazon EBS is limited by Amazon

EC2 bandwidth

• EBS throughput =

EBS IOPS × Block size

– Ex: 64 MB/s = 4000 IOPS × 16 KB

Instance vCPU EBS

Optimized Max MB/s Max 16k IOPS

t1 micro 1 No 32MB/s 2000

m1.small 1 No 64MB/s 4000

m1.medium 1 No 64MB/s 4000

m1.large 2 Yes 64MB/s 4000

m1.xlarge 4 Yes 128MB/s 8000

m3.xlarge 4 Yes 64MB/s 4000

m3.2xlarge 8 Yes 128MB/s 8000

c1.medium 2 No 32MB/s 2000

c1.xlarge 8 Yes 128MB/s 8000

cc2.8xlarge 32 NA 800MB/s 50,000

m2.xlarge 2 No 64MB/s 4000

m2.2xlarge 4 Yes 64MB/s 4000

m2.4xlarge 8 Yes 128MB/s 8000

cr1.8xlarge 32 NA 800MB/s 50,000

hi1.4xlarge 16 NA 800MB/s 50,000

cg1.4xlarge 16 NA 800MB/s 50,000

Max 8k =

2x

Max 4k =

4x*

Max 2k =

8x*

*Maximum IOPS is also limited to ~100,000 per 32 vCpu,

irrespective of block size/throughput.

EBS-Optimized • EBS-optimized offers a “SAN-like” experience

• Network interference results:

No impact on IOPS or

Amazon EBS

throughput

Row Labels AvgBW AvgIOPs

m3.2xlarge (EBS-optimized)

no network load

random

read 57,542 3,596

write 61,713 3,857

rw (70/30) 66,997 4,186

sequential

read 61,708 3,856

write 61,651 3,853

rw (70/30) 66,996 4,187

with network load-test1

random

read 59,835 3,739

write 63,407 3,962

rw (70/30) 68,859 4,303

sequential

read 61,736 3,858

write 63,360 3,959

rw (70/30) 68,859 4,302

Network interference tests

I/O Characteristics • I/O size

– 4 KB to 64 MB

• I/O pattern – Sequential and random

• I/O type – Read and write

• PIOPS always measures I/O in terms of 16 KB or smaller

• PIOPS delivers same number of IOPS for sequential and random I/O

• PIOPS delivers same number of IOPS for reads or writes

PIOPS is optimized for database workloads

Smaller I/O (4 KB, 16 KB)

• Why are 4 KB I/O size in sequential operations driving greater than 4000 IOPS?

• Why is m1.large and m3.xlarge IOPS at 16 KB less than 4000 IOPS?

• Database needs 5000 ops/second. How many IOPS do I need to provision?

• What happens when customers want to burst beyond provisioned IOPS?

IOPS and BW

performance at QD

8

m1.large M3.xlarge m3.2xlarge

AvgIOPs

Avg BW

( KB) AvgIOPs

Avg BW

( KB) AvgIOPs

Avg

BW

( KB)

Write

sequential

4K 4146 16,587 5997 23,990 7767 31,068

16K 3712 59,402 4157 55,461 4153 60,332

Write random

4K 4082 16,329 4433 17,733 4178 16,712

16K 3713 59,422 3743 53,813 4153 60,332

Read

sequential

4K 5301 21,205 9232 36,929 13450 53,802

16K 3533 56,535 4796 56,824 4153 60,332

Read random

4K 4538 18,154 5864 23,457 4177 16,711

16K 3510 56,168 3583 51,246 4153 60,332

Results for 400 GB volume with 4000 IOPS at QD 8; EBS-optimized instances

Larger I/O (128 KB, 512 KB)

• Why am I seeing only 462

IOPS on a volume?

• Why there is no difference

in performance for

random and sequential

workloads?

• How should I configure

500 MB/s read or write

throughput using PIOPS

volumes

IOPS and BW

performance at QD 8

m1.large M3.xlarge m3.2xlarge

AvgIOPs

Avg

BW(KB) AvgIOPs

Avg

BW(KB) AvgIOPs

Avg

BW(KB)

Write sequential

128K 462 59,268 462 59,145 522 66,843

512K 115 59,292 115 59,278 130 66,804

Write random

128K 462 59,265 462 59,241 522 66,843

512K 115 59,291 115 59,272 130 66,843

Read sequential

128K 455 58,240 454 58,225 522 66,843

512K 113 58,003 114 58,589 130 66,843

Read random

128K 455 58,236 454 58,215 522 66,843

512K 113 57,960 114 58,496 130 66,805

4000, 16 KB read/write per second, or 2000 32 KB read/write per second, or 1000 64 KB read/write per second…

Results for 400 GB volume with 4000 IOPS at QD 8

Write Latency • Database applications care

about latency as much as IOPS

delivered

• There is an Interdependency

among IOPS, queue depth, and

latency

• Current guidance is queue

depth of 1 for every 200 IOPS,

but if latency-bound and write-

heavy, 1:500 – 1:1000 is better.

1 4 8 12 16 20 24 28 32

AvgIOPS ( Count) 845 4152 4153 4177 4152 4176 4177 4177 4151

AvgTP90 ( ms) 3.13 1.47 2.03 3.56 3.62 5.54 6.18 7.48 7.71

845

4152

3.13

1.47

2.03

3.56 3.62

5.54

6.18

7.48 7.71

0

1

2

3

4

5

6

7

8

9

0

500

1000

1500

2000

2500

3000

3500

4000

4500

Wri

te IO

PS

16 KBk random WRITE- M3.2Xlarge EBS-optimized

L

a

t

e

n

c

y

QD

Read Latency

• Reads can take advantage of a

deeper queue

• Current guidance is queue

depth of 1 for every 250 IOPS

• EBS-optimized provides

predictable latency

1 4 8 12 16 20 24 28 32

AvgIOPS ( Count) 1864 4153 4153 4177 4120 2800 1965 1213 1089

AvgTP90 ( ms) 0.68 1.46 2.15 3.43 3.88 5.18 91.14 93.18 93.70

1864

4153 4120

1965

0.68 1.46 2.15 3.43 3.88 5.18

91.14 93.18 93.70

0

10

20

30

40

50

60

70

80

90

100

0

500

1000

1500

2000

2500

3000

3500

4000

4500

Read

IO

PS

16 KB random READ - M3.2Xlarge EBS-optimized

L

a

t

e

n

c

y

QD

• Performance requirements may be driven by IOPS or latency or both

• Recommendation is to start with queue depth of 4 and tune based on IOPS and latency requirement – Some customers may need lowest possible latency; this can be

achieved at queue depth of 1 or 2

• Very high queue depths ( >24) may decrease IOPS count as well as increase latency

Architecting for Performance: Latency

• Typically 5%, extreme worst case of 50% performance reduction in IOPS and latency when volumes are used without pre-warming

– Performance is as provisioned when all the chunks are accessed

• Recommendation if testing or you have spare setup time: – Write to every 4 MB block before using new volumes

• Linux: DD

• Windows: NTFS Full format

– Takes roughly an hour to pre-warm 1TB 4KB PIOPS volume

– Be warned, can take up to a day for a 1 TB standard EBS volume

Pre-warming EBS volumes

What about RAID?

Amazon Elastic

Block Storage

(EBS)

Amazon Elastic

Compute Cloud

(EC2)

Amazon Elastic

Block Storage

(EBS)

Amazon Elastic

Compute Cloud

(EC2)

Architecting for Performance: RAID • Customers stripe number of

volumes to drive higher

IOPS and throughput – RAID 0 or RAID 10

• How should customers think

about taking snapshots on a

striped volume?

– Quiesce file systems and

take snapshot

– Unmount file system and

take snapshot

– Use OS-specific tools

12×400 GB PIOPS volumes, pre-warmed,

RAID 0 LVM, Stripe size 64 KB, attached to

CR1 instance

IO Pattern Block Sizes Thread Write IOPS Write BW (MB) Read IOPS Read BW (MB)

Sequential

4K 8 33,500 134 48,250 193

16K 8 13,875 222 48,063 769

1M 1 247 247 823 823

Random

4K 8 35,250 141 48,250 193

16K 8 13,875 222 42,125 674

1M 1 496 496 795 795

Check out STG402 for new advanced snapshot tools!!

Workload/

Software

Typical block

size

Random

/Seq?

Typical

correct

EBS type

Max EBS @

500 MB/s

Max EBS @

1 GB/s

Max EBS @ CC/H

Oracle DB

Configurable:2 KB –

16 KB

Default 8 KB

random PIOPS RAID 0, 2 × 4000

PIOPS:

~7800 IOPS

RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 24 × 4000 PIOPS:

~96,000 IOPS

Microsoft

SQL Server

8 KB w/ 64 KB

extents

random PIOPS RAID 0, 2 × 4000

PIOPS:

~7800 IOPS

RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 20 × 4000 PIOPS:

~80,000 IOPS

MySQL

16 KB random PIOPS 1 x 4000 PIOPS:

~4000 IOPS

RAID 0, 2 × 4000

PIOPS:

~7,800 IOPS

RAID 0, 12 × 4000 PIOPS:

~48,000 IOPS

PostgreSQL 8 KB random PIOPS RAID 0, 2 × 4000

PIOPS:

~7,800 IOPS

RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 24 × 4000 PIOPS:

~96,000 IOPS

IBM DB/2 Configurable:2 KB –-

256 KB

Default 32 KB

random PIOPS 1 x 4000 PIOPS:

~2000 IOPS

RAID 0, 4 × 4000

PIOPS:

~8,000 IOPS

RAID 0, 12 × 4000 PIOPS:

~24,000 IOPS

NuoDB 8 KB random PIOPS RAID 0, 2 × 4000

PIOPS:

~7800 IOPS

RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 24 × 4000 PIOPS:

~96,000 IOPS

Clustrix 4 KB random PIOPS RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 8 × 4000

PIOPS:

~31,000 IOPS

RAID 0, 24 × 4000 PIOPS:

~96,000 IOPS

GenieDB 16 KB random PIOPS 1 × 4000 PIOPS:

~4000 IOPS

RAID 0, 2 × 4000

PIOPS:

~7,800 IOPS

RAID 0, 12 × 4000 PIOPS:

~48,000 IOPS

N

ew

SQ

L

RD

BM

S

Cheat Sheet! Storage Workloads on AWS

Workload/

Software

Typical

block

size

Random

/Seq?

Typical

correct

EBS type

Max EBS @

500 MB/s

Max EBS @ 1 GB/s Max EBS @ CC/H

MongoDB 4 KB serialized PIOPS RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 8 × 4000

PIOPS:

~31,000 IOPS

RAID 0, 24 × 4000

PIOPS:

~96,000 IOPS

Cassandra 4 KB random PIOPS RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 8 × 4000

PIOPS:

~31,000 IOPS

RAID 0, 24 × 4000

PIOPS:

~96,000 IOPS

CouchBase 4 KB random PIOPS RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 8 × 4000

PIOPS:

~31,000 IOPS

RAID 0, 24 × 4000

PIOPS:

~96,000 IOPS

Hadoop /

Hbase

64 KB random PIOPS 1 × 4000 PIOPS:

~1000 IOPS

RAID 0, 2 × 4000

PIOPS:

~2000 IOPS

RAID 0, 12 × 4000

PIOPS:

~12,000 IOPS

Riak /

LevelDB

4 KB random PIOPS RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 8 × 4000

PIOPS:

~31,000 IOPS

RAID 0, 24 × 4000

PIOPS:

~96,000 IOPS

NoS

QL

Cheat Sheet! Storage Workloads on AWS

Cheat Sheet! Storage Workloads on AWS

Workload/

Software

Typical

block

size

Random

/Seq?

Typical

correct

EBS type

Max EBS @

500 MB/s

Max EBS @ 1 GB/s Max EBS @ CC/H

AeroSpike 128 KB random PIOPS 1 × 4000

PIOPS:

~500 IOPS

RAID 0, 2 x 4000

PIOPS:

~1000 IOPS

RAID 0, 12 x 4000 PIOPS:

~6,000 IOPS

(768MB/s)

Vertica 1 MB sequential PIOPS 1 × 4000

PIOPS:

~68 IOPS

RAID 0, 2 × 4000

PIOPS:

~125 IOPS

RAID 0, 12 × 4000 PIOPS:

~750 IOPS

(768MB/s)

TeraData 128 KB sequential PIOPS 1 × 4000

PIOPS:

~500 IOPS

RAID 0, 2 × 4000

PIOPS:

~1000 IOPS

RAID 0, 12 × 4000 PIOPS:

~6,000 IOPS

(768MB/s)

GlusterFS 128 KB sequential PIOPS 1 × 4000

PIOPS:

~500 IOPS

RAID 0, 2 × 4000

PIOPS:

~1000 IOPS

RAID 0, 12 × 4000 PIOPS:

~6,000 IOPS

(768MB/s)

OrangeFS 256 KB sequential PIOPS 1 × 4000

PIOPS:

~250 IOPS

RAID 0, 2 × 4000

PIOPS:

~500 IOPS

RAID 0, 12 × 4000 PIOPS:

~3,000 IOPS

(768MB/s)

Riak (CS) 1 MB sequential PIOPS 1 × 4000

PIOPS:

~68 IOPS

RAID 0, 2 × 4000

PIOPS:

~125 IOPS

RAID 0, 12 × 4000 PIOPS:

~750 IOPS

(768MB/s)

DN

File

Syste

m W

are

ho

use

Architecture – What Else Uses Network?

Performance – Extra-large Production Scale

• Leverage SSD instance type

(hi1.4xlarge)

o 2 × 1 TB SSD storage (ephemeral

storage)

o Perfect for replicas

• If replicas on SSD instance types, disable

integrity features such as fsync and

full_page_writes on those hosts to

improve performance

Sta

ble

Testing Random 4 KB Reads

EBS

PIOPS

+ SSD

What About Performance Cost? cc2.8xlarge

24 @ 4 KB

PIOPS

VS.

hi1.4xlarge hi1.4xlarge

$11773 on-demand,

$10589 effective 3 YR reserved

$4538 on-demand,

$1539 effective 3 YR reserved

If >20 KB IOPS read, choose hi1

If 3 YR, and >8 KB IOPS, choose hi1

If >10 KB write IOPS, TEST,

but probably choose PIOPS

On demand, If <20 KB read IOPS, choose PIOPS

What about Capacity Cost? cc2.8xlarge

48x

1TB

EBS

VS.

hs1.8xlarge hs1.8xlarge

$7312 on-demand,

$6128 effective 3 YR reserved

$6734 on-demand,

$2408 effective 3 YR reserved

If >43TB, or > 800MB/s, choose hs1

If 3 year, and >18TB, choose hs1

Performance / Stability Tips

• Ext4 or XFS (understand journal impact!)

• nobarrier, noatime, noexec, nodiratime

• Raise file descriptor limits

• Set read-aheads low

• SNAPSHOT SNAPSHOT SNAPSHOT

From 1000 to 98,000: What does that mean?

So.. all storage workloads!

Please give us your feedback on this

presentation

As a thank you, we will select prize

winners daily for completed surveys!

STG 302

@milesward