understanding ssd performance: using the sss pts to ... · understanding ssd performance: using the...

Understanding SSD Performance:

Using the SSS PTS to Evaluate and Compare SSD Performance

Esther Spanjer, SMART Storage Systems Easen Ho, CTO, Calypso Systems

Author: Esther Spanjer, SMART Storage Systems

Author: Easen Ho, CTO, Calypso Systems

Understanding SSD Performance © 2011 Storage Networking Industry Association. All Rights Reserved. 2 2

SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA. Member companies and individual members may use this material in presentations and literature under the following conditions:

Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.

This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.

Understanding SSD Performance © 2011 Storage Networking Industry Association. All Rights Reserved.

Abstract

A variety of parameters can influence the performance behavior of a solid state drive: current and previous workloads, block size, R/W mix, queue depth, just to name a few… SNIA’s Solid State Storage (SSS) Performance Test Specification (PTS) provides performance benchmarking that result in repeatable and consistent test results. This presentation will provide an overview of the SNIA SSD Performance Test Specification for both client and enterprise SSDs, and provide examples of how to use the standard report formats to facilitate comparison of SSD performance.

3


Outline

Why an SSD Testing Standard is Needed SNIA’s Solid Stage Storage (SSS) Performance Test Specification (PTS) for Enterprise and Client SSDs SNIA’s SSS PTS Common Reporting Format Using the SNIA SSS PTS to Compare SSD Performance

4


Definition of SSS

5

SSS = Solid State Storage

Understanding SSD Performance © 2011 Storage Networking Industry Association. All Rights Reserved. 6

The Testing Application Landscape

IOMeter 2006

ATTO Disk Benchmark

PC Mark Vantage

HD Tune

SYS Mark

IOMeter 2008

IOMeter 20xx

VD Bench FIOS

CrystalDiskMark

HD Tach

PassMark

SiSoft Sandra

IOzone

AS SSD

Xbench Storage Bench

PC Mark 7

Bonnie ++

Anvil Storage Utilities

…...

…… ……


The Testing Platform Landscape

Windows 7 Intel Core i5

ICH10R

Intel Core i7

Intel Xeon Intel Dual Xeon

Windows 8 Fedora X

Marvell 9128

Intel AHCI Microsoft AHCI

RHEL X AMD Opteron

AMD A6

SUSE X

JMicron 36X

SATA 3.0

SATA 6.0

SAS 3.0 SAS 6.0

Mac OS

CentOS X Solaris

…...

…… ……


Performance Specification Landscape

Random or Sustained?

Up to?

Random Precondition Sustained Speed?

IOPS? MB/s or Mb/s?

Block Size?

Write Cache On? Short Stroked?

Random Data?

Active Footprint?

Uniform Data?


Outline


9


Two specifications released

10


Factoring Influencing Performance

Write History

• What was previously written

PC Active Range

• Where data was previously written

• Trim effects

Test Active Range

• Where data is currently written

• Temporal or spatial coherency?

Data Content and Size

• What is the nature of the data

Access Pattern

• Manner in which data is being accessed

Demand Intensity

• How hard the application is driving the device

Throttling

• How fast is data being written vs warranty

Others

• ?


SSSI Reference Test Platform (RTP)

12

Intel S5520HC

Single Intel W5580, 3.2GHz, Quad-core CPU

12GB, 1333MHz, ECC DDR3 RAM

LSI 9212-4e4i 6Gb/s SAS HBA

Intel ICH10R 3Gb/s SATA

8X Gen-II PCI-e

CentOS 5.X

Calypso RTP Backend V1.5

Calypso Test Suite (CTS) V6.5


Tests Contained In PTS-E 1.0 SPEC

Enterprise Performance Test Specification (PTS-E) V1.0 encompasses:

A suite of basic SSS performance tests

Preconditioning and Steady State requirements

Standard test procedures and reporting requirements

13

Write Saturation

• Random Access • R/W: • 100% Writes

• BS: • 4KiB

Enterprise IOPS

• Random Access • R/W: • 100/0, 95/5, 65/35,

50/50, 35/65, 5/95, 0/100

• BS: • 1024KiB, 128KiB,

64KiB, 32KiB, 16KiB, 8KiB, 4KiB, 0.5KiB

Enterprise TP

• Sequential Access

• R/W: • 100/0, 0/100 • BS: • 1024KiB, 128KiB

Enterprise Latency

• Random Access • R/W: • 100/0, 65/35, 0/100

• BS: • 8KiB, 4KiB, 0.5KiB


Tests Contained In PTS-C 1.0 SPEC

14

Client IOPS

• Random Access • R/W: • 100/0, 95/5, 65/35, 50/50, 35/65,

5/95, 0/100 • BS: • 1024KiB, 128KiB, 64KiB, 32KiB,

16KiB, 8KiB, 4KiB, 0.5KiB • Range Restriction: • 100% & 75% LBA • 2048 Segments • Active Footprint

Restriction: • 8 & 16 GiB

Client TP

• Sequential Access • R/W: • 100/0, 0/100 • BS: • 1024KiB • Range Restriction: • 100% & 75% LBA • 2048 Segments • Active Footprint


Client Latency

• Random Access • R/W: • 100/0, 65/35, 0/100

• BS: • 8KiB, 4KiB, 0.5KiB • Range Restriction: • 100% & 75% LBA • 2048 Segments • Active Footprint



Basic Test Flow

15

• Security Erase, Sanitize, Format Unit, other proprietary methods where indicated 1. Purge

• Set user selectable test parameters, such as Active Range, Data Pattern, Demand Intensity 2. Set Conditions

• Workload independent (WIPC) • Workload dependent (WDPC) 3. Pre-Condition

• Reiterate loops until Steady State is reached, or run to a prescribed maximum number of loops 4. Run Until SS

• Collect data from Steady State Measurement Window 5. Collect Data

• Use standard report formats and include required and optional elements 6. Generate Reports

!

!


Key Concepts in the SNIA SSS PTS

16

Common Reporting Steady State Pre-

Conditioning


Key Concept: Pre-Conditioning

Pre-Conditioning is a key requirement in getting repeatable, representative results Goal is to put drive into “Steady State”, using:

Workload independent – PTS v1.0 Section 3.3

Use a prescribed workload unrelated to the test loop Write 2X user capacity using SEQ/128KiB blocks

Workload dependent – PTS v1.0 Section 3.3

Run test workload itself as pre-conditioning (self pre-conditioning)

17


The need for Preconditioning


Key Concept: Steady State

Premise is that reported data should be take only AFTER the test results show the drive has reached and MAINTAINED “Steady State” The Measurement Window is the interval, measured in Rounds, when the test results have entered and maintained Steady State for 5 Rounds

19


Key Concept: Steady State

Steady State is reached only if BOTH of the following conditions are satisfied (assuming “y” is the variable being tracked): 1. Variation of y within the Measurement Windows is

within 20% of the Average “ Max(y)-Min(y) within the Measurement Window is no more than 20% of

the Ave(y) within the Measurement Window; and ”

2. Trending of y within the Measurement Windows is within 10% of the Average

“ [Max(y) as defined by the linear curve fit of the data within the Measurement Window] – [Min(y) as defined by the best linear curve fit of the data within the Measurement Window] is within 10% of Ave(y) within the Measurement Window. “

20

…not too much data variation…

…not too much

trending…


SS Measurement Window

21


SS Measurement Window

22


IOPS RW/BS Sequence

23

Round


TP RW/BS Sequence

24


Outline


25


Common Reporting Format (IOPS - Test Set-Up Pages 1-3 of 6)

Steady State Convergence Steady State Window Steady State Measurement Calculations

MLC-A


Common Reporting Format (IOPS - Results Pages 4-6 of 6)

Tabular Data 2D Plot 3D Bar Plot

MLC-A


PTS 1.0 Report Format Individual Report Page Headers • Required on Each Individual Test Report Page • Lists the Settings pertaining to the reported Test

28

Key Header Information • Device & Test System • DUT Preparation: PURGE & Preconditioning • Test Loop Parameters: Data Pattern, TC & QD • Steady State: Convergence, Rounds, PC AR, AR

Amount, Number of Segments


Client IOPS MLC-A

RND 4KiB IOPS 100% W: • IOPS = 3,147 IOPS • WSAT = 2,714 IOPS

Header Information: • WIPC: 2X SEQ 128KiB • WDPC: IOPS Loop • OIO: 8 (QD=8 / TC=1) • SS Rounds: 1-5 • AR Amount = 16GiB • AR Segments = 2048

IOPS


The full SNIA SSS PTS is available for download at:

http://www.snia.org/tech_activities/standar

ds/curr_standards/pts

30


Outline


31


Comparing Client WSAT - IOPS v TGBW

MLC-A 256 GB

MLC-B 160 GB

WSAT IOPS v TGBW MLC-A MLC-B

FOB 4KiB 55,896 20,364

SS 4KiB 2,714 658

Peak Drop 250 TGBW 38 TGBW

Steady State

1,400 TGBW

80 TGBW

Peak Drop User Capacity

1 Drive Fill 0.38 Drive Fill

Steady State User Capacity

5 Drive Fills

0.8 Drive Fill

NOTE: MLC-B x & y axis are scaled to match MLC-A for comparison NOTE: MLC-B TGBW is limited by slower RND 4KiB W speed (1440 minute test) MLC-A interface is 6Gb/s MLC-B interface is 3Gb/s

WSAT

Manufacturer’s FOB Claim


Comparing Enterprise IOPS

SLC-A 100 GB

SLC-B 100 GB

IOPS


Comparing Enterprise IOPS

SLC-A 100 GB

SLC-B 100 GB

RND IOPS

Steady State SLC-A SLC-B

0.5KiB 100% R 95,924 43,368

4KiB 100% R 93,707 46,365 4KiB 65:35 R/W 28,019 41,460

4KiB 100% W 16,563 19,561

128KiB 65:35 1,709 1,389

NOTE: SLC-B y-axis is scaled to match SLC-A for comparison Key Points: • SLC-A:

• Higher small block Reads • Lower small block Writes.

• SLC-B • More balanced Block Size optimization • Generally higher W performance

• Writes have a disproportionately strong influence in any R/W mix performance (note 65/35 R/W comparisons)

IOPS


January 2012

Comparing Client Throughput

MLC-A 256 GB

MLC-B 160 GB

SEQ 1024KIB

TP

Steady State MLC-A MLC-B

100% R 417 MB/s 264 MB/s

100% W 267 MB/s 99 MB/s

Key Points: • MLC-A:

• DUT Interface 6Gb/s SATA • Higher large block SEQ Reads & Writes

• MLC-B • DUT Interface 3 Gb/s SATA • Slower large block SEQ Reads & Writes

TP


January 2012

Comparing Client Latency AVE

MLC-A 256 GB

MLC-B 160 GB

Latency RND 4KiB AVE

Steady State

MLC-A MLC-B

100% R AVE 0.20 mS 0.19 mS

65/35 R/W AVE 0.33 mS 0.65 mS

100% W AVE 0.35 mS 1.00 mS


• More stable & faster RND 4KiB AVE Latency • Higher RND 0.5KiB 100% W AVE Latency

• MLC-B • Slower overall RND 4KiB AVE Latency • Higher RND 8KiB 100% W AVE Latency

LAT


Comparing Client Latency MAX

MLC-A 256 GB

MLC-B 160 GB

Latency RND 4KiB MAX

Steady State

MLC-A MLC-B

100% R MAX 1.59 mS 1.43 mS

65/35 R/W MAX 50.02 mS 283.21 mS

100% W MAX 51.05 mS 288.29 mS


• More stable & faster RND 4KiB MAX Latency • Smaller RND 0.5KiB & 8KiB MAX Latency

• MLC-B • Slower overall RND 4KiB MAX Latency • Higher RND 0.5KiB & 8KiB MAX Latency

LAT


PTS Follow-On Work (PTS-C/E 1.1…)

38

• See how the drive responds to host idle time amidst continuous access Host Idle Test

• See how drive handles switching between sustained access patterns

Cross Stimulus Recovery

• See how drive responds to increasing host demands Demand Intensity

• Get detailed response time statistics during specific stimulus Response Time Histogram

• Measures power efficiency of the device IOPS/W

• Synthetic composite workload for Enterprise environments similar to JEDEC workload for endurance testing

Enterprise Composite Synthetic Workload

• Captures or uses captured workloads traces and provide a consistent way to playback such traces Trace-Based Workloads


39

SSD Client & Enterprise PTS Comparison Data, Whitepapers, and Webcasts

Available at

www.snia.org/forums/sssi/pts

Visit the “Solid State Storage in the Enterprise” Hands-On Lab at SNW

Understanding SSD Performance © 2011 Storage Networking Industry Association. All Rights Reserved. 40 40

Q&A / Feedback

Please send any questions or comments on this presentation to SNIA: [email protected]

Many thanks to the following individuals

for their contributions to this tutorial. - SNIA Education Committee

Eden Kim Easen Ho Esther Spanjer

mailto:[email protected]�

understanding ssd performance: using the sss pts to ... · understanding ssd performance: using the...

Documents