an in-depth look at snia's enterprise solid state storage … · 2020. 9. 16. · pts-e 1.0...
TRANSCRIPT
An In-Depth Look at SNIA's Enterprise Solid State Storage Performance Test Specification (PTS-E V1.1)
Easen Ho, CTO, Calypso Systems
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved. 2 2
SNIA Legal Notice
The material contained in this tutorial is copyrighted by the SNIA. Member companies and individual members may use this material in presentations and literature under the following conditions:
Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.
This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Abstract
Many things influence the performance of a SSD: current and previous workloads, block size, R/W mix, queue depth, just to name a few… SNIA’s Solid State Storage (SSS) Performance Test Specification (PTS) provides methodology that ensures fair and consistent performance test results. This tutorial will provide an overview of the PTS-E V1.1 for enterprise SSDs as well as usage examples
3
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Some Definitions and Clarifications
SNIA’s Solid State Storage Performance Test Specification (short hand) = PTS
Enterprise = PTS-E Client= PTS-C PTS-E 1.0 released 2011-04-26 PTS-C 1.0 released 2011-08-06
Solid State Storage (SSS) & Solid State Drive (SSD) in this presentation refers to NAND flash-based storage devices in all form factors and interfaces
4
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Outline
Why a SSD Testing Standard is Needed Review of What’s in the PTS-E V1.0 What’s New in the PTS-E V1.1
Host Idle Recovery Test (HIR) Cross Stimulus Response Test(XSR) Demand Intensity/Response Time Histogram Test (DIRTH) Enterprise Composite Workload Test (ECW)
Other Work In Progress
5
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
SSD Testing - Need For Standards
No standard test methodology Everyone has a favorite benchmark program Pre-conditioning? Steady State? Demand Intensity? Active Range? Data Pattern? ….
No standard testing platform Platforms (software+hardware) becomes more relevant as SSDs gets faster & faster Can tests results be compared?
No standard performance metrics IOPS? TP? Block Size? R/W Mix? Response Time? ….
6
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Where Do We Report Performance?
7
PCIe SLC MLC HDD
Log Scale!
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved. 8
Factoring Influencing Performance
Write History
• What was previously written
PC Active Range
• Where data was previously written
• Trim effects
Test Active Range
• Where data is currently written
• Temporal or spatial coherency?
Data Content and Size
• What is the nature of the data
Access Pattern
• Manner in which data is being accessed
Demand Intensity
• How hard the application is driving the device
Throttling
• How fast is data being written vs warranty
Others
• ?
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Two specifications released
9
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
SSSI Reference Test Platform (RTP)
10
Intel S5520HC
Single Intel W5580, 3.2GHz, Quad-core CPU
12GB, 1333MHz, ECC DDR3 RAM
LSI 9212-4e4i 6Gb/s SAS HBA
Intel ICH10R 3Gb/s SATA
8X Gen-II PCI-e
CentOS 6.X
Calypso RTP Backend V1.5
Calypso Test Suite (CTS) V6.5
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Outline
Why a SSD Testing Standard is Needed Review of What’s in the PTS-E V1.0 What’s New in the PTS-E V1.1
Host Idle Recovery Test (HIR) Cross Stimulus Response Test(XSR) Demand Intensity/Response Time Histogram Test (DIRTH) Enterprise Composite Workload Test (ECW)
Other Work In Progress
11
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Basic Test Flow
12
• Security Erase, Sanitize, Format Unit, other proprietary methods where indicated 1. Purge
• Set user selectable test parameters, such as Active Range, Data Pattern, Demand Intensity 2. Set Conditions
• Workload independent (WIPC) • Workload dependent (WDPC) 3. Pre-Condition
• Reiterate loops until Steady State is reached, or run to a prescribed maximum number of loops 4. Run Until SS
• Collect data from Steady State Measurement Window 5. Collect Data
• Use standard report formats and include required and optional elements 6. Generate Reports
!
!
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Tests Contained In PTS-E 1.0 SPEC
Enterprise Performance Test Specification (PTS-E) V1.0 encompasses:
A suite of basic SSS performance tests
Preconditioning and Steady State requirements
Standard test procedures and reporting requirements
13
Write Saturation
• Random Access • R/W: • 100% Writes
• BS: • 4KiB
Enterprise IOPS
• Random Access • R/W: • 100/0, 95/5, 65/35,
50/50, 35/65, 5/95, 0/100
• BS: • 1024KiB, 128KiB,
64KiB, 32KiB, 16KiB, 8KiB, 4KiB, 0.5KiB
Enterprise TP
• Sequential Access
• R/W: • 100/0, 0/100 • BS: • 1024KiB, 128KiB
Enterprise Latency
• Random Access • R/W: • 100/0, 65/35, 0/100
• BS: • 8KiB, 4KiB, 0.5KiB
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Tests Contained In PTS-C 1.0 SPEC
14
Client IOPS
• Random Access • R/W: • 100/0, 95/5, 65/35, 50/50, 35/65,
5/95, 0/100 • BS: • 1024KiB, 128KiB, 64KiB, 32KiB,
16KiB, 8KiB, 4KiB, 0.5KiB • Range Restriction: • 100% & 75% LBA • 2048 Segments • Active Footprint
Restriction: • 8 & 16 GiB
Client TP
• Sequential Access • R/W: • 100/0, 0/100 • BS: • 1024KiB • Range Restriction: • 100% & 75% LBA • 2048 Segments • Active Footprint
Restriction: • 8 & 16 GiB
Client Latency
• Random Access • R/W: • 100/0, 65/35, 0/100
• BS: • 8KiB, 4KiB, 0.5KiB • Range Restriction: • 100% & 75% LBA • 2048 Segments • Active Footprint
Restriction: • 8 & 16 GiB
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved. 15
Key Concept: Pre-Conditioning
Transition
F O B
Steady State
Pre-Conditioning
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Key Concept: Steady State
Premise is that reported data should be take only AFTER the test results show the drive has reached and MAINTAINED “Steady State” The Measurement Window is the interval, measured in Rounds, when the test results have entered and maintained Steady State for 5 Rounds
16
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Key Concept: Steady State
Steady State is reached only if BOTH of the following conditions are satisfied (assuming “y” is the variable being tracked): 1. Variation of y within the Measurement Windows is
within 20% of the Average “ Max(y)-Min(y) within the Measurement Window is no more than 20% of
the Ave(y) within the Measurement Window; and ”
2. Trending of y within the Measurement Windows is within 10% of the Average
“ [Max(y) as defined by the linear curve fit of the data within the Measurement Window] – [Min(y) as defined by the best linear curve fit of the data within the Measurement Window] is within 10% of Ave(y) within the Measurement Window. “
17
…not too much data variation…
…not too much
trending…
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
SS Measurement Window
18
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
SS Measurement Window
19
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved. 20
Common Reporting Format (IOPS - Test Set-Up Pages 1-3 of 6)
Steady State Convergence Steady State Window Steady State Measurement Calculations
MLC-A
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved. 21
Common Reporting Format (IOPS - Results Pages 4-6 of 6)
Tabular Data 2D Plot 3D Bar Plot
MLC-A
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Outline
Why a SSD Testing Standard is Needed Review of What’s in the PTS-E V1.0 What’s New in the PTS-E V1.1
Host Idle Recovery Test (HIR) Cross Stimulus Response Test(XSR) Demand Intensity/Response Time Histogram Test (DIRTH) Enterprise Composite Workload Test (ECW)
Other Work In Progress
22
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Tests Contained In PTS-E 1.1
PTS-E 1.1 adds:
23
Host Idle Recovery
• Examines effect of idle (no IO) on small block RND writes
• RND/4KiB Writes
Cross Stimulus Response
• Examines switching between large block SEQ and small block RND writes
• SEQ/1024KiB &
RND/4KiB Writes
Demand Intensity – Response Time
Histograms
• Performance and detailed response time statistics under various workload types
• R/W=65/35 %, RND/8K
• R/W=90/10 %, RND/128K
• Response Time Histograms at various operating points
Enterprise Composite Workload
• Performance and detailed response time in a mixed IO Enterprise environment
• R/W=60/40 % • BS from 0.5-
64KiB • Three LBA
Groups
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Host Idle Recovery Test (HIR) Purpose Examines Effect of Host Idle Period (No Host IO) On
The Performance of RND Small Block Writes
Test Setup
Preconditioning RND/4KiB Writes to Steady State
Test Insert various amount of idle time (no IO from host) between periods of 5 second RND/4KiB writes: Segment 1 (Wait State 1): 360 x (5S Write + 5S Idle) + 360 x (5S Write) Segment 2 (Wait State 2): 360 x (5S Write + 10S Idle) + 360 x (5S Write) Segment 3 (Wait State 3): 360 x (5S Write + 15S Idle) + 360 x (5S Write) Segment 4 (Wait State 5): 360 x (5S Write + 25S Idle) + 360 x (5S Write) Segment 5 (Wait State 10): 360 x (5S Write + 50S Idle) + 360 x (5S Write)
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
HIR Example: MLC/SATA
25
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
HIR Example: SLC/SAS
26
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Cross Stimulus Response Test (XSR) Purpose Examines Switching Between Sustained Large Block SEQ
and Sustained Small Block RND Writes
Test Setup
Preconditioning None
Test Apply three Access Groups: Access Group 1 (Large Block SEQ): 100% SEQ Write, Block Size=1024 KiB Access Group 2 (Small Block RND): 100% RND Write, Block Size=4 KiB Access Group 3 (Large Block SEQ): 100% SEQ Write, Block Size=1024 KiB
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Example: MLC/SATA
28
1
2 3
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Example: MLC/SAS
29
1
2 3
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Demand Intensity, Response Time Histogram Test (DIRTH)
Purpose Examines IOPS and Response Time Characteristics of Various Enterprise Workloads
Test Setup
Preconditioning Access Pattern, 100% Writes, until Steady State
Test 1. Using TC=[1-32] and OIO/Thread= [1-32], apply ECW using order of decreasing total OIO, until Steady State is reached for (32,32)
2. Manually determine the following operating points:
MaxIOPS: operating point with maximum IOPS while maintaining an ART < 5 mS MinIOPS: operating point with minimum measured IOPS MidIOPS: a minimum of one or more operating point(s) that has IOPS values between and equally divides the IOPS value spanned by MaxIOPS and MinIOPS 3. Perform Response Time Histograms, capturing all IO completion times for 10 Min at each operating points.
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
DIRTH Test
Currently there are two Access Patterns specified for the DIRTH test:
OLTP-Like: BS= 8 KiB R/W= 65/35 % Random Access, Random Data Full Drive Access
Video-Server-Like BS= 128 KiB R/W= 90/10 % Random Access, Random Data Full Drive Access
31
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
DIRTH (OLTP): Demand Intensity
32
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
DIRTH (OLTP): Demand Intensity
33
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
DIRTH (OLTP): MaxIOPS Point
34
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
DIRTH (OLTP): MidIOPS Point
35
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
DIRTH (OLTP): MinIOPS Point
36
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Enterprise Composite Workload (ECW) Test
Purpose Examines IOPS and Response Time Characteristics Using a Mixed IO Workload
Test Setup
Preconditioning ECW, 100% Write, to Steady State
Test 1. Using TC=[1-32] and OIO/Thread= [1-32], apply ECW using order of decreasing total OIO, until Steady State is reached for (32,32)
2. Manually determine the following operating points:
MaxIOPS: operating point with maximum IOPS while maintaining an ART < 5 mS MinIOPS: operating point with minimum measured IOPS MidIOPS: a minimum of one or more operating point(s) that has IOPS values between and equally divides the IOPS value spanned by MaxIOPS and MinIOPS 3. Perform Response Time Histograms, capturing all IO completion times for 10 Min at each operating points.
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
The Enterprise Composite Workload
The ECW is a R/W=40/60%, random access pattern with a distribution of Block Sizes, each with a pre-defined Access Probability, plus restrictions on Access Range Probability Distribution
38
Block Size in Bytes (KiB) Access Probability Within Each Measurement Period
512 bytes (0.5 KiB) 4% 1024 bytes (1 KiB) 1% 1536 bytes (1.5 KiB) 1% 2048 bytes (2 KiB) 1% 2560 bytes (2.5 KiB) 1% 3072 bytes (3 KiB) 1% 3584 bytes (3.5 KiB) 1% 4096 bytes (4 KiB) 67% 8192 bytes (8 KiB) 10% 16,384 bytes (16 KiB) 7% 32,768 bytes (32 KiB) 3% 65,536 bytes (64 KiB) 3%
Total 100%
% of Access within
1 Measurement Period
Active Range Restriction
Label
50%
First 5% LBA Group A
30%
Next 15% LBA Group B
20%
Remaining 80% LBA Group C
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
ECW: Demand Intensity (PCIe, MLC)
39
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
ECW: TRSP Histograms (PCIe,MLC)
40
A
B
C
A
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
ECW: Demand Intensity (SAS, MLC)
41
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
ECW: TRSP Histograms (SAS,MLC)
42
A
B
C
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Released SNIA SSS PTS is available for download at:
http://www.snia.org/tech_activities/sta
ndards/curr_standards/pts
43
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
Outline
Why a SSD Testing Standard is Needed Review of What’s in the PTS-E V1.0 What’s New in the PTS-E V1.1
Host Idle Recovery Test (HIR) Cross Stimulus Response Test(XSR) Demand Intensity/Response Time Histogram Test (DIRTH) Enterprise Composite Workload Test (ECW)
Other Work In Progress
44
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
PTS Follow-On Work (PTS-E/C)
45
• Conduct workload summary capture and see if different common use- cases can be found
Workload Capture/Distillation
• Test and metrics related to power usage and efficiency Power Efficiency
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.
46
SSD Client & Enterprise PTS Comparison Data, Whitepapers, and Webcasts
Available at
www.snia.org/forums/sssi/pts
An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved. 47 47
Q&A / Feedback
Please send any questions or comments on this presentation to SNIA: [email protected]
Many thanks to the following individuals
for their contributions to this tutorial.
- SNIA Education Committee
Eden Kim Easen Ho