1 - 56

Adam Chunn

15 March 2012

Lugano - Switzerland

• Company overview

• Architecture & Performance

• Reliability

• Maximizing SSD

• Q&A

BUILDING HIGH AVAILABILITY SSD

2 - 56

Sent a text message

Placed online bet

Booked a cruise or flight

Used an ATM

Conducted a financial trade

Shopped online

Used pre-paid wireless

Gamed online

…RamSan is Everywhere

The largest SSD installations in production in the world

Currently operating in 10 major financial exchanges worldwide

Used today by 7 out of 11 of the world’s largest telecoms

Installed and in production in over 34 countries

Select RamSan Facts…

3 - 56

The largest SSD installations in production in the world

Currently operating in 10 major financial exchanges worldwide

Used today by 7 out of 11 of the world’s largest telecoms

Installed and in production in over 35 countries

…RamSan is Everywhere

Sent a text message

Placed online bet

Booked a cruise or flight

Used an ATM

Conducted a financial trade

Shopped online

Used pre-paid wireless

Gamed online

Select RamSan Facts…

4 - 56

Background on TMS

Solid State Storage

Leader

Global Enterprise

Customers

Strong Financial

Performance

World Class

Team

Deep Domain

Expertise

• Industry’s highest performance, highest reliability,

lowest latency, lowest power SSD solutions

• Growing enterprise customer base in over 34

countries

• No Venture Capital/Long Term Debt

• Strong management and engineering teams

• Over 400 man-years of SSD experience

• 33 years experience designing SSDs; 30+ patents

granted and pending; many trade secrets

5 - 56

Key References

See all of these and more in the Success Stories section

of our web site at www.ramsan.com.

6 - 56

ARCHITECTURE & PERFORMANCE

7 - 56

L = λWThe long-term average number of customers in a stable system L is equal to the long-term average effective arrival rate, λ, multiplied by the average

time a customer spends in the system, W 1

Above is Little’s Law which is just a fancy

way to say that performance is based on

Latency and Parallelism

1 Paraphrased from Little’s Law, John D.C. Little and Stephen C. Graves, MIT

8 - 56

Flash Controller Design Basics

Write Buffer

I/O Interface

Flash

Controller

FPGA

FLASH Media

Lookup

Tables

CPU

RAM

CPU

• Each controller handles 10 flash chips

• The Lookup Tables and Write Buffer is RAM accessible from the controller only.

• The I/O Interface and controller are both separate FPGAs

• The CPU is an embedded processor that handles all out-of-band operations

• DMAs are all processed completely in FPGA hardware

9 - 56

DMAs are hardware only

Write Buffer

I/O Interface

Flash

Controller

FPGA

FLASH Media

Lookup

Tables

CPU

RAM

CPU

•DMAs are all

processed

completely in

FPGA hardware

10 - 56

Decreasing Latency

Write Buffer

I/O Interface

Flash

Controller

FPGA

FLASH Media

Lookup

Tables

• Remove from the DMA path, all non-critical flash memory book-keeping

• Write setup

• Garbage collection

• Error handling

• Health calculation

• Wear Leveling

• Statistics collection

• Formatting

• Backup/Restore

• Key Generation

The Embedded CPU

CPU

RAM

CPU

11 - 56

I/O Interface

CPU

RAM

CPU

Write Buffer

Flash

Controller

FPGA

FLASH Media

Lookup

Tables

Increasing Parallelism

• Increasing the number of flash chips that can run concurrently

• Which is done by increasing the number of flash chip controllers

• Each TMS flash chip controller can do 36 4KB DMAs in parallel

• (40 if you include the background chip RAID, or VSR, operations)

• A RamSan-70 has 8 controllers, so it can do 288 4KB operations simultaneously

• A RamSan-810 has 40 controllers, so it can do 1440 4KB operations simultaneously

12 - 56

L = λW

So, what else effects Latency and

Parallelism?

13 - 56

L = λWWhat else effects Latency?

• CPU Speed

• not number of cores

• not number of chips

• Bus architecture

• North/south bridges

• PCIe hierarchy

• PCIe controller

• CPU Usage (so in a convoluted way, cores and chip counts do matter)

14 - 56

L = λWWhat else effects Latency?

• Operating system and file system

• OSes and file systems optimized for disks tend to count on slow data access to hide processing

• Modern OSes and file systems are now written to maximize SSD

• The driver, the bridge between the OS and the hardware

• It must be thin or else adds latency

• Linux, Windows, Solaris, VMWare, OSX, AIX

• We are actively trying to push the driver into the Linux kernel

• If measuring at the application layer, middleware (for example, databases) can inject latency

15 - 56

L = λWWhat else effects Parallelism?

• Large Blocks

• RamSan products break apart large block DMAs into multiple, parallel DMAs

• For example, a 64kB DMA is converted into 16 parallel 4kB DMAs

• A single application can be written to either have multiple threads of synchronous I/O or a single thread that allows multiple outstanding asynchronous I/O

• Most high-performance middleware does just this (such as Microsoft SQL, Oracle, et cetera)

• Running multiple applications can provide the same effect as a single application running multiple threads

• CPU becomes more and more of a bottleneck, however

16 - 56

CSCS Benchmark

• CSCS = Swiss National Computing Centre

• Independent evaluation of PCIe SSDs

• RamSan-70 results:

– “…by far the best IOPS result we have ever measured…” (300K+ random 4K IOPS)

– “Unlike the FusionIO and Virident TachIOn devices, the bandwidth is almost independent of block size…”

17 - 56

RamSan Flash Product Portfolio

RamSan-70 RamSan-710/810 RamSan-720/820 RamSan-630

SLC Flash SLC/eMLC Flash SLC/eMLC Flash SLC Flash

900GB 5/10TB 12/24TB 10TB

1.2M IOPS 400K/320K IOPS 500K/450K IOPS 1M IOPS

2.5GB/s 5/4GB/s 5/4GB/s 10GB/s

Full-height, half-

length PCIe x8 2.01U rackmount, 4x IB or FC ports

3U rackmount, 10x

IB or FC ports

Single Server Apps;

Distributed

filesystems

Clustered Server Apps; Shared-storage filesystems (GPFS, GFS2,

etc)

18 - 56

SPC Price/Performance Leader

TMS

RamSan-630

0

5

10

15

20

25

5.000 6.000 7.000 8.000 9.000 10.000

SP

C-2

MB

PS

™ x

1k

/ T

ota

l T

SC

Pri

ce (

US

D)

SPC-2 MBPS™

Top 10 SPC-2 MBPS™

TMS

RamSan-630

TMS

RamSan-

400

0,00

0,20

0,40

0,60

0,80

1,00

1,20

1,40

1,60

250.000 350.000 450.000 550.000

SP

C-1

IO

PS

™ /

To

tal

TS

C P

rice

(U

SD

)

SPC-1 IOPS™

Top 10 SPC-1 IOPS™

19 - 56

Keys to Performance

• Hardware-only Data Path

– FPGA & Hardware Logic

– Faster than software-shared memory

• Software cannot add performance

– Virtualization is a software overhead to utilizing

additional hardware

– QoS is a software overhead to give applications

priority over another on shared hardware

20 - 56

RELIABILITY

21 - 56

Flash Quality

• Flash type matters!

– SLC in most RamSans

– Enterprise MLC (eMLC) in RamSan-8x0

• SLC is best but most expensive/least dense

• eMLC chips last 10x longer vs. normal MLC

• TMS technologies like Variable Stripe RAID™ lengthen system life

0

10

20

30

40

50

60

70

80

90

100

MLC eMLC SLCP

/E C

ycl

es

(Th

ou

san

ds)

Flash Type

Typical Chip Endurance

22 - 56

Combat Endurance

• Endurance of system is calculated:

BandwidthWriteMedia

QualityFlashCapacityFlash ×

23 - 56

Combat Endurance

5TB RamSan-710 (SLC Flash)

EnduranceYearsGBps

TB8.15

1

000,1005 =×

10TB RamSan-810 (eMLC Flash)

EnduranceYearsGBps

TB5.9

1

000,3010 =×

24 - 56

Combat Endurance

• Fight endurance with increased capacity

• eMLC has 2x Capacity for same cost– 2/3rd endurance of SLC

• MLC is 3000 Writes where eMLC is 30000 Writes

• MLC is ~1/4th price of eMLC storage– Sustained writes do not make sense for MLC

– MLC will last less than a year from sustained writes at same cost and half the write workload

yearathanLessMBps

TB =×500

000,31

25 - 56

Flash Problems and TMS Solutions

Problem Solution

Limited write-erase cycles Wear leveling

Bit errors ECC

Block/plane/device failuresBlock remapping, RAID,

Variable Stripe RAID™

Disturb errors(read, write, erase)

Voltage and timing adjustments

Erases need big blocks and

take a long timeOverprovisioning

26 - 56

Four Layers of Data Correction

Layer Protection

System-level RAID 5managed by centralized RAID controllers

Module failure

Module-level Variable Stripe RAID™managed by each module across its chips

Sub-chip failure, System Longevity

Module-level RAID 5managed by each module across its chips

Chip failure

Chip-level ECCmanaged by each module using its chips

Bit and block errors

RamSan-720/820 introduce System-Level RAID 5 across

Flash modules, plus the other mechanisms found on all

RamSan Flash storage systems.

RamSan-720/820 only

27 - 56

Variable Stripe RAID™ (VSR)

• Patented VSR allows RAID stripe sizes to vary.

• If one die fails in a ten-chip stripe, only the failed die is bypassed, and then data is restriped across the remaining nine chips.

…

…

16 Planes

10 Chips

FAIL

28 - 56

2D Flash RAID™ (RS-720/820)

RAID Controller RAID Controller

RAID 5 across Flash Modules (10 data + 1 parity + 1 hot spare)

Interface Interface

External

Interfaces

(FC, IB)

RAID

Controllers

RAID 5 within

Flash Modules

(9 data + 1 parity)

TMS

2D Flash RAID™

29 - 56

RamSan-70 Overview

11

1. PCIe 2.0 x8

2. PowerPC CPU

3. Xilinx FPGAs

4. 900GB usable SLC Flash

(1374GB raw)

5. 4GB DRAM

6. Super-Capacitors

7. Half-length card

22 33

33

44

44

55

55

55

5566

77

• Usable 450-900GB

• 650,000 4K IOPS

• 2.5 GB/s Bandwidth

• 30 µs sustained 4K Write Latency / 100us 4K Read

Latency

• 10 Years Life Expectancy

• Series-7™ Flash Controller

30 - 56

MAXIMIZING SSD

31 - 56

Segregation of Workload

But the question is, what data

makes sense to store on SSD?

• Metadata, Working Data, Archived Data

• Metadata is typically accessed the most, but takes up the least space

• Archived Date is accessed the least, but takes up the most space

• Moving high-access data into a high-performance medium has the greatest impact

32 - 56

Performance per Capacity

• Historically, TMS has designed DRAM-based

SSD devices that performed GB/sec per GB of

storage [Metadata]

• Our flash-based SSD devices perform GB/sec

per TB of storage [Metadata, Working Data]

• Disk-based products typically grossly under-

perform SSD, but economical performance at

>>TB of storage [Archive, Large Working Data]

33 - 56

Algorithm Matrix

Low CPU Utilization +

Low I/O Wait=

Algorithm needs to

provide more work

Low CPU Utilization +

High I/O Wait= Great fit for SSD!!

High CPU Utilization +

Low I/O Wait= In-memory work

High CPU Utilization +

High I/O Wait=

Using Asynchronous I/OAdd disks for growing capacity

Add SSD for same size capacity

34 - 56

Q & A

35 - 56

128 - 512 GB capacity

600,000 IOPS

4.5 GB/s throughput

Latency 15 µs

2-8 FC Ports

Industry Firsts:

512GB Non-volatile RAM storage

RAM SSD with Flash backup

RAID protected RAM and Flash modules

TMS patented IO2 Instant-on Input-Output option.

RamSan-440 Overview

36 - 56

Management Control

Processor

Hot Swappable Redundant

Power Supplies

Redundant Batteries

RAID Protected Backup Flash

Redundant Fans

4U Chassis

RAID Protected RAM Boards

4 Dual-ported Fibre Channel or

InfiniBand Interfaces

RamSan-440 Architecture

37 - 56

Series-7 Flash Controller Design

LookupTables

Flash Controller

FPGA(Process all of the “IN DATA”

activities)

I/O Interface

WriteBuffer CPU – (out of Primary Data Path)

Write setup, Garbage collection, Error handling

Super Capacitors

Memory Backup

Out of the data path activities

Best Performance:4K aligned I/O

4 GB RAM Cache

4 GB RAM Cache

38 - 56

RamSan-630 Overview1-10TB capacity

1 Million IOPS

10 GB/s throughput

Latency 80-250 µs

Highest density SLC Flash SSD system available.

Leverages proven flash core from the RamSan-20 and RamSan-620

Easily shared and multipathed through ten 8 GbitFibre Channel ports or QDR InfiniBand ports

Enterprise Reliability

Single Layer Cell (SLC) Flash

Fault Tolerant Flash (FTF) Architecture

Active Spare Flash

39 - 56

RamSan-630 Architecture

Management

Control

Processor

Redundant

Power

Supplies

5 Dual-ported FC or

IB

Interfaces

Redundant

Fans

1-10TB of SLC

Flash Boards

3U Chassis

40 - 56

RamSan-630 Flash Board

Super capacitors

RAID-5 Protected Flash480 GB usable, 640 GB RAW

4 Flash Controllers

Embedded PowerPC

On Board RAMECC Protected

Gateway

FPGA

41 - 56

RamSan-710 Overview

Highest density SLC Flash SSD system available in a 1U

Series-7™ Flash Controller

Four 8 Gbit Fibre Channel ports or QDR InfiniBand ports

Enterprise reliability

Single Layer Cell (SLC) Flash

Variable Stripe RAID (VSR)™

Active Spare

1-5 TB Usable capacity (6.8 TB Raw)

400,000 IOPS

5 GB/s throughput

35-175 µs latency

150K+ Write/Erase Cycles per Cell

42 - 56

management

control processor

redundant

power

supplies

2 dual-ported 8Gb FC or

QDR IB interfaces

N+1

batteries

4-20 Flash modules

+ 1 “Active Spare”

1U chassis

redundant fans

RamSan-710 Overview

43 - 56

RamSan-710 Overview

44 - 56

RamSan-810 Overview

Highest density eMLC Flash SSD system available in a 1U

Series-7™ Flash Controller

Four 8 Gbit Fibre Channel ports or QDR InfiniBand ports

Enterprise reliability

enterprise Multi-Level-Cell (eMLC) Flash

Variable Stripe RAID (VSR)™

Active Spare

2-10 TB Usable capacity (13.7 TB Raw)

320,000 IOPS

5 GB/s throughput

70-225 µs latency (est.)

30K+ Write/Erase Cycles per Cell

45 - 56

management

control processor

redundant

power

supplies

1-2 interface modules

N+1

batteries

4-20 Flash modules

+ 1 “Active Spare”

1U chassis

redundant fans

RamSan-810 Architecture

46 - 56

Motherboard

47 - 56

Toshiba eMLC Flash

Series-7 Flash

Controller FPGAs

Gateway FPGA

DDR DRAM

PowerPC CPU @

400 MHz

48 - 56

• Data Warehousing

• Web Content Hosting

• Low Bandwidth Log Files

• READ Intensive, Low WRITE Application

For Users Writing at 600 MB/s, the Lifetime of the eMLC

RamSan-810 is rated at 10 years*.– *2TB =10TB WRITES per Day

– *6TB =30TB WRITES per Day

– *10TB =50TB WRITES per Day

Applications Suited for eMLC

49 - 56

RamSan-70 Overview

50 - 56

RamSan-70 Architecture

11

33 44

44

22

22

55

55

55

5566

77

• 450-900GB

• 650.000 IOPS (4K)

• 2,5GB/s Bandwidth

• 30 µs Write Latency

• 10 Years Life Expectancy (25% writes)

• Series-7™ Flash Controller

1. PCIe 2.0 x8

3. Power PC CPU 333 mHz

4. Xilinx FPGAs

2. 900GB usable SLC Flash

(1374GB raw)

5. 4GB DRAM

6. Super-Capacitors

7. Half-length card

51 - 56

RamSan-720 Overview

Highest density SLC Flash SSD system available in a 1U

Series-7™ Flash Controller

Four 8 Gbit Fibre Channel ports or QDR InfiniBand ports

High Enterprise reliability

Single-Level-Cell (SLC) Flash

Variable Stripe RAID (VSR)™

2D Flash RAID™

6 or 12 TB Usable capacity (~ 7.8 or ~15.6 TB Raw)

500,000 IOPS (4K)

5 GB/s throughput

<100µs latency

No Single Point of Failure (nSPoF)

Hot Swappable Flash Cards

52 - 56

RamSan-820 Overview

Highest density eMLC Flash SSD system available in a 1U

Series-7™ Flash Controller

Four 8 Gbit Fibre Channel ports or QDR InfiniBand ports

High Enterprise reliability

enterprise Multi-Level-Cell (eMLC) Flash

Variable Stripe RAID (VSR)™

2D Flash RAID™

12 or 24TB Usable capacity (~ 15.6 or ~31.2 TB Raw)

450,000 IOPS (4K)

5 GB/s throughput

<100µs latency

No Single Point of Failure (nSPoF)

Hot Swappable Flash Cards

53 - 56

RamSan-Green IT

H A H N S T Ä T T E N • M Ü N C H E N

CONFIDENTIAL • COPYRIGHT BY PSP

Speed & IOPS

Latency

Bandwidth

H A H N S T Ä T T E N • M Ü N C H E N

CONFIDENTIAL • COPYRIGHT BY PSP

Pure SSD-Racepower

World‘s Fastest Storage Since 1978

56 - 56

Thanks for your attention…