new implementing nvme in high-performance flash memory systems · 2018. 12. 5. · • december...

Implementing NVMe in High-Performance Flash Memory Systems

Chris Dennett Product Architect

IBM FlashSystems

NVMe Developer Days 2018 San Diego, CA

1

IBM NVMe Timeline

2

•  May 2017: IBM announced NVMe strategy optimizing the entire storage system stack, from applications and workloads through to flash technology and extending to the cloud

•  December 2017: IBM exhibited the IBM FlashSystem 900 using NVMe over an InfiniBand fabric with an IBM POWER9 Systems server

•  February 2018: IBM announced general availability of NVMe over Fabrics (NVMe-oF) for IBM FlashSystem 900 with an InfiniBand fabric and software support for NVMe fabrics for the IBM Spectrum Accelerate and IBM Spectrum Virtualize families

•  July 2018: IBM announces FlashSystem 9100, NVMe-

accelerated, multi-cloud enabled storage system

•  August 2018: IBM demoed the IBM FlashSystem 9100 using NVMe/FC

•  December 2018: IBM announced general availability of NVMe over Fabrics (NVMe-oF) for IBM FlashSystem 900 with FC-NVMe.

Connection Technologies


Media



Bringing NVMe over Fabrics to the FlashSystems 900

•  IBM selects FlashSystems 900 as the first platform to support NVMe-oF

•  August 2017: Work begins in earnest to bring NVMe to existing Mellanox ConnectX-3 InfiniBand controllers

•  February 2018: NVMe-oF for IB released on FS900 •  New FC Adapter goes into development to enable FC-NVMe

shortly thereafter •  December 2018: FC-NVMe released on FS900


3

FlashSystems 900 RAID Datapath


4

XBAR/RAID Controller

XBAR/RAID Controller

Interface Interface Interface Interface

Flash Module 2

Flash Module 1

Flash Module 3

Flash Module

12

The FlashSystem architecture is one of massive parallelism, with every component generally having unique data paths to every downstream component. Notice there are no direct paths between components at the same level.

All intra-system connections are a proprietary NVM Hardware Data Path.

FlashSystems Storage Interface


5

PPC / USIC Application

Interface FPGA

IO ASIC

XB

AR

s

Buffer RAM

CPU RAM H

osts

Data

Control

5

§  Embedded PPC runs a proprietary storage application USIC that handles the translation from host commands to internal system commands

§  All storage commands and IO ASIC control goes directly to firmware

§  All user data bypasses the embedded processor and goes directly through the Interface FPGA

§  System transfers uses a HW DMA engine that requires no additional input from firmware

Universal Storage Interface Controller (USIC)

•  Originally developed in 2005 •  Code is structured with abstraction layers such as Hardware,

Transport, and ULP (Upper Layer Protocol) •  Has been ported across 6 generations of HW architectures •  Has supported storage transports of Fibre Channel, InfiniBand

SRP, FCoE, and iSCSI •  For 12 years “SCSI” was the only directory in ULP, now shared

with “NVMe” •  Unique architecture allows single embedded storage application

to support multiple protocols simultaneously NVMe Developer Days 2018 San Diego, CA

6

Advantages to the “Front-End” Approach

•  Support for NVMe-oF was added primarily through firmware/software updates

•  Allows for concurrent support of both SCSI and NVMe protocols, even on the same physical links

•  Allows customers to slowly move to new storage interconnect technology


7

FlashSystem 9100


8

Redundant Controllers •  Dual Socket Skylake •  24 DIMMs •  3 HBAs. •  Battery for persistent

write Cache Up to 24 2.5” U.2 SSDs •  25W •  NVMe •  Dual ported

Bringing FlashCore Technology to the FlashSystem 9100

•  Flash team tasked with bringing FlashCore to a new standardized platform with NVMe back-end

•  Wanted maximum re-use of existing Flash technology, but needed to fit into a 2.5” form factor


9

FlashCore in 2.5” U.2 NVMe


10

ü Updated to Micron 64L 3D TLC ü Still leveraging features and value prop of FlashCore, such as Compression and Endurance ü Currently, highest density 2.5” Enterprise SSD on market (even before compression)

Fitting within the 2.5” & 25W envelope: Consolidated multiple chips into single FPGA •  PPC •  Flash FPGA •  System Interface FPGA With embedded ARM cores •  NVMe core •  Flash Management core

FCM NVMe Architecture


11

NVMe Gateway

•  Embedded ARM runs a proprietary storage application that handles the translation from host NVMe commands to internal Flash Controller commands

•  All storage commands and user data goes to buffers/queues in CPU RAM •  Data transfers uses a HW XDMA engine that requires no additional input from firmware •  Host commands are all translated to internal 16k flash commands using a flat address scheme

CPU RAM

PC

Ie

XDMA Engine

ARM / NVMe Application

PC

Ie

PCIe

PCIe HDMA Engine

HDMA Engine

NVMe Control

NVMe Data

NVMe Controller Registers

NVMe Controller Registers

FlashCore

Advantages to the “Back-End” Approach

•  Makes use of industry standard components •  Allows for an architecture that supports

standard SSDs •  Opens the door to bring FlashCore

Technologies onto more platforms


12

new implementing nvme in high-performance flash memory systems · 2018. 12. 5. · • december...

Documents