Building Your Own Robust and Powerful Software Defined Storage with VMware vSAN Tips on Choosing Hardware for vSAN Deployment

2

Agenda

1 Overview of VSAN

2 VSAN VCG at a Glance

3 VSAN Hardware Guidance (Ready Nodes, Post Deployment Experience)

4 Ready Node Configurator 2.0 - Demo

5 VSAN Certification Under the Hood

6 Q & A

Overview of VSAN

4

Powered by VMware VSAN and vSphere

Runs on any standard x86 server

Pools HDD/SSD into a shared datastore

Delivers enterprise-grade scale and performance

Managed through per-VM storage policies

Deeply integrated with the VMware stack

vSphere VSAN

VSAN Datastore

5

Supporting a Broad Variety of Use Cases

Virtual San

Business Critical Apps Virtual Desktops (VDI)

DR / DA

Test/Dev DMZ

ROBO Management

Staging

6

VSAN is the Market Leader

VSAN Customers 5,000 Y-O-Y

Growth >200%

Countries have VSAN Presence

109

Industry Verticals use VSAN

100%

Penetration in Fortune 1000 (US)

40%

5 of top 7 Healthcare companies in US

20 of top 25 Finance Companies in US

20 of top 25 High Tech Companies in US

12 of top 15 Retail companies in US

vSAN VCG at a Glance

8

8

164 Ready

Nodes

65 Unique Server

Platforms

15 Server

Partners

VSAN Offers the Widest Diversity of Server Choices

9

9

164 Ready

Nodes

65 Unique Server

Platforms

15 Server

Partners

VSAN Offers the Most Flexibility of Configuration Choices

✔ Widest Diversity of Platform Choices Most Flexibility ✔

10

10

164 Ready

Nodes

65 Unique Server

Platforms

15 Server

Partners

We are Continuously Refreshing and Keeping Up to Date….

70

Ready Nodes

AUGUST 2015

11

11

We are Continuously Refreshing and Keeping Up to Date….

163 NVMe

August 2016

? NVMe

August 2015

12

12

We are Continuously Refreshing and Keeping Up to Date….

163 NVMe

August 2016

0 NVMe

August 2015

13

We support the latest NVMe Devices

13

P3500, 3600, 3700

PM1725

SN150

14

14

164 Ready Nodes

65 Unique Server Platforms

15 Server Partners

163/ 220 NVMe / PCIe devices

122 IO controllers

Broadest Deployment Options

Deployment Choices

Broadest Deployment Choices

VxRail Ready Node Build Your Own

15

15

164 Ready Nodes

65 Unique Server Platforms

15 Server Partners

163 / 220 NVMe / PCIe devices

122 IO controllers

Broadest Deployment Options

Factory Install Broadest Deployment Choices

VxRail Ready Node Build Your Own

Software Flexibility

Ready to Run Ready Nodes

16

16

164 Ready Nodes

65 Unique Server Platforms

15 Server Partners

163 / 220 NVMe / PCIe devices

122 IO controllers

Broadest Deployment Options

Agility Broadest Deployment Choices

VxRail Ready Node Build Your Own

Out of Box Experience

Ready to Run Ready Nodes + Auto CONFIG

17

17

164 Ready Nodes

65 Unique Server Platforms

15 Server Partners

163 / 220 NVMe / PCIe devices

122 IO controllers

Broadest Deployment Options

Choice Flexibility Broadest Deployment Choices

VxRail Ready Node Build Your Own

Out of Box Experience

Ready to Run Ready Nodes + Auto CONFIG

Real Time Validation

Ready Node Configurator 2.0

Hardware Guidance

19

HCI is Pushing the Server IOs like Never Before!

19

CPU

DRAM

DRAM

PCIe Bus

Compute / Memory IO Subsystem

VSAN CERT

20

1. NVMe and PCIe Devices on the PCIe bus (Extension of Memory)

20

CPU

DRAM

DRAM NVMe PCIe

PCIe Bus

Compute / Memory IO Subsystem 10’s µs 10s ns – 1µs

21

2. Network Adaptors – Inter Cluster Communication

21

CPU

DRAM

DRAM

NVMe PCIe

PCIe Bus

Compute / Memory

IO Subsystem

NIC 10GbE

10’s ns – 1µs

10’s µs

22

3. IO Controllers to Propagate the IOs to the Drives

22

CPU

DRAM

DRAM

NVMe PCIe

PCIe Bus

IO Controller

SATA / SAS

Compute / Memory

IO Subsystem

NIC 100µs – 1ms

>10s ms

1 ns – 1µs

10’s µs

23

Design Guidance

23

1.  All Components must be on the HCL/VCG

2.  Turn off caching for HDDs ; turn off local drive cache

24

HCI is pushing the IO Subsystem like Never Before!

24

IO Controllers

NIC

NVMe / PCIe devices

Drives

1 2

3 4

VSAN CERT

VSAN CERT

VSAN CERT

IO CERT

25

VSAN Hardware Guidance Focus on 3 Components

25

HBA / IO Controllers NVMe / PCIe devices

Drives

1 2

3

26

Considerations for NVMe and PCIe devices

26

PCIe NVMe

PCIe Bus

Choosing the right form factor - HHHL, 2.5” and M.2

Better performance with NVMe as VSAN cache layer for higher IOPs

1

2

27

Considerations for NVMe and PCIe devices (and SSDs)

27

PCIe NVMe

PCIe Bus

Choose the correct Endurance and Performance Rating

Endurance: TB written in 5 years

Performance: Writes per second

3

*

http://www.vmware.com/resources/compatibility/vsan_profile.html

*

28

Considerations for NVMe and PCIe devices

28

PCIe NVMe

PCIe Bus

Choose the correct Endurance and Performance Rating

Endurance: TB written in 5 years

Performance: Writes per second

3

4

http://www.vmware.com/resources/compatibility/vsan_profile.html

Search for VSAN Quick Reference

Guide

29

3. IO Controllers to Propagate the IOs to the Drives

29

CPU

DRAM DRAM

NVMe PCIe

PCIe Bus

IO Controller

SATA / SAS

Compute / Memory

IO Subsystem

NIC

30

IO Controller Modes of Operation

30

Firmware Commands

RAID Acceleration Block

PCIe Interface

PCIe Host Interface

Link 0 Link 1

SAS / SATA Interface

(1) Passthrough

(2) RAID

31

VSAN prefers Passthrough Over RAID

Firmware Commands RAID Block

PCIe Interface

PCIe Host Interface

Link 0 Link 1

VSAN prefers passthrough mode over RAID mode

For best performance use controllers with Queue Depth of 512 or higher

1

2 SAS SATA Interface

32

1 Most Optimized VSAN Experience

2 Simplicity of Initial set up and Ongoing Operations

3 Consistency of Experience

Why Passthrough is Preferred

Passthrough mode is also called HBA mode

33

Do Not Share non VSAN and VSAN Drives on Same Controller

vSphere Virtual SAN

Boot Devices and Log Devices, should not be behind the same controller backing VSAN drives

VMFS (other no VSAN file systems) have different IO completion expectation

1

2

34

Make sure you use the latest Driver Firmware Listed ?

1.  Running the correct combination of driver and firmware is critical for any IO Device

2.  Don’t use a later version of driver and firmware if it is not on the VSAN VCG …no matter what anyone tells you!

35

Boot Options

SD / USB 4 – 8 GB

SATADOM Min: 16GB

SSD / HDD

§  Direct to Syslog server or use network dump Collector

§  Persist or use network dump collector

§  Endurance 384 TBW for

VSAN 6.2 and beyond

§  Persist or use network dump collector

§  Separate Controller compared to VSAN

Drives Connect to the PHYs and Arranged into Disk Groups

CONFIDENTIAL 36

Firmware Commands

RAID Acceleration Block

PCIe Interface

PCIe Host Interface

Link 0 Link 1 Disk Group:1

Disk Group:2

PHYs

37

Max Disk Group is 5 per HOST

•  Cluster: 2-64 hosts •  Host: 1-5 disk groups •  Disk Group

–  1 SSD / NVMe / PCIe for cache

–  1-7 SSD or HDD for capacity

Host 1 Host 2 Host 3 Host 64 …

Disk Group

1

Disk Group

2

SSD or NVMe or

PCIe 1

SSD or

HDD1

SSD or

HDD2

SSD or

HDD7

Disk Group

5 …

…

Cache

Capacity

10GbE

1

7

38

Firmware Commands

RAID Acceleration Block

PCIe Interface

PCIe Host Interface

Link 0 Link 1

NVMe Caching Tier with SSD Capacity Tier

NVMe / PCIe

PCIe Interface

Disk Group

39

Disk Groups for VSAN

40 Max Drives per Host

3 - 4 Disk Groups

Balanced Disk Groups

More Drives / Disk Group

40

Read / Write Patterns for All Flash vs. Hybrid

SSD or NVMe or

PCIe 1

…

Cache

Capacity

All Flash Hybrid

CACHE R/W Split 100% Writes 70% Reads ,

30% Writes

Reads Capacity Tier Both Tiers

Writes Cache Tier Both Tiers

NVMe or

SSD

HDD HDD HDD

NVMe or

SSD NVMe

or SSD

Much lower endurance Capacity Device

Ready Nodes

42

Firmware Processing Unit

RAID Processing Unit

PCIe Interface

PCIe Host Interface

Link 0 Link 1

Ready Nodes Abstracts the Complexities

Ready Nodes abstract the complexities and provide a simple consumption model

BYO doesn’t factor in additional server hardware and Firmware Expanded Backplane

Chassis FW

Drives SAS Expander

Boot Options

43

Controller + Expander Certifications only available on Ready Nodes

Expanded Backplane

Chassis FW Drives SAS Expander Boot Options

Post Deployment Experience

45

Errors Do Happen…Don’t Worry!

Health Check

Burn In Tests

Flag incompatible Firmware, Drivers

Network, Performance, Capacity

46

Errors Do Happen…Don’t Worry! – Future Release

Health Check

Burn In Tests

Update Controller’s Firmware and Driver

Auto Config

Ready Node Configurator

48

DEMO

vSAN Certification Under the Hood

50

Principle 1 Identify the different types of I/O: Application IO

APPLICATION I/O

HOUSEKEEPING I/O

51

Principle 1 Identify the different types of I/O : Rebuild / Resync IO

APPLICATION I/O

RESYNC / REBUILD I/O

52

Principle 1 Identify the different types of I/O

APPLICATION I/O

ESX MANAGEMENT I/O RESYNC / REBUILD

I/O

HOUSEKEEPING I/O

53

Principle 2 Account for Worst Case Scenario

APPLICATION I/O

RESYCN / REBUILD I/O

HOUSEKEEPING I/O

✔

✔

✔

Worst case load on the system

54

Principle 3 Initiate Controller Task Management Activities during IO fulfillment

Virtual SAN

IO Controller

IO controller may initiate its own Error Handling Operations

This is over an above the regular IO Operations

55

Certification Suite Tests

Performance

Different combinations of

Read / Write profiles*

Steady State Analysis

Sustained period of time with

Heavy I/O

Worst Case Behavior

Planned and Unplanned tests

Drive or Disk Group

Failure

Stress Hot Plug

Sequential / Random

Both drive and entire disk group failure

Constantly Evolving and Enhancing our Certification Suite

Formulating Rigorous Equivalency model to reduce certification efforts

*100r0w, 70r30w, 0r100w, 50r50w

Q & A