enabling converged networks with iscsi over data center bridging

Enabling Converged Networks with iSCSI over Data Center Bridging

Joe GervaisSenior Director, Product Marketing, Emulex

Dell Storage Forum 2011

Storage2

Agenda

• iSCSI over Data Center Bridging (DCB) Defined• Enabling Convergence• Companion Technologies For 10G iSCSI over DCB• Sample Topology


Storage3

iSCSI over DCB Defined


Storage4

iSCSI over DCB

• iSCSI over Data Center Bridging (DCB) protocols a.k.a. Enterprise iSCSI

– Data Center Bridging Capabilities eXchange Protocol (DCBX)› Supports centralized configuration of DCB and related protocols› Initiated by endpoints (hosts/storage) announce themselves with iSCSI

Type/Length Value (TLV)

– Enhanced Transmission Selection› Provides priority groups with bandwidth controls

– Priority Flow Control (PFC)› Enable lossless Ethernet operation to provide deterministic performance

• What’s needed– iSCSI HBA supporting iSCSI over DCB

› Could be a NIC – some limitations– 10G Ethernet Switch supporting DCB– 10G Storage Array supporting iSCSI over DCB


Storage

Data Center Bridging Exchange Protocol (DCBX)

Automatic “handshake” to allow device to exchange Enhanced Ethernet capabilities• Simplifies configuration• Identify misconfigurations• Allows switch to override server parameters

Based on Link Layer Discovery Protocol (LLDP)


Storage

Benefit Hardware-offload iSCSI

Software iSCSI

Performance iSCSI uses separate I/O stack than NIC traffic. Maintains consistent storage performance under varying LAN traffic conditions.

LAN congestion leads to PFC pause frames affecting iSCSI traffic.

• Essential to have lossless, consistent, reliable storage when sharing the same network regardless of LAN traffic conditions

• DCB designed to address this concern using PFC for different traffic types

– Uses VLAN priorities - each can be paused

– Ensures zero data loss when receive queues are full

• Implementation specifics may limit efficiency of PFC

– Hardware iSCSI vs. Software iSCSI

Lossless Ethernet

• PAUSE

• Ethernet Link• Receive

Queues• Transmit Queues

• Zero

• One

• Two

• Three

• Four

• Five

• Six

• Seven

• Priority

• Zero

• One

• Two

• Three

• Four

• Five

• Six

• Seven

• Priority

• STOP


Storage

Server Hardware

OS

Hypervisor

App

OS

App

OS

App

OS

App

OS

iSCSI Target

LAN

NIC Buffer Pool

App

OS

App

OS

CN

A P

ort

Adapter Hardware

LAN app trafficHardware iSCSI

iSCSI Buffer Pool

Initiates iSCSI transactions, instructs OS to pre-allocate buffers for transaction

PG1PG2

Lossless Ethernet - Hardware iSCSI


Storage

Server Hardware

OS

Hypervisor

App

OS

App

OS

App

OS

App

OS

No iSCSI session awareness, hence cannot pre-allocate buffers in shared OS buffer pool. Buffer over-run results in upstream Pause Frames QOS is

compromised

iSCSI Target

LAN

NIC Buffer PoolShared between LAN and iSCSI traffic

App

OS

App

OS

CN

A P

ort

Adapter Hardware

PriorityPauseFrame

Software iSCSI LAN app traffic

PG1PG2

iSCSI traffic suffers from unnecessary pause frames especially with concurrent TCP/UDP traffic… therefore not eliminating jitter

Lossless Ethernet - Software iSCSI


Storage9

Enabling Convergence


Storage10

Barriers to Convergence

• iSCSI on Gigabit Ethernet best practices architected separate Ethernet network for iSCSI traffic– Servers - iSCSI on separate NICs– Switches – iSCSI on separate switches

• Rational– Congestion causes packet loss, creating non-deterministic performance– TCP/IP error recovery expensive from a predictable latency perspective

• In virtualized environments, best practices employ multiple NICs for traffic isolation (iSCSI, vMotion, production, management, etc.)– Enables effective bandwidth caps between services, ensures available

bandwidth at server


StorageEmulex Confidential

• Large Enterprise– Provide Deterministic

Performance› Requires Lossless Ethernet

(DCB)– Support Convergence

› DCB support› Multichannel Support

– Enterprise class management

• Small / Medium Enterprise– Support 1 -> 10 migration

› Multichannel support› DCB support

– Support 1G backwards compatibility› 10GBASE-T support

– Easy-to-use tools for configuration/management› DCBX based network

configuration

Enabling Converged iSCSI

User Experience

Lossless Ethernet

Multichannel

10GBaseT



User Experience

• Hardware iSCSI enhances other aspects of the user experience

Benefit Hardware-offload iSCSI Software iSCSIScalability Support of more VMs in virtualized

environmentsNot performance-optimized

Reliability Attachments to iSCSI targets are maintained in hardware

Disruption in the host could lead to the loss of sessions

Manageability Emulex OneCommand™ Manager provides a single, uniform management interface; procedures for attaching to iSCSI targets are the same on all supported host operating systems. Graphical and command-line interfaces are provided.

The software iSCSI initiator runs on top of the TCP/IP stack and is managed differently for every operating system.

OneConnect™ iSCSI adapters can be managed across the enterprise from a single console.

Software iSCSI is managed locally on each host.



User Experience - Scalability and Reliability

• Hardware iSCSI demonstrated support for more VM’s per server, given performance objective of 5k IOPs per VM

• Reduced need for physical servers translate to power and cost savings• Attachments to iSCSI targets maintained in adapter hardware – immunity to host disruptions that

could lead to session loss in software initiators

App

OS

App

OS

App

OS

App

OS

App

OS

App

OS

App

OS

App

OS… …

iSCSI storage

iSCSI storage

EthernetSwitch

EthernetSwitch

VM1 VM2 VM3 VM26 VM1 VM2 VM3 VM17

Hardware iSCSI Software iSCSI

53% more

Software iSCSI

Hardware iSCSI

0

5

10

15

20

25

30

VMs


Storage

• OneCommand Manager provides a single, uniform management interface

– Procedures for attaching to iSCSI targets are the same across supported OS

• Alternatively, can use VMware vSphere Client to manage OneConnect iSCSI adapter

– SCSI Management API (IMA) plugin provided for VMware

– Hardware iSCSI device viewed in vSphere Client under Storage Adapters in the Configuration tab

• Less complicated than software iSCSI, which requires configuring a Vmkernel connection to ESX, then binding to the physical NIC

User Experience - Manageability



Boot from SAN

• OneConnect iSCSI adapter can attach to an iSCSI target at boot time• iSCSISelect Utility is used to specify and configure a boot device on a

particular LUN– Invoked at boot-time by pressing CTRL-S– Configures iSCSI initiator to attach to desired target– LUN can be configured as boot device


Storage

• Current versions of vSphere don’t directly support DCB protocols– No APIs for a NIC to support

DCBX and PFC.

• Since HBAs/CNAs offload iSCSI, DCB transparent to OS– DCBX and PFC all handled by

device– Host to Device interaction all at

SCSI layer

iSCSI over DCB with VMware

App

OS

App

OS

App

OS

App

OS…

iSCSI storage

EthernetSwitch

VM1 VM2 VM3 VM26

Hardware iSCSI


Storage17

Companion Technologies For 10G iSCSI over DCB


Storage

• Technology– Universal Multi-Channel (UMC)

presents multiple (typically 4) PCI functions per port to the OS

› Partitions the port into independent channels of user-defined bandwidth

– Switch-agnostic› Multiple channels can be

segmented based on MAC address and/or customer-provided VLAN ID (C-VLAN)

• Benefits– Retains networking best practices

in 1G to 10G migration› Physical 1G port becomes logical

10G port– Transparent to operating systems

› OS still sees dedicated NIC devices

– Complementary to DCB traffic shaping with ETS

› Bandwidth controls at both logical NIC and priority group level

Multi-Channel

Physical Port Channels

MultifunctionAdapters

Network/IP

Storage/iSCSI

Network/IP

Network/IP

Unique PCI function per logical port



Low Cost Media Options for iSCSI over DCB

• 10GBASE-T– Backwards compatible with

Gigabit Ethernet– Uses Cat6/Cat6A unshielded

twisted pair cabling with RJ-45 connectors

– Supports 100m distances› Top of Rack, End of Row (EOR)

• SFP+ Direct Attach Copper– iSCSI adapter with SFP cage– Twinaxial Cable with SFP

modules on ends – Supports 5-7m distances

› Top of Rack› Optical versions available

supporting 100m or more


Storage20

Sample iSCSI over DCB Topology



Enterprise iSCSI Solution Example

• Sample Solution Overview– PowerEdge servers (R710)

in VMware cluster– EqualLogic array

(PS6010XVS)– PowerConnect switch (B-

8000e)– Emulex Enterprise iSCSI

HBAs› vCenter iSCSI IMA Plug-in

IMAPlugin

iSCSI Configuration today


Storage

Enterprise iSCSI Solution Example

DCBX iSCSI TLV

DCBX iSCSI TLV

DCBX Enable PFCDCBX Enable 9000B MTUDCBX Enable ETS PG2 BW 1G

DCBX Enable PFCDCBX Enable 9000B MTUDCBX Enable ETS PG2 BW 1G


Storage

Emulex OneConnect

• 10 Gigabit Ethernet products for Dell PowerEdge Servers– NIC, iSCSI and FCoE models

• iSCSI over DCB support with OCe11000 family of products– Built in support for DCB with Microsoft Windows,

VMware vSphere, Red Hat and Novell SUSE Linux operating systems

• SFP+ and 10GBASE-T models


Storage24

iSCSI over DCBBenefits Summary

• Enables convergence of LAN and SAN traffic– DCB enables mixed traffic– Multichannel enables multiple logical NICs in one 10G link

• Deterministic performance in converged, mixed traffic environment– Lossless Ethernet using Priority Flow Control enables support

• Ease of use– Centralized configuration via switch with DCBX support

› Hosts, storage announce capabilities with iSCSI TLV› Switch enables PFC, Jumboframes, other parameters via DCBX

• Easy server integration– iSCSI HBAs such as Emulex OneConnect provide transparent support


Storage

• See http://www.emulex-dell.com/

– To register for the Emulex quarterly Dell newsletter

– For information on Emulex OneConnect products supporting iSCSI over DCB

– For information about other Emulex solutions for Dell

– For iSCSI over DCB white paper

Resources & Call to Action

http://www.emulex-dell.com/


Storage26

Thank you

enabling converged networks with iscsi over data center bridging

Technology

iscsi trafficservers

g iscsi

iscsi targets

traffic isolation iscsi

separate nicsswitches

vmwarehardware iscsi

user experiencehardware

hbascnas offload iscsi