brksan-2821

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

1

© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 1BRKSAN-282114572_05_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved. Cisco PublicBRKSAN-282114572_05_2008_c1 2

I/O Consolidationin the Data Center

BRKSAN-2821


2


Agenda

Section 1: What is I/O Consolidation

Section 2: Enabling Technologies

Section 3: FCoE (Fibre Channel over Ethernet)

Section 4: I/O Consolidation Use Cases

Challenges

Closing Remarks


Section 1What Is I/O Consolidation


3


What Is I/O Consolidation

IT organizations operate multiple parallel networksIP and other LAN protocols over an Ethernet network

SAN over a Fibre Channel network

HPC/IPC over an InfiniBand network

I/O consolidation supports all three types of traffic onto a single network

Servers have a common interface adapter that supports all three types of traffic

IPC: Inter-Process Communication


Processor

Memory

I/O Consolidation in the Network

LAN

Stor

age

IPC

Processor

Memory

I/O Subsystem

LAN

Stor

age

IPC

I/O I/O I/O

IPC: Inter-Process Communication


4


FC TrafficFC HBA

I/O Consolidation in the Host

Fewer CNAs (Converged Network Adapters) instead of NICs, HBAs, and HCAs

Limited number of interfaces for Blade Servers

All Traffic Goes over

10 GE

CNA

CNA

FC TrafficFC HBA

NIC Enet Traffic

NIC Enet TrafficNIC Enet Traffic

HCA IPC Traffic

IPC TrafficHCA


Cabling and I/O Consolidation


5


I/O Consolidation: Benefits to Customers

Fewer CNAs and Cables Storage Keeps the SameManagement Model as Native FC

No Storage Gateway Less Power and Cooling

FC TrafficFC Traffic

Enet TrafficEnet Traffic

FCoEFCoE

FC Storage FC Switch FCoESwitch

DisplayFCoE

Adapter

Server

FCoE SAN

SAN A

SAN BFCoE

FCoE


Merging the Requirements

LAN/IP

Must be EthernetToo much investment

Too many applications that assume Ethernet

Must follow the Fibre Channel model

Losing frames is not an option

StorageIPC

(Inter-Process Communication)

Doesn’t care of the underlying network, provided that:

It is cheap

It is low latency

It supports APIs like OFED, RDS, MPI, sockets


6


Why Has It Not Succeeded Yet?

Previous attemptsFibre Channel

Never credible

InfiniBand

Not Ethernet

iSCSI

Not Fibre Channel

Before PCI-Express there was not enough I/O bandwidth in the servers

It needs to be Ethernet, but…1 GE didn’t have enough bandwidth


PCI-Express

PCI Express (PCI-E or PCIe)A computer expansion card interface format designed to replace PCI, PCI-X, and AGP

PCIe 1.1Serial links at 2.5 Gbps (2 Gbps at the Datalink)

Speeds from 2 Gbps (1x) to 32 Gbps (16x)

8x is required for 10 GE

PCIe 2.0 (aka PCIe Gen 2) Doubles the bandwidth per serial link from 2 Gbit/s to 4 Gbit/s

Spec available since January 2007

Products are making their way into the market


7


10 GE

2008 will be the year of 10GE

10 GE has enough bandwidth

Merging example2 x 1 GE Ethernet NIC

1 x 4 Gbps FC (really 3.2 Gbps)

Total 5.2 Gbps over a 10 Gbps link

CNAs will all be dual-ported for HA20 Gbps usable bandwidth per server with a single CNA


CableTransceiver

Latency (Link)Power

(Each Side)DistanceTechnology

Twinax 0.1 μs0.1W10 mSFP+ CUCopper

MM 62.5 μmMM 50 μm 01W82 m

300 mSFP+ SRShort Reach

MM OM2MM OM3 01W10 m

100 mSFP+ USR

Ultra Short Reach

Cat6Cat6a/7Cat6a/7

2.5 μs2.5 μs1.5 μs

8W8W4W

55 m100 m30 m

10GBASE-T

100 Mb 1 Gb 10 Gb

UTP Cat 5 UTP Cat 5SFP Fiber

10 Mb

Mid-1980s Mid-1990s Early-2000s Late-2000s

SFP+ CuSFP+ to SFP+

Evolution of Ethernet Physical MediaRole of Transport in Enabling These Technologies


8


Twin-ax Copper Cable

Low power consumption

Low cable cost

Low transceiver latency

Low error rate (10–17)

Thinner cable with higher bend radius

Easier to manage cabling solution reduces deployment time

All copper cables are contained within rack

Application Server

Application Server

Application Server

Application Server

Application Server

Application Server

Application Server

Application Server

Application Server

Application Server

Application Server

Application Server

Application Server

Application Server

16x1

0 G

E C

able

s

16x1

0 G

E C

able

s

Application Server

Application Server

SAN BSAN A LAN


Multicore CPU Architectures Allowing Bigger and Multiple Workloads on the Same Machine

Server Virtualization Driving the Need for More Bandwidth per Server Due to Server Consolidation

Growing Need for Network Storage Driving the Demand for Higher Network Bandwidth to the Server

Drivers for 10GE to the Servers

Multicore CPUs and Server Virtualization Driving the Demand for Higher Bandwidth Network Connections


9


Section 1Enabling Technologies


Three Challenges + One


10


Why Are Frames Lost?

Collision

No longer present in full duplex Ethernet

Very rare in the data center

Transmission Error

Most common cause

Congestion is a switch issue, not a link issue

A full duplex IEEE 802.3 link does not lose frames

It must be dealt with in the bridge/switch

By IEEE 802.1

Congestion


Can Ethernet Be Lossless?

Yes, with Ethernet PAUSE Frame

PAUSESTOP

Ethernet Link

Switch A Switch B

Queue Full

Defined in IEEE 802.3—Annex 31BThe PAUSE operation is used to inhibit transmission of data frames for a specified period of time

Ethernet PAUSE transforms Ethernet into a lossless fabric


11


A

How PAUSE Works

Threshold

PAUSEFrame

Stop SendingFrames for ThisInterval of Time

PAUSEFrame

Start SendingFrames Again

B


Let’s Compare PAUSE with FC Buffer to Buffer Credit

Eight credits preagreed

R_RDYR_RDY

A B


12


PAUSE

How PAUSE Propagates

Threshold

PAUSE

S1 S2 S3


PAUSE Frame Format

A standard Ethernet frame, not tagged

EtherType = 0x8808 means MAC Control Frame

Opcode = 0x0101 means PAUSE

Pause_Time is the time the link needs to remain paused in Pause Quanta (512-bits time)

There is a single Pause_Time for the whole link

CRC

Pad42 Bytes

01:80:C2:00:00:01

Source Station MAC

EtherType = 0x8808

…

PAUSE Frame

Pause_TimeOpcode = 0x0001


13


Why Is PAUSE Not Widely Deployed?

Inconsistent implementations Standard allows for asymmetric implementations

Easy to fix

PAUSE applies to the whole linksSingle mechanism for all traffic classes

This may cause “traffic interference”e.g., Storage traffic paused due to a congestion on IP traffic


Priority Flow Control (PFC)

a.k.a. PPP (Per Priority Pause)PFC enables PAUSE functionality per Ethernet priority

IEEE 802.1Q defines eight prioritiesTraffic classes are mapped to different priorities:

No traffic interferenceIP traffic may be paused while storage traffic is being forwardedOr, vice versa

Requires independent resources per priority (buffers)

High level of industry supportCisco distributed proposalStandard track in IEEE 802.1Qb

EtherType = IEEE 802.1Q Priority CFI VLAN ID

IEEE 802.1Q Tag

16 3 1 12 Bits


14


Priority Flow Control in Action

EightPriorities

Switch A Switch B

Transmit QueuesEthernet Link

Receive Queues

One

Two

Three

Four

Five

Seven

Eight

Six

One

Two

Three

Four

Five

Seven

Eight

SixSTOP PAUSE


PFC Frame Format

Similar to the PAUSE frame

Opcode = 0x0101 is used to distinguish PFC from PAUSE

Class vector indicates for which priorities the frame carries valid Pause information

There are eight Time fields, one per priority

Class Enable VectorTime (Class 0)

CRC

Pad28 Bytes

01:80:C2:00:00:01

Source Station MAC

EtherType = 0x8808Opcode = 0x0101

Time (Class 1)Time (Class 2)Time (Class 3)Time (Class 4)Time (Class 5)Time (Class 6)Time (class 7)

…

Priority Flow Control


15


Is Lossless Better?

Frames are not dropped

FC over lossless Ethernet works well

TCP relies on losses

We can run it on a priority where we do not enable Pause

Congestion spreading and head of line blocking


In Order to Build a Deployable I/O Consolidation Solution, the Following Additional Components Are Required:

Is Anything Else Required?

Discovery protocol (DCBX)

Bandwidth manager

Congestion management


16


Discovery Protocol

DCBX: Data Center

Bridging eXchange

DCBX

Data CenterEthernet Links

Data Center Ethernet Links with Partial Enhancements

Data Center Ethernet Cloud

DCBC

XPLegacy Ethernet Network

Legacy Ethernet Links

DCBX


DCBX

Hop-by-hop negotiation for:Priority Flow Control (PFC)

Bandwidth management

Congestion management (BCN/QCN)

Applications

Logical link-down

Based on LLDP (Link Level Discovery Protocol)Added reliable transport

Allows either full configuration or configuration checkingLink partners can choose supported features and willingness to accept configuration from peer


17


Bandwidth Management

IEEE 802.1Q defines priorities, but not a simple, effective, and consistent scheduling mechanism

Products typically implement some form of Deficit Weighted Round Robin (DWRR)

Configuration and interworking is problematic

Proposal for HW-efficient, two-level DWRR with strict priority support

Consistent behavior and configuration across network elements

Standard track in IEEE 802.1Qaz


Priority Groups

Priorities Are Assigned to Individual

Traffic Classes

PriorityGroups

Priority Groups Are Then

Scheduled

First Level of Scheduling Inside Each Group

Final Link Behavior

LAN

SAN

IPC


18


Goals

BW assignment for each “Priority Group”

Example: 40% LAN, 40% SAN, 20% IPC

Should allow multiple traffic classes within a “Priority Group”

Allow these traffic classes to share BW without hard configuration

Example: VoIP and bulk traffic to share 40% LAN BW

Cannot compromise low-latency application due to convergence

Allow strict, high priority scheduling of IPC (and equivalent) traffic

Should provide management infrastructure (MIBs)

Defining scheduling algorithms is too restrictive and not necessary

Interoperability for management is important


Example of Link Bandwidth Allocation10 GE Link Realized Traffic Utilization

T1 T2 T3

LAN Traffic(40%)

Storage Traffic(30%)

(30%) HPC Traffic(30%)

(30%)

(30%)

(20%)

(50%)

(30%)

HPC Traffic—Priority Class “High”—20% Guaranteed BandwidthLAN Traffic—Priority Class “Medium”—50% Guaranteed BandwidthStorage Traffic—Priority Class “Medium-High”—30% Default Bandwidth

Offered Traffic

3 Gbs 4 Gbs 6 Gbs

3 Gbs 3 Gbs

3 Gbs 3 Gbs 3 Gbs

2 Gbs

T1 T2 T3


19


Congestion Management

Layer 2, end-to-end congestion management

Standards track in IEEE 802.1Qau

a.k.a. BCN (Backward Congestion Notification) or QCN (Quantized Congestion Notification)

SwitchThrottle

Switch

Transmit Queue

Switch

Receive Buffer

Throttle

Switch

Switch


Congestion Management Principles

Move congestion to network edges to avoid congestion spreadingUse rate-limiters at the edge to shape flows causing congestion

Tune rate-limiter parameters based on feedback coming from congestion points

Inspired by:TCP

AIMD (Additive Increase, Multiplicative Decrease) rate controlTCP window increases linearly in absence of congestionDecreases exponentially (gets halved) at every congestion indication (either implicit or explicit)

FCC (Fibre Channel Congestion Control)A feature on Cisco MDS switches


20


Congestion Point and Reaction Point

Roles and responsibilitiesReaction Points (RP) shape traffic entering the network

Congestion Points (CP) indicate congested state of queuing points

CPRP

RP


DCB: Data Center Bridging

Industry consensus term to indicatePriority flow control

Bandwidth management

Congestion management

Discovery (DCBX)


21


DCE: Data Center Ethernet

Cisco term used to indicate Cisco switches that implement the DCB features, plus

Layer 2 multipathing

Fibre Channel over Ethernet


Layer 2 Multipathing

Increase bandwidth of L2 networks via multiple active links


22


L2 Multipathing

Multiple paths are used, reclaiming network bandwidth

L3 multipathing is common in IP networks

Important when there is limited or no differentiation in speed between access links and backbone links

Reduces latency

L2 multipathingEliminates Spanning Tree from the backbone

No packet flooding

Small forwarding tables


Layer 2 Multipathing

Cisco DCE is:A precursor of TRILL, an IETF project for Layer 2 multipath

Inspired to FSPF (Fibre Channel Shortest Path First)

Cisco DCEComputes topology and forwarding via IS-IS

Provides optimal pair-wise unicast forwarding

Provides multipathing for unicast and multicast frames

Provides seamless interoperability with existing devices


23


FCoE: Fibre Channel over Ethernet

FCoE is the protocol used to carry Fibre Channel over CEE/DCE

Allows storage I/O consolidation

It’s in an advanced state of definition in INCITS T11 FC-BB-5 WG

FCoE

SANLANSAN


Delayed Drop

Delayed Drop is a mechanism that:Allows a switch buffer to virtually extend to previous hop

This reduces packet drop for transient congestions

Is enabled per priority

It is implemented by asserting PFC on the priority for a short time

After that time, traffic can flow again or can be dropped

Delayed Drop Is a Means of Using PFC to Mitigatethe Effects of Short-Term Traffic Bursts While

Maintaining Packet Drop for Long-Term Congestion


24


Actual Queue

Proxy Queue

TrafficFlow

PAUSE

UNPAUSE

Delay Drop and Proxy Queue

During short-term congestion, both queues drain fast enough that the actual queue releases the PAUSE on its own

During long-term congestion, the proxy queue fills to its high-water mark, and it releases the PAUSE; the actual queue begins dropping packets, and the congestion is managed through higher-level protocols


Section 3FCoE: Fibre Channel over Ethernet


25


FCoE: FC over Ethernet

FCoE is I/O consolidation of FC storage traffic over Ethernet

FC traffic shares Ethernet links with other traffics

Requires a lossless Ethernet fabric

Fibre Channel Traffic

Ethernet


1, 10…Gbps

FCoE Protocol Stack

From a Fibre Channel standpoint, its FC connectivity over a new type of cable called an Ethernet cloud

From an Ethernet standpoint, it’s yet another ULP (Upper Layer Protocol) to be transported


26


FCoE Benefits

FCoE benefits are the same of any I/O consolidation solution

Fewer cables

Both block I/O and Ethernet traffic coexist on same cable

Fewer adapters needed

Overall less power

Plus additional advantages of being FCSeamless integration with existing FC SANs

No gateway


FCoE Benefits

16 Servers Enet FC Total

Adapters 16 16 32

Switches 2 2 4

Cables 36 36 72

Mgmt Pts 2 2 4

16 Servers Enet FC Total

Adapters 16 0 16

Switches 2 0 2

Cables 36 4 40

Mgmt Pts 2 0 2

4

2

Nearly Halfthe Cables

LAN SAN-BSAN-A

4

2

LAN SAN-BSAN-A


27


FCoE Is Fibre ChannelFCoE Is Fibre Channel at the Host and Switch Level

Same Operational Model

Same Techniques ofTraffic Management

Same Managementand Security Models

Easy to Understand

Completely Based on the FC Model

Same Host-to-Switch and Switch-to-Switch Behavior of FC

e.g., in Order Delivery or FSPF Load Balancing

WWNs, FC-IDs, Hard/Soft Zoning, DNS, RSCN


The Two Protocols Have:Two different EtherTypesTwo different frame formats

Protocol Organization

FCoE Itself FIP (FCoE Initialization Protocol)

Is the data plane protocol

It is used to carry most of the FC frames and all the SCSI traffic

It is the control plane protocol

It is used to discover the FC entities connected to an Ethernet cloud

It is also used to login to and logout from the FC fabric


28


FCoE Frame SizeEthernetHeader

FCoEHeader

FCHeader

FC Payload

CRCEOFFCS

12 Bytes (MAC Addresses) + 4 Bytes (802.1Q Tag)

16 Bytes

24 Bytes

Up to 2112 Bytes

4 Bytes

1 Byte (EOF) + 3 Bytes (Padding)

4 Bytes

Total: 2180 Bytes


FCoE Frame FormatDestination MAC Address

Source MAC AddressIEEE 802.1Q Tag

ET = FCoE Ver Reserved

Reserved

Reserved SOF

Encapsulated FC Frame(Including FC-CRC)

EOF ReservedFCS

Reserved


29


ENode: Simplified Model

ENode (FCoE Node): a Fibre Channel HBA implemented within an Ethernet NIC

a.k.a. CNA (Converged Network Adapter)

EnetPort

EnetPort

FC Node

FCoEFCoE


FCoE Switch: Simplified Model

FCF (Fibre Channel Forwarder), the forwarding entity inside an FCoE switch

EthPort

EthPort

EthPort

EthPort

EthPort

EthPort

EthPort

EthPort

Ethernet Bridge

FCPort

FCPort

FCPort

FCPort

FCFFCoE

FCoE Switch


30


FC-BB-5 Terminology

Unchanged from previous FC standardVN_Port: Virtual N_Port

VF_Port: Virtual F_Port

VE_Port: Virtual E_Port

Added to support FCoEFCoE_LEP (FCoE link endpoint): The data forwarding component that handles FC frame encapsulation/decapsulation, and transmission/reception of FCoE frames

FCoE Controller: the entity that implement the FIP protocol


ENode: Complete Model

FCoEController

VN_Port

FCoE_LEP

FCEntity

FCoEEntity

FC-3/FC-4s

VN_Port

FCoE_LEP

FCEntity

FCoEEntity

FCoEController

VN_Port

FCoE_LEP

FCEntity

FCoEEntity

VN_Port

FCoE_LEP

FCEntity

FCoEEntity

Lossless Ethernet MAC Ethernet_Port Lossless Ethernet MAC Ethernet_Port

FC-3/FC-4s FC-3/FC-4s FC-3/FC-4s

FLOGIFDISC


31


FCoE Switch: Complete ModelE_Port E_Port E_Port F_Port F_Port F_PortFC Fabric Interface

FC Switching Element

FCoE_LEP FCoEEntity

FCoEController

FCoEEntity

FCoE_LEPFCoE_LEPFCoE_LEP FCoE_LEP FCoEEntity

FCoEController

FCoEEntity

FCoE_LEPFCoE_LEPFCoE_LEP

Ethernet_Port

Ethernet_Port

Lossless EthernetBridging Element

Ethernet_PortEthernet_Port Ethernet_Port

MeansOptional

Lossless Ethernet MAC Ethernet_Port Lossless Ethernet MAC Ethernet_Port

Ethernet_Port

Ethernet_Port

Lossless EthernetBridging Element

Ethernet_PortEthernet_Port Ethernet_Port

VE_Port FCEntity VF_Port FC

Entity VE_Port FCEntity VF_Port FC

Entity


FCoE: Initial Deployment

SAN A SAN B

10 GE4/8 Gbps FC

VF_Ports

VN_Ports

10 GEBackbone


32


10 GEBackbone

FCoE: Adding Blade Servers

SAN A SAN B

VF_Ports

VN_Ports

10 GE4/8 Gbps FC


10 GEBackbone

FCoE: Adding Native FCoE Storage

SAN B

VF_Ports

VN_Ports

SAN A

VN_Ports

10 GE4/8 Gbps FC


33


10 GEBackbone

FCoE: Adding VE_Ports

VF_Ports

VE_Ports

VN_Ports

10 GE4/8 Gbps FC

SAN BSAN A


FCoE Addressing and Forwarding

FCoE frames have:MAC addresses (hop-by-hop)

FC addresses (end-to-end)

EthernetFabricFC Fabric

FC Domain 7FC Domain 3

MAC AFCID 7.1.1

FCID 1.1.1MAC C

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

FC Frame

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

FC Frame

EthernetFabric

FC Domain 1MAC B

FC Storage

FCoE Frame

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

Dest. = MAC BSrce. = MAC A

D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)

Dest. = MAC CSrce. = MAC B

FC Fabric


34


FCoE MAC Addresses

VE_Ports and VF_Ports always use MAC addresses derived from the switch pool

VN_Ports may use two types of MAC addressesSPMA (Server Provided MAC Addresses)

FPMA (Fabric Provided MAC Addresses)

MAC Addresses are negotiated in FIP

Initial deployment will use FPMA only


The Mapped MAC Addresses

A dedicated MAC address for each FC-IDAssigned by the FC fabrics

Consistent with the FC modelOUIs with U/L = 1 (Local addressing), called FC-MAPsMultiple FC-MAPs may be supported (one per FC fabric)

48 Bits

24 Bits 24 Bits

FC-MAP(ex 02-12-34)

FC-ID7.8.9

MACAddress

FC-MAP(ex 02-12-34)

FC-ID7.8.9


35


Initial Login Flow LadderENode FCoE Switch

DiscoverySolicitation

FLOGI/FDISC FLOGI/FDISC Accept

FC Command FC CommandResponses

FIP:FCoEInitialization Protocol

FCOEProtocol

DiscoveryAdvertisement


FIP Frame: Contains FIP Operation

Destination MAC Address

Source MAC Address

IEEE 802.1Q Tag

ET = FIP Ver Reserved

Encapsulated FIP Operation (Self-Describing Length)

Ethernet FCS

PAD to Minimum Length or Mini-Jumbo Length


36


FIP Descriptors (1)

MAC AddressLen = 8Type = 2

FC-MAPReservedLen = 8Type = 3

Switch_Name

ReservedLen = 12Type = 4

Fabric_Name


PriorityReservedLen = 4Type = 1


FIP Descriptors (2)

Port_Name


FLOGI Request, FLOGI LS_ACC/LS_RJT

NPIV FDISC Request, FDISC LS_ACC/LS_RJT

Fabric LOGO Request, LOGO LS_ACC/LS_RJT

(No SOF/EOF / FC-CRC?)

ReservedLen = XXType = 7


37


FCF A

All-MACs

MAC(H2)

Solicitation (FIP)

[F=0, S=0, MAC(H2),Capability, Other]

FIPMulticast Solicitation from H2

Solicitation identifies VF_Port capable FCF-MACs with compatible addressing capabilities

Other parameters may include ENode’s Port_Name for optional duplicate MAC address detection

FCF-MAC (B)

LosslessEthernetBridge

H1

H2

MAC (H2)

MAC (H1)

FCF A

FCF-MAC (A)

FCFabric



H1

H2

MAC (H2)

MAC (H1)

FIPUnicast Advertisements from A and B

H2’s FCF list:FCF-MAC(A) [J]

FCF-MAC(B) [J]

FCF not meeting capability of ENode does not reply

MAC(H2)

FCF-MAC(A)

Mini-jumbo Advertisement (FIP)

[S=1, F=1, Priority, FC-MAP, FCF-MAC(A), Switch_Name,

Fabric_Name, Capability, Other]

MAC(H2)

FCF-MAC(B)

Mini-jumbo Advertisement (FIP)

[S=1, F=1, Priority, FC-MAP, FCF-MAC(B), Switch_Name,

Fabric_Name, Capability, Other]

FCF A

FCF-MAC (A)

FCF A

FCF-MAC (B)

FCFabric


38



FIPFLOGI Request

Capability agreed during discovery

FCF-MAC(A)

MAC (H2)

FLOGI Request (FIP)

[FC Header, FLOGI data,Proposed MAC’(H2)]

FCF-MAC(B)

MAC(H2)

FLOGI Request (FIP)

[FC Header, FLOGI data,Proposed MAC’’(H2)]

FCF A

FCF A

FCF-MAC (A)

FCF-MAC (B)

H1

H2

MAC (H2)

MAC (H1)

FCFabric



FIPFLOGI LS_ACC

ENode uses MAC addressin FIP FLOGI LS_ACC as the VN_Port MAC address for the FC-ID contained in the FLOGI data for subsequent FCoE frames

MAC (H2)

FCF-MAC(A)

FLOGI LS-ACC (FIP)

[FC Header, LS_ACC data, Approved MAC(H2)’]

MAC(H2)

FCF-MAC(B)

FLOGI LS-ACC (FIP)

[FC Header, LS_ACC data,Approved MAC(H2)’’]

H1

H2

MAC (H2)

MAC (H1)

FCF A

FCF A

FCF-MAC (A)

FCF-MAC (B)

FCFabric


39



FCoEData Transfer

All subsequent FCoE frames use granted MAC address and assigned FC-ID

FCF-MAC(A)

MAC(H2)’

Fibre Channel Frame (FCoE)

[FC SOF, FC Header, FC Data, FC CRC FC EOF]

FCF-MAC(B)

MAC(H2)’’

Fibre Channel Frame (FCoE)

[FC SOF, FC Header, FC Data, FC CRC, FC EOF]

FIP frames continue to use MAC(H2)For SPMA, MAC(H2)’ = MAC(H2)’’ = MAC(H2)For FPMA, MAC(H2)’ and MAC(H2)’’ useFC-IDs as low order 24 bits and FC-MAP for upper 24 bits

H1

H2

MAC (H2)MAC (H2)’MAC (H2)’’

MAC (H1)

FCF A

FCF A

FCF-MAC (A)

FCF-MAC (B)

FCFabric


1, 2, 4, (8), 10 Gbps 1, 10 . . . Gbps 10, 20 Gbps

The Most Asked Question: Is FCoE Routable?

FCoE


40


FCIP

Is FCoE Routable?

Most folks mean “Is FCoE IP-routable”The answer is NO, there is no IP layer in FCoE

This was a design goal to keep FCoE simple

FC-BB-5 contains FCIP that is “IP-routable”

FCoE is FC-routableFCoE switches may forward FC frames across different Ethernet clouds

FCoE switches may forward FC frames over the Internet using FCIP

FCoE FCIP FCoEIP Cloud


PCIe

Ethernet10G

bE

10GbE

Link

PCIe

Fibre Channel

EthernetH

BA

HB

A

Link

Fibre Channel Drivers

Ethernet Drivers

Operating System

Fibre Channel Drivers

Ethernet Drivers

Operating System

PCIe

Fibre Channel

Ethernet10G

bEE

10GbEE

Link

CNA: Converged Network AdapterLAN CNAHBA


41


View from Operating System

Standard drivers

Same management

Operating system sees:Dual-port, 10 Gigabit Ethernet adapter

Dual-port, 4 Gbps Fibre Channel HBAs


Open-FCoE Software

SCSI Layer

HBA Driver

Linux Kernel

HBA

HBA

HBA Mgmt Plane

File System layers

Fibre

SCSI Layer

FCoE Layer

Linux Kernel

FCoE

Net Device

FCoE Mgmt Plane

File System layers

Ethernet

Ethernet Driver

OpenFC Layer

Ethernet

Server Server


42


Open-FCoE Software: How to Get It

Open source project

Open-FCoE.org—source Git trees

Or Open-FCoE source package—TBD

Install a Linux Red Hat EL5, Fedora Core 7, or SuSE 10 distribution

Update kernel to 2.6.23 or later

Install: see Quick Start Guide at open-fcoe.org

Use switch, soft-target, or gateway


Wireshark

Once known as Ethereal

Captures and displays network traffic

Available from: http://wireshark.org/

Sample trace file/common/openfc/traces/fcoe-t11.cap

Use tcpdump to capturetcpdump –i eth0 –s 0 –w /tmp/fcoe.cap

Screenshots/demo


43


Wireshark Screenshot


Section 4Case Studies


44


Server Cabinet 1

4 x 4G FC 1 x 10 GE

MDS 9500Cisco Catalyst® 6509

Access

4

POD 1

Server Cabinet 1 Server Cabinet N Server Cabinet N

Discrete 1 GE NICsandFC HBA

Distribution

NIC TeamingActive/Standby

STP BLK

Current Data Center Environment

MDS 9500Cisco Catalyst 6509

POD N

SAN-B

LAN Core

SAN-A


POD 1

8

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

8

Top-of-Rack Consolidated I/O I/O Consolidation at Access

STP BLK

Adapter: CNA ConvergedNetworkAdapter10 GE/FCoE

POD N


Access

Distribution

Nexus 5000

SAN-B

LAN Core

SAN-A


45


POD 1 POD N

8

8

Ethernet Host VirtualizerActive/Active

Top-of-Rack Consolidated I/OEthernet Host Virtualizer




Access

Distribution

Nexus 5000

SAN-B

LAN Core

SAN-A


POD 1 POD N

VSS Supportat Aggregation

Top-of-Rack Consolidated I/OVSS Support at Aggregation

VSS8

8



Access

Distribution

Nexus 5000


SAN-B

LAN Core

SAN-A


46


Physical Topology—40 Servers, ToR

6x4 GFC

40x10 GE Ports per Switch

4x10 GEPorts

4x10 GEPorts

6x4 GFC

Nexus 5000

SAN-A SAN-BLAN


SAN-BSAN-A

Physical Topology—200 Servers40 4GFC

LAN

6x4 GFC4x10 GEPorts


47


POD 1 POD N



Access

Distribution

Nexus Family

8

Blade Servers with Copper Pass-Through

8

Blade Server NBlade Server 1Blade Server 1 Blade Server N

Blade ServerCopper Pass-Through

DC CoreMDS

SAN-CoreMDS

SAN-Core


POD 1 POD N

Blade Servers with Ethernet Switches



Access

Distribution

Nexus Family

Blade ServerEthernet-Only Switch

8

8

Blade Server 1 Blade Server 1 Blade Server 1Blade Server 1

DC CoreMDS

SAN-CoreMDS

SAN-Core


48


SAN-A SAN-B

Physical Topology—20 Blade Server, ToR

4x4 GFC2x10 GEPorts

2x10 GEPorts

4x4 GFC

Nexus 5000

LAN


SAN-A SAN-B

Topology—100 Blade Servers, ToR

8x4 GFC

4x10 GEPorts

4x10 GEPorts

8x4 GFC

LAN


49


POD 1 POD N


Layer 2 Multipath

Access Nexus 5000Distribution Nexus 7000

8

8MDS 9500Nexus 7000

Access

Distribution

Nexus 5000


SAN-B

LAN Core

SAN-A


POD 1 POD N

Consolidation in the Distribution Layer


MDS 9500Nexus 7000

Access

Distribution

Nexus 5000

8

8

Layer 2 Multipath


SAN-B

LAN Core

SAN-A


50


Conclusions


Challenges

FCoE redefines consolidated scenariosEthernet switch manufacturers will try to enter the FC switching market

FC switch manufacturers will try to enter the Ethernet switching market

HBA manufacturers will try to enter the NIC market

NIC manufacturers will try to enter the HBA market

Deep integration with virtualization will take some time


51


Web Pointers

PCI Expresshttp://en.wikipedia.org/wiki/Pci_express

IEEE 802.3http://standards.ieee.org/getieee802/802.3.html

Improvements to Ethernethttp://www.nuovasystems.com/EthernetEnhancements-Final.pdf

IEEE 802.1 activitieshttp://www.ieee802.org/1/files/public/docs2007/new-cm-barrass-pause-proposal.pdfhttp://www.ieee802.org/1/files/public/docs2007/new-cm-pelissier-enabling-block-storage-0705-v01.pdfhttp://www.ieee802.org/1/files/public/docs2007/au-ko-fabric-convergence-0507.pdfhttp://www.ieee802.org/1/pages/802.1au.htmlhttp://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbcxp-overview-rev0.2.pdf

FCoEhttp://www.fcoe.com/http://www.t11.org/http://www.open-fcoe.org/http://www.fibrechannel.org/OVERVIEW/FCIA_SNW_FCoE_WP_Final.pdf

TRILLhttp://www.ietf.org/html.charters/trill-charter.html


Thank You


52


Recommended Reading

Continue your Cisco Live learning experience with further reading from Cisco Press

Check the Recommended Reading flyer for suggested books

Available Onsite at the Cisco Company Store


Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Passport points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008.

Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.


53


brksan-2821

Documents