fibre channel over ethernet (fcoe) data center bridging (dcb) case studies techbook

Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB) Case Studies

Version 6.0

• EMC Connectrix B Setup Examples

• Cisco Nexus Series, vPC Setup Examples

• FCoE-Based Blade Server Solutions

• Juniper QFX3500 Setup Examples

• Brocade VDX Switches Setup Examples

Mark LippittErik SmithErik PaineBenjamin Heydary

Fibre Channel over Ethernet (FCoE) Case Studies TechBook2

Copyright © 2008–2014 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information issubject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NOREPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THISPUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY ORFITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicablesoftware license.

EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the UnitedState and other countries. All other trademarks used herein are the property of their respective owners.

For the most up-to-date regulator document for your product line, go to EMC Online Support(https://support.emc.com).

Part number H10524.6

Contents

Preface

Chapter 1 Introduction to Fibre Channel over EthernetIntroduction ....................................................................................... 26Benefits ............................................................................................... 28Terminology....................................................................................... 29Management tools............................................................................. 31Cable management recommendations .......................................... 32Protocols ............................................................................................. 33Supported connectivity options...................................................... 34

Configuration rules ...................................................................34Connectivity options ..................................................................36

Chapter 2 EMC Connectrix B Setup ExamplesMP-8000B ........................................................................................... 46

MP-8000B overview....................................................................46Management................................................................................48POST and boot specifications....................................................48

MP-8000B supported features and topologies.............................. 49FCoE interface .............................................................................50Ethernet uplink interface ...........................................................50FC interface..................................................................................52Ethernet end devices ..................................................................52

MP-8000B direct-connect topology................................................. 53Existing topology .......................................................................54Target topology...........................................................................56Prerequisites ................................................................................57Configuring the MP-8000B........................................................58

Fibre Channel over Ethernet (FCoE) Case Studies TechBook 3

Contents

MP-8000B firmware upgrade procedure....................................... 90Assumptions ............................................................................... 90Upgrading firmware .................................................................. 90

Troubleshooting the MP-8000B....................................................... 93Fibre Channel FOS...................................................................... 93CMSH........................................................................................... 99Connectivity troubleshooting ................................................. 102VLAN troubleshooting ............................................................ 102DCBX troubleshooting............................................................. 102FCoE troubleshooting .............................................................. 103QoS troubleshooting ................................................................ 103Link Aggregation (LAG) troubleshooting ............................ 103

ED-DCX-B........................................................................................ 104Topology .................................................................................... 105Compatibility ............................................................................ 105Supported topologies............................................................... 106Installing the PB-DCX-24FCoE............................................... 106Configuring the PB-DCX-24FCoE.......................................... 107Useful commands..................................................................... 109

Chapter 3 Nexus Series Switches Setup ExamplesNexus Series switches .................................................................... 113

Nexus 6004................................................................................. 113Nexus 6001................................................................................. 114Nexus 2248PQ Fabric Extender (FEX) ................................... 115Nexus 5596................................................................................. 116Nexus 5548................................................................................. 116Nexus 5020................................................................................. 117Nexus 5010................................................................................. 118Nexus 4000................................................................................. 118Nexus 2232PP............................................................................ 119Management.............................................................................. 121VPN Routing and Forwarding (VRF).................................... 122

Nexus 6004 supported features and topologies ......................... 123Interfaces.................................................................................... 123Features ...................................................................................... 125

Nexus 6001 supported features and topologies ......................... 128Interfaces.................................................................................... 128Features ...................................................................................... 130

Nexus 2248PQ FEX supported features and topologies ........... 133Interfaces.................................................................................... 133Features ...................................................................................... 134


Contents

Nexus 5000 supported features and topologies.......................... 136FCoE interface ...........................................................................137Ethernet uplink interface .........................................................138FC interface ................................................................................139Ethernet end-device interface..................................................140

Nexus 2000 supported features and topologies.......................... 141FCoE interface ...........................................................................141Ethernet uplink interface .........................................................142Ethernet end-device interface..................................................142

Nexus 6000 Series and Fabric Extenders topologies .................. 143Multihop FCoE ..........................................................................143Leaf and Spine (Layer-2/Layer-3) fabric ...............................144High Performance Compute (HPC) and HighFrequency Trading (HFT) ........................................................145Scalable data center access and aggregation withFabric Extenders........................................................................146Fabric Extenders (FEX).............................................................147

Configuring 40GbE FCoE connectivity with Nexus 6000Series ................................................................................................. 149

Prerequisites...............................................................................149Summary of configuration steps.............................................150Individual reference topologies ..............................................152

Nexus 5000 direct-connect topology ............................................ 173Summary of configuration steps.............................................173Existing topology ......................................................................174Target topology .........................................................................176Prerequisites...............................................................................177Configuring the virtual interfaces on Nexus 5020 ...............178Etherchannel configuration on a Nexus 7000 switch...........203Spanning tree configuration and verification on Nexus5020..............................................................................................206

Nexus 7000, Nexus 5000, and MDS 9500 series topology ......... 210Summary of configuration steps.............................................210Target topology .........................................................................212Prerequisites...............................................................................214Configuring the Nexus 7000....................................................214Configuring the Nexus 5548....................................................231Configuring the MDS 9513 ......................................................238

Nexus 5000/Nexus 2232PP topology........................................... 242Summary of configuration steps.............................................242Existing topology example ......................................................243Target topology .........................................................................244Prerequisites...............................................................................245

5Fibre Channel over Ethernet (FCoE) Case Studies TechBook

Contents

Configuring the Nexus 5000 ................................................... 246Configuring Cisco VE_Ports ......................................................... 272Configuring Cisco FCoE NPV ports ............................................ 274Complex FCoE topologies ............................................................. 277

Fixed port chassis ..................................................................... 277Modular chassis ........................................................................ 278

Nexus Series switches firmware upgrade procedure................ 280Assumptions ............................................................................. 280Upgrading firmware ................................................................ 280

Troubleshooting the Nexus Series switches................................ 284Show tech-support selections ................................................. 284General connectivity troubleshooting ................................... 285Fibre Channel troubleshooting............................................... 286Hardware-related errors.......................................................... 288High CPU issue troubleshooting............................................ 288Gatos Port ASIC troubleshooting........................................... 289Altos Fabric ASIC troubleshooting ........................................ 289FCoE troubleshooting .............................................................. 289DCBX troubleshooting............................................................. 290Verify zoning configuration and that CNA WWPN iszoned active............................................................................... 290Verify CNA WWPN is registered in nameserver in yourVSAN.......................................................................................... 290Ethernet PortChannel troubleshooting ................................. 290Verify PortChannels................................................................. 291Spanning-tree troubleshooting............................................... 291Multicast troubleshooting ....................................................... 292

Virtual PortChannel........................................................................ 293Introduction............................................................................... 293vPC overview............................................................................ 299vPC topologies overview ........................................................ 306vPC topology considerations.................................................. 309vPC case studies ....................................................................... 324

Chapter 4 Blade Server Solutions Setup ExamplesIntroduction..................................................................................... 420

Cisco UCS .................................................................................. 421IBM BladeCenter....................................................................... 423

Cisco UCS (B-Series)....................................................................... 426Cisco UCS supported features and topology ....................... 426UCS fabric interconnect initial configuration....................... 432Cisco UCS Fibre Channel Switch Mode configuration


Contents

example.......................................................................................434Cisco UCS Fibre Channel End Host mode configurationexample.......................................................................................452Directly attaching EMC storage to Cisco UCS FabricInterconnect ...............................................................................477Cisco UCS Fibre Channel Server Mode: Boot from SANexample.......................................................................................508Setting up and implementing VE_Port on Cisco UCS.........519Setting up and implementing VF_Port on Cisco UCS.........545Configuring UCS 10G iSCSI using VNX-2 arrays ................574

IBM BladeCenter ............................................................................. 599Brocade Converged 10 GbE Switch Module supportedfeatures and topologies ............................................................599Brocade Converged 10 GbE configuration example............603Cisco Nexus 4000 supported features and topologies.........620Cisco Nexus 4001I configuration example ............................622

HP Virtual Connect FlexFabric and Flex-10 ................................ 654Virtual Connect FlexFabric solution.......................................654Flex-10 technology ....................................................................658FlexFabric components and data center communication....659Using FlexHBAs for FC SAN connectivity............................666Creating Server Profiles ...........................................................668Case study..................................................................................671References ..................................................................................687

Chapter 5 Juniper QFX3500 Switches Setup ExamplesJuniper QFX3500 overview............................................................ 690Juniper QFX3500 supported features and topologies................ 692

FCoE Interface ...........................................................................693Ethernet uplink Interface .........................................................693FC Interface ................................................................................694Ethernet end device Interface..................................................694

QFX3500 Gateway and Transit mode topology.......................... 695Summary of configuration steps.............................................695Existing topology ......................................................................697Target topology .........................................................................697

Configuration steps......................................................................... 699Configure Gateway mode switch (54.80) ..............................699Configure transit mode switch (54.81)...................................706

Useful JUNOS commands.............................................................. 711


Contents

Chapter 6 Brocade VDX Switches Setup ExamplesBrocade VDX data center switches .............................................. 714

Ethernet fabrics ......................................................................... 714VDX 6710 ................................................................................... 715VDX 6720 ................................................................................... 715VDX 6730 ................................................................................... 715Modes of operation .................................................................. 716

Brocade VCS fabric technology .................................................... 718Introduction............................................................................... 718Terminology .............................................................................. 720VCS Ethernet fabric .................................................................. 720

Brocade VDX switches supported and unsupportedfeatures ............................................................................................. 722

Supported features and configurations................................. 722Unsupported features and configurations............................ 724

Brocade VDX switches in a VCS fabric case study .................... 726Summary of configuration steps ............................................ 726VCS LAN/SAN Convergence case study............................. 726

References ........................................................................................ 741

Appendix A Installation NotesInstalling the Nexus 5000............................................................... 744Installing the MP-8000B................................................................. 749

Prerequisites ............................................................................. 749Installing power to the MP-8000B......................................... 751Interpreting POST results....................................................... 751

Installing the MLX .......................................................................... 758

Glossary


Title Page

Figures

1 Typical topology versus FCoE example using Cisco Nexus 5000........... 272 Nexus 2000 supported connectivity options .............................................. 363 Nexus 2248PQ supported connectivity options ........................................ 374 Nexus 4001i supported connectivity options............................................. 375 Nexus 5000 supported connectivity options .............................................. 386 Nexus 6000 supported connectivity options .............................................. 397 Nexus 61x0 supported connectivity options .............................................. 408 Nexus 7000 supported connectivity options .............................................. 419 MDS 9000 supported connectivity options ................................................ 4210 Brocade 8000/DCX supported connectivity options ................................ 4311 Juniper QFX3500/3600 supported connectivity options.......................... 4412 Switch layout example................................................................................... 4713 Topology example.......................................................................................... 4914 Existing topology example............................................................................ 5415 Target topology .............................................................................................. 5616 switchshow output......................................................................................... 9617 ED-DCX-4S-B switch and PB-DCX-24FCoE module topology

example............................................................................................................10518 Straight-through configuration example .................................................. 12019 Cross-connected configuration example................................................... 12020 Topology example........................................................................................ 13621 Nexus 2000 supported features and topologies....................................... 14122 Cisco Nexus 6004/6001 40GbE Multihop FCoE topology example ..... 14423 Cisco Nexus 6004/6001 Leaf and Spine topology example ................... 14524 Cisco Nexus 6004/6001 HPC and HFT topology example .................... 14625 Scalable data center access and aggregation with Fabric Extenders

example............................................................................................................14726 Cisco Nexus 2000 Series Fabric Extenders design scenarios.................. 14827 Reference topologies .................................................................................... 15228 Existing topology example.......................................................................... 174


Figures

29 Target topology example ............................................................................ 17630 Virtual FC interface...................................................................................... 17931 Target topology example ............................................................................ 18032 Target topology example ............................................................................ 21233 Existing topology example ......................................................................... 24334 Target topology example ............................................................................ 24435 VE_Ports topology ....................................................................................... 27236 VN_Ports topology ...................................................................................... 27437 Fixed port chassis configuration example ................................................ 27838 Modular chassis configuration example................................................... 27939 STP example.................................................................................................. 29540 PortChannel example .................................................................................. 29741 vPC example ................................................................................................. 30042 Switch vPC/straight-through topology example ................................... 30743 Switch vPC/cross-connect topology example......................................... 30844 Host vPC direct-connect topology example............................................. 30945 Switch vPC/straight-through Nexus 5000 to Nexus 2000

configuration................................................................................................... 31146 Supported FCoE topology example .......................................................... 31347 Non-supported topology for FCoE example ........................................... 31448 Switch vPC/straight-through Nexus 5000 to Nexus 4000

configuration................................................................................................... 31549 Switch vPC/cross-connect Nexus 5000 to Nexus 2000 configuration.... 31850 Switch vPC/cross-connect Nexus 5000 to Nexus 4000 configuration.... 32051 Supported FCoE topology .......................................................................... 32252 Straight-through topology example .......................................................... 32653 Existing topology example ......................................................................... 32854 Target topology example ............................................................................ 32955 Ethernet connectivity with vPC and FEX Straight-Through topology

example............................................................................................................ 33656 SAN connectivity with vPC and FEX Straight-Through topology

example............................................................................................................ 34857 Cross-connect topology example............................................................... 35858 Existing topology example ......................................................................... 36159 Target topology example ............................................................................ 36260 Direct-connect topology example .............................................................. 37061 Existing topology example ......................................................................... 37262 iSCSI target topology................................................................................... 37463 FCoE vPC direct connect target topology ................................................ 37864 Unisphere navigation and user interface.................................................. 39365 Unisphere navigation and user interface.................................................. 41066 Cisco Unified Computer System (UCS).................................................... 42167 Brocade Converged 10 GbE Switch Module (Brocade 8470) ................. 423


Figures

68 IBM H series chassis, rear view .................................................................. 42469 Target topology example............................................................................. 42770 Connection between Fabric Extender and Fabric Interconnect ............. 42871 Existing topology.......................................................................................... 43672 Target topology............................................................................................. 43773 Change the mode.......................................................................................... 43874 Enable port channeling and Trunking....................................................... 43975 Create VSAN ................................................................................................. 44076 Disable default zoning ................................................................................. 44177 Create Storage VSAN................................................................................... 44278 SAN tab .......................................................................................................... 44379 Add Ports screen........................................................................................... 44480 Enable Port-Channel .................................................................................... 44581 Assign VSAN to vHBAs .............................................................................. 44682 Assign VSAN to storage FC and FCoE interfaces.................................... 44783 Existing topology.......................................................................................... 45584 Target topology............................................................................................. 45685 Set chassis discovery policy ........................................................................ 45786 Change connectivity policy......................................................................... 45887 Create VSAN in SAN cloud ........................................................................ 45988 Create VSAN ................................................................................................. 45989 Assign VSAN to vHBA................................................................................ 46090 Assign uplink FC interface.......................................................................... 46191 Enable port channel and trunking ............................................................. 46292 Create FC port channels .............................................................................. 46393 Add Ports ....................................................................................................... 46394 Assign VSAN to port channel..................................................................... 46495 CLARiiON Connectivity Status window.................................................. 47196 Register Initiator Record dialog box .......................................................... 47197 EMC Symmetrix Manager Console, Tasks ............................................... 47398 Masking Wizard Summary window ......................................................... 47599 Existing topology example.......................................................................... 478100 Target topology example............................................................................. 479101 Existing case topology example ................................................................. 509102 Target topology............................................................................................. 520103 Target topology............................................................................................. 546104 Configuring UCS 10G iSCSI using VNX-2 arrays example.................... 575105 Topology example ........................................................................................ 600106 Existing topology.......................................................................................... 604107 Target topology, example 1......................................................................... 605108 Target topology, example 2......................................................................... 607109 AMM I/O Module Power/Restart window ............................................ 609


Figures

110 I/O Module Configuration window......................................................... 610111 Nexus 4000 topology example ................................................................... 620112 Existing topology ......................................................................................... 624113 Target topology ............................................................................................ 625114 AMM I/O Module Admin/Power/Restart window............................. 629115 AMM I/O Module Configuration window ............................................. 630116 FlexFabric interconnect module................................................................. 655117 FlexFabric adapters ...................................................................................... 657118 FlexNIC.......................................................................................................... 658119 Communication between FlexFabric technology in HP BladeServer

enclosure and the data center....................................................................... 660120 Mapping mode ............................................................................................. 662121 Mapping Mode used for Shared Uplinks example ................................. 663122 Tunneling mode ........................................................................................... 664123 Tunneling Mode used for Dedicated Uplinks example ......................... 665124 High-level target topology example.......................................................... 671125 Specific target topology example............................................................... 672126 Edit SAN Fabric screen for defining B300_b5X1X2 fabric ..................... 674127 Edit SAN Fabric screen for defining B300_b6X1X2 fabric ..................... 675128 Edit SAN Fabric screen for defining C9148_b5X3X4 fabric ................... 676129 Edit SAN Fabric screen for defining C9148_b6X3X4 fabric ................... 677130 All SAN fabrics and link status.................................................................. 678131 Defining the Server Profile for Blade 2 ..................................................... 679132 Configuring the Server Profile for Blade 2 screen................................... 680133 Defining the Server Profile for Blade 15 screen ....................................... 681134 Configuring the Server Profile for Blade 15 screen................................. 681135 Status of internal/external ports on FlexFabric module, Bay 5 ............ 682136 Status of internal/external ports on FlexFabric module, Bay 6 ............ 683137 Status between FlexFabric adapter ports in Server 2 and the external Bro-

cade SAN switch 684138 Status between FlexFabric adapter ports in Server 15 and the

external Cisco SAN switch............................................................................ 685139 Brocade Name Server FlexHBA port logins, Server 2 ............................ 686140 Cisco Name Server FlexHBA port logins, Server 15 ............................... 686141 Juniper QXF3500 overview......................................................................... 692142 Existing topology ......................................................................................... 697143 Target topology ............................................................................................ 698144 Classic Ethernet and corresponding VCS Fabric architecture............... 719145 Source topology example............................................................................ 727146 Target topology ............................................................................................ 728147 Detailed topology......................................................................................... 730148 Attaching a front rackmount bracket to a Nexus 5020 switch .............. 745


Figures

149 Installing the slider rails .............................................................................. 746150 Sliding the chassis into the rack.................................................................. 747151 Attaching the switch to the rack................................................................. 748152 LEDs on the port side of MP-8000-B.......................................................... 754153 LEDs on the non-port side of MP-8000-B.................................................. 754


Figures


Title Page

Tables

1 Acronyms .........................................................................................................292 IP address and Domain ID assignment ......................................................2133 Host 1 information ........................................................................................2134 Storage 1 information ...................................................................................2135 Port groups .....................................................................................................2166 Port groups .....................................................................................................2247 LACP compatibility ......................................................................................2998 Parameters ......................................................................................................3049 Default values for keepalive messages ......................................................30610 Supported protocols ......................................................................................32511 Supported protocols ......................................................................................35712 Supported protocols ......................................................................................36913 Supported switch modes and port types ...................................................42214 Brocade and IBM Converged Module part numbers ...............................42315 Mapping and Tunneling mode comparison .............................................66616 VCS terminology ...........................................................................................72017 Nexus 5020 switch rackmount kit ...............................................................74418 Power supply specifications ........................................................................75019 LED patterns ..................................................................................................75520 LEDs on non-port side of switch .................................................................756


Tables


Preface

The former EMC Engineering Fibre Channel over Ethernet (FCoE) DataCenter Bridging (DCB) Concepts and Protocol TechBook has beendivided into two TechBooks. This TechBook focuses on case studies andprovides a brief overview of Fibre Channel over Ethernet (FCoE), thesupported configurations and features, and case studies to show how toincorporate EMC Connectrix B switches, Cisco Nexus switches, and BladeServers utilizing FCoE into an existing data center.

The current Fibre Channel over Ethernet (FCoE), Data Center Bridging(DCB) Concepts and Protocols TechBook provides basic information tobetter understand the various aspects and protocols involved with a typicalEthernet environment. FCoE connectivity and storage in an FCoEenvironment are discussed. Information on RecoverPoint and Celerra MPFSas solutions in an FCoE environment is included. Basic troubleshootingtechniques are also provided. This TechBook can be found athttp://elabnavigator.EMC.com.

E-Lab would like to thank all the contributors to this document, includingEMC engineers, EMC field personnel, and partners. Your contributions areinvaluable.

As part of an effort to improve and enhance the performance and capabilitiesof its product lines, EMC periodically releases revisions of its hardware andsoftware. Therefore, some functions described in this document may not besupported by all versions of the software or hardware currently in use. Forthe most up-to-date information on product features, refer to your productrelease notes. If a product does not function properly or does not function asdescribed in this document, please contact your EMC representative.


https://elabnavigator.emc.com

18

Preface

Note: This document was accurate at publication time. New versions of thisdocument might be released on EMC Online Support athttps://support.emc.com. Check to ensure that you are using the latestversion of this document.

Audience This TechBook is intended for EMC field personnel, includingtechnology consultants, and for the storage architect, administrator,and operator involved in acquiring, managing, operating, ordesigning a networked storage environment that contains EMC andhost devices.

EMC Support Matrixand E-Lab

InteroperabilityNavigator

For the most up-to-date information, always consult the EMC SupportMatrix (ESM), available through E-Lab Interoperability Navigator athttp://elabnavigator.EMC.com. Included in the ELN are:

◆ The EMC Support Matrix, a complete guide to interoperable, andsupportable, configurations.

◆ Host connectivity guides for complete, authoritative informationon how to configure hosts effectively for various storageenvironments.

Consult the Internet Protocol pdf for EMC's policies and requirementsfor the EMC Support Matrix.

Relateddocumentation

◆ The following documents, including this one, are availablethrough the E-Lab Interoperability Navigator athttp://elabnavigator.EMC.com.

These documents are also available at the following location:

http://www.emc.com/products/interoperability/topology-resource-center.htm

• Backup and Recovery in a SAN TechBook• Building Secure SANs TechBook• Extended Distance Technologies TechBook• Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB)

Concepts and Protocols TechBook• Fibre Channel SAN Topologies TechBook• iSCSI SAN Topologies TechBook• Networked Storage Concepts and Protocols TechBook• Networking for Storage Virtualization and RecoverPoint TechBook• WAN Optimization Controller Technologies TechBook• EMC Connectrix SAN Products Data Reference Manual• Legacy SAN Technologies Reference Manual• Non-EMC SAN Products Data Reference Manual

Fibre Channel over Ethernet (FCoE) Case Studies TechBook

http://elabnavigator.EMC.com






Preface

◆ EMC Support Matrix, available through E-Lab InteroperabilityNavigator at http://elabnavigator.EMC.com

◆ RSA security solutions documentation, which can be found athttp://RSA.com > Content Library

All of the following documentation and release notes can be found atEMC Online Support at https://support.emc.com.

Hardware documents and release notes include those on:

◆ Connectrix B series◆ Connectrix M series◆ Connectrix MDS (release notes only)◆ Symmetrix◆ VMAX◆ VNX series◆ CLARiiON◆ Celerra

Software documents include those on:

◆ EMC Ionix ControlCenter◆ RecoverPoint◆ TimeFinder◆ PowerPath

The following E-Lab documentation is also available:

◆ Host Connectivity Guides◆ HBA Guides

For Cisco, Brocade, and HP documentation, refer to the vendor’swebsite.

◆ http://cisco.com◆ http://brocade.com◆ http://hp.com




http://RSA.com

20

Preface

Authors of thisTechBook

This TechBook was authored by Mark Lippitt, Erik Smith, Erik Paine,and Shreedhan Nikam, with contributions from the following EMCemployees: Kieran Desmond, Ger Halligan, and Ron Stern, alongwith other EMC engineers, EMC field personnel, and partners.

Mark Lippitt is a Technical Director in EMC E-Lab with over 30 yearsexperience in the storage industry, including Engineering andMarketing roles at Data General, Tandem Computers, and EMC.Mark initiated and led the Stampede project in 1997, which becameEMC's first Connectrix offering. Mark is an active T11 participant, acommittee within the InterNational Committee for InformationTechnology Standards, responsible for Fibre Channel Interfaces.

Erik Smith is a Consulting Technologist for the Connectrix businessunit within EMC Engineering. Over the past 13 years, Erik has heldvarious technical roles in both EMC Engineering and TechnicalSupport. Erik has authored and coauthored several EMC TechBooks.Erik is also a member of T11.

Erik Paine is a Principal Systems Integration Engineer and has beenwith EMC for over 12 years. Erik transferred to E-Lab with a strongnetworking background spanning over 20 years, including timespent at BBN Inc., Tufts University, and numerous engineering roleswithin EMC. Erik is using his networking and Ethernet knowledge tohelp qualify and integrate the emerging storage technologies utilizingEthernet as a medium.

Benjamin Heydary is a Principal Systems Integration Engineer withover 30 years of experience in the storage industry. Ben has been withEMC for over 16 years, 14 with EMC E-Lab. Ben has worked on manydifferent storage technologies integrations and has authorednumerous White Papers, Technical Notes, and TechBooks.


Preface

Conventions used inthis document

EMC uses the following conventions for special notices:

IMPORTANT

An important notice contains information essential to software orhardware operation.

Note: A note presents information that is important, but not hazard-related.

Typographical conventionsEMC uses the following type style conventions in this document.

Normal Used in running (nonprocedural) text for:• Names of interface elements, such as names of

windows, dialog boxes, buttons, fields, andmenus

• Names of resources, attributes, pools, Booleanexpressions, buttons, DQL statements, keywords,clauses, environment variables, functions, andutilities

• URLs, pathnames, filenames, directory names,computer names, links, groups, service keys, filesystems, and notifications

Bold Used in running (nonprocedural) text for names ofcommands, daemons, options, programs, processes,services, applications, utilities, kernels, notifications,system calls, and man pages

Used in procedures for:• Names of interface elements, such as names of

windows, dialog boxes, buttons, fields, andmenus

• What the user specifically selects, clicks, presses,or types

Italic Used in all text (including procedures) for:• Full titles of publications referenced in text• Emphasis, for example, a new term• Variables

Courier Used for:• System output, such as an error message or script• URLs, complete paths, filenames, prompts, and

syntax when shown outside of running text

Courier bold Used for specific user input, such as commands


22

Preface

Where to get help EMC support, product, and licensing information can be obtained onthe EMC Online Support site as described next.

Note: To open a service request through the EMC Online Support site, youmust have a valid support agreement. Contact your EMC sales representativefor details about obtaining a valid support agreement or to answer anyquestions about your account.

Product informationFor documentation, release notes, software updates, or forinformation about EMC products, licensing, and service, go to theEMC Online Support site (registration required) at:

https://support.EMC.com

Technical supportEMC offers a variety of support options.

Support by Product — EMC offers consolidated, product-specificinformation on the Web at:

https://support.EMC.com/products

The Support by Product web pages offer quick links toDocumentation, White Papers, Advisories (such as frequently usedKnowledgebase articles), and Downloads, as well as more dynamiccontent, such as presentations, discussion, relevant CustomerSupport Forum entries, and a link to EMC Live Chat.

EMC Live Chat — Open a Chat or instant message session with anEMC Support Engineer.

Courier italic Used in procedures for:• Variables on the command line• User input variables

< > Angle brackets enclose parameter or variable valuessupplied by the user

[ ] Square brackets enclose optional values

| Vertical bar indicates alternate selections — the barmeans “or”

{ } Braces enclose content that the user must specify,such as x or y or z

... Ellipses indicate nonessential information omittedfrom the example


https://support.EMC.com

http://support.EMC.com

https://support.EMC.com/products

Preface

eLicensing supportTo activate your entitlements and obtain your Symmetrix license files,visit the Service Center on https://support.EMC.com, as directed onyour License Authorization Code (LAC) letter e-mailed to you.

For help with missing or incorrect entitlements after activation (thatis, expected functionality remains unavailable because it is notlicensed), contact your EMC Account Representative or AuthorizedReseller.

For help with any errors applying license files through SolutionsEnabler, contact the EMC Customer Support Center.

If you are missing a LAC letter, or require further instructions onactivating your licenses through the Online Support site, contactEMC's worldwide Licensing team at [email protected] or call:

◆ North America, Latin America, APJK, Australia, New Zealand:SVC4EMC (800-782-4362) and follow the voice prompts.

◆ EMEA: +353 (0) 21 4879862 and follow the voice prompts.

We'd like to hear from you!Your suggestions will help us continue to improve the accuracy,organization, and overall quality of the user publications. Send youropinions of this document to:

[email protected]

Your feedback on our TechBooks is important to us! We want ourbooks to be as helpful and relevant as possible. Send us yourcomments, opinions, and thoughts on this or any other TechBook to:

[email protected]




24

Preface


1

This chapter provides an introduction to Fibre Channel over Ethernet(FCoE) and includes the following information:

◆ Introduction ........................................................................................ 26◆ Benefits ................................................................................................ 28◆ Terminology........................................................................................ 29◆ Management tools.............................................................................. 31◆ Cable management recommendations............................................ 32◆ Protocols .............................................................................................. 33◆ Supported connectivity options....................................................... 34

Note: More detailed Fibre Channel over Ethernet information can be found inthe Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts andProtocols TechBook at http://elabnavigator.EMC.com, under the TopologyResource Center tab. Information in this TechBook includes FCoE andEthernet basics, EMC storage, RecoverPoint and Celerra MPFS as solutions inan FCoE environment, and troubleshooting basic FCoE and CEE problems.

Introduction to FibreChannel over Ethernet

Introduction to Fibre Channel over Ethernet 25


26

Introduction to Fibre Channel over Ethernet

IntroductionI/O consolidation has been long sought by the IT industry to unifythe multiple transport protocols in the data center. This sectionprovides a basic introduction to Fibre Channel over Ethernet (FCoE),which is an approach to I/O consolidation that has been defined bythe FC-BB-5 T11 work group.

I/O consolidation, simply defined, is the ability to carry differenttypes of traffic, having different traffic characteristics and handlingrequirements, over the same physical media. I/O consolidation’smost difficult challenge is to satisfy the requirements of differenttraffic classes within a single network. Since Fibre Channel is thedominant storage protocol in the data center, any viable I/Oconsolidation solution for storage must allow for the FC model to beseamlessly integrated. FCoE meets this requirement in part byencapsulating each Fibre Channel frame inside an Ethernet frame.

The goal of FCoE is to provide I/O consolidation over Ethernet,allowing Fibre Channel and Ethernet networks to share a single,integrated infrastructure, thereby reducing network complexities inthe data center. An example is shown in Figure 1 on page 27.

FCoE consolidates both SANs and Ethernet traffic onto oneConverged Network Adapter (CNA), eliminating the need for usingseparate Host Bus Adapters (HBAs) and Network Interface Cards(NICs).



.

Figure 1 Typical topology versus FCoE example using Cisco Nexus 5000

Note: More detailed Fibre Channel over Ethernet information can be found inthe Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts andProtocols TechBook at http://elabnavigator.EMC.com, under the TopologyResource Center tab.

LAN SAN A

Today

SAN B LAN

Data Centre Ethernet and FCoE Ethernet

SAN A

I/O Consolidation with FCoE

SAN B

FC GEN-001008

Introduction 27


28


BenefitsThe Fibre Channel portion of FCoE appears as normal Fibre Channelto a host or a switch, and therefore to a user. It is based completely onthe FC model, which makes it easy to understand, manage, andtroubleshoot. A major value is that FCoE uses Ethernet hardware todeliver an enterprise storage solution, while also using the existingFC management infrastructure.

The benefits of FCoE include:

◆ Becomes part of the Fibre Channel architecture, allowing for:

• Seamless integration with existing FC SANs

• Uses existing FC SAN admin tools and workflows

◆ Requires no gateway

• Since the FC frame is untouched, the operation is completelystateless

◆ Provides the following current functions and services, allowingfor a smooth transition:

• Zoning• dNS (distributed Name Server)• RSCN (Registered State Change Notification)• FSPF (Fibre Channel Shortest Path First)• Management tools• Storage and server virtualization

Further benefits include:

◆ Fewer cables, simplifying cable management

◆ Fewer adapters and switch ports, saving in power, equipment,and cooling costs



TerminologyTable 1 provides commonly used acronyms.

Table 1 Acronyms (page 1 of 2)

Acronym Definition

ACL access control list

CEE Converged Enhanced Ethernet (Deprecated, see “DCB”)

CNA Converged Network Adapter

CRC Cyclical Redundancy Check

DA Destination MAC Address

DCB Data Center Bridge

DCBX Data Center Bridging Capability eXchange Protocol

DCFM Data Center Fabric Manager

ETHv2 Ethernet Version 2

EOF End of Frame

ETS Enhanced Transmission Selection

FCF FCoE Forwarder

FCoE Fibre Channel over Ethernet

FPMA Fabric Provided MAC Address

IP Internet Protocol

LACP Link Aggregation Control Protocol

LAG Link Aggregation Group

LAN Local Area Network

LLDP Link Layer Discovery Protocol

MAC Media Access Control

MSTP Multiple Spanning Tree Protocol

NPV N_Port virtualization

Terminology 29

30


NPIV N_Port ID virtualization

NIC Network Interface Card

PFC Priority Flow Control

QoS Quality of Service

SA Source MAC Address

SAN Storage Area Network

SPMA Server Provided MAC Address

STP Spanning Tree Protocol

RSTP Rapid Spanning Tree Protocol

VE Virtual E port

VF Virtual Fabric port

VFC Virtual Fibre Channel

VLAN Virtual LAN (ID)

WAN Wide Area Network

Table 1 Acronyms (page 2 of 2)

Acronym Definition



Management toolsThe management tools used to manage FCoE and Fibre Channelenvironments are similar.

CNA management tools include:

◆ Emulex — HBAnywhere/OneCommand Manager

◆ QLogic — SANSurfer

◆ Brocade — HCM

Switch management tools include:

◆ Brocade

• Fabric OS CLI

– For configuration of Fibre Channel features• CMSH (CEE management shell) CLI

– For configuration of Converged Enhanced Ethernet (CEE)features

• CMCNE - Connectrix Manager Converged Network Edition

◆ Cisco

• Fabric Manager

– For configuration of Fibre Channel features• NX-OS CLI

– For configuration of CEE features• Data Center Network Manager (DCNM)

Management tools 31

32


Cable management recommendationsConsider the following recommendations for cable management.

The minimum bend radius for a 50 micron cable is 2 inches under fulltensile load and 1.2 inches with no tensile load.

Cables can be organized and managed in a variety of ways, forexample, using cable channels on the sides of the cabinet or patchpanels to minimize cable management. Following is a list ofrecommendations:

Note: You should not use tie wraps with optical cables because they are easilyovertightened and can damage the optic fibers.

◆ Plan for rack space required for cable management beforeinstalling the switch.

◆ Leave at least 1 m (3.28 ft) of slack for each port cable. Thisprovides room to remove and replace the switch, allows forinadvertent movement of the rack, and helps prevent the cablesfrom being bent to less than the minimum bend radius.

◆ If you are using Brocade ISL Trunking, consider grouping cablesby trunking groups. The cables used in trunking groups mustmeet specific requirements, as described in the Fabric OSAdministrator’s Guide.

◆ For easier maintenance, label the fiber optic cables and record thedevices to which they are connected.

◆ Keep LEDs visible by routing port cables and other cables awayfrom the LEDs.

◆ Use hook and loop style straps to secure and organize fiber opticcables.



ProtocolsFCoE is two different protocols:

◆ FCoE — Data plane protocol

This protocol is data intensive and requires lossless Ethernet. It istypically implemented in hardware and is used to carry most ofthe FC frames and all the SCSI traffic.

◆ FIP (FCoE Initialization Protocol) — Control plane protocol

This is not data intensive. It is typically implemented in software,and used to discover FCoE capable devices connected to anEthernet network and to negotiate capabilities.

For more information on FIP, refer to the “FCoE InitializationProtocol (FIP)” section, "Introduction to Fibre Channel over Ethernet"chapter, in the Fibre Channel over Ethernet (FCoE) Data Center Bridging(DCB) Concepts and Protocols TechBook athttp://elabnavigator.EMC.com, under the Topology ResourceCenter tab.

The following list contains a few of the technologies and protocolsrequired to enable I/O consolidation:

◆ Converged Network Adapters

◆ Fibre Channel Forwarders

◆ Priority Flow Control and PAUSE

◆ Data Center Bridging (DCB)

◆ Data Center Bridging Capability eXchange Protocol (DCBX)

Each of these are discussed further in the "Introduction to FibreChannel over Ethernet" chapter in the Fibre Channel over Ethernet(FCoE) Data Center Bridging (DCB) Concepts and ProtocolsTechBook athttp://elabnavigator.EMC.com, under the Topology ResourceCenter tab. This document also provides information onencapsulation and connectivity options.

Protocols 33



34


Supported connectivity optionsThis section provides the current configuration rules as well asgraphical illustrations of the connectivity options currentlysupported by EMC:

◆ “Configuration rules ” on page 34

◆ “Connectivity options” on page 36

Configuration rulesThe following configuration rules were used to create the topologydiagrams used in this TechBook. If there is a discrepancy between thediagrams and the configuration rules, the configuration rules takeprecedence.

Note: These rules are numbered for referencing convenience only.

Cisco Nexus and MDSswitches

1. The Nexus 2232 requires a connection to a Nexus 5000 in order tofunction.

2. The Nexus 2232 does not support native Fibre Channel ports.

3. The Nexus 4001i requires a connection to a Nexus 5000 in order tosupport FCoE.

4. The Nexus 4001i does not support native Fibre Channel ports.

5. The Nexus 4001i does not support external VN_Port connections.

6. The Fibre Channel ports on the Nexus 5000 cannot be used forend device connectivity while the switch is in NPV mode.

7. Storage ports cannot be connected to the Nexus 5000 while it isrunning in NPV mode.

8. The Nexus 7000 does not support native Fibre Channel ports.

9. As of November 1, 2011, FCoE is supported on the MDS 95xx andNexus 7000 platforms.

10. All host connectivity to the UCS 6100 is via FCoE.

11. The Fibre Channel ports on the UCS 6100 cannot be used for enddevice connectivity while the switch is in NPV mode.



12. The UCS 6100 does not currently provide an FC zoning interface.As a result, when the 6100 is running in FC-SW mode, it must beconnected to another Cisco switch that is also running in switchmode.

13. UCS does not currently support VE_Ports.

14. The Nexus 2248PQ requires an upstream connection to a parentswitch such as the Nexus 6000 series.

15. The Nexus 6000 series does not support native Fibre Channelports.

Brocade 8000 andDCX switches

16. The Brocade 8000 and DCX do not support VE_Ports.

17. The Fibre Channel ports on the Brocade 8000 cannot be used forend device connectivity while the switch is in NPV mode.

Brocade/Ciscointeroperability

18. FC-SW interoperability between Brocade and Cisco switches isnot supported. When attaching a Brocade switch to a Cisco FCFand vice versa, either NPV mode must be used on the Ciscoswitch or AG mode must be used on the Brocade switch.

19. FCoE NPV uplinks between Cisco and Brocade switches are notsupported.

20. Cascading switches that are running in NPV/AG mode is notsupported.

Juniper QFX3500/3600 21. Juniper QFX3500 and QFX3600 switches that are running inGateway mode must be connected to either a Brocade or Ciscoswitch that supports the NPIV protocol.

22. Juniper QFX3500 and QFX3600 switches that are running inTransit switch mode must be connected to either a QFX3500 orQFX3600 that is running in Gateway mode or to an FCoE-capableCisco switch.

23. From a supported topologies perspective, a Juniper QFabric isequivalent to a Juniper QFX3500 or QFX3600.

24. EMC Storage may not be directly connected to Juniper QFX3500or QFX3600 switches running in either Gateway or Transit switchmodes.

Supported connectivity options 35

36


Connectivity optionsThis section contains information on the connectivity options for thefollowing connectivity components:

◆ “Nexus 2000” on page 36

◆ “Nexus 2248PQ” on page 37

◆ “Nexus 4001i” on page 37



◆ “Nexus 61x0” on page 40


◆ “MDS 9000” on page 42

◆ “Brocade 8000/DCX” on page 43

◆ “Juniper QFX3500/3600” on page 44

Nexus 2000 Figure 2 shows configurations specific to the Nexus 2000.

Figure 2 Nexus 2000 supported connectivity options

Host StorageFCoE

Host StorageFCoE FCoE

FCNexus2232 FCoE

Nexus5000 (NPV)FCoE

Nexus2232

Host StorageFCoE

Nexus5000 (NPV)FCoE

Nexus2232

Host StorageFCoE

Nexus5000 (NPV)FCoE

Nexus2232


FCNexus5000 (NPV)FCoE

Nexus2232 FCoE

FCoE FCoE

FCNexus 5000

(FC-SW)

FC FCMDS (FC-SW)

FCFCoE

FCBrocade8000 /DCX

FC FCBrocadeFOS/EOS

Nexus5000

(FC-SW)

EMC

EMC

EMC

EMC

EMC



Nexus 2248PQ Figure 3 shows configurations specific to the Nexus 2248PQ.

Figure 3 Nexus 2248PQ supported connectivity options

Nexus 4001i Figure 4 shows configurations specific to the Nexus 4001i.

Figure 4 Nexus 4001i supported connectivity options

HostFCoE FCoEFCoE

Nexus2248PQ

Nexus6004

Storage

EMC

HostFCoE FCoEFCoE

Nexus2248PQ

Nexus6001

Storage

EMC

GEN-002129

Host StorageFCoE


FCNexus4001i FCoE

Nexus5000 (NPV)FCoE

Nexus4001i

Host StorageFCoE

Nexus5000 (NPV)FCoE

Nexus4001i

Host StorageFCoE

Nexus5000 (NPV)FCoE

Nexus4001i


FCoE

FCNexus5000 (NPV)FCoE

Nexus4001i FCoE

FCoE

FCNexus 5000

(FC-SW)

FC FCMDS (FC-SW)

FCFCoE

FCBrocade8000 /DCX

FC FCBrocadeFOS/EOS

Nexus5000

(FC-SW)

EMC

EMC

EMC

EMC

EMC


38








HostFCoE FCoE

Nexus6000

Storage

EMC

HostFCoE FCoEFCoE

Nexus5000

Nexus6000

Storage

EMC

HostFCoE FCoEFCoE

Nexus6000

Nexus6000

Storage

EMC

HostFCoE FCoEFCoE

Nexus6000

Nexus5000

Storage

EMC

GEN-0


40


Nexus 61x0 Figure 7 shows configurations specific to the Nexus 61x0.

Figure 7 Nexus 61x0 supported connectivity options

Host Storage

Nexus61x0(NPV)FCoE

FC

Host StorageFCoE

Host StorageFCoE

Host Storage

Nexus61x0

(FC-SW)

Nexus61x0

(FC-SW)

Nexus61x0 (NPV)FCoE

FCFCoE

FCNexus 5000

(FC-SW)

FC FCMDS (FC-SW)

FCFCoE

FCBrocade8000 /DCX

Host Storage

Nexus61x0 (NPV)FCoE

FC FCBrocadeFOS/EOS

FCoEFCNexus

5000 (FC-SW)

Host StorageFCoE

Nexus61x0 (NPV)

FC FCMDS (FC-SW)

Host StorageFCoE

Nexus5000

(FC-SW)

FCFCoE

FCNexus 61x0

(FC-SW)

Host StorageFCoE

Nexus5000

(FC-SW)

FCFCoE

Nexus 61x0 (NPV)

Host Storage

Nexus61x0(NPV)FCoE FCoE

This configuration requires that that

61x0 be connected to another switch

running in FC-SW mode.

Host StorageFCoE

Nexus61x0

(FC-SW)

FCFCoE

Nexus 5000 (NPV)

EMC

EMC

EMC

EMC

EMC

EMC

EMC

EMC

EMC

EMC





Host Storage

Nexus 7000

Host Storage

FCoE FCoE

FCFCoE FCoE FCoE

Host Storage

FCFCoE FCoE FCoE

Host Storage

FCFCoE

MDSFCoE

Host StorageFCoE

FCoE

FCoENexus 7000 FCoE

EMC

EMC

EMC

EMC

EMC

Nexus 7000

Nexus 5000

Nexus 7000

Nexus 7000

Nexus 7000

(FC-SW)

Nexus 5000(NPV)


42


MDS 9000 Figure 9 shows configurations specific to the MDS 9000 series.

Figure 9 MDS 9000 supported connectivity options

Host Storage

MDSFCFC

Host Storage

MDSNexus 5000

(FC-SW)

FCFC

Host Storage

MDSFCFC

Host Storage

MDS MDSFC

Host Storage

MDSNexus5000(NPV)

FCFCoE

FC

Nexus 61x0

(FC-SW)

FC

FC FC

FC

Host Storage

MDSNexus5000

(FC-SW)

FCFCoE

FC FC

Host Storage

MDSFCFC Nexus

61x0 (NPV) FCoE

FCoE

FCoE

EMC

EMC

EMC

EMC

EMC

EMC

EMC

FCoE

FCoE FCoE

FCoE FCoE

FCoE FCoE

FCoE FCoE

FCoE

FCoE

FCoE

Host Storage

MDS Nexus7000

FCFCoE

EMCFCoE FCoE

FCoE

FCoE



Brocade 8000/DCX Figure 10 shows configurations specific to the Brocade 8000/DCX.

Figure 10 Brocade 8000/DCX supported connectivity options


44


Juniper QFX3500/3600 Figure 11 shows configurations specific to the Juniper QFX3500/3600.

Figure 11 Juniper QFX3500/3600 supported connectivity options


2

This chapter provides basic information, supported features,topologies, and detailed setup steps to create the configurationshown in Figure 15, “Target topology,” on page 56. Information onthe ED-DCX-B topology, Figure 17 on page 105, is also included.

◆ MP-8000B............................................................................................. 46◆ MP-8000B supported features and topologies ............................... 49◆ MP-8000B direct-connect topology.................................................. 53◆ MP-8000B firmware upgrade procedure ........................................ 90◆ Troubleshooting the MP-8000B ........................................................ 93◆ ED-DCX-B ......................................................................................... 104

Note: More detailed Fibre Channel over Ethernet information can be found inthe Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts andProtocols TechBook at http://elabnavigator.EMC.com, under the TopologyResource Center tab. Information in this TechBook includes FCoE andEthernet basics, EMC storage, RecoverPoint and Celerra MPFS as solutions inan FCoE environement, andtroubleshooting basic FCoE and CEE problems.

EMC Connectrix BSetup Examples

EMC Connectrix B Setup Examples 45


46

EMC Connectrix B Setup Examples

MP-8000BThe MP-8000B is a 24-port 10 GbE line-rate, low latency, losslessConverged Enhanced Ethernet (CEE) and an 8-port auto-sensing 1, 2,4, or 8 Gb/s Fibre Channel switch that delivers the latest BrocadeASIC technology and architecture for Fibre Channel Storage AreaNetworks (SANs).

The MP-8000B enables the Fibre Channel over Ethernet (FCoE)protocol and is a high performance 8 Gb/s Fibre Channel switchdesigned for the needs of enterprise environments.

The MP-8000B supplies Reliability, Availability, and Serviceability(RAS) performance and scalability requirements of an enterpriseswitch along with the interoperability and ease-of-use advantagesfound only in the EMC® Connectrix® product family.

This section contains the following information:

◆ “MP-8000B overview” on page 46

◆ “Management” on page 48

◆ “POST and boot specifications” on page 48

MP-8000B overviewThe MP-8000B internal architecture consists of a Fibre Channel switchand an Ethernet switch, with an FCoE Gateway in between them.This means that there are separate processors, ASICs, andconfiguration files for the Fibre Channel and Ethernet portions of theswitch. Each time you access the switch, whether it is through a serialcable or through a telnet/SSH session you will be entering the switchthrough the Fibre Channel portion of the switch.

As a result of the partitions within the switch, there are two separateoperating systems that need to be configured: the Fibre Channel sideof the switch (FOS) and the Ethernet side of the switch (CMSH), eachwith its own type of operating system, as noted below:

◆ FOS – (Fabric Operating System) for Fibre Channel switch

◆ CMSH – (CEE Management Shell) for Ethernet switch



Figure 12 is a high level diagram showing the switch layout. Noticethe designation of the FC switch and the enhanced Ethernet switchwith the FCoE gateway in the middle.

Figure 12 Switch layout example

The following configuration steps will show how to configure bothsides of the switch as well as the initial configuration to get the switchready for remote access.

IMPORTANT

Follow these steps in order to ensure a smooth installation. Afterthe initial configuration is completed and remote access is tested,all the physical connections will need to be connected and verified.

FC SwitchCondor Chipset

FCoE GatewayZeus Chipset

Ethernet SwitchAnvil Chipset

Ethernet Interfaces0 - 23

Fibre Channel Interfaces0 - 7

CEEPort

0

CEEPort23

SYM-002069

FCPort

0

FCPort

7

MP-8000B 47

48


ManagementThe MP-8000B provides two different operating environments, FibreChannel and Ethernet. Both need to be configured. The operatingenvironments and how to configure each are:

◆ Command Line Interface (CLI)

• FOS – (Fabric Operating System) for Fibre Channel switch

• CMSH – (CEE Management Shell) for Ethernet switch

◆ HTTP - (Web Tools)

• FOS

• CMSH

– Minimal configurations

POST and boot specificationsWhen the switch is turned on or rebooted, the switch performs aPOST (Power-on self test). Total boot time with POST can be severalminutes. POST can be omitted after subsequent reboots by using thefastboot command or entering the diagDisablePost command topersistently disable POST.

For more information about these commands, refer to the BrocadeConverged Enhanced Ethernet Command Reference manual.

POST The success or failure results of the diagnostic tests that run duringPOST can be monitored through LED activity, the error log, or thecommand line interface.

POST includes the following tasks:

◆ Conducts preliminary POST diagnostics.

◆ Initializes the operating system.

◆ Initializes hardware.

◆ Runs diagnostic tests on several functions, including circuitry,port functionality, memory, statistics counters, and serialization.

Refer to “Interpreting POST results” on page 751 for moreinformation.



MP-8000B supported features and topologiesThe supported features and topologies have been broken down intothe following four areas, shown in Figure 13, and discussed further inthis section.

◆ “FCoE interface” on page 50

◆ “Ethernet uplink interface” on page 50

◆ “FC interface” on page 52

◆ “Ethernet end devices ” on page 52

Figure 13 is for clarification purposes only. The preferred method ofincorporating the MP-8000B into an existing topology can be found inthe target topology.

Figure 13 Topology example

MP-8000B supported features and topologies 49

50


FCoE interfaceThe following interfaces are supported:

◆ CNAs directly connected via physical media to the MP-8000B

Note: Connecting the CNA to the MP-8000B via a stand-alone Top ofRack (TOR) or End Of Row (EOR) intermediate 10G Ethernet Bridge(either DCB capable or not) is not supported unless stated otherwisebelow.

◆ Physical connectivity

• EMC direct-attached native FCoE storage devices

• Fabric A /Fabric B connectivity is supported

• An example of Fabric A /Fabric B connectivity is twoMP-8000Bs, each connected to one of the two sides of amirrored fabric

• SFP+ optical (short reach only)

• SFP+ cu (Twinax - Molex)

• 1 m, 3 m, and 5 m are available

◆ Hardware-based FCoE CNAs from Brocade, Emulex, QLogic

Note: Refer to the EMC Support Matrix for the most up-to-date supportinformation.

◆ Software-based FCoE CNAs from Intel and Broadcom

◆ Native FCoE Storage:

• EMC VNX®

• EMC CLARiiON® CX4

• EMC Symmetrix® VMAX®

Ethernet uplink interfaceThe following distribution switches are preferred:

◆ Brocade BIGIron series

◆ Brocade NETIRON series

◆ Cisco Catalyst 6500




◆ Cisco Nexus 7000

Note: The MP-8000B does not support 1 GbE, 10 GbE only.

◆ Any 10 GbE switch that supports 802.3ae-2002 (10 GbE optical,10GBASE-SR and 10GBASE-LR) and 802.3ak-2004 (SFP+ DirectAttach, 10GSFP+Cu, Twinax copper connections) and is also IEEE802.1ad and 802.1Q compliant

Layer 2 featuresThe following Layer 2 features are supported:

◆ PAUSE 802.3x will not be required.

◆ Etherchannels for Link Aggregation (LAG)

◆ Link Aggregation Control Protocol (LACP)

◆ Active mode – initiates LACP

◆ Passive mode – responds to LACP, does not initiate

◆ On – forces the LAG on, bypasses LACP

◆ Spanning Tree

◆ 802.1d - STP

◆ 802.1.s - MSTP

◆ 802.1w - RSTP, one instance for all VLANs

◆ Ethernet Edge Ports

◆ PAUSE frames

◆ Spanning Tree Edge

◆ Spanning Tree Auto Edge

◆ Spanning Tree Link Type

◆ Spanning Tree Root Guard

◆ Spanning Tree BPDU Guard

◆ Network Interface Card (NIC) teaming

• FCoE interface except with EMC iSCSI

◆ Jumbo frames up to 9208 bytes

MP-8000B supported features and topologies 51

52


FC interfaceThe following Fibre Channel features are supported:

◆ For connectivity to Connextrix B-Series switches, the MP-8000Bcan be running in FC-SW mode (E_Ports) or in Access Gateway(AG) mode

◆ For connectivity to Connectrix MDS-Series switches, theMP-8000B can only be running in AG mode

◆ Access Gateway (NPV)

◆ NPIV support

◆ 8 FC ports with 1/2/4/8G FC

◆ Universal E, F, and FL ports

◆ Builds on top of FOS 6.1+ release

◆ Hot-pluggable, Brocade branded SFP+

◆ Short-Wave Laser (SWL), Long-Wave Laser (LWL)

◆ Default Licenses

• Full Fabric

• 8Gb

• Enhanced Group Management

• FCoE

◆ Optional licenses

• Fabric Watch (FW)

• Advanced Performance Monitoring (APM)

• ISL Trunking (TRK)

Ethernet end devicesThe following are default licenses:

◆ Generic 10 GbE NICs that conform to media requirements

◆ Celerra 10 GbE (NAS+iSCSI+MPFS)

◆ Symmetrix, VNX series, or CLARiiON iSCSI interfaces

◆ NIC Teaming



MP-8000B direct-connect topologyThis section contains the following information:

◆ “Existing topology ” on page 54

◆ “Target topology” on page 56

◆ “Prerequisites” on page 57

◆ “Configuring the MP-8000B” on page 58

Refer to “Installing the MP-8000B” on page 749 for information oninstalling the MP-8000B.

MP-8000B direct-connect topology 53

54


Existing topologyIt is assumed that the customer has an existing LAN and mirroredSAN, as shown in Figure 14, and is adding the MP-8000B to both.

Figure 14 Existing topology example

Existing LAN The existing LAN has the following features/configurationparameters:

◆ Etherchannel (802.1q)◆ LACP (Link Aggregation Control Protocol)◆ RSTP (Rapid Spanning Tree Protocol)◆ MST (Multiple Spanning Tree)◆ IGMP Snooping◆ VLANs

2/11/1 1/1 2/1

Distribution-1 Distribution-21/21/31/41/5

1/21/31/41/5

Fabric A Fabric BStorage

1 2 3 4

1/1

1/1

DCX_1Domain 1

172.23.185.24

DCX_2Domain 2

172.23.185.25

1/11/21/31/41/5

1/11/21/31/41/5

MDS 9513_1VSAN 100Domain 1

172.23.185.22


172.23.185.23

CNA_1 CNA_2 CNA_1 CNA_2

Host_1(Windows)

Host_2(Linux)

CNA_1:QLogicWWPN = 21:01:00:1b:32:2a:c0:b8 Storage = Storage port 2

CNA_1:EmulexWWPN = 10:00:00:00:c9:3c:f8:3c Storage = Storage port 1

CNA_2:EmulexWWPN = 10:00:00:00:c9:3c:f7:c0 Storage = Storage port 3

CNA_2:QLogicWWPN = 21:00:00:1b:32:0a:c0:b8 Storage = Storage port 4

Storage port 1: 50:06:04:82:D5:2E:69:79

Storage port 2: 50:06:04:82:D5:2E:69:49

Storage port 3: 50:06:04:8A:D5:2E:69:79

Storage port 4: 50:06:04:82:D5:2E:69:59 SYM-002072

10 GbE

4 Gb/s FC



Existing SAN The existing LAN has the following features/configurationparameters:

Fabric A

◆ DCX_1

• Domain ID 1

◆ DCX_2

• Domain ID 2

Fabric B

◆ 9513_1

• Domain ID 1

◆ 9513_2

• Domain ID 2


56


Target topologyFigure 15 shows the target topology.

Figure 15 Target topology



PrerequisitesNote the following prerequisites:

FCF prerequisites The MP-8000B requires:

◆ 2u of rack space, preferably at the top of the rack where theservers that will be connecting to the MP-8000B are located

◆ Two power drops of either 110V or 220V

◆ An IP address for management purposes and a network port towhich to connect the 8000 management port

◆ A customer-supplied password for the admin user

Laptop prerequisites Note: The laptop used in this example is needed only to initially configurethe MP-8000B and will be disconnected after the installation is complete.

Laptop prerequisites include:

◆ Communications port 1 with a DB-9 connector

◆ Download and install PuTTY

Note: Installing PuTTY is recommended but not mandatory.

◆ Ensure that HyperTerminal is installed

Note: The operating system used for this example was Windows XPprofessional and HyperTerminal was installed by default. Some of thelater Windows-based operating systems do not have HyperTerminalinstalled. This will need to be done before attempting to configure theMP-8000Bs. See the help feature for the particular version of Windowsyou are using for more information on installing HyperTerminal.

MLX considerations Before connecting the MP-8000Bs to the MLX switches, ensure theyare properly configured. Refer to “ED-DCX-B” on page 104 for moreinformation on completing link aggregation configurations on theBrocade NetIron MLX-16 switching router.


58


Configuring the MP-8000BThis section contains the following information:

◆ “Configuring MP-8000B-1” on page 58

◆ “Configuring MP-8000B-2” on page 73

◆ “Configuring the Windows host” on page 88

◆ “Configuring the Linux host” on page 88

◆ “Configuring the VMware host” on page 89

◆ “Configuring EMC storage” on page 89

Configuring MP-8000B-1

Initial configurationThe initial configuration will be from the Fibre Channel (FOS) side ofthe switch. You can enter the help command while within FOS,which will show all available configuration commands. You can alsouse the help command to see detailed descriptions on a specificconfiguration parameter. For example:

MP_8000B:admin> help

This command shows all available configuration commands.

MP_8000B:admin> help ipaddrset

This command shows a detailed description for the ipaddrsetcommand.

To complete the initial configuration you will need to connect to theMP-8000B using the supplied serial cable.

1. Connect the rj45 connector into the console port on the front ofthe switch.

2. Attach the db9 connector to COM port 1 of the laptop.

3. Open up the Hyperterminal. Select Start > Programs >Accessories > Communications > Hyperterminal.

The New Connection dialog is opened with the entry namedialog highlighted.

4. Enter a name such as MP-8000B, and then click OK.

The Connect to dialog box displays.



5. Select COM 1 under the Connect using pull-down list, and thenclick OK.

6. In the COM 1 Properties dialog box, use the following settings:

• Bits per second: = 9600• Data bits: = 8• Parity: = none• Stop bits: = 1• Flow control: = None

Note: The first time that you boot up the MP- 8000B, you will need toenter some information that will give the switch its basic configurationneeded to access it remotely.

Prior to the initial boot sequence the following information isneeded to complete the process.

7. At the prompt enter "admin".

MP_8000B:admin> admin

8. Enter the default password, which is "password".

9. Enter the following information, which is required:

• Ip addressIpaddrset – This command initiates the script to walk youthrough setting the ip address, subnet mask, and defaultgateway. At the FOS switch prompt, enter the command“ipaddrset” and you will be prompted for the belowinformation.

MP_8000B:admin> ipaddrset

Ethernet IP Address 172.23.185.114Ethernet Subnetmask 255.255.255.0Gateway IP Address 172.23.185.2DHCP [Off]: - leave off

• Subnet mask• Default gateway• Switch name• Time Zone information

10. Enter the following information:

• Switch name

switchname – Add name of switch after the command


60


MP_8000B:admin>switchname MP-8000B-1

• Time zone

tsTimeZone – Time zone you are in

MP_8000B:admin> tsTimeZone --interactive

Note: This will guide you through an interactive menu to completethe time zone configuration.

• Date

date – Date and time

The date value of 0610123009 has a format ofmmddHHMMyy, where:

– mm is the month; valid values are 01 through 12.– dd is the date; valid values are 01 through 31.– HH is the hour; valid values are 00 through 23.– MM is minutes; valid values are 00 through 59.– yy is the year; valid values are 00 through 99 (values

greater than 69 are interpreted as 1970 through 1999, andvalues less than 70 are interpreted as 2000-2069).

MP_8000B:admin>date 0610123009

Help command

Within the Ethernet side of the switch (CMSH) you can enter the ?command to see all available configuration commands. Thefollowing are example descriptions from the MP-8000B:

◆ Full help is available when you are ready to enter a commandargument (e.g., show ?) and describes each possible argument.

◆ Partial help is provided when an abbreviated argument is enteredand you want to know what arguments match the input (e.g.,show ve?).

Configuration mode

This section describes how to navigate the configuration mode.Configuration mode is the name of the process in which you configurethe Ethernet portion of the MP-8000B, whether configuring theMP-8000B directly through a serial connection or remotely through atelnet session. In either case, you will enter the MP-8000B through theFOS first. To get the CMSH, enter the cmsh command.

To make any type of configuration changes within the CMSH youmust enter configuration mode. Enter the configure terminal



command, which will bring you to the configuration prompt. Thefollowing is an example of the output.

switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.switch(config)#

From this prompt you can enter the ? command to see availableoptions.

The running-configuration is the active configuration on the device.Any changes made to the switch while in the CMSH are made to thehelp command. The configuration stored in nonvolatile memory(nvram) is called the startup-configuration.

To copy the running-configuration to nvram, enter the commandcopy running-config startup-config.

switch# copy running-config startup-configOverwrite the startup config file (y/n): y

Global configuration refers to anything within therunning-configuration that is system wide to the Ethernet switch.

The following is a truncated example of the running configurationoutput and how the different configuration modes are broken down.Interfaces TenGigabitEthernet 0/2 through TenGigabitEthernet 0/22have been left out to keep the example short. All of the sameproperties apply to these interfaces.

switch# sho run

!hostname switch ------ Global Configuration begins here-----!vlan classifier rule 1 proto fcoe encap ethv2vlan classifier rule 2 proto fip encap ethv2vlan classifier group 1 add rule 1vlan classifier group 1 add rule 2no protocol spanning-tree!cee-map defaultpriority-group-table 1 weight 40 pfcpriority-group-table 2 weight 60priority-table 2 2 2 1 2 2 2 2

! ------ Global Configuration ends here-----!! ------ Interface Configuration begins here-----interface Vlan 1!interface Vlan 1002


62


fcf forward!interface TenGigabitEthernet 0/0shutdown!interface TenGigabitEthernet 0/1switchportswitchport mode accessno shutdowncee default

||interface TenGigabitEthernet 0/23shutdown! ------ Interface Configuration ends here-----!! ------ Global Configuration resumes here-----protocol lldpadvertise dcbx-fcoe-app-tlvadvertise dcbx-fcoe-logical-link-tlv

!line console 0login

line vty 0 31login

!end

CMSH configuration1. Enter the CMSH to begin configuring the Ethernet portion of the

switch.

MP-8000B-1:admin> cmsh

Note: MP-8000Bs sold through EMC may have a default configurationalready loaded. The default configuration parameters included are:Spanning -tree, VLAN Classifiers, VLAN Classifier Group, CEE Maps,and LLDP. To view these default configurations or to verify theparameters are included, enter the show running-config command.

2. Enter the show running-config command to check theconfiguration. The following shows a a truncated output of theshow running-config command with the default parameterspresent. If the parameters are not included in the configuration,then Step 3 through Step 7 will need to be applied to therunning-configuration.

MP-8000B-1# show running-config!



protocol spanning-tree mstp <– Spanning-tree!!vlan classifier rule 1 proto fcoe encap ethv2 <– Vlan Classifier rule 1vlan classifier rule 2 proto fip encap ethv2 <– Vlan Classifier rule 2vlan classifier group 1 add rule 1 <– Vlan Classifier group 1vlan classifier group 1 add rule 2 <– Vlan Classifier group 1!cee-map default <– CEE Mappriority-group-table 1 weight 40 pfcpriority-group-table 2 weight 60priority-table 2 2 2 1 2 2 2 2

!.NOTE: output truncated.protocol lldp <– LLDPadvertise dcbx-fcoe-app-tlvadvertise dcbx-fcoe-logical-link-tlv

3. Configure Spanning Tree.

Note: By default, Spanning Tree should be enabled and MSTP should beconfigured.

The Spanning Tree Protocol (STP) is an algorithm that works toprevent Layer 2 loops in the network by ensuring there is onlyone active forwarding port between Ethernet switches at a time.

MP-8000B-1# config tMP-8000B-1(config)# protocol spanning-tree mstpMP-8000B-1(conf-mstp)# end

Verify MSTP is configuredMP-8000B-1# sho spanning-tree brief

Spanning-tree Mode: Multiple Spanning Tree Protocol ← Ensure MSTP is configured

4. Configure the VLAN Classifier.

The VLAN Classifier is used to apply the FCoE VLAN ID to theinbound frames from the CNA that are either the FIP or FCoEprotocol. Inbound packets that are “untagged” (Ethernet framesthat do not have a VLAN ID assigned) and match either rule #1 orrule #2 will be assigned the correct FCoE VLAN at the interfaceconfiguration. The interface configuration steps are covered laterin the document.


64


The steps that follow will classify any FCoE traffic encapsulatedwithin Ethernet with rule #1 and any FIP traffic encapsulatedwithin Ethernet with rule #2.

Note: This is an MP-8000B configuration only, and is done to classify theFCoE traffic to ensure that both the FCoE and FIP protocols will beguaranteed at least 40% of the bandwidth in the event of congestion onthe interface.

MP-8000B-1# config tMP-8000B-1(config)# vlan classifier rule 1 proto fcoe encap ethv2MP-8000B-1(config)# vlan classifier rule 2 proto fip encap ethv2

Show configured classifier rulesMP-8000B-1(config)# do sho vlan classifier rule

vlan classifier rule 1 proto fcoe encap ethv2vlan classifier rule 2 proto fip encap ethv2

The commands provided above have classified the two types oftraffic, FCoE and FIP, both encapsulated within Ethernet Version2.

5. Configure the VLAN Classifier Group.

Use the following configuration steps to put both rule #1 and rule#2 into the same VLAN classifier group - 1.

MP-8000B-1(config)# vlan classifier group 1 add rule 1MP-8000B-1(config)# vlan classifier group 1 add rule 2

Show configured classifier groups.MP-8000B-1(config)# do sho vlan classifier group

vlan classifier group 1 add rule 1vlan classifier group 1 add rule 2

At this point we have classified FCoE and FIP traffic encapsulatedwithin Ethernet Version 2 and assigned them to Group- 1.

6. Configure CEE-MAP.

MP-8000B-1(config)# cee-map defaultMP-8000B-1(conf-ceemap)# priority-group-table 1 weight 40 pfcMP-8000B-1(conf-ceemap)# priority-group-table 2 weight 60MP-8000B-1(conf-ceemap)# priority-table 2 2 2 1 2 2 2 2MP-8000B-1(conf-ceemap)# exit

Show configured cee mapsMP-8000B-1(conf-ceemap)# do sho cee maps



CEE Map defaultPrecedence 1Priority Group Table

1: Weight 40, PFC Enabled, TrafficClass 3, BW% 402: Weight 60, PFC Disabled, TrafficClass 4, BW% 6015.0: PFC Disabled15.1: PFC Disabled15.2: PFC Disabled15.3: PFC Disabled15.4: PFC Disabled15.5: PFC Disabled15.6: PFC Disabled15.7: PFC Disabled

Priority TableCoS: 0 1 2 3 4 5 6 7

---------------------------------------------PGID: 2 2 2 1 2 2 2 2

Enabled on the following interfaces

CEE Map testPrecedence 1

Enabled on the following interfaces

7. Configure LLDP.

The LLDP portion of the configuration is at the bottom of therunning-configuration. The LLDP configuration is needed so theswitch will advertise the appropriate Type Length Values (TLVs).

MP-8000B-1(config)# protocol lldpMP-8000B-1(conf-lldp)# advertise dcbx-fcoe-app-tlvMP-8000B-1(conf-lldp)# advertise dcbx-fcoe-logical-link-tlv

Show global LLDP information

MP-8000B-1(conf-lldp)# do sho lldp

LLDP Global Informationsystem-name: MP-8000B-1system-description: CEE Switchdescription:State: EnabledMode: Receive/TransmitAdvertise transmitted: 30 secondsHold time for advertise: 120 secondsRe-init Delay Timer: 2 secondsTx Delay Timer: 1 secondsTransmit TLVs: Chassis ID Port ID

TTL IEEE DCBxDCBx FCoE App DCBx FCoE Logical Link

Link Prim Brocade Link


66


DCBx FCoE Priority Bits: 0x8DCBx iSCSI Priority Bits: 0x10

Show interface specific LLDP information

MP-8000B-1(conf-lldp)# do sho lldp int t 0/1

LLDP information for Te 0/1State: EnabledMode: Receive/TransmitAdvertise Transmitted: 30 secondsHold time for advertise: 120 secondsRe-init Delay Timer: 2 secondsTx Delay Timer: 1 secondsDCBX Version : CEEAuto-Sense : Yes

Transmit TLVs: Chassis ID Port IDTTL IEEE DCBxDCBx FCoE App DCBx FCoE Logical LinkLink Prim Brocade Link

8. Configure VLAN.

On the MP-8000B switch you can create VLANs with VLAN IDsup to 3583. You can only configure one VLAN acting as the FibreChannel Forwarder (FCF).

VLANs are created to segment broadcast domains and differenttypes of user traffic. A VLAN allows for a more dynamicenvironment while providing the segmentation of a physicalLAN. This is accomplished by configuring VLAN assignmentsthrough software rather than moving physical interfaces on theswitch.

Note: VLAN 1 is the default VLAN on the MP-8000Bs. Currently, it is notpossible to change the value of the default VLAN from 1.

MP-8000B-1(conf-lldp)# int vlan 100

9. Configure FCoE VLAN.

There can only be one VLAN configured as the FCF on aMP-8000B.

The following example shows VLAN 1002 as the FCF forwarder.

MP-8000B-1(conf-if-vl-100)# int vlan 1002MP-8000B-1(conf-if-vl-1002)#fcf forward



MP-8000B-1 Interface configurationsThe following example shows how to configure configured interfaceTenGigabitEthernet 0/0.

Perform Step 1 through Step 7 on all interfaces that will use FCoE.This example uses interfaces 0/0 and 0/4.

1. Run the switchport command to enable the layer 2 switchingcapability.

MP-8000B-1(conf-if-vl-1002)# int t 0/0MP-8000B-1(conf-if-te-0/0)# switchport

2. Configure the interfaces so they can handle both types of Ethernettraffic. Brocade has created the “converged” mode which willforward both the untagged (Ethernet) and tagged (FCoE) packets.

MP-8000B-1(conf-if-te-0/0)# switchport mode converged

3. Assign the default VLAN for the interfaces. This configurationuses VLAN 100 as the default VLAN.

This is the VLAN used for Ethernet traffic, not FCoE.

MP-8000B-1(conf-if-te-0/0)# switchport converged vlan 100

Note: By default, VLAN 1 is used as the default VLAN if there is not oneconfigured.

4. Configure the interfaces to handle the FCoE traffic.

For this configuration step it is necessary to apply the VLANclassifier created in the Global Configuration and apply it to theFCoE VLAN, 1002.

MP-8000B-1(conf-if-te-0/0)# vlan classifier activate group 1 vlan 1002

5. Apply STP configurations to the interfaces.

These commands work in conjunction with the GlobalConfiguration command – protocol spanning-tree RSTP.

The following commands will configure the ports as an edgeport.This will allow the ports to begin forwarding traffic immediatelyafter a STP event as well as guard it against incoming BPDUs.

MP-8000B-1(conf-if-te-0/0)# spanning-tree edgeportMP-8000B-1(conf-if-te-0/0)# spanning-tree edgeport bpdu-guard


68


6. Perform the next configuration step to apply the defaultCEE-MAP to the interface that you created earlier.

The syntax for this command is cee <name>.

The default CEE-MAP is named “default”. The command is:

MP-8000B-1(conf-if-te-0/0)# cee default

7. Enable the interfaces so that they will be ready to pass traffic.

MP-8000B-1(conf-if-te-0/0)# no shut

Note: The no shut command will need to be completed on any of theEthernet interfaces being used. By default all interfaces are in the“shutdown” state and will not pass any traffic.

8. Repeat Step 1 through Step 7 for other FCoE interfaces.

9. Perform the following to verify the configuration steps have beencompleted successfully:

MP-8000B-1(conf-if-te-0/0)# do sho run!hostname MP-8000B-1!protocol spanning-tree rstp!vlan classifier rule 1 proto fcoe encap ethv2vlan classifier rule 2 proto fip encap ethv2vlan classifier group 1 add rule 1vlan classifier group 1 add rule 2!cee-map defaultpriority-group-table 1 weight 40 pfcpriority-group-table 2 weight 60priority-table 2 2 2 1 2 2 2 2

!interface Vlan 1!interface Vlan 100!interface Vlan 1002fcf forward

!interface TenGigabitEthernet 0/0switchportswitchport mode convergedswitchport converged vlan 100vlan classifier activate group 1 vlan 1002no shutdown



spanning-tree edgeportspanning-tree edgeport bpdu-guardcee default!interface TenGigabitEthernet 0/1shutdown

!interface TenGigabitEthernet 0/2shutdown


!interface TenGigabitEthernet 0/4switchportswitchport mode convergedswitchport converged vlan 100vlan classifier activate group 1 vlan 1002

spanning-tree edgeportspanning-tree edgeport bpdu-guardno shutdown|.... NOTE: output truncated..interface TenGigabitEthernet 0/23shutdown

!protocol lldpadvertise dcbx-fcoe-app-tlvadvertise dcbx-fcoe-logical-link-tlv


line vty 0 31login

!end

10. Perform Save configuration – copy running-config tostartup-config.

Note: At this point, you will need to exit out of configuration mode. Youcan either enter "exit" to back out one level at a time, or "end" to exitcompletely out of configuration mode.

MP-8000B-1(conf-if-te-0/0)# endMP-8000B-1# copy run startOverwrite the startup config file (y/n): yBuilding configuration...


70


11. Create the port channel interface. The port channel interface isused to configure a generic configuration that can be applied tothe appropriate interfaces for link aggregation (LAG). LAG, orEther-Channel is a way in which to combine the interfaces into asingle virtual link, therefore combining its bandwidth.

MP-8000B-1(config)# interface port-channel 1MP-8000B-1(conf-if-po-1)# switchportMP-8000B-1(conf-if-po-1)# switchport mode trunkMP-8000B-1(conf-if-po-1)# switchport trunk allowed vlan add 100MP-8000B-1(conf-if-po-1)# no shutMP-8000B-1(conf-if-po-1)# exit

12. Configure Uplink interfaces t 0/15 and t 0/16 as LAG ports.

MP-8000B-1(config)# interface t 0/15MP-8000B-1 (conf-if-te-0/15)# channel-group 1 mode on type standardMP-8000B-1 (conf-if-te-0/15)# no shutMP-8000B-1(config)# interface t 0/16MP-8000B-1 (conf-if-te-0/16)# channel-group 1 mode on type standardMP-8000B-1 (conf-if-te-0/16)# no shut

13. Verify the port channel interface is configured.

MP-8000B-1(conf-if-te-0/16)#do sh run int port-channel 1!interface Port-channel 1switchportswitchport mode trunkno shutdown


MP-8000B-1(conf-if-te-0/16)# endMP-8000B-1# copy runn startOverwrite the startup config file (y/n): yBuilding configuration...

MP-8000B-1 Zoning configurationAfter the configuration steps for the CMSH portion of the switchhave been completed and you have saved the configuration you willneed to complete the zoning configurations.

Note: At this point, the traditional GUI-based tools should be able to accessthe MP-8000B to complete the zoning configuration, but for this example, wewill complete the zoning configurations using the FOS CLI.

At the Ethernet prompt enter the command exit to return to the FOS.

MP-8000B-1#exit



You will notice that the prompt has changed and you are now readyto configure the Fibre Channel portion.

MP-8000B-1:admin>

The Fibre Channel portion of the configuration will include the FibreChannel attached storage devices. At this point the CNAs areinstalled into the Servers with the correct drivers loaded, andphysically connected to the MP-8000B.

After you plug the storage devices into the MP-8000B, they shouldlog in to the fabric. EMC allows only one Fibre Channel Forwarder(FCF) VLAN per switch so any zone configurations within the FOSare going to be to the same Fabric.

The steps required to complete the Fibre Channel portion of theconfiguration are:

• Identify WWPNs

• Create zones

• Create zone configuration

• Enable the zone configuration

• Save the zone configuration

These steps are further detailed as follows:

1. Identify the WWPNs associated to the CNAs.

Print local Name Server informationMP-8000B-1:admin> fcoe --loginshow=============================================================================Port Te port Device WWN Device MAC Session MAC=============================================================================8 Te 0/0 10:00:00:00:c9:3c:f8:3c 00:00:c9:3c:f8:3c 0e:fc:00:01:14:0110 Te 0/2 21:01:00:1b:32:2a:c0:b8 00:1b:32:2a:c0:b8 0e:fc:00:01:18:01

2. Create applicable zones.

a. Create each zone by using the following command:

zonecreate "<zone name a>", "WWPN1; WWPN2; .... WWPNn"

b. Create a zone named Host1_CNA1 with the above WWPNsusing the zonecreate command.


72


Note: In the following examples the direct attached CNAs are in theoutput from the nsshow command because they were directly attachedto the switch. The second WWPN in the examples are from Figure 15 onpage 56.

MP-8000B-1:admin> zonecreate"Host1_CNA1","10:00:00:00:c9:3c:f8:3c;50:06:04:82:d5:2e:69:79"

c. Create a zone named Host1_CNA2 with the above WWNsusing the zonecreate command.

MP-8000B-1:admin> zonecreate "Host1_CNA2","10:00:00:00:c9:3c:f7:c0;0:06:04:82:d5:2e:69:49"

3. Create the zone configuration.

a. Create the zone configuration by using the followingcommand:

cfgcreate "<cfg name>", "<zone name a>; <zone name b>; .... <zone name n>".

b. Create a zone configuration using the newly created zones,Host1_CNA1 and Host1_CNA2.

MP-8000B-1:admin> configcreate "Host1_FCoE", "Host1_CNA1; Host1_CNA2"

4. Enable the zone configuration.

a. Enable the zone config by using by using the followingcommand:

cfgenable "<cfg name>".b. Enable the new zone configuration.

MP-8000B-1:admin> cfgenable “Host1_FCoE”

5. Save the configuration by using the cfgsave command:

MP-8000B-1:admin> cfgsave

6. Verify the zone configuration.

Show the zone configuration by using the cfgshow command.

MP-8000B-1:admin> cfgshow

Defined configuration:cfg: Host1_FCoE Host1_CNA1; Host1_CNA2zone: Host1_CNA1

10:00:00:00:c9:3c:f8:3c; 50:06:04:82:d5:2e:69:79



zone: Host1_CNA210:00:00:00:c9:3c:f7:c0; 50:06:04:82:d5:2e:69:49

Effective configuration:cfg: Host1_FCoEzone: Host1_CNA1

10:00:00:00:c9:3c:f8:3c 50:06:04:82:d5:2e:69:79

zone: Host1_CNA210:00:00:00:c9:3c:f7:c0 50:06:04:82:d5:2e:69:49

Configuring MP-8000B-2

Initial configurationThe initial configuration will be from the Fibre Channel (FOS) side ofthe switch. You can enter the help command while within FOS,which will show all available configuration commands. You can alsouse the help command to see detailed descriptions on a specificconfiguration parameter. For example:

MP_8000B:admin> help

This command shows all available configuration commands.

MP_8000B:admin> help ipaddrset

This command shows a detailed description for the ipaddrsetcommand.

To complete the initial configuration, you will need to connect to theMP-8000B using the supplied serial cable by following the next steps:

1. Connect the rj45 connector into the console port on the front ofthe switch.

2. Connect the db9 connector to COM port 1 of the laptop.

3. Open up the Hyperterminal. Select Start > Programs >Accessories > Communications > Hyperterminal.

The New Connection dialog box is opened with the entry namedialog highlighted.

4. Enter a name, such as MP-8000B, and then click OK.


5. Select COM 1 under the Connect using pull-down list, and thenclick OK.

6. In the COM 1 Properties dialog box, use the following settings:


74


• Bits per second: = 9600

• Data bits: = 8

• Parity: = none

• Stop bits: = 1

• Flow control: = None

Note: The first time that you boot up the MP-8000B you will need toenter some information that will give the switch its basicconfiguration needed to access it remotely. You will need the belowinformation prior to the initial boot sequence to complete the process.

7. At the prompt, enter admin.

MP_8000B:admin> admin

8. Enter the default password, which is password.

9. Enter the following information, which is required:

• Ip address

The Ipaddrset command initiates the script to walk youthrough setting the ip address, subnet mask, and defaultgateway. At the FOS switch prompt, enter the ipaddrsetcommand and you will be prompted for the belowinformation.

MP_8000B:admin> ipaddrsetEthernet IP Address 172.23.185.115Ethernet Subnetmask 255.255.255.0Fibre Channel IP Address [none] - leave blankFibre Channel Subnetmask [none]: - leave blankGateway IP Address 172.23.185.2DHCP [Off]: - leave off

• Subnet mask• Default gateway• Switch name• Time Zone information

10. Enter the following information:

• Switch name

switchname – Add name of switch after the command

MP_8000B:admin>switchname MP-8000B-2



• Time zone

tsTimeZone – Time zone you are in

MP_8000B:admin> tsTimeZone US/Eastern (case sensitive)

• Date

date – Date and time

The date value of 0610123009 has a format ofmmddHHMMyy, where:

– mm is the month; valid values are 01 through 12.– dd is the date; valid values are 01 through 31.– HH is the hour; valid values are 00 through 23.– MM is minutes; valid values are 00 through 59.– yy is the year; valid values are 00 through 99 (values

greater than 69 are interpreted as 1970 through 1999, andvalues less than 70 are interpreted as 2000-2069).

MP_8000B:admin>date 0610123009

Enabling Access Gateway (MP-8000B-2 only)For details on Access Gateway (AG), refer to the “Brocade AccessGateway” section in the Non-EMC SAN Products Data ReferenceManual, available through the E-Lab Interoperability Navigator,Topology Resource Center tab, at http://elabnavigator.EMC.com.

To enabling Access Gateway, complete the following steps:

1. Disable the switch. The switch must be disabled before you canenable AG mode.

MP_8000B:admin> switchdisable

2. Enable AG mode.

MP_8000B:admin> ag --modeenable

Warning: Access Gateway mode changes the standardbehavior of the switch. Please check the Access GatewayAdministrator's Guide before proceeding.Enabling agmode will remove all the configuration dataon the switch, including zoning configuration andsecurity database. Back up your configuration usingtheconfigupload command. This operation will reboot theswitch.Do you want to continue? (yes, y, no, n): [no] yAccess Gateway mode was enabled successfullySwitch is being rebooted...



76


IMPORTANT

If you are connected to the switch with a console cable, you willneed to wait until the switch comes back online.

If you are logging into the switch remotely, you will need toestablish a new telnet session after the switch comes backonline.

3. Verify the switch is in AG mode.

After the switch has come back online, enter the followingcommand to verify that the switch is now in AG mode:

MP_8000B:admin> ag --modeshowAccess Gateway mode is enabled.

AG commands

The following are useful AG commands:

ag --modeshow

ag --show

ag --help

Configuration mode

This section describes how to navigate the configuration mode.Configuration mode is the name of the process in which you configurethe Ethernet portion of the MP-8000B, whether configuring theMP-8000B directly through a serial connection or remotely through atelnet session. In either case, you will enter the MP-8000B through theFOS first. To get the CMSH, enter the cmsh command.

Within the Ethernet side of the switch (CMSH) you can enter the ?command to see all available configuration commands. Thefollowing are example descriptions from the MP-8000B:

◆ Full help is available when you are ready to enter a commandargument (e.g., show ?) and describes each possible argument.

◆ Partial help is provided when an abbreviated argument is enteredand you want to know what arguments match the input (e.g.,show ve?).



To make any type of configuration changes within the CMSH, youmust enter configuration mode. Enter the configure terminalcommand, which will bring you to the configuration prompt. Thefollowing is an example of the output.

switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.switch(config)#

From this prompt you can enter ? to see available options.

The running-configuration is the active configuration on the device.Any changes made to the switch while in the CMSH are made to therunning-configuration. The configuration stored in Non volatilememory (nvram) is called the startup-configuration.

To copy the running-configuration to nvram, enter the commandcopy running-config startup-config.

switch# copy running-config startup-configOverwrite the startup config file (y/n): y

Global configuration refers to anything within the runningconfiguration that is system wide to the Ethernet switch.

The following is a truncated example of the running configurationoutput and how the different configuration modes are broken down.Interfaces TenGigabitEthernet 0/2 through TenGigabitEthernet 0/22have been left out to keep the example short. All of the sameproperties apply to these interfaces.

switch# sho run

!hostname switch ------ Global Configuration begins here-----!vlan classifier rule 1 proto fcoe encap ethv2vlan classifier rule 2 proto fip encap ethv2vlan classifier group 1 add rule 1vlan classifier group 1 add rule 2no protocol spanning-tree!cee-map defaultpriority-group-table 1 weight 40 pfcpriority-group-table 2 weight 60priority-table 2 2 2 1 2 2 2 2

! ------ Global Configuration ends here-----!! ------ Interface Configuration begins here-----interface Vlan 1!


78


interface Vlan 1002fcf forward

!interface TenGigabitEthernet 0/0shutdown!interface TenGigabitEthernet 0/1switchportswitchport mode accessno shutdowncee default

||interface TenGigabitEthernet 0/23shutdown! ------ Interface Configuration ends here-----!! ------ Global Configuration resumes here-----protocol lldpadvertise dcbx-fcoe-app-tlvadvertise dcbx-fcoe-logical-link-tlv


line vty 0 31login

!end

CMSH configuration1. Enter the CMSH to begin configuring the Ethernet portion of the

switch.

MP-8000B-1:admin> cmsh

Note: MP-8000Bs sold through EMC may have a default configurationalready loaded. The default configuration parameters included are:Spanning -tree, VLAN Classifiers, VLAN Classifier Group, CEE Maps,and LLDP. To view these default configurations or to verify theparameters are included, enter the show running-config command.

2. Enter the show running-config command to check theconfiguration. The following shows a a truncated output of theshow running-config command with the default parameterspresent. If the parameters are not included in the configuration,then Step 3 through Step 8 will need to be applied to therunning-configuration.



MP-8000B-1# show running-confighostname MP8000B-1!protocol spanning-tree mstp <– Spanning-tree!!vlan classifier rule 1 proto fcoe encap ethv2 <– Vlan Classifier rule 1vlan classifier rule 2 proto fip encap ethv2 <– Vlan Classifier rule 2vlan classifier group 1 add rule 1 <– Vlan Classifier group 1vlan classifier group 1 add rule 2 <– Vlan Classifier group 1!cee-map default <– CEE Mappriority-group-table 1 weight 40 pfcpriority-group-table 2 weight 60priority-table 2 2 2 1 2 2 2 2

!.NOTE: output truncated.protocol lldp <– LLDPadvertise dcbx-fcoe-app-tlvadvertise dcbx-fcoe-logical-link-tlv

3. Configure Spanning Tree.

Note: By default, Spanning Tree should be enabled and MSTP should beconfigured.

The Spanning Tree Protocol (STP) is an algorithm that works toprevent Layer 2 loops in the network by ensuring there is onlyone active forwarding port between Ethernet switches at a time.

If necessary, enable the STP:

MP-8000B-1# config tMP-8000B-1(config)# protocol spanning-tree mstpMP-8000B-1(conf-mstp)# end

Verify MSTP is configuredMP-8000B-2# sho spanning-tree brief

Spanning-tree Mode: Multiple Spanning Tree Protocol

Root ID Priority 32768Address 0005.1e76.81a0Hello Time 2, Max Age 20, Forward Delay 15

Bridge ID Priority 32768Address 0005.1e76.81a0Hello Time 2, Max Age 20, Forward Delay 15, Tx-HoldCount 6Migrate Time 3 sec


80


4. Configure the VLAN Classifier.

The VLAN Classifier is used to apply the FCoE VLAN ID to theinbound frames from the CNA that are either the FIP or FCoEprotocol. Inbound packets that are “untagged” (Ethernet framesthat do not have a VLAN ID assigned) and match either rule #1 orrule #2 will be assigned the correct FCoE VLAN at the interfaceconfiguration. The interface configuration steps are covered laterin the document.

The steps that follow will classify any FCoE traffic encapsulatedwithin Ethernet with rule #1 and any FIP traffic encapsulatedwithin Ethernet with rule #2.

Note: This is an MP-8000B configuration only, and is done to classify theFCoE traffic to ensure that both the FCoE and FIP protocols will beguaranteed at least 40% of the bandwidth in the event of congestion onthe interface.

Show configured classifier rulesMP-8000B-2(config)# do sho vlan classifier rule

vlan classifier rule 1 proto fcoe encap ethv2vlan classifier rule 2 proto fip encap ethv2

The commands provided above have classified the two types oftraffic, FCoE and FIP, both encapsulated within Ethernet Version2.

5. Configure the VLAN Group.

VLAN Group groups multiple VLAN Classifiers into a singlegroup so they can be applied in a simpler fashion to an interface.

Now that the traffic is classified, you need to apply this to a groupfor further configuration. In the next configuration steps, we aregoing to put both rule #1 and rule #2 into the same VLANclassifier group - 1.

Show configured classifier groupsMP-8000B-2(conf-lldp)# do sho vlan classifier group

vlan classifier group 1 add rule 1vlan classifier group 1 add rule 2

At this point we have classified FCoE and FIP traffic encapsulatedwithin Ethernet Version 2 and assigned them to Group 1.

6. Configure the CEE-MAP.



The CEE-Map prioritizes the Ethernet traffic. This is done byputting the Ethernet interface into a “trusted” state so it willhonor any priority bits within the Ethernet frames. A framecoming into the interface that has a priority within it will be“trusted”. A frame coming into the Ethernet interface that doesnot contain a priority will be given a default value of 0, whichtranslates to a Priority Group ID (PGID) of 2 in the defaultCEE-MAP configuration.

Any FCoE traffic will have a priority within the frame and thepriority is trusted. Next is an example of a CEE-Map when allFCoE traffic is given a priority of 1 while any other type of trafficis given a priority of 2. This will guarantee that any FCoE trafficwill get 40% of the available bandwidth.

MP-8000B-2(config)# cee-map defaultMP-8000B-2(conf-ceemap)# priority-group-table 1 weight 40 pfcMP-8000B-2(conf-ceemap)# priority-group-table 2 weight 60MP-8000B-2(conf-ceemap)# priority-table 2 2 2 1 2 2 2 2

Show configured cee mapsMP-8000B-2(conf-ceemap)# do sho cee maps

CEE Map defaultPrecedence 1Priority Group Table

1: Weight 40, PFC Enabled, TrafficClass 3, BW% 402: Weight 60, PFC Disabled, TrafficClass 4, BW% 6015.0: PFC Disabled15.1: PFC Disabled15.2: PFC Disabled15.3: PFC Disabled15.4: PFC Disabled15.5: PFC Disabled

15.6: PFC Disabled15.7: PFC Disabled

Priority TableCoS: 0 1 2 3 4 5 6 7

---------------------------------------------PGID: 2 2 2 1 2 2 2 2

FCoE CoS: NoneEnabled on the following interfaces

MP-8000B-2(conf-ceemap)# exit

7. Configure the LLDP.

The LLDP portion of the configuration is at the bottom of therunning-configuration. The LLDP configuration is needed so theswitch will advertise the appropriate Type Length Values (TLVs).


82


MP-8000B-2(config)# protocol lldpMP-8000B-2(conf-lldp)# advertise dcbx-fcoe-app-tlvMP-8000B-2(conf-lldp)# advertise dcbx-fcoe-logical-link-tlv

Show global lldp information

MP-8000B-2(conf-lldp)# do sho lldp

LLDP Global Informationsystem-name: MP-8000B-1system-description: CEE Switchdescription:State: EnabledMode: Receive/TransmitAdvertise transmitted: 30 secondsHold time for advertise: 120 secondsRe-init Delay Timer: 2 secondsTx Delay Timer: 1 secondsTransmit TLVs: Chassis ID Port ID

TTL IEEE DCBxDCBx FCoE App DCBx FCoE Logical Link

Link Prim Brocade LinkDCBx FCoE Priority Bits: 0x8DCBx iSCSI Priority Bits: 0x10

Show interface specific LLDP information

MP-8000B-2(conf-lldp)# sho lldp int t 0/1LLDP Interface statistics for Te 0/1Frames transmitted: 25652Frames Aged out: 0Frames Discarded: 0Frames with Error: 0Frames Recieved: 25159TLVs discarded: 0TLVs unrecognized: 0

8. Configure the VLAN configuration.

On the MP-8000B switch, you can create VLANs with VLAN IDsup to 3583, but you can only configure one VLAN acting as theFibre Channel Forwarder (FCF).

IMPORTANT

VLAN 1 is the default VLAN on the MP-8000Bs and should notbe altered

MP-8000B-2(conf-lldp)# exitMP-8000B-2(config)# int vlan 100



9. Configure the FCoE VLAN.

There can only be one VLAN configured as the FCF perMP-8000B. The next example uses VLAN 1002 as the FCFforwarder.

MP-8000B-2(conf-if-vl-1002)# int vlan 1002MP-8000B-2(conf-if-vl-1002)# fcf forward

MP-8000B-2 Interface configurationsThe following example shows how to configure configured interfaceTenGigabitEthernet 0/0.

Note: These same steps will need to be performed on interfaceTenGigaibitEthernet 0/4 to complete this installation.

1. Run the switchport command to enable the layer 2 switchingcapability.

MP-8000B-2(conf-if-vl-1002)# int t 0/0MP-8000B-2(conf-if-te-0/0)# switchport

2. Configure the interfaces so they can handle both types of Ethernettraffic. Brocade has created the “converged” mode which willforward both the untagged (Ethernet) and tagged (FCoE) packets.

MP-8000B-2(conf-if-te-0/0)# switchport mode converged

3. Assign the default VLAN for the interfaces. This configurationuses VLAN 100 as the default VLAN.

This is the VLAN used for Ethernet traffic, not FCoE.

MP-8000B-2(conf-if-te-0/0)# switchport converged vlan 100

Note: By default, VLAN 1 is used as the default VLAN if there is not oneconfigured.

4. Configure the interfaces to handle the FCoE traffic.

For this configuration step it is necessary to apply the VLANclassifier created in the Global Configuration and apply it to theFCoE VLAN, 1002.

MP-8000B-2(conf-if-te-0/0)# vlan classifier activate group 1 vlan 1002

5. Apply STP configurations to the interfaces.


84


These commands work in conjunction with the GlobalConfiguration command – protocol spanning-tree RSTP.

The following commands will configure the ports as an edgeport.This will allow the ports to begin forwarding traffic immediatelyafter a STP event as well as guard it against incoming BPDUs.

MP-8000B-2(conf-if-te-0/0)# spanning-tree edgeportMP-8000B-2(conf-if-te-0/0)# spanning-tree edgeport bpdu-guard

6. Perform the next configuration step to apply the defaultCEE-MAP to the interface.

The syntax for this command is cee <name>.

The default CEE-MAP is named “default”. The command is:

MP-8000B-2(conf-if-te-0/0)# cee default

7. Enable the interfaces so that they will be ready to pass traffic.

MP-8000B-2(conf-if-te-0/0)# no shut

Note: The no shut command will need to be completed on any of theEthernet interfaces being used. By default all interfaces are in the“shutdown” state and will not pass any traffic.

8. Perform the following to verify the configuration steps have beencompleted successfully:

MP-8000B-2(conf-if-te-0/0)# do sho run!hostname MP-8000B-2!protocol spanning-tree mstp!vlan classifier rule 1 proto fcoe encap ethv2vlan classifier rule 2 proto fip encap ethv2vlan classifier group 1 add rule 1vlan classifier group 1 add rule 2!cee-map defaultpriority-group-table 1 weight 40 pfcpriority-group-table 2 weight 60priority-table 2 2 2 1 2 2 2 2

!interface Vlan 1!interface Vlan 100!interface Vlan 1002fcf forward

!



interface TenGigabitEthernet 0/0switchportswitchport mode convergedswitchport converged vlan 100vlan classifier activate group 1 vlan 1002

spanning-tree edgeportspanning-tree edgeport bpdu-guardno shutdown




!interface TenGigabitEthernet 0/4switchportswitchport mode convergedswitchport converged vlan 100vlan classifier activate group 1 vlan 1002

spanning-tree edgeportspanning-tree edgeport bpdu-guardno shutdown|.... NOTE: output truncated..interface TenGigabitEthernet 0/23shutdown



line vty 0 31login

!end


MP-8000B-2(conf-if-te-0/0)# endMP-8000B-2 # copy run startOverwrite the startup config file (y/n): yBuilding configuration...


86


10. Create the port channel interface. The port channel interface isused to configure a generic configuration that can then be appliedto the appropriate interfaces for link aggregation (LAG).

MP-8000B-2(config)# interface port-channel 1MP-8000B-2(conf-if-po-1)# switchportMP-8000B-2(conf-if-po-1)# switchport mode trunkMP-8000B-2(conf-if-po-1)#switchport trunk allowed vlan 100MP-8000B-2(conf-if-po-1)# no shut

11. Configure interfaces t 0/15 and t 0/16 as LAG ports.

MP-8000B-2(config)# interface t 0/15MP-8000B-2(conf-if-te-0/15)# channel-group 1 mode on type standardMP-8000B-2(conf-if-te-0/15)# no shutMP-8000B-2(config)# interface t 0/16MP-8000B-2(conf-if-te-0/16)# channel-group 1 mode on type standardMP-8000B-2(conf-if-te-0/16)# no shut

12. Verify the port channel interface is configured.

MP-8000B-2(conf-if-te-0/16)# do sh run int port-channel 1!interface Port-channel 1switchportswitchport mode trunkno shutdown

13. Perform the save configuration by issuing the copyrunning-config startup-config command.

MP-8000B-2# copy run startOverwrite the startup config file (y/n): yBuilding configuration...

MP-8000B-2 Zoning configurationAfter the configuration steps for the CMSH portion of the switchhave been completed and you have saved the configuration you willneed to complete the zoning configurations. At this point, the CNAsare installed into the Servers with the correct drivers loaded andphysically connected to the MP-8000B.

Since MP-8000B-2 is running in AG mode, all zoning will need to beperformed from the MDS-9513-2. It is strongly recommended you cutand paste the CLI to be used for zoning to minimize transcriptiontypes of errors when manually typing in the WWPNs. Plug thecorrect storage devices into the corresponding DCX, at which pointthey should log in to the switch.



The steps required to complete the zoning of the CNAs are:

1. Log in to the switch.

2. Create applicable zones.

a. zone name <ZONE_NAME> vsan <VSAN ID>

b. member pwwn <PWWN_1>

c. member pwwn <PWWN_2>

3. Create zoneset.

a. zone set name <ZONE_SET_NAME> vsan <VSAN ID>

b. member <ZONE_NAME>

4. Active zoneset.

a. zoneset activate name <ZONE_SET_NAME> vsan <VSANID>

5. Distribute zoneset.

a. zoneset distribute full vsan <VSAN>

6. Confirm zoneset activation.

a. show zoneset active vsan <VSAN>

7. Save configuration.

a. Copy running startup

Each of these steps are further detailed as follows:

1. Log in to the switch 172.23.185.22.

2. Create applicable zones:

MDS-9513-1# config terminalMDS-9513-1 (config)# zone name host1_CNA2 vsan 100MDS-9513-1 (config-zone)# member pwwn 10:00:00:00:c9:3c:f7:c0MDS-9513-1 (config-zone)# member pwwn 50:06:04:8A:D5:2E:69:79MDS-9513-1(config)# zone name host2_CNA2 vsan 100MDS-9513-1 (config-zone)# member pwwn 21:00:00:1B:32:0A:C0:B8MDS-9513-1 (config-zone)# member pwwn 50:06:04:82:D5:2E:69:59

3. Create zoneset:

MDS-9513-1 (config)# zoneset name FCoE_100 vsan 100MDS-9513-1 (config-zoneset)# member host1_CNA2MDS-9513-1 (config-zoneset)# member host2_CNA2


88


4. Activate zoneset:

MDS-9513-1 (config-zoneset)# zoneset activate name FCoE_100 vsan 100Zoneset activation initiated. check zone statusMDS-9513-1(config)# exit

5. Distribute zoneset:

MDS-9513-1 (config)# zoneset distribute full vsan 100

6. Confirm zoneset activation:

MDS-9513-1(config)# show zoneset active vsan 100zoneset name FCoE_100 vsan 100

zone name host1_cna2 vsan 100*fcid 0x920006 [pwwn 10:00:00:00:c9:3c:f7:c0]*fcid 0x100400 [pwwn 50:06:04:8a:d5:2e:69:79]

zone name host2_cna2 vsan 100* fcid 0x920033 [pwwn 21:00:00:1b:32:0a:c0:b8]* fcid 0x100300 [pwwn 50:06:04:82:d5:2e:69:59]

7. Save configuration:

MDS-9513-1(config)# exitMDS-9513-1# copy running startup-config

Configuring the Windows hostTo configure the Windows host, refer to the following documentation,available at EMC Online Support at https://support.emc.com:

◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel andiSCSI Host Bus Adapters (HBAs) and Converged Network Adapters(CNAs) in the Windows Environment P/N 300--001-164

◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel HostBus Adapters (HBAs) and Converged Network Adapters (CNAs) in theWindows Environment P/N 300-001-157

◆ EMC Host Connectivity Guide for Windows, P/N 300-000-603

Configuring the Linux hostTo configure the Linux host, refer to the following documentation,available at EMC Online Support at https://support.emc.com:

◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel andiSCSI Host Bus Adapters (HBAs) and Converged Network Adapters(CNAs) for the Linux Environment, P/N 300-002-803



◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel HostBus Adapters (HBAs) and Converged Network Adapters (CNAs) for theLinux v2.6.x Kernel Environment and the v8.x-Series Driver, P/N300-002-583

◆ EMC Host Connectivity Guide for Linux, P/N 300-003-805

Configuring the VMware hostTo configure the VMware host, refer to the following documentation,available at EMC Online Support at https://support.emc.com:

◆ EMC Host Connectivity Guide for VMware ESX Server,P/N 300-002-304

Configuring EMC storageOnce the switch, hosts, and adapters have been configured, thestorage will need to be configured to all each initiator to access anumber of LUNs.


90


MP-8000B firmware upgrade procedureDownloading firmware to the MP-8000B from a remote FTP server isdone from the Fibre Channel (FOS) side of the switch.

This process is used to upgrade the firmware on the MP-8000B. Thisprocedure is completed only from the FOS and will upgrade both theFOS and CMSH portions of the switch. The following is an exampleof how to complete this operation.

IMPORTANT

This procedure will reloaded the switch, during which time alltraffic will be disrupted for 3-10 minutes.

AssumptionsThis procedure assumes the following:

◆ The firmware files are stored in the outgoing/firmware directoryof an FTP server at the IP address of 168.159.216.19

◆ The name of the image is v6.1.2_cee1

Upgrading firmware

Note: The text in bold is input by the user.

To upgrade firmware, complete the following steps:

1. From the CMSH, create a backup of your existing configurationfile. (Ensure the running configuration is correct.)

Note: This command needs to be run from the CMSH only; the FOS willsave any configuration changes automatically.

MP-800B-1# copy running-config startup-config

2. Back up the existing configuration file. This can be done throughthe ftp server using the following format:

copy startup-config ftp://[username[:password]@server/path]



MP-800B-1# copy startup-configftp://anonymous:[email protected]@168.159.216.19/MP-8000B-1_startup-config

Building configuration...3. Download and install new firmware.

MP-800B-1# firmwaredownloadServer Name or IP Address: 168.159.216.19User Name: anonymousFile Name: v6.1.2_cee1Network Protocol(1-auto-select, 2-FTP, 3-SCP) [1]: 2Password: [email protected] system settings for firmwaredownload...Server IP: <ip address of FTP sever>, Protocol IPv4System settings check passed.

You can run firmwaredownloadstatus to get the statusof this command.

This command will cause a cold/disruptive reboot and will require that existingtelnet, secure telnet or SSH sessions be restarted.

Do you want to continue [Y]: yFirmware is being downloaded to the switch. This step may take up to 30 minutes.Preparing for firmwaredownload...Start to install packages...dir ##################################################ldconfig ##################################################glibc ##################################################glibc-linuxthreads ##################################################bash ##################################################NOTE: output truncated

prom-8548 ##################################################Please avoid powering off the system during prom update.awk ##################################################Removing unneeded files, please wait ...Finished removing unneeded files.All packages have been downloaded successfully.Firmware has been downloaded to the secondary partition of the switch.HA Reboot not supportedRebooting system now..

Broadcast message from root (pts/0) Tue Aug 25 17:55:19 2009...

The system is going down for reboot NOW !!

4. Reconnect to the switch after the reload has completed. Verify thestatus of Firmware download.

MP-8000B-1# firmwaredownloadstatus[1]: Tue Aug 25 17:50:53 2009

MP-8000B firmware upgrade procedure 91

92


Firmware is being downloaded to the switch. This step may takeup to 30 minutes.

[2]: Tue Aug 25 17:55:18 2009

Firmware has been downloaded to the secondary partition of theswitch.

[3]: Tue Aug 25 17:56:07 2009

The firmware commit operation has started. This may take up to10 minutes.

[4]: Tue Aug 25 17:56:48 2009

The commit operation has completed successfully.

[5]: Tue Aug 25 17:56:48 2009

Firmwaredownload command has completed successfully. Usefirmwareshow to verify the firmware versions.

5. Verify the status of Firmware download.

MP-800B-1# firmwareshow

Appl Primary/Secondary Versions------------------------------------------FOS v6.1.2_cee1

v6.1.2_cee1



Troubleshooting the MP-8000BThere are two sides to the MP-8000B, each briefly discussed in thissection:

◆ The Fibre Channel side of the switch that uses the Fibre ChannelOS (FOS). Commands and information are included in “FibreChannel FOS” on page 93.

◆ The Ethernet side of the switch that uses the CMSH (CEEManagement Shell). Commands and information are included in“CMSH” on page 99.

Troubleshooting information is provided for the following:

◆ “Connectivity troubleshooting” on page 102

◆ “VLAN troubleshooting” on page 102

◆ “DCBX troubleshooting” on page 102

◆ “FCoE troubleshooting” on page 103

◆ “QoS troubleshooting” on page 103

◆ “Link Aggregation (LAG) troubleshooting” on page 103

Fibre Channel FOSThis section includes command information for the Fibre ChannelFOS.

help This command, without a parameter, lists all available commandswithin the FOS where help is available. With a parameter, it displayshelp about the specified command only.

The following is an example of how this command works.

MP-8000B:admin> help fcoe

Administrative commands include:

fcoe(1m) NAMEfcoe - Manages and displays FCoE configuration.

SYNOPSISfcoe --cfgshow [port]

fcoe --disable port

Troubleshooting the MP-8000B 93

94


fcoe --enable port

fcoe --loginshow port

fcoe --fcmapset -vlan vid fcmapid

fcoe --fcmapunset -vlan vid

fcoe --fipcfg -advintvl intvl

fcoe --fipcfgshow

fcoe --resetlogin [-teport slot/port | -device wwn ]

fcoe --help

supportsave This command retrieves information on the switch involving zoning,performance, reboots, CP failovers, fabric-wide problems, etc.

If no operands are provided, then the command is interactive.

Note: The Supportsave command is a minimum step in troubleshooting andis normally required with any type of escalation process.

MP-8000B-1:admin> supportsave

This command collects RASLOG, TRACE, supportShow, core file,FFDC data and other support information and then transfer it to aFTP/SCP server or a USB device. This operation can take severalminutes.

Note: SupportSave transfers existing trace dump file first, then automaticallygenerates and transfers the latest one. There will be two trace dump filestransferred after this command.

OK to proceed? (yes, y, no, n): [no] yesHost IP or Host Name: [x.x.x.x]User Name: [ftp username]Password: [ftp password]Protocol (ftp or scp): ftpRemote Directory: [ftp directory of server]

Below is an example of the output:

Saving support information for switch:MP-8000B-1, module:CONSOLE0......rtSave_files/MP-8000B-1-S0-200908111724-CONSOLE0.gz: 6.05 kB 180.38 kB/sSaving support information for switch:MP-8000B-1, module:RASLOG......tSave_files/MP-8000B-1-S0-200908111724-RASLOG.ss.gz: 41.34 kB 1.04 MB/sSaving support information for switch:MP-8000B-1, module:TRACE_OLD......les/MP-8000B-1-S0-200908111725-old-tracedump.dmp.gz: 1.60 MB 10.26 MB/s



Saving support information for switch:MP-8000B-1, module:TRACE_NEW......les/MP-8000B-1-S0-200908111725-new-tracedump.dmp.gz: 1.69 MB 10.29 MB/sSaving support information for switch:MP-8000B-1, module:ZONE_LOG......ave_files/MP-8000B-1-S0-200908111725-ZONE_LOG.ss.gz: 26.71 kB 846.44 kB/sSaving support information for switch:MP-8000B-1, module:RCS_LOG......Save_files/MP-8000B-1-S0-200908111725-RCS_LOG.ss.gz: 873.00 B 23.39 kB/s

(output truncated)

switchshow This command prints switch and port status. This gives output of theFC interfaces and Virtual Fibre Channel interfaces.

MP-8000B-1:admin> switchshow

switchName: MP-8000B-1switchType: 76.7switchState: OnlineswitchMode: NativeswitchRole: SubordinateswitchDomain: 1switchId: fffc01switchWwn: 10:00:00:05:1e:76:77:00zoning: ON (FCoE)switchBeacon: OFF

Area Port Media Speed State Proto=====================================0 0 id N4 Online FC F-Port 50:06:04:82:D5:2E:69:791 1 id N4 Online FC F-Port 50:06:04:82:D5:2E:69:492 2 id N2 Online FC F-Port 50:06:04:8A:D5:2E:69:793 3 id N8 No_Light FC4 4 id N2 No_Light FC5 5 id N2 Online FC F-Port 50:06:04:82:D5:2E:69:596 6 id AN No_Sync FC7 7 id N4 No_Light FC E-Port 10:00:00:05:1e:04:31:67(upstream)(Trunk master)8 8 -- 10 Online FCoE F-Port 2 NPIV public}9 9 -- 10 Online FCoE F-Port 2 NPIV public}

10 10 -- 10 Online FCoE F-Port 2 NPIV public} FCoE Ports11 11 -- 10 Online FCoE F-Port 3 NPIV public} (VF_PORTS)12 12 -- 10 Online FCoE F-Port 2 NPIV public}13 13 -- 10 Online FCoE F-Port 3 NPIV public}

Note: In the above output from the switchshow command, ports 0 through 7are the Fibre Channel interfaces. Ports 8 through 13 are the six 10gigFCoE_Ports connected between the Fibre Channel switch and the FCoEGateway. Each one of the 6 FCoE-Ports translate to 4 ports on the Ethernetswitch. See Figure 16 on page 96.


96


Figure 16 switchshow output

nsshow Use this command to display local Name Server information aboutdevices connected to this switch. The output from this command willinclude all Fibre Channel and FCoE devices logged in to the switch.This command allows you to correlate the WWPN and FCoE-Port,which you can use to find the Ethernet interface bound to a particularFCoE device. The following is an example of the nsshow command.

MP-8000B-1:admin>nsshow

{Type Pid COS PortName NodeName TTL(sec)N 010000; 3; 50:06:04:82:D5:2E:69:79; 50:06:04:82:D5:2E:a4:00; na

FC4s: FCPPortSymb: [94] "SYMMETRIX::000192601257::SAF- 8fB::FC::5874_173+::EMUL

B80F0000 2C1385CD 647680 07.29.09 13:22"NodeSymb: [38] "SYMMETRIX::000192601257::FC::5874_173+"Fabric Port Name: 20:00:00:05:1e:76:77:00Permanent Port Name: 50:06:04:82:D5:2E:69:79

EthernetSwitch

CMSH

FCoEGateway

FC Switch

FOS

Interfaces t 0/0 - 0/3






0

8 x 8 G FCFE Ports

SYM-002070

24 x 10 GEFE Ports

6 x 10 GFCoE-Ports(VF_Ports)

7

8

9

10

11

12

13



Port Index: 0Share Area: NoDevice Shared in Other AD: NoRedirect: No

N 010100; 3; 50:06:04:82:D5:2E:69:49;50:06:04:82:D5:2E:ac:00; naFC4s: FCPPortSymb: [94] "SYMMETRIX::000192601259::SAF- 8gB::FC::5874_173+::EMUL

B80F0000 2C1385CD 647680 07.29.09 13:22"NodeSymb: [38] "SYMMETRIX::000192601259::FC::5874_173+"Fabric Port Name: 20:01:00:05:1e:76:77:00Permanent Port Name: 50:06:04:82:D5:2E:69:49


N 010804; 3;10 10:00:00:00:c9:3c:f8:3c;20:00:00:05:1e:9a:95:14; naFC4s: FCPPortSymb: [89] "BR-1020 | 2.1.0.005 | WIN-9I2KM0KQSSS | Windows Server (R)

2008 Standard | Service Pack 1"Fabric Port Name: 20:08:00:05:1e:76:77:00Permanent Port Name: 20:08:00:05:1e:76:77:00


N 010902; 3; 10:00:00:00:c9:3c:f7:c0; 20:00:00:00:c9:3c:f7:c0; naFC4s: FCPPortSymb: [89] "BR-1020 | 2.1.0.005 | WIN-9I2KM0KQSSS | Windows Server (R)

2008 Standard | Service Pack 1"Fabric Port Name: 20:09:00:05:1e:76:77:00Permanent Port Name: 20:09:00:05:1e:76:77:00


In the above output, each device successfully logged in to the switchas shown. The first two devices are logged in with Port Indexes of 0and 1, which translates to Fibre Channel ports of 0 and 1.

The second two devices are Brocade CNAs (BR-1020) and areshowing Port Indexes of 8 and 9, which are FCoE ports bound toEthernet interfaces. To see what Ethernet interfaces each device isbound to, complete the fcoe --loginshow [port] command, explainednext.


98


fcoe --loginshow[port]

The [port] is the Port Index portion of the nsshow commandexplained previously. This is only valid for the FCoE-Ports (8 - 13). Ifyou try to complete this command using a Fibre Channel port youwill receive the following error:

error: loginshow failed

Invalid slot/port number

MP-8000B-1:admin> fcoe -loginshow 8Number of connected devices: 1================================================================================Peer Type Connect Info Device WWN Device MAC SessionMAC FCoE Port MAC Te port

================================================================================FCOE_DEVICE Direct 10:00:00:00:c9:3c:f8:3c 00:00:c9:3c:f8:3c

0e:fc:00:01:08:04 00:05:1e:76:77:00 Te 0/0

MP-8000B-1:admin> fcoe --loginshow 9Number of connected devices: 1================================================================================Peer Type Connect Info Device WWN Device MAC SessionMAC FCoE Port MAC Te port

================================================================================FCOE_DEVICE Direct 10:00:00:00:c9:3c:f7:c0 00:00:c9:3c:f7:c0

0e:fc:00:01:09:02 00:05:1e:76:77:06 Te 0/4

(output truncated)

Note: The output from this command is showing only one successful login toeach of the FCoE ports, 8 and 9. Te 0/0 is bound to FCoE-Port 8 and Te 0/4 isbound to FCoE-Port 9.

fcoe--cfgshow [port] — Displays the configuration of a specifiedembedded FCoE port. If a port is not specified, the commanddisplays all port configurations.

This command gives a quick representation of how many devices arelogged in to a specific FCoE port.

MP-8000B-1:admin> fcoe --cfgshowUser Port Status Port WWN DeviceCount Port Type Peer MAC================================================================================8 ENABLED 20:08:00:05:1e:76:77:00 1 FCOE F-Port9 ENABLED 20:09:00:05:1e:76:77:00 1 FCOE F-Port10 ENABLED 20:0a:00:05:1e:76:77:00 0 FCOE F-Port11 ENABLED 20:0b:00:05:1e:76:77:00 0 FCOE F-Port12 ENABLED 20:0c:00:05:1e:76:77:00 0 FCOE F-Port13 ENABLED 20:0d:00:05:1e:76:77:00 0 FCOE F-Port



Note: In the above command there was no FCoE port specified, therefore theoutput is showing all of the FCoE ports. In this output, ‘User Port’ is the FCoEport, the ‘Port WWN’ is the address associated to the FCoE port, and the‘DeviceCount’ is the number of devices logged into the FCoE port. In thiscase you can see that the ‘User Ports’ 8 and 9 each have one successful login,which correlates to the previous fcoe–loginshow command.

fcoe – enable/disable <port> — Enable or disable the specified FCoEport.

MP-8000B-1:admin> fcoe -disable 13

The FCoE port 13 is now disabled.

If a FCoE port is disabled it will not perform any of the FCoE portfunctions and will show up in the above outputs as disabled, asshown in the below example using the fcoe-cfgshow command:

MP-8000B-1:admin> fcoe --cfgshowUser Port Status Port WWN DeviceCount Port Type Peer MAC================================================================================8 ENABLED 20:08:00:05:1e:76:77:00 1 FCOE F-Port9 ENABLED 20:09:00:05:1e:76:77:00 1 FCOE F-Port10 ENABLED 20:0a:00:05:1e:76:77:00 0 FCOE F-Port11 ENABLED 20:0b:00:05:1e:76:77:00 0 FCOE F-Port12 ENABLED 20:0c:00:05:1e:76:77:00 0 FCOE F-Port13 DISABLED 20:0d:00:05:1e:76:77:00 0 FCOE F-Port

Note: The above output is an example only and is not relevant to theconfigurations used within this document.

CMSHTo help in the troubleshooting process of the CEE Management Shell(CMSH), you need to have a basic understanding of how this shellactually works. This section contains information on some of the keyfeatures and commands that will help you when working with thedifferent modes within the CMSH.

Key features of the CMSH include:

◆ Commands are NOT case sensitive.

◆ Supports abbreviated commands. you need to enter only enoughcharacters for the switch to recognize the command as unique.

• MP-8000B-1(config)# show interface brief - sho int bri


100


• MP-8000B-1(config)# configure terminal - config t

◆ Configuration commands show errors only if command fails or ifthere is an error in syntax.

◆ Configuration commands have a [no] option to negate or removeconfiguration.

• MP-8000B-1(config)# no int VLAN 100

Key commands include the following:

config t Enter configuration mode.

exit Go to previous mode, back one level from current mode.

end Exit completely from configuration mode.

do Execute a command from configuration mode.

MP-8000B-1(config)# do sho run

fos Execute a FOS command from within the CMSH

MP8000B-1# fos nsshow

To execute a fos command from within the configuration mode youwill also need to enter the do command.

MP-8000B-1(config)# do fos nsshow

? Provides a list of available commands from the current mode. Thiscan also be used to help finish a command string.

show The following are show command options:

MP8000B-1# show ?

calendar Show the calendar information

cee QoS Converged Enhanced Ethernet (CEE)

clock Show clock information

cores Display Core dump information

debug Debugging functions (see also 'undebug')

environment Show environment information

file Displays the contents of the file

history Display the session command history



interface The layer2 interfaces

ip Internet Protocol (IP)

lacp LACP commands

line Display information about terminal lines

lldp Link Layer Discovery Protocol

logging Display raslog messages

mac MAC Access-list information

mac-address-tableDisplay Mac Address table information

port Port commands

port-channel LACP port-channel

power Show power information

privilege Show current privilege level

processes Show processes information

qos Quality of Service (QoS)

rmon Remote Monitoring Protocol (RMON)

running-config Current Operating configuration

spanning-tree Display spanning tree information

startup-config Contents of startup configuration

statistics Accounting information

system Show system information

tech-support Show system information for Tech-Support

users Display information about terminal lines

version Display FOS version

vlan Vlan commands

zlog Debugging functions (see also 'undebug')


102


Connectivity troubleshootingThe following commands help troubleshoot connectivity issues.

VLAN troubleshootingThe following commands help troubleshoot VLAN issues.

DCBX troubleshootingThe following commands help troubleshoot DCBX issues.

MP8000B-1# show logging Show logging messages

MP8000B-1# sho running-config Show current configuration

MP8000B-1# sho system Show system information

MP8000B-1# sho ip interface brief Summary of IP status and configuration

MP8000B-1# sho interface switchport Display the modes of the Layer2 interfaces

MP8000B-1# sho interface [x/y] Display specific Layer2 interfaceconfiguration and statistics

MP8000B-1# sho mac-address-table Show entire Mac Address table

MP8000B-1# sho mac-address-table int [x/y] Show Mac Address for specific interface

MP8000B-1# sho mac-address-table vlan [x] Show Mac Address table for specific vlan

MP8000B-1# sho vlan [x] Show specific vlan

MP8000B-1# sho vlan brief Show all vlan information

MP8000B-1# sho vlan fcoe Show FCoE Vlan and active ports

MP8000B-1# sho lldp interface t [x/y] Show interface status and configuration

MP8000B-1# sho lldp neighbors Show neighbor information

MP8000B-1# sho lldp neighbors detail Show neighbor detail information byinterface



FCoE troubleshootingThe following commands help troubleshoot FCoE issues.

QoS troubleshootingThe following commands help troubleshoot QoS issues.

Link Aggregation (LAG) troubleshootingThe following commands help troubleshoot Etherchannel (LAG)issues.

MP8000B-1# sho vlan classifier group Show vlan classifier group information

MP8000B-1# sho vlan classifier rule Show vlan classifier rule information

MP8000B-1# sho vlan classifier interface group Show vlan classifier group interfaces

MP8000B-1# sho cee maps Show active CEE Map information

MP8000B-1# sho qos flowcontrol interface t [x/y] Show Per Priority Pause Flow

MP8000B-1# sho qos queue interface t [x/y] Show queueing information

MP8000B-1# sho port-channel Show Portchannel information

MP8000B-1# sho lacp counter ShowLACP counters information


104


ED-DCX-BThe topology discussed in this section, as illustrated in Figure 17 onpage 105, shows the PB-DCX-24FCoE module plugged into aConnectrix ED-DCX-B switch, enabling FCoE.

With FOS version v6.4.1_fcoe the DCX/DCX-4s (model numbersED-DCX-B and ED-DCX-4S-B) are capable of running FCoE throughthe FCoE-capable line cards (PB-DCX-24FCoE). The line cards are 24port FCoE 10 Gb/s Ethernet modules that plug directly into the DCXswitches. In either switch, up to four modules are usable within asingle chassis.

Although there are other versions of FOS that will run FCoE,v6.4.1_fcoe is the only one capable of High Availability between CPfailovers and is the only current version that supports ISSU (InService Software upgrade) for FCoE traffic.


◆ “Topology” on page 105

◆ “Compatibility” on page 105

◆ “Supported topologies” on page 106

◆ “Installing the PB-DCX-24FCoE” on page 106

◆ “Configuring the PB-DCX-24FCoE” on page 107

◆ “Useful commands” on page 109



TopologyThe topology illustrated in Figure 17 shows the PB-DCX-24FCoEmodule plugged into an ED-DCX-4S-B switch, enabling FCoE.

Figure 17 ED-DCX-4S-B switch and PB-DCX-24FCoE module topology example

CompatibilityThe DCX chassis is not capable of supporting other intelligentapplication blades while running the PB-DCX-24FCoE modules.These include:

◆ Fibre Channel application blade PB-48K-AP4-18◆ Fibre Channel router blade PB-48K-18i◆ Fibre Channel over IP blade PB-DCX-FX8-24◆ Storage encryption blade PB-DCX-16EB

ED-DCX-B 105

106


Supported topologiesIn effect, the PB-DCX-24FCoE module is the MP-8000B stand aloneswitch with the sheet metal removed. Once installed into a DCXchassis, it has all of the same properties of the MP-8000B and isconfigured in the same manner.

The FCoE module supports all of the same topologies as theMP-8000B and will perform the same functions as the MP-8000B.

For more information on the supported topologies, refer to“MP-8000B supported features and topologies” on page 49.

Installing the PB-DCX-24FCoETo install the FCoE module(s) into a DCX chassis, complete thefollowing steps.

1. Install the module(s) into the DXC chassis.

2. Enable the Ethernet service.

DCX_4s_admin> fosconfig -enable ethswWARNING: This operation will enable the Ethernet

Switch Service on this switch.Would you like to continue [Y/N]: yEnabling the Ethernet Switch Service. Please wait ...

The Ethernet switch service has been enabled. Issue theslotpoweroff and slotpoweron commands on all of the bladeswith ID 74 to complete the process.

3. Verify the switch modules.

In the following example there are FCoE modules in slot 1 andslot 8.

DCX_4s_admin> slotshowDCX_4s_Edison:FID128:admin> slotshow

Slot Blade Type ID Status-----------------------------------

1 AP BLADE 74 FAULTY (9)2 SW BLADE 51 ENABLED3 CORE BLADE 46 ENABLED4 CP BLADE 50 ENABLED5 CP BLADE 50 ENABLED6 CORE BLADE 46 ENABLED7 SW BLADE 55 ENABLED8 AP BLADE 74 FAULTY (9)



4. Issue the slotpoweroff for all new FCoE modules.

DCX_4s_admin> slotpoweroff 1Slot 1 is being powered offDCX_4s_admin> slotpoweroff 8Slot 8 is being powered off

5. Issue the slotpoweron command for the FCoE modules.

DCX_4s_admin> slotpoweron 1Powering on slot 1DCX_4s_admin> slotpoweron 8Powering on slot 8

6. Verify the installation of FCoE modules was successful.

Note: It will take several minutes for the modules to come up andcomplete the POST diagnostics.

DCX_4s_admin> slotshowSlot Blade Type ID Status-----------------------------------

1 AP BLADE 74 ENABLED2 SW BLADE 51 ENABLED3 CORE BLADE 46 ENABLED4 CP BLADE 50 ENABLED5 CP BLADE 50 ENABLED6 CORE BLADE 46 ENABLED7 SW BLADE 55 ENABLED8 AP BLADE 74 ENABLED

The FCoE modules are now installed and ready to configure.

Configuring the PB-DCX-24FCoEThe configuration of the FCoE modules is identical to that of theMP-8000B. You need to enter the cmsh through the FOS side of theswitch. This will bring you to the Ethernet side of the switch whereyou will complete all Ethernet and FCoE configurations.

DCX_4s_admin> cmshDCX_4s_admin#

ED-DCX-B 107

108


Enhanced FCoEconfiguration

With v.6.4.1_fcoe, there are some provisioning steps that have beenoptimized and require fewer steps for interface configurations. Thefollowing parameters are now enabled by default and will not needto be configured manually.

◆ FCoE VLAN

• VLAN 1002 is now preconfigured.

This can be changed but only one FCoE vlan is allowed.

◆ CEE map

• cee-map default

This is now configured by default and cannot be deleted

◆ VLAN classifiers

• FCoE/FIP VLAN classifier

These parameters are now part of the default fcoe-map. The fcoe-mapapplies the FCoE-specific configurations to one place holder. With thefcoe-map, a user now only has to apply it to an interface using thefcoeport command, rather than configure each of the parametersseparately.

◆ LLDP

• LLDP - configured by default

– FCoE TLVs

Interfaceconfiguration

With the new parameters applied by default, the configuration stepson an interface have been greatly reduced. Complete the followingsteps to apply the FCoE-specific information to an interface using thenew fcoeport command.

1. Enter the configuration mode.

DCX_4s_admin# config tDCX_4s_admin(config)# int t 1/0DCX_4s_admin(conf-if-te-1/0)# fcoeportDCX_4s_admin(conf-if-te-1/0)# no shut

2. Verify the new configurations.

DCX_4s_admin(conf-if-te-1/0)# do sho run int t 1/0!interface TenGigabitEthernet 1/0fcoeportno shutdown!

The above interface is now ready for FCoE traffic.



Additionalconfiguruations

For more detailed steps and configuration options, refer to“Configuring the MP-8000B” on page 58.

Useful commandsThe following are some useful commands and applicable output.

Show running-config DCX_4s_admin# sho run!no protocol spanning-tree <- this will need to be

configured accordingly!cee-map default <- default cee-mappriority-group-table 1 weight 40 pfcpriority-group-table 2 weight 60priority-table 2 2 2 1 2 2 2 2!fcoe-map default <- fcoe mapfcoe-vlan 1002NOTE: output truncated!interface TenGigabitEthernet 1/0fcoeport <- command to apply fcoe-mapno shutdown

NOTE: output truncated

protocol lldp <- default lldp parametersadvertise dcbx-fcoe-app-tlvadvertise dcbx-fcoe-logical-link-tlv

Show cee maps DCX_4s_admin# sho cee mapsCEE Map default

Precedence 1Priority Group Table

1: Weight 40, PFC Enabled, TrafficClass 3, BW% 402: Weight 60, PFC Disabled, TrafficClass 6, BW% 6015.0: PFC Disabled15.1: PFC Disabled15.2: PFC Disabled15.3: PFC Disabled15.4: PFC Disabled15.5: PFC Disabled15.6: PFC Disabled15.7: PFC Disabled

Priority TableCoS: 0 1 2 3 4 5 6 7---------------------------------------------PGID: 2 2 2 1 2 2 2 2

ED-DCX-B 109

110


Enabled on the following interfacesTe 1/0

Show fcoe maps DCX_4s_admin# sho fcoe-mapfcoe-map default

cee-map : defaultfcoe-vlan : 1002interface(s) : Te 1/0


3

This chapter provides basic information, supported features,topologies, and detailed setup steps for Cisco Nexus Series switches.Model numbers and troubleshooting information are also provided.

Note: This section provides information for the Nexus 5596, 5548, 5020, and5010 switches. When the information presented is applicable to only specificswitches, the model number is used. Nexus 5000 is used when information isrelevant to all switches.

Note: The terms Nexus 4000 and Nexus 4001I are used interchangeably.

◆ Nexus Series switches............................................................................ 113◆ Nexus 6004 supported features and topologies................................. 123◆ Nexus 6001 supported features and topologies................................. 128◆ Nexus 2248PQ FEX supported features and topologies................... 133◆ Nexus 5000 supported features and topologies................................. 136◆ Nexus 2000 supported features and topologies................................. 141◆ Configuring 40GbE FCoE connectivity with Nexus 6000 Series..... 149◆ Nexus 5000 direct-connect topology ................................................... 173◆ Nexus 7000, Nexus 5000, and MDS 9500 series topology ................ 210◆ Nexus 5000/Nexus 2232PP topology.................................................. 242◆ Configuring Cisco VE_Ports................................................................. 272◆ Configuring Cisco FCoE NPV ports.................................................... 274◆ Complex FCoE topologies .................................................................... 277◆ Nexus Series switches firmware upgrade procedure ....................... 280◆ Troubleshooting the Nexus Series switches ....................................... 284◆ Virtual PortChannel ............................................................................... 293

Nexus Series SwitchesSetup Examples

Nexus Series Switches Setup Examples 111

112

Nexus Series Switches Setup Examples

Note: More detailed Fibre Channel over Ethernet information can be found inthe Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts andProtocols TechBook at http://elabnavigator.EMC.com, under the TopologyResource Center tab. Information in this TechBook includes FCoE andEthernet basics, EMC storage, RecoverPoint and Celerra MPFS as solutions inan FCoE environment, and troubleshooting basic FCoE and CEE problems.

Note: More details on Cisco Nexus switches can be found in the Non-EMCConnectrix Products Data Reference Manual, located on the E-Lab Navigator, inthe Topology Resource Center tab.




Nexus Series switchesThis section provides basic information for the following Nexusswitches, along with information on management and VPN Routingand Forwarding (VRF):



◆ “Nexus 2248PQ Fabric Extender (FEX)” on page 115





◆ “Nexus 2232PP” on page 119


◆ “VPN Routing and Forwarding (VRF)” on page 122

Note: Some of the information for the Nexus Series switches comes fromCisco documentation. For more detailed information about any Ciscoproduct, refer to http://www.cisco.com.

Nexus 6004The Cisco Nexus 6004 is a 4RU 10 and 40 Gigabit Ethernet switchoffering wire-speed performance for up to ninety-six 40 GigabitEthernet ports or three hundred eighty-four 10 Gigabit Ethernet ports(using QSFP breakout cables) for Ethernet and FCoE traffic with anoverall throughput of 7.68 Tb/s.

The Cisco Nexus 6004 offers 48 fixed ports of 40 Gigabit Ethernetwith the base of the chassis and 4 line-card expansion module (LEM)slots. Although the base chassis has 48 fixed 40 Gigabit Ethernetports, customers can purchase a twenty-four 40 Gigabit Ethernet portentry option to address the small-scale deployments. With twoadditional 12-port 40 Gigabit Ethernet software licenses, theremaining 24 ports can be enabled to use all 48 ports of 40 GigabitEthernet on the base chassis. Each LEM supports 12 ports of 40Gigabit Ethernet in a QSFP form factor. For example, by adding fourLEMs to the base chassis, 48 additional 40 Gigabit Ethernet ports can

Nexus Series switches 113

114


be achieved. Each 40 Gigabit Ethernet port can be split into four 10Gigabit Ethernet ports using QSFP breakout cables.

The Cisco Nexus 6004 Switch as a high-density 10 and 40 GigabitEthernet and FCoE platform can be deployed in multiple scenarios:direct-attach 10 and 40 Gigabit Ethernet server-access andhigh-density FEX aggregation deployments, leaf and spinearchitectures, or compact aggregation to build a scalable CiscoUnified Fabric across a diverse set of physical and virtual serverenvironments in data centers.

The Cisco Nexus 6004 supports up to 4 additional expansion modulesof the following offering:

N6K-C6004-M12: 12 ports of 40-Gb/s Gigabit Ethernet and FCoEports using a QSFP interface. These ports can also be configuredas breakout ports consisting of four 10GbE ports per physicalQSFP (up to 48 10GbE ports per LEM).

The supported features and topologies for the Nexus 6004 are listedin “Nexus 6004 supported features and topologies” on page 123.

For additional information regarding the Cisco Nexus 6004, refer tothe Cisco Nexus 6000 Series Switches on the Cisco website athttp://www.cisco.com.

Nexus 6001The Cisco Nexus 6001 is a 1RU 10 and 40 Gigabit Ethernet switchoffering wire-speed performance for up to sixty-four 10 GigabitEthernet ports (using Quad Small Form-Factor Pluggable [QSFP]breakout cables) for Ethernet and FCoE traffic, with an overallthroughput of 1.28Tbp/s.

The Cisco Nexus 6001 offers 48 fixed 10 Gigabit Ethernet EnhancedSmall Form-Factor Pluggable (SFP+) ports and four 40 GigabitEthernet QSFP+ ports. Each 40 Gigabit Ethernet port can be split intofour 10 Gigabit Ethernet ports using a QSFP breakout cable. TheCisco Nexus 6001 delivers low port-to-port latency of approximately1 microsecond and low jitter independent of packet size usingcut-through switching architecture and with features enabled.

The Cisco Nexus 6001 can be deployed in multiple scenarios -direct-attach 10 and 40 Gigabit Ethernet server access andhigh-density fabric extender aggregation deployments, leaf and spinedesigns, and compact aggregation - to build scalable Cisco Unified



Fabric across a diverse set of physical and virtual serverenvironments in the data center.


For additional information regarding the Cisco Nexus 6001, refer tothe Cisco Nexus 6000 Series Switches on the Cisco website athttp://www.cisco.com.

Nexus 2248PQ Fabric Extender (FEX)The Cisco Nexus 2248PQ is a 1RU 10 Gigabit Ethernet FabricExtender. It supports high-density 10 Gigabit Ethernet environmentswith 48 1/10 Gigabit Ethernet SFP+ host ports and 4 QSFP+ fabricports (16 x 10 GE fabric ports). The Cisco Nexus 2248PQ 10GE FabricExtender also supports FCoE and a set of network technologiesknown collectively as Data Center Bridging (DCB). These featuresallow support for multiple traffic classes over a lossless Ethernetfabric, thus enabling consolidation of LAN, storage area network(SAN), and cluster environments.

The Cisco Nexus 2248PQ Fabric Extenders connect to a parent CiscoNexus switch through their fabric links and behave like remote linecards for a parent Cisco Nexus 5000, Nexus 6000, or Nexus 7000Series Switch. Working in conjunction with Cisco Nexus switches, theCisco Nexus 2248PQ Fabric Extenders extend the capabilities andbenefits offered by the parent Cisco Nexus switch.

Like other Cisco Nexus Fabric Extenders (FEX), the Nexus 2248PQdoes not switch any frames locally. All frames are forwarded to theparent switch through its uplink ports and forwarded from there tothe Destination MAC Address. Because of this, the Nexus 2248PQmust remain connected to at least one Nexus series parent switch inorder to function. In addition to providing all switching, the parentNexus switch also downloads firmware to the Nexus 2248PQ. Allmanagement functions for the Nexus 2248PQ are performed on thesupported parent Nexus switch.

The supported features and topologies for the Nexus 2248PQ ESX arelisted in “Nexus 2248PQ FEX supported features and topologies” onpage 133.

For additional information regarding the Cisco Nexus 2248PQ FabricExtender (FEX), refer to the Cisco Nexus 2000 Series Fabric Extenderson the Cisco website at http://www.cisco.com.


116


Nexus 5596The Cisco Nexus 5596UP switch is a 2RU 10 Gigabit Ethernet, FibreChannel, and FCoE switch offering up to 1920 Gb/s of throughputand up to 96 ports. The switch has 48 unified ports and threeexpansion slots.

Unified ports support traditional Ethernet, Fibre Channel, and FibreChannel over Ethernet. Connectivity options include 1 Gb Ethernetand 10 Gig Ethernet with FCoe, and 1/2/4/8 Gb Native FibreChannel.

The Cisco Nexus 5596UP supports three expansion modules from thefollowing offerings:

◆ Ethernet module that provides sixteen 1 and 10 Gigabit Ethernetand FCoE ports using the SFP+ interface.

◆ Fibre Channel plus Ethernet module that provides eight 1 and 10Gigabit Ethernet and FCoE ports using the SFP+ interface, andeight ports of 8/4/2/1-Gb/s native Fibre Channel connectivityusing the SFP+/SFP interface.

◆ A unified port module that provides up to sixteen 1 and 10Gigabit Ethernet and FCoE ports using the SFP+ interface or up tosixteen ports of 8/4/2/1-Gb/s native Fibre Channel connectivityusing the SFP+ and SFP interface. The use of 1 and 10 GigabitEthernet or 8/4/2/1-Gbps Fibre Channel on a port is mutuallyexclusive but selectable for any of the 16 physical ports permodule.

◆ A Layer 3 module provides up to 160 Gb/s of Layer 3 forwardingcapability (240 mpps) that can be shared by all the I/O ports inthe chassis.


Nexus 5548The Nexus 5548 is a one rack-unit (1U), 10 GbE, Cisco Data CenterEthernet, and FCoE 1/2/4/8 Gb/s Fibre Channel switch built toprovide 960 Gb/s throughput with very low latency.

The Nexus 5548 has 32 fixed 1 GbE and 10 GbE ports, capable ofsupporting Cisco Data Center Ethernet and FCoE Small Form FactorPluggable Plus (SFP+).



Note: Expansion modules used in the Nexus 5020 and Nexus 5010 switchesare not compatible with the Nexus 5548 switch.

The switch has a serial console port and an out-of-band 10/100/1000Mb/s Ethernet management port. Two N+N redundant,hot-pluggable power supplies and two N+N redundant,hot-pluggable fan modules provide highly reliable front-to-backcooling.

Nexus 5548 delivers unified fabric at the network access layer, oredge, where servers connect to the LAN, SAN, and server clusters. Tointegrate with existing infrastructure, the Nexus 5548 Series providesnative Fibre Channel uplinks to facilitate connection with installedSANs and available SAN switches.


Nexus 5548UP The Nexus 5548UP is the same as a Nexus 5548 switch, except that allof the interfaces are unified ports (UPs). The Nexus 5548UP is a 1rack-unit chassis that provides 32 fixed unified ports on the basechassis along with 1 expansion slot, scaling up to a total of 48 ports.

Nexus 5020The Nexus 5020 is a two rack-unit (2U), 10 GbE, Cisco Data CenterEthernet, and FCoE 1/2/4/8 Gb/s Fibre Channel switch built toprovide 1.04 Tb/s (terabits) throughput with very low latency.

The Nexus 5020 has 40 fixed 10 GbE ports capable of supportingCisco Data Center Ethernet, and FCoE Small Form Factor PluggablePlus (SFP+).

The switch has a serial console port and an out-of-band 10/100/1000Mb/s Ethernet management port. The switch is powered by 1+1redundant, hot-pluggable power supplies, and 4+1 redundant,hot-pluggable fan modules to provide highly reliable front-to-backcooling.



118



Nexus 5010The Nexus 5010 is a one rack-unit (1U), 10 GbE, Cisco Data CenterEthernet, and FCoE 11/2/4/8 Gb/s Fibre Channel switch built toprovide 520 Gb/s throughput with very low latency.

I/O consolidation using the Nexus 5010 is shown in Figure 20 onpage 136. It has 20 fixed 10 GbE ports capable of supporting CiscoData Center Ethernet, and FCoE Small Form Factor Pluggable Plus(SFP+).

The switch has a serial console port and an out-of-band 10/100/1000Mb/s Ethernet management port. The switch is powered by 1+1redundant, hot-pluggable power supplies, and 4+1 redundant,hot-pluggable fan modules to provide highly reliable front-to-backcooling.



Nexus 4000The Cisco Nexus 4000 is a 20-port, 10 G FCoE aware Ethernet switchmodule intended for use in IBM Blade Servers and is fully compliantwith the IEEE 802.1 Data Center Bridging (DCB) specification. Thisblade switch is also referred to as the Nexus 4001I Switch Module.

The term FCoE aware indicates that the switch is capable ofperforming FIP snooping and supports the creation of dynamicACLs. It also indicates that the Nexus 4000 does not contain a FibreChannel Forwarder (FCF) and as a result FCoE frames received fromCNAs connected to the Nexus 4000 that are destined to a native FCdevice must be forwarded to a switch that does contain an FCF beforethe FC frame can be de-encapsulated and forwarded onto the SAN.Because the CNAs are not directly connected to a switch that containsan FCF, the resulting topology is actually referred to as a DCB Cloud.



Due to the nature of a DCB Cloud topology, some of the setup stepswill differ from steps used in a direct connect environment.

Note: For more information on direct connect and DCB Cloud, refer tothe “FCoE Initialization Protocol (FIP)” section in the Fibre Channel overEthernet (FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook athttp://elabnavigator.EMC.com, under the Topology Resource Center tab.

Fourteen of the 10 G ports are used to connect to the server bladesand six of the ports are available to connect to the Nexus 5020.

The six external interfaces can all be bundled into a port channel.

The Nexus 4000 can be managed via Telnet.

Nexus 2232PPThe Cisco Nexus 2232PP is effectively an expansion blade for theNexus 5000 series switches. The Nexus 2232PP does not switch anyframes locally. All frames are forwarded to the parent 2232(s) throughits uplink ports and forwarded from there to the Destination MACAddress. Because of this, the Nexus 2232PP must remain connectedto at least one Nexus 5000 series switch in order to function.

In addition to providing all switching, the parent Nexus 5000 alsodownloads firmware to the Nexus 2232.

All management functions for the Nexus 2232PP are performed onthe parent Nexus 5000.

The Cisco Nexus 2232PP provides 32 10 Gb Ethernet and FibreChannel Over Ethernet (FCoE) Small Form-Factor Pluggable Plus(SFP+) server ports and eight 10 Gb Ethernet and FCoE SFP+ uplinkports in a one rack unit (1RU) form factor.

The Nexus 2232PP can be connected either in a straight-throughconfiguration, as shown in Figure 18 on page 120, or in across-connected configuration, as shown in Figure 19 on page 120.



120


Figure 18 Straight-through configuration example

Figure 19 Cross-connected configuration example

The obvious difference between the two configurations is the numberof parent switches.

N5k

N4k / N2k

N5k

N4k / N2k

Host

ICO-IMG-000854

N5k

N4k / N2k

N5k

N4k / N2k

Host

ICO-IMG-000855



◆ In the straight-through configuration, each Nexus 2232PP is onlyconnected to a single parent Nexus 5000 switch.

◆ In the cross-connected configuration, each Nexus 2232PP isconnected to two parent Nexus 5000 switches.

Another difference between the two configurations is the use of theVirtual Port Channel (vPC) feature on the Nexus 5000 switches.

◆ In the straight-through topology, vPC can be configured but is notrequired.

◆ In the cross-connected topology, vPC must be configured.

Note: Refer to “Virtual PortChannel” on page 293 for more information onvPC.

IMPORTANT

FCoE is not currently supported and will not function properly in across-connected topology.

For additional information on the Nexus 5000/2232 refer to the DataCenter Access Design with Cisco Nexus 5000 Series Switches and 2000Series Fabric Extenders and Virtual Port Channels document, located athttp://www.cisco.com.

ManagementThe management tools used to manage FCoE and non-FCoEenvironments are similar. In all cases, the FC portion of the CNAappears as a FC adapter to the OS, and the IP portion appears as a 10GbE adapter. Other management tools include:

◆ Fabric Manager (FM)• FM will properly display

– FCoE interface– FC interface– CNA logins

◆ EMC Ionix™ Network Configuration Manager (NCM)• Discover/Identify• Configuration management• Device access• Credential management


122


◆ Ethernet Uplink Interface will be shown in Data Center NetworkManager (DCNM)

◆ Device Manager

◆ CLI

◆ SNMP

VPN Routing and Forwarding (VRF)VPN Routing and Forwarding (VRF) is used on the managementinterface of the Nexus 5000. VRF increases functionality by allowingmultiple instances of a routing table to co-exist within the sameNexus 5000 at the same time. It also increases functionality byallowing network paths to be segmented without using multipledevices.

VRF acts like a logical router, but while a logical router may includemany routing tables, a VRF instance uses only a single routing table.

If you are familiar with SAN-OS, the presence of VRF in NX-OS mayrequire that you slightly alter the commands you are used to whenperforming actions involving the Management interface such ascopying files and pinging other IP addresses. Refer to “Nexus Seriesswitches firmware upgrade procedure” on page 280 for an exampleof using VRF management.



Nexus 6004 supported features and topologiesThe supported interfaces, features and topologies have been brokendown into multiple areas, and are discussed in this section. Actualreference topologies are provided in “Nexus 6000 Series and FabricExtenders topologies” on page 143.

InterfacesThis section contains information on the following interfaces:

◆ “Physical interfaces” on page 123

◆ “FCoE interfaces” on page 124

◆ “Ethernet Fabric Extension interfaces and technologies” onpage 124

◆ “Ethernet end-device interfaces” on page 124

◆ “FC interfaces” on page 125

Physical interfaces The following interfaces speed configurations are supported:

◆ 40GbE (True 40GbE)

◆ 40GbE (4x10GbE)

◆ 10GbE

The following optic types are supported:

◆ QSFP-40G-SR4 (MMF – 850nm)

◆ QSFP-40G-CSR4 (MMF – 850nm)

◆ QSFP-40G-LR4 (SMF – 1310nm)

The following cable types are supported:

◆ 40G QSFP+ to 4x 10G SFP+ fibre breakout (1xMTP-to-4xLC)

◆ 40G QSFP+ to 4x 10G SFP+ copper breakout

◆ 40G QSFP+ direct-attach copper (Twinax CU 1M, 3M, 5M -passive)

◆ 40G QSFP+ direct-attach copper (Twinax CU 7M, 10M - active)

Refer to the Cisco documentation on the Cisco website athttp://www.cisco.com for a current and fully updated list ofsupported optics and cables.

Nexus 6004 supported features and topologies 123

http://www.cisco.com


http://support.emc.com


https://elabnavigator.emc.com/

124


FCoE interfaces The following interfaces are supported:

◆ Native FCoE CNA directly connected to Cisco Nexus 6004 QSFP+via breakout cable (4x10GbE)

◆ Native FCoE Storage directly connected to Cisco Nexus 6004QSFP+ via breakout cable (4x10GbE)

• EMC VNX Series

• EMC VMAX Series

◆ FC Enhanced port types: VE, TE and VF

Note: Refer to the EMC Support Matrix available through E-LabInteroperability Navigator at http://elabnavigator.EMC.com, under thePDFs and Guides tab for a current list of supported 10GbE iSCSI, FCoE andCNA adapters.

Ethernet FabricExtension interfaces

and technologies The following are supported:

◆ Cisco Adapter FEX

◆ Cisco Data Center Virtual Machine FEX (VM-FEX)

◆ Support for up to 24 fabric extenders (Layer 2) with each CiscoNexus 6004 switch

◆ Cisco 40GbE Line Expansion Modules (LEM)

Note: Refer to the EMC Support Matrix available through E-LabInteroperability Navigator at http://elabnavigator.EMC.com, under thePDFs and Guides tab for a current list of supported fabric extenders (FEX).

Ethernet end-deviceinterfaces

The following interfaces are supported:

◆ 10GbE CNA directly connected to Cisco Nexus 6004 QSFP+ viabreakout cable (4x10GbE)

◆ 10GbE iSCSI directly connected to Cisco Nexus 6004 QSFP+ viabreakout cable (4x10GbE)

◆ 10GbE FCoE and iSCSI targets directly connected to Cisco Nexus6004 QSFP+ via breakout cable (4x10GbE)

• EMC VNX Series

• EMC VMAX Series





◆ 10GbE NICs, which adhere to industry standards, via QSFP+breakout cable (4x10GbE)

◆ PortChannels (PC)

◆ Virtual Port Channels (vPC)

◆ NIC Teaming

FC interfaces


◆ Direct connect FC Interfaces are not supported on the Nexus 6004

◆ FC Enhanced port types: VE, TE and VF

FeaturesThis section lists the information on the following features.

◆ “Data Center Bridging (DCB)” on page 125

◆ “FCoE ” on page 125

◆ “Ethernet (Layer-2)” on page 126


Data Center Bridging(DCB)

◆ CEE- and IEEE-compliant PFC (per-priority Pause frame support)

◆ PFC link distance support: 300m

◆ CEE-compliant DCBX Protocol

◆ CEE- and IEEE-compliant Enhanced Transmission Selection

FCoE

◆ T11 standards-compliant FCoE (FC-BB-5)

◆ T11 FCoE Initialization Protocol (FIP) (FC-BB-5)

◆ Any 10 or 40 Gigabit Ethernet port configurable as FCoE

◆ Fibre Channel forwarding (FCF)

◆ Fibre Channel enhanced port types: VE, TE, and VF

◆ Direct attachment of FCoE targets

◆ Up to 32 virtual SANs (VSANs) per switch

◆ Fabric Device Management Interface (FDMI)


126


◆ Fibre Channel ID (FCID) persistence

◆ In-order delivery

◆ Port tracking

◆ Cisco FCoE_NPV technology

◆ N-port identifier virtualization (NPIV)

◆ Fabric services: Name server, registered state change notification(RSCN), login services, and name-server zoning

◆ Per-VSAN fabric services

◆ Cisco Fabric Services

◆ Fabric Shortest Path First (FSPF)

◆ Storage Management Initiative Specification (SMI-S)

◆ Boot from SAN over vPC and Enhanced vPC (EvPC)

◆ FCP

◆ Native Interop Mode 2 & 3

◆ VSAN trunking

◆ Cisco NPV technology

Ethernet (Layer-2) ◆ Layer 2 switch ports and VLAN trunks

◆ IEEE 802.1Q VLAN encapsulation

◆ Rapid Per-VLAN Spanning Tree Plus (PVRST+) (IEEE 802.1wcompatible)

◆ Multiple Spanning Tree Protocol (IEEE 802.1s) instances: 64instances

◆ Spanning Tree PortFast

◆ Spanning Tree root guard

◆ Spanning Tree Bridge Assurance

◆ Cisco EtherChannel technology (up to 16 ports per EtherChannel)

◆ Cisco vPC technology

◆ vPC configuration synchronization

◆ Link Aggregation Control Protocol (LACP): IEEE 802.3ad

◆ Jumbo Frames on all ports (up to 9216 bytes)

◆ Pause frames (IEEE 802.3x)



◆ Storm control (unicast, multicast, and broadcast)

◆ Private VLANs

◆ Private VLAN over trunks (isolated and promiscuous)

◆ Private VLANs over vPC and EtherChannels

◆ VLAN Remapping

◆ Cisco FabricPath

◆ PC and vPC+ with FabricPath



◆ Support for up to 24 fabric extenders (Layer 2) with each CiscoNexus 6004 platform

Ethernet (Layer-3) ◆ Layer 3 interfaces: Routed ports on Cisco Nexus 6004 platforminterfaces, switched virtual interface (SVI), PortChannels,subinterfaces, and PortChannel subinterfaces

◆ Sixteen-way Equal-Cost Multipathing (ECMP)*

◆ Routing protocols: Static, RIPv2, EIGRP, OSPFv2, and BGP

◆ IPv6 routing protocols: Static, OPFv3, BGPv6, and EIGRPv6

◆ IPv6 VRF-Lite

◆ HSRP and VRRP

◆ ACL: Routed ACL with Layer 3 and 4 options to match ingressand egress ACL

◆ Multicast: PIMv2 sparse mode, Source Specific Multicast (SSM),BiDir PIM, Multicast Source Discovery Protocol (MSDP), IGMPv2 and v3, and Multicast VLAN Registration (MVR)

◆ VRF: VRF-Lite (IP VPN); VRF-aware unicast; and BGP-, OSPF-,RIP-, and VRF-aware multicast

◆ URFP with ACL; strict and loose modes

◆ Jumbo Frame support (up to 9216 bytes)

◆ Support for up to 24 fabric extenders on each Cisco Nexus 6004


128


Nexus 6001 supported features and topologiesThe supported interfaces, features and topologies have been brokendown into multiple areas, and are discussed in this section. Actualreference topologies are provided in “Nexus 6000 Series and FabricExtenders topologies” on page 143.




◆ “Ethernet Fabric Extension interfaces and technologies” onpage 129


◆ “FC interfaces” on page 130


◆ 40GbE (True 40GbE)

◆ 40GbE (4x10GbE)

◆ 10GbE

◆ 1GbE




◆ SFP-10G-SR4 (MMF - 850nm)

◆ SFP-10G-LR4 (SMF)

◆ SFP-10G-ER4 (SMF)













◆ 10G SFP+ direct-attach copper (Twinax CU 1M, 3M, 5M - passive)

◆ 10G SFP+ direct-attach copper (Twinax CU 7M, 10M - active)

◆ Standard OM3 / OM4 Fibre with SFP+ Type LC connectors


FCoE interfaces The following interfaces are supported:

◆ Native FCoE CNA directly connected to Cisco Nexus 6001 SFP+

◆ Native FCoE Storage directly connected to Cisco Nexus 6001SFP+

• EMC VNX Series

• EMC VMAX Series

◆ FC Enhanced port types: VE, TE, and VF


Ethernet FabricExtension interfaces

and technologies

The following are supported:



◆ Support for up to 24 fabric extenders (Layer 2) with each CiscoNexus 6001 switch

◆ Cisco 40GbE Line Expansion Modules (LEM)

Note: Refer to the EMC Support Matrix available through E-LabInteroperability Navigator at http://elabnavigator.EMC.com, under thePDFs and Guides tab for a current list of supported fabric extenders (FEX).



◆ 10GbE CNA directly connected to Cisco Nexus 6001 SFP+ viabreakout cable (4x10GbE)

◆ 10GbE iSCSI directly connected to Cisco Nexus 6001 SFP+ viabreakout cable (4x10GbE)




130


◆ 10GbE FCoE and iSCSI targets directly connected to Cisco Nexus6001 SFP+ via breakout cable (4x10GbE)

• EMC VNX Series

• EMC VMAX Series

◆ 10GbE NICs, which adhere to industry standards, via SFP+breakout cable (4x10GbE)



◆ NIC Teaming

FC interfaces The following interfaces are supported:

◆ Direct connect FC Interfaces are not supported on the Nexus 6004

◆ FC Enhanced port types: VE, TE, and VF



◆ “FCoE ” on page 130




◆ CEE- and IEEE-compliant PFC (per-priority Pause frame support)

◆ PFC link distance support: 300m

◆ CEE-compliant DCBX Protocol

◆ CEE- and IEEE-compliant Enhanced Transmission Selection

FCoE ◆ T11 standards-compliant FCoE (FC-BB-5)

◆ T11 FCoE Initialization Protocol (FIP) (FC-BB-5)

◆ Any 10 or 40 Gigabit Ethernet port configurable as FCoE

◆ Fibre Channel forwarding (FCF)

◆ Fibre Channel enhanced port types: VE, TE, and VF

◆ Direct attachment of FCoE targets

◆ Up to 32 virtual SANs (VSANs) per switch



◆ Fabric Device Management Interface (FDMI)

◆ Fibre Channel ID (FCID) persistence

◆ In-order delivery

◆ Port tracking

◆ Cisco FCoE_NPV technology

◆ N-port identifier virtualization (NPIV)

◆ Fabric services: Name server, registered state change notification(RSCN), login services, and name-server zoning

◆ Per-VSAN fabric services

◆ Cisco Fabric Services

◆ Fabric Shortest Path First (FSPF)

◆ Storage Management Initiative Specification (SMI-S)

◆ Boot from SAN over vPC and Enhanced vPC (EvPC)

◆ FCP

◆ Native Interop Mode 2 & 3

◆ VSAN trunking

◆ Cisco NPV technology

Ethernet (Layer-2) ◆ Layer 2 switch ports and VLAN trunks


◆ Rapid Per-VLAN Spanning Tree Plus (PVRST+) (IEEE 802.1wcompatible)

◆ Multiple Spanning Tree Protocol (IEEE 802.1s) instances: 64instances

◆ Spanning Tree PortFast

◆ Spanning Tree root guard

◆ Spanning Tree Bridge Assurance

◆ Cisco EtherChannel technology (up to 16 ports per EtherChannel)

◆ Cisco vPC technology

◆ vPC configuration synchronization




132



◆ Storm control (unicast, multicast, and broadcast)

◆ Private VLANs

◆ Private VLAN over trunks (isolated and promiscuous)

◆ Private VLANs over vPC and EtherChannels

◆ VLAN Remapping

◆ Cisco FabricPath

◆ PC and vPC+ with FabricPath



◆ Support for up to 24 fabric extenders (Layer 2) with each CiscoNexus 6001 platform

Ethernet (Layer-3) ◆ Layer 3 interfaces: Routed ports on Cisco Nexus 6004 platforminterfaces, switched virtual interface (SVI), PortChannels,subinterfaces, and PortChannel subinterfaces

◆ Sixteen-way Equal-Cost Multipathing (ECMP)*

◆ Routing protocols: Static, RIPv2, EIGRP, OSPFv2, and BGP

◆ IPv6 routing protocols: Static, OPFv3, BGPv6, and EIGRPv6

◆ IPv6 VRF-Lite

◆ HSRP and VRRP

◆ ACL: Routed ACL with Layer 3 and 4 options to match ingressand egress ACL

◆ Multicast: PIMv2 sparse mode, Source Specific Multicast (SSM),BiDir PIM, Multicast Source Discovery Protocol (MSDP), IGMPv2 and v3, and Multicast VLAN Registration (MVR)

◆ VRF: VRF-Lite (IP VPN); VRF-aware unicast; and BGP-, OSPF-,RIP-, and VRF-aware multicast

◆ URFP with ACL; strict and loose modes

◆ Jumbo Frame support (up to 9216 bytes)

◆ Support for up to 24 fabric extenders on each Cisco Nexus 6001



Nexus 2248PQ FEX supported features and topologiesThe supported interfaces, features and topologies have been brokendown into multiple areas, and are discussed in this section. Actualreference topologies are provided in “Nexus 6000 Series and FabricExtenders topologies” on page 143.






◆ 40GbE (4x10GbE)

◆ 10GbE

◆ 1GbE




◆ SFP-10G-SR4 (MMF - 850nm)

◆ SFP-10G-LR4 (SMF)






◆ 10G SFP+ direct-attach copper (Twinax CU 1M, 3M, 5M - passive)

◆ 10G SFP+ direct-attach copper (Twinax CU 7M, 10M - active)

◆ Standard OM3 / OM4 Fibre with SFP+ Type LC connectors

Nexus 2248PQ FEX supported features and topologies 133






134



FCoE interfaces The following interface is supported:

◆ Native 10GbE FCoE CNA directly connected to Cisco Nexus2248PQ SFP+




◆ 10GbE CNA directly connected to Cisco Nexus 2248PQ

◆ 10GbE iSCSI directly connected to Cisco Nexus 2248PQ

◆ 10GbE FCoE and iSCSI targets directly connected to Cisco Nexus2248PQ

◆ 10GbE NICs, which adhere to industry standards



◆ NIC Teaming





◆ DCB - Data Center Bridging

Ethernet (Layer-2) ◆ IEEE 802.1p: CoS prioritization

◆ IEEE 802.1Q: VLAN tagging

◆ IEEE 802.3: Ethernet

◆ IEEE 802.3ae: 10 Gigabit Ethernet Layer

◆ Layer 2 VLAN trunks





◆ Cisco EtherChannel technology on uplinks

◆ PortChannel on server ports on Cisco Nexus 2200 platforms

◆ Advanced PortChannel hashing


◆ Pause frames (priority flow control [PFC] and IEEE 802.3x)

◆ Private VLANs (promiscuous only on uplinks)

Nexus 2248PQ FEX supported features and topologies 135

136


Nexus 5000 supported features and topologiesThe supported features and topologies have been broken down intofour areas, as shown in Figure 20, and are discussed in this section.

◆ “FCoE interface” on page 137◆ “Ethernet uplink interface” on page 138◆ “FC interface” on page 139◆ “Ethernet end-device interface” on page 140

Figure 20 is for clarification purposes only. Actual referencetopologies are provided in the following sections.


MDS Brocade

FCInterface

Ethernet uplinkInterface

Ethernet enddevice Interface

FCoEInterface

Nexus 5000

Ethernet Bridge(Switch)

10 GbENIC

ServerServer

CNA

Storage(e.g., VMAX,VNX, CX4)

Storage(e.g., VMAX,VNX, CX4)

Storage(e.g., VNXe)

GEN-001009a




◆ CNAs directly connected via physical media to the Nexus 5000

Note: Connecting the CNA to the Nexus via a stand-alone Top of Rack(TOR) or End Of Row (EOR) intermediate 10G Ethernet Bridge (eitherDCB capable or not) is not supported unless stated otherwise in thissection.

◆ IBM Blade Server Nexus 4001I FIP Snooping connections

◆ VE_Ports are supported starting with NX-OS 5.0(2)N2(1)

• VE_Ports allow for FCoE ISLs to be created and enablemulti-hop FCoE


• Fabric A /Fabric B connectivity is supported• An example of Fabric A /Fabric B connectivity is two Nexus

5020s, each connected to one of the two sides of a mirroredfabric.

• SFP+ optical (short reach only)• SFP+ cu (Twinax - Molex)• 1 m, 3 m, 5 m, 7 m, and 10 m are available

Note: Refer to "Enabling technologies" section, "Introduction to FibreChannel over Ethernet" chapter in the Fibre Channel over Ethernet(FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook athttp://elabnavigator.EMC.com, under the Topology ResourceCenter tab.

Note: Refer to the “Twinax interoperability constraints” table,"Introduction to Fibre Channel over Ethernet" chapter in the FibreChannel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts andProtocols TechBook at http://elabnavigator.EMC.com, under theTopology Resource Center tab, for information on interoperabilityconstraints.







138



◆ Native FCoE storage

• VNX• CX4• VMAX

Ethernet uplink interfaceThe following are supported:

◆ Preferred Ethernet connectivity to Catalyst 6k or Nexus 7k

◆ Supported UNH interop approved Ethernet for 10 GbE

• 1 GigE is supported on the first eight ports of the Nexus 5010and the first 16 ports of the Nexus 5020 when 1 Gb/soptical/Cu interface SFPx modules are used

◆ PAUSE 802.3x is supported but will not be required on Nexus5020 uplink

◆ Etherchannels for link aggregation

• Support up to 256 VLANs

◆ 802.1w, Per VLAN Rapid Spanning Tree (PVRST+)

• Preferred, may be better if available

◆ 802.1s Multiservice Transport Platform (MSTP)

• Most widely deployed

◆ Spanning Tree PortFast and PortFast Guard

◆ Spanning Tree UplinkFast and BackboneFast

◆ Bridge Protocol Data Unit (BPDU) Guard


◆ Internet Group Management Protocol (IGMP) snooping

◆ IGMP snooping querier

◆ Protocol Independent Multicast (PIM)

◆ Cisco Etherchannel

• 8 members per channel

• 12 channels per chassis



• VSAN truck pruning done through configuration; no VTPsupport on Nexus 5000


◆ Advanced port channel hashing


◆ Storm control

◆ Private VLANs / Private VLANs over trunks

◆ Virtual PortChannel (vPC) is supported

• See “Virtual PortChannel” on page 293 for configurationinformation

◆ VE_Ports are supported

FC interface

Note: The Nexus 5000 can be configured to run in NPV mode to connect toeither Brocade (FOS or EOS) or MDS products, but not at the same time.

◆ NPV — Brocade and Cisco

• Minimum number of NP_Ports per Nexus 5000 is 1

• Maximum number of NP_Ports per Nexus 5000 is 16

• Maximum distance is shortwave

– Longwave is currently not supported◆ FC-SW — Cisco MDS only

• Maximum number of ISLs per Nexus 5000 is 16

• Max domains is per the EMC Support Matrix

• Maximum distance is shortwave

– Longwave is not currently supported• Max hops is per the EMC Support Matrix

• Interopmode is native

• Port channels (link aggregation) is supported

• Trunking (multiple VSANs) is supported

• Scalability

– Same as MDS (except VSAN up to 32)




140


• HBA and storage direct connect to the Nexus FC ports aresupported starting with the 4.0(1a)N1(1a) firmware release.

• The supported HBA and storage ports for the Nexus are thesame as those for the MDS.

Note: For the most up-to-date support information, refer to the EMCSupport Matrix.

Ethernet end-device interfaceThe following are supported:


◆ Celerra® 10 GbE (NAS+iSCSI+MPFS)


◆ NIC Teaming

◆ Virtual PortChannels (vPC)





Nexus 2000 supported features and topologiesThe supported features and topologies have been broken down intothree areas as shown in Figure 21, each discussed further:



◆ “Ethernet end-device interface” on page 142


Figure 21 Nexus 2000 supported features and topologies


◆ CNAs directly connected via physical media to the Nexus 5000

Note: Connecting the CNA to the Nexus via a stand-alone Top of Rack(TOR) or End Of Row (EOR) intermediate 10G Ethernet Bridge (eitherDCB capable or not) is not supported unless stated otherwise in thissection.



FCoEInterface

Nexus 2232PP

Nexus 5000Parent switch

Celerra

10 GbENIC

10 GbENIC

ServerServer

CNA CNA

Server

ICO-IMG-000853


142


◆ IBM Blade Server Nexus 4001I FIP Snooping connections


• SFP+ optical (short reach only)• SFP+ cu (Twinax - Molex)• 1 m, 3 m, and 5 m are available




Ethernet uplink interfaceThe Nexus 2232PP must be directly connected to a Nexus 5000.


◆ Features

• Static Pinning Fabric Interface Connections• EtherChannel Fabric Interface Connections• Virtual Port Channel (VPC)


• Fabric Extender Transceivers (FET)• SFP+ optical (short reach only)• SFP+ cu (Twinax - Molex)• 1 m, 3 m, and 5 m are available

Ethernet end-device interfaceThe following are supported:




◆ NIC Teaming




Nexus 6000 Series and Fabric Extenders topologiesThis section contains information for the following Nexus 6000 seriestopologies.

◆ “Multihop FCoE” on page 143

◆ “Leaf and Spine (Layer-2/Layer-3) fabric” on page 144

◆ “High Performance Compute (HPC) and High FrequencyTrading (HFT)” on page 145

◆ “Scalable data center access and aggregation with FabricExtenders” on page 146

◆ “Fabric Extenders (FEX)” on page 147

Multihop FCoEThe Cisco Nexus 6004 can support end-to-end data centerconvergence from server to storage by delivering multihop FCoEcapability in the data center. The FCoE function complements theexisting FCoE function on the Cisco Nexus 5500 Switch platform.

With the addition of the Nexus 6000 series switches, Cisco UnifiedFabric supports both the access and core network layers, supportingall storage traffic (FCoE, Small Computer System Interface over IP[iSCSI], and network attached storage (NAS) over simplifiedinfrastructure based on lossless 10 and 40 Gigabit Ethernet.

Nexus 6000 Series and Fabric Extenders topologies 143

144


Figure 22 shows an example of a Nexus 6004/6001 40GbE multihopFCoE topology.

Figure 22 Cisco Nexus 6004/6001 40GbE Multihop FCoE topology example

Leaf and Spine (Layer-2/Layer-3) fabric

Note: Some information in this section comes from Cisco documentation. Formore detailed information, refer to http://www.cisco.com.

The Cisco Nexus 6004 can be deployed as a leaf or spine node in aLayer 2 or Layer 3 fabric design. Leaf and spine layer designs usinghigh-density and low-latency switches lead to flatter networkarchitectures, allowing connections scaling from hundreds to morethan 10,000 servers with large bisectional bandwidth. The Leaf andSpine design helps ensure low-latency fabric with minimal hopcount.

◆ The spine switches create a non-blocking, low-latency fabric,forwarding packets between leaves.

◆ The leaf switches provide connectivity to servers.



Using a highly-meshed architecture helps ensure the highest possiblenetwork availability with minimal effect on traffic if a failure occurs.Cisco Nexus 6004 can be deployed as a Layer 2 or Layer 3 spine orleaf switch, providing a high degree of design flexibility.

Figure 23 shows an example of a Leaf and Spine topology with Nexus6004/6001 switches.

Figure 23 Cisco Nexus 6004/6001 Leaf and Spine topology example

High Performance Compute (HPC) and High Frequency Trading (HFT)


The Cisco Nexus 6004 can be deployed as a high-density access layerswitch to consolidate a large number of 10 and 40 Gigabit Ethernetservers where the deployment calls for a minimal number of hopsfrom the server to the upstream network to minimize latency.

The Cisco Nexus 6004 has a high density of 10 and 40 GigabitEthernet ports per rack unit, ~1 microsecond latency port to port forany packet size and integrated line-rate Layer 2 and Layer 3 featureswith scale.


146


Figure 24 shows an example of a High Performance Compute (HPC)and High Frequency Trading (HFT) topology with Nexus 6004/6001switches.

Figure 24 Cisco Nexus 6004/6001 HPC and HFT topology example

Scalable data center access and aggregation with Fabric Extenders


Data center designs are increasingly moving toward lessoversubscription starting from the server access layer. Moreover,growth of 10 Gigabit Ethernet server deployments is accelerating theneed for top-of-rack (ToR) solutions that can provide 40 GigabitEthernet uplink capabilities to meet the oversubscriptionrequirement.

The Cisco Nexus 6004 with high-density 40 Gigabit Ethernet portscan be utilized as a compact aggregation platform for the ToRswitches with 40 Gigabit Ethernet uplinks.



Figure 25 shows an example of scalable data center access andaggregation using Fabric Extenders.

Figure 25 Scalable data center access and aggregation with Fabric Extendersexample

Fabric Extenders (FEX)


The Cisco Nexus 2000 Series can be used in conjunction with a CiscoNexus parent switch in two main design scenarios (shown inFigure 22 on page 144). Note the following deployment scenarios:

◆ Cisco Nexus 2000 Series Fabric Extenders single-connected to oneupstream Cisco Nexus 5000, Nexus 6000, or Nexus 7000 SeriesSwitch

In this deployment scenario, access-layer redundancy is achievedthrough redundant server connections to two upstreamdistributed modular systems using vPC (Cisco Nexus 5000 orNexus 6000 Series) or server NIC teaming to two Cisco Nexus2000 Series Fabric Extenders.

◆ Cisco Nexus 2000 Series Fabric Extenders dual-connected to twoupstream Cisco Nexus 5000 or Nexus 6000 Series Switches (vPC)


148


In this deployment scenario, access-layer redundancy is achievedthrough a combination of Cisco Nexus 2000 Series FabricExtenders dual-connected to an upstream parent switch andserver NIC teaming.

◆ Enhanced vPC (EvPC):

In this deployment scenario, access layer redundancy is achievedin two ways: through redundant connections between the CiscoNexus 2000 Fabric Extenders and the Cisco Nexus parentswitches using vPC, and through redundant server connectionsto two fabric extenders using vPC and active-active server NICteaming.

◆ vPC+

In this deployment scenario, access-layer redundancy is achievedthrough server vPC, FEX vPC, and EvPC. In addition, a vPC+domain allows the Cisco Nexus parent switch and the fabricextenders to be viewed as a single virtual switch in a CiscoFabricPath network.

Figure 26 shows an example of these four scenarios.

Figure 26 Cisco Nexus 2000 Series Fabric Extenders design scenarios



Configuring 40GbE FCoE connectivity with Nexus 6000 SeriesThis section describes the setup steps required to implement 40GbEFCoE connectivity between the Nexus 6004 and the three different40GbE supported options (switch, fabric extender, and LEM).

Note: Cisco Nexus 6000 switches can be used in place of the Cisco Nexus5000 switches used in the other case studies defined in this TechBook. Thissection focuses only on the connectivity options being introduced by theCisco Nexus 6000.

This section is not intended to address a fully high-availabilityfunctional fabric, rather 40GbE connectivity.

Refer to Figure 27 on page 152 for the configuration used in thissection.

This section includes the following information:


◆ “Summary of configuration steps” on page 150

◆ “Individual reference topologies” on page 152

PrerequisitesThis section contains prerequisites for the Cisco Nexus 6004, 6001,5596UP, and 2248PQ FEX switches and fabric extender.

Nexus 6004 ◆ Cisco NX-OS 6.0(2)N1(2a) minimum release

◆ Appropriate cable for 40GbE ISL connectivity (Ex.QSFP-H40G-CU1M, etc.)

◆ Appropriate QSFP+ optics for breakout QSFP+ to 4x10GbE cablefor host or storage 10GbE connectivity (Ex. QSFP-40G-SR4)

◆ Appropriate QSFP-to-4x10GbE breakout cable

◆ 12 port 40GbE line expansion modules (LEM), if needed

◆ All required license keys are installed

Nexus 6001 ◆ Cisco NX-OS 6.0(2)N1(2a) minimum release


Configuring 40GbE FCoE connectivity with Nexus 6000 Series 149

150



Nexus 5596UP ◆ Cisco NX-OS 6.0(2)N1(2a) minimum release


◆ Cisco 40GbE expansion module: N55-M4Q


Nexus 2248PQ FEX ◆ Cisco NX-OS 6.0(2)N1(2a) minimum release (installed via parentswitch)


Summary of configuration stepsThis section summarizes the configuration steps detailed in thefollowing sections:

◆ “Connecting and configuring a Cisco Nexus 6004 to another CiscoNexus 6004” on page 152

◆ “Connecting and configuring a Cisco Nexus 6004 to Cisco Nexus6001” on page 157

◆ “Connecting and configuring a Cisco Nexus 6004 to a CiscoNexus 5596UP with LEM N55-M4Q” on page 162

◆ “Connecting and configuring a Cisco Nexus 6004 to a CiscoNexus 2248PQ FEX” on page 168

Connecting and configuring a Cisco Nexus 6004 to another Cisco Nexus 60041. Enable features.

2. Create VLAN/VSAN.

3. Configure 40GbE PC.

4. Create vfcs.

5. Configure 40GbE (4x10GbE) breakout ports for storage/hosts10GbE.

6. Create vfcs.

7. Connect cables.



Connecting and configuring a Cisco Nexus 6004 to Cisco Nexus 60011. Enable features.


3. Configure 40GbE PC.

4. Create vfcs.


6. Create vfcs.

7. Connect cables.

Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 5596UP with LEMN55-M4Q

1. Enable features.


3. Configure 40GbE breakout ports for PC.

4. Configure 40GbE (4x10GbE) PC.

5. Create vfcs.


7. Create vfcs.

8. Connect cables.

Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 2248PQ FEX1. Enable features.


3. Configure 40GbE (4x10GbE) PC.

4. Create and associate FEX.


6. Create vfcs.

7. Connect cables.


152


Individual reference topologiesFigure 27 shows the reference topologies used in the following fourconfigurations discussed in this section.

◆ “Connecting and configuring a Cisco Nexus 6004 to another CiscoNexus 6004” on page 152

◆ “Connecting and configuring a Cisco Nexus 6004 to Cisco Nexus6001” on page 157

◆ “Connecting and configuring a Cisco Nexus 6004 to a CiscoNexus 5596UP with LEM N55-M4Q” on page 162

◆ “Connecting and configuring a Cisco Nexus 6004 to a CiscoNexus 2248PQ FEX” on page 168

Figure 27 Reference topologies

Connecting and configuring a Cisco Nexus 6004 to another Cisco Nexus 6004

On both Cisco Nexus6004 switches

Complete the following steps:

1. Enable the FCoE feature.

a. feature fcoe

2. Enable LACP.

a. feature lacp



3. Enable LLDP.

a. feature lldp

4. Enable vPC.

a. feature vpc

5. Create VLAN/VSAN and map FCoE LAN to VSAN.

a. vsan database

b. vsan 2

c. vlan 2

d. fcoe vsan 2

e. exit

6. Create Port-Channel.

a. interface port-channel 4

b. switchport mode trunk

c. switchport trunk allowed vlan 2,100,200

d. no shutdown

e. exit

7. Configure interfaces for port-channel 4.

a. interface ethernet 2/1-2



d. channel-group 4

e. exit

8. Create vFC for port-channel.

a. interface vfc 104

b. bind interface port-channel4

c. switchport mode E

d. switchport trunk allowed vsan 2

e. no shutdown

f. exit


154


9. Associate vFC of port-channel to VSAN.

a. vsan database

b. vsan 2 interface vfc 104

c. exit

10. Validate.

a. show port-channel database

b. show interface vfc 103

c. show interface ethernet 2/1-2

d. show vsan membership

e. show vsan 2

f. show fcoe database

g. show vlan fcoe

h. show interface brief

i. show running-config

11. Save the configuration.

a. copy running-config startup-config

12. Breakout True 40GbE interface ports into 4x10GbE for 10GbEstorage connectivity, if 40GbE storage ports not available.

a. show interface brief

b. show module

c. interface breakout slot 1 port 7-9 map 10g-4x

d. poweroff module 1

e. show module

f. no poweroff module 1

g. show module


13. Configure breakout 10GbE port for storage connectivity.

a. interface ethernet 1/7/1

b. spanning-tree port type edge trunk



c. switchport mode trunk

d. switchport trunk allowed vlan 2

e. no shutdown

f. exit

14. Create vFC for storage interface.

a. interface vfc 1

b. bind interface ethernet 1/7/1

c. no shutdown

d. exit

15. Associate storage vFC to VSAN.

a. vsan database


c. exit

16. Validate.

a. show interface vfc 1

b. show fcoe database

c. show interface fcoe

d. show flogi database

e. show interface Ethernet 1/7/1

f. show vsan membership

g. show vlan




18. Configure 10GbE interface port for host CNA connectivity.






156


e. no shutdown

f. exit


a. interface vfc 1


c. no shutdown

d. exit


a. vsan database


c. exit

21. Validate.







g. show vlan




23. Create zones, zoneset and activate on FCoE VSAN.

Refer to Cisco zoning documentation

24. Verify.

a. VSAN zone mode (basic, enhanced) matches across switches

– show zone statusb. zoneset is active on FCoE VSAN

– show zoneset active vsan 2



c. Verify port-channel and vfc are up

– show port-channel database– show interface port-channel 3– show interface vfc 103

d. Verify switches are merged and zoneset is active onappropriate VSAN across all switches in the fabric

– show fcdomain– show fcs database– show fcns database

On the storage array 1. Check Storage array to ensure that host initiator is logged intoarray.

2. Configure device masking.

On the host 1. Rescan host bus.

2. Ensure host discovers LUNs.

Connecting and configuring a Cisco Nexus 6004 to Cisco Nexus 6001

On both Nexus 6004and 6001



a. feature fcoe

2. Enable LACP.

a. feature lacp

3. Enable LLDP.

a. feature lldp

4. Enable vPC.

a. feature vpc


a. vsan database

b. vsan 2

c. vlan 2

d. fcoe vsan 2

e. exit


158






d. no shutdown

e. exit


a. interface ethernet 3/1-2



d. channel-group 3

e. exit



b. bind interface port-channel 3



e. no shutdown

f. exit


a. vsan database


c. exit

10. Validate.



c. show interface ethernet 3/1-2


e. show vsan 2




g. show vlan fcoe





On Nexus 6004 only 1. Breakout True 40GbE interface ports into 4x10GbE for 10GbEstorage connectivity, if 40GbE storage ports not available.


b. show module



e. show module


g. show module







e. no shutdown

f. exit


a. interface vfc 1


c. no shutdown

d. exit



160


a. vsan database


c. exit

5. Validate.







g. show vlan




On Nexus 6001 only 1. Configure 10GbE interface port for host CNA connectivity.





e. no shutdown

f. exit


a. interface vfc 1


c. no shutdown

d. exit


a. vsan database




c. exit

4. Validate.







g. show vlan




On Nexus 6004 only 1. Create zones, zoneset and activate on FCoE VSAN.

Refer to Cisco zoning documentation.

On Nexus 6004 and6001

1. Verify.



– show zoneset active vsan 2c. Verify port-channel and vfc are up






162




2. Ensure host discovers LUNs.

Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 5596UP with LEMN55-M4Q

On both Nexus 6004and 5596

Complete the following steps.


a. feature fcoe

2. Enable LACP.

a. feature lacp

3. Enable LLDP.

a. feature lldp

4. Enable vPC.

a. feature vpc


a. vsan database

b. vsan 2

c. vlan 2

d. fcoe vsan 2

e. exit

On Nexus 6004 only 1. Breakout True 40GbE interface ports into 4x10GbE forconnectivity to the Nexus 5596 N55-M4Q 40GbE line expansionmodule.


b. show module



e. show module




g. show module






d. no shutdown

e. exit


a. interface ethernet 1/1/1-4



d. channel-group 1

e. exit






e. no shutdown

f. exit


a. vsan database


c. exit

6. Validate.



c. show interface ethernet 1/1/1-4


164



e. show vsan 2


g. show vlan fcoe





On Nexus 5596 only 1. Create Port-Channel.




d. no shutdown

e. exit





d. channel-group 1

e. exit






e. no shutdown

f. exit


a. vsan database




c. exit

5. Validate.



c. show interface ethernet 4/1/1-4


e. show vsan 2


g. show vlan fcoe





On Nexus 6004 only 1. Breakout True 40GbE interface ports into 4x10GbE for 10GbEstorage connectivity, if 40GbE storage ports not available.


b. show module



e. show module


g. show module








166


e. no shutdown

f. exit


a. interface vfc 1


c. no shutdown

d. exit


a. vsan database


c. exit

5. Validate.







g. show vlan




On Nexus 5596 only 1. Configure breakout 10GbE interface port for host CNAconnectivity.

a. interface ethernet 1/31




e. no shutdown



f. exit


a. interface vfc 1

b. bind interface ethernet 1/31

c. no shutdown

d. exit


a. vsan database


c. exit

4. Validate.





e. show interface Ethernet 1/31


g. show vlan




On Nexus 6004 only 1. Create zones, zoneset and activate on FCoE VSAN.


On Nexus 6004 and5596

1. Verify.





168








Ensure host discovers LUNs.

Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 2248PQ FEX

On Nexus 6004 with2248PQ FEX



a. feature fcoe

2. Enable LACP.

a. feature lacp

3. Enable LLDP.

a. feature lldp

4. Enable vPC.

a. feature vpc

5. Enable FEX

b. feature fex


a. vsan database

b. vsan 2

c. vlan 2

d. fcoe vsan 2



e. exit



b. switchport

c. switchport mode fex-fabric


e. fex associate 100

f. no shutdown

g. exit



b. switchport mode fex fabric

c. switchport trunk allowed vlan 2

d. fex associate 100

e. channel-group 2

f. no shutdown

g. exit

9. Validate.


b. show fex detail

c. show module fex all

d. show fex

e. show fex 100

f. show fex 100 detail

g. show interface fex-fabric

h. show port-channel database

i. show interface port-channel 2

j. show interface ethernet 4/1/1-4

k. show running-config


170




11. Breakout True 40GbE interface ports into 4x10GbE for 10GbEstorage connectivity, if 40GbE storage ports not available.


b. show module



e. show module


g. show module







e. no shutdown

f. exit


a. interface vfc 1


c. no shutdown

d. exit


a. vsan database


c. exit

15. Validate.









g. show vlan




On Nexus 6004 and22488PQ

Complete the following steps.

1. Configure breakout 10GbE port for host CNA connectivity.





e. no shutdown

f. exit


a. interface vfc 5


c. no shutdown

d. exit


a. vsan database


c. exit

4. Validate.



172







g. show vlan




On Nexus 6004 and22488PQ

1. Create zones, zoneset and activate on FCoE VSAN.


2. Verify.










Ensure host discovers LUNs.



Nexus 5000 direct-connect topologyThis section details the setup steps required to create theconfiguration shown in Figure 29 on page 176. It includes thefollowing information:


◆ “Existing topology” on page 174



◆ “Configuring the virtual interfaces on Nexus 5020” on page 178

◆ “Etherchannel configuration on a Nexus 7000 switch” onpage 203

◆ “Spanning tree configuration and verification on Nexus 5020” onpage 206

Refer to “Installing the Nexus 5000” on page 744 for information oninstalling the Nexus switch.

Summary of configuration stepsThis section provides a summary and examples of the steps to createthe configuration shown in the target topology, Figure 29 onpage 176.

To create a vFC (virtual) in Nexus FCoE switch, complete thefollowing steps:

1. Create Virtual FC interfaces.

2. Create a VSAN.

3. Create a VLAN to map to a VSAN.

4. Configure the physical Ethernet port as a trunk port and allowthe VLAN.

Nexus 5000 direct-connect topology 173

174


Existing topologyIt is assumed that the customer has an existing LAN and mirroredSAN, as shown in Figure 28, and is adding the Nexus 5020 to both.


Core-N7K-1

CNA_1

Host_1 (Windows)

CNA_1:EmulexVSAN = 100WWPN= 10:00:00:00:c9:3c:f8:3cStorage = Storage port 1

CNA_2:EmulexVSAN = 200WWPN = 10:00:00:00:c9:3c:f7:c0Storage = Storage port 3

Storage port 4:50:06:04:82:D5:2E:69:59VSAN = 400

Storage port 3:50:06:04:8A:D5:2E:69:79VSAN = 200

Storage port 2:50:06:04:82:Da5:2E:69:49

Storage port 1:50:06:04:82:D5:2E:69:79

CNA_2

Core-N7K-2

1/1 2/1 1/1 2/1

1/2

1/3

1/4

1/5

StorageFabric BFabric A

1 2 3 4

1/2

1/2

1/3

1/4

1/5

1/2

1/2

1/1

1/3

1/4

1/5

1/2

1/1

1/3

1/4

1/5

CNA_1

Host_1 (Linux)

VSAN 100

VSAN 300

VSAN 200

VSAN 400

4Gb/s FC

GEN-001013

CNA_1:QLogicVSAN = 300WWPN = 21:01:00:1b:32:2a:c0:b8 Storage = Storage port 2

CNA_2:QLogicVSAN = 400WWPN = 21:00:00:1b:32:0a:c0:b8Storage = Storage port 4

CNA_2

DCX_1Domain 1

172.23.185.24

DCX_2Domain 2

172.23.185.25

MDS-9513-1VSAN 400Domain 1

172.23.185.22


172.23.185.23



Existing LAN The existing LAN has the following features/configurationparameters:

◆ Etherchannel (802.1q)


◆ Rapid Spanning Tree Protocol (RSTP)

◆ Multiple Spanning Tree (MST)

◆ IGMP Snooping

◆ VLANs

Existing SAN The existing SAN has the following features/configurationparameters:

◆ DCX_1

• Domain ID 1

◆ DCX_2

• Domain ID 2

◆ 9513_1

• Domain ID 1

• VSAN 400

◆ 9513_2

• Domain ID 2

• VSAN 200


176



Figure 29 Target topology example

2/11/1 1/1 2/1

Core-N7K-1 Core-N7K-2

Port-channel 1

Port-channel 2

1/21/31/41/5

1/21/31/41/5

Nexus-5020-1 Nexus-5020-2

1/21/1 1/21/1

SAN A SAN BStorage

1 2 3 4

1/1

1/1

DCX_1Domain 1

172.23.185.24

DCX_2Domain 2

172.23.185.25

SAN PortChannels

1/11/21/31/41/5

1/11/21/31/41/5


172.23.185.22


172.23.185.23


Host_1(Windows)

Host_2(Linux)

VLAN / VSAN 100

VLAN / VSAN 200

VLAN / VSAN 300

VLAN / VSAN 400

10 GbE

4 Gb/s FC

IP = 172.23.185.112 Domain = N/A (NPV mode)

CNA_1:QLogicVSAN = 300 WWPN = 21:01:00:1b:32:2a:c0:b8 Storage = Storage port 2

CNA_1:EmulexVSAN = 100 WWPN = 10:00:00:00:c9:3c:f8:3c Storage = Storage port 1

CNA_2:

Nexus-5020-1

IP = 172.23.185.113 Domain = 3

Nexus-5020-2EmulexVSAN = 200 WWPN = 10:00:00:00:c9:3c:f7:c0 Storage = Storage port 3


Storage port 1: 50:06:04:82:D5:2E:69:79

Storage port 2: 50:06:04:82:D5:2E:69:49

Storage port 3: 50:06:04:8A:D5:2E:69:79VSAN = 200

Storage port 4: 50:06:04:82:D5:2E:69:59VSAN = 400 GEN-001122



PrerequisitesFCoE requires the use of specialized host adapters, referred to as aConverged Network Adapter (CNA), and specialized switchhardware known as a Fibre Channel Forwarder (FCF). CNAs areinstalled in the host and then attached to an Ethernet port thatprovides FCF functionality. The connection between the CNA andEthernet switch port can be either optical or copper based.

CNA prerequisites The first generation of CNAs requires a server that has a PCIe slotand can contain a full length and height PCIe expansion card. Refer tothe EMC Support Matrix for information on the supported hostplatforms.

FCF prerequisites The Nexus 5020 requires:

◆ 2 U of rack space, preferably at the top of the rack where theservers that will be connecting to the Nexus 5020 are located

◆ Two power drops of either 110 V or 220 V

◆ An IP address for management purposes and a network port towhich to connect the Nexus 5020 management port

◆ A customer-supplied password for the admin user

◆ An available Domain ID in the fabric to which the Nexus 5020will be connected

Note: If the Nexus 5020 will be carved into multiple VSANs, a DomainID will need to be reserved for each VSAN.

◆ For each Nexus 5020, at least one expansion module containing 4G FC ports will need to be installed

Laptop prerequisites Laptop prerequisites include:

Note: The laptop used in this example is needed only to initially configurethe Nexus 5020 and will be disconnected after the installation is complete.

◆ Com port with a DB-9 connector

◆ Download and install PuTTY





178


Note: The operating system used for this example was Windows 2000Professional and HyperTerminal was installed by default. Some of thelater Windows-based operating systems do not have HyperTerminalinstalled. This will need to be done before attempting to configure theNexus 5020s. See the help feature for the particular version of Windowsyou are using for more information on installing HyperTerminal.

Configuring the virtual interfaces on Nexus 5020This section contains the following information:

◆ “Configuring Nexus-5020-1” on page 180






◆ “Configuring network parameters” on page 203

By default, the Ethernet interfaces on the Nexus 5020 do not forwardEthernet frames with an Ethertype of FCoE. To allow the forwardingof FCoE frames, a Virtual Fabric Channel (VFC) interface will need tobe created and bound to the port where the FCoE frames will bereceived. An example of a Virtual FC interface is shown in Figure 30on page 179.



Figure 30 Virtual FC interface

The basic steps to configure a switch port to support FCoE are:

1. Create a VLAN for FCoE traffic.

2. Associate a VSAN with the FCoE VLAN.

3. Create the Virtual Fibre Channel interface.

4. Configure the Ethernet port settings.

Once these steps have been completed, the CNA should be able toperform Fabric Login (FLOGI) with the FCF functionality within theNexus 5020.

As shown in Figure 31 on page 180, each Nexus 5020 has two CNAsconnected to it, one on interface 1/1 and the other on interface 1/2.Due to the port-to-ASIC layout on the Nexus 5020, it is recommendedthat you connect the first CNA to interface 1/1, then connectsubsequent CNAs by starting at interface 1/2, and then proceeding inorder up to interface 1/40. This ensures that each ASIC remainsequally loaded.

PC

Is

Lin

ks10Gb

10Gb

Fibre ChannelEthernet

LOGIC

Eth1/1

vfc1

Eth1/29

vfc29

Virtual FC interface:Maps FC vHBA on server

GEN-001022Converged Network Adapter (CNA)


180



Configuring Nexus-5020-1Before configuring the Nexus-5020-1, consider the following:

The configuration of Nexus-5020-1 will be slightly different fromNexus-5020-2 for the following reasons:

◆ Nexus-5020-1 will be attached to a Brocade fabric consisting oftwo Brocade ED-DCX-B switches. Since the Nexus 5020 does notcurrently support FC-SW interop to an EMC ED-DCX-B,Nexus-5020-1 will need to run in NPV mode.

2/11/1 1/1 2/1


Port-channel 1

Port-channel 2

1/21/31/41/5

1/21/31/41/5


1/21/1 1/21/1

SAN A SAN BStorage

1 2 3 4

1/1

1/1

DCX_1Domain 1

172.23.185.24

DCX_2Domain 2

172.23.185.25

SAN PortChannels

1/11/21/31/41/5

1/11/21/31/41/5


172.23.185.22


172.23.185.23


Host_1(Windows)

Host_2(Linux)

VLAN / VSAN 100

VLAN / VSAN 200

VLAN / VSAN 300

VLAN / VSAN 400

10 GbE

4 Gb/s FC




CNA_2:

Nexus-5020-1

IP = 172.23.185.113 Domain = 3



Storage port 1: 50:06:04:82:D5:2E:69:79

Storage port 2: 50:06:04:82:D5:2E:69:49





◆ NPV mode allows the Nexus 5020 to act as an N_Port and passlogin requests directly from the CNA to the Fabric by using theNPIV protocol. For more information on NPV and NPIV, refer tothe “NPIV” section in the Networked Storage Concepts and ProtocolsTechBook available on the E-Lab Interoperability Navigator,Topology Resource Center tab, athttp://elabnavigator.EMC.com. Further information on how toconfigure NPV mode and the supported features with NPV isavailable at http://www.cisco.com.

Since Nexus-5020-1 will be connected to two different ED-DCX-Bs,VSAN assignments to the Nexus 5020 VFC and FC interfaces will beused to allow the CNA to attach to the proper fabric.

To configure Nexus-5020-1:

1. Connect to the console port of Nexus-5020-1.

a. Attach the DB-9 connector of the serial cable to COM port 1 onthe laptop.

b. Attach the RJ-45 connector of the serial cable to the consoleport on the Nexus-5020-1.

c. Select Start > Programs > Accessories > Communications >Hyperterminal.

The new connection dialog box is opened with the entry namedialog highlighted.

d. Type a name such as Nexus, and then click OK.


e. Select COM 1 in the Connect using pull-down list, and thenclick OK.

The COM1 Properties dialog box displays.

f. In the COM 1 Properties dialog box, use the followingsettings:

– Bits per second: = 9600– Data bits: = 8– Parity: = none– Stop bits: = 1– Flow control: = None

g. Click OK. You will be connected and should see the followingdisplayed:



182


Enter the password for "admin":

Note: You may need to press Enter several times before any text isdisplayed.

h. Type the customer-supplied password for admin, which inthis case is nbv12345.

Note: It may be helpful to log the output by using the capture textfeature under the Transfer: pull-down list.

i. Re-type the password for the admin user.

The Basic System Configuration dialog box displays.

2. Complete the basic configuration:

a. Answer the prompts as follows:

Would you like to enter the basic configuration dialog (yes/no):Y

Note: If the installer does not use the install startup script, managementvrf requires manual installation.

Create another login account (yes/no) [n]: NConfigure read-only SNMP community string (yes/no) [n]: NConfigure read-write SNMP community string (yes/no) [n]: NEnter the switch name: Nexus-5020-1Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: YMgmt0 IPv4 address: 172.23.185.112Mgmt0 IPv4 netmask: 255.255.255.0Configure the default gateway? (yes/no) [y]: YIPv4 address of the default gateway: 172.23.185.2Enable the telnet service? (yes/no) [y]: YEnable the ssh service? (yes/no) [n]: YType of ssh key you would like to generate (dsa/rsa): dsaNumber of key bits <768-2048>: 1024Configure the ntp server? (yes/no) [n]: NEnter basic FC configurations (yes/no) [n]: YConfigure default physical FC switchport interface state (shut/noshut) [shut]:

shutConfigure default physical FC switchport trunk mode (on/off/auto) [on]: onConfigure default zone policy (permit/deny) [deny]: denyEnable full zoneset distribution? (yes/no) [n]: y

A summary of the configuration displays:

The following configuration will be applied:switchname Nexus-5020-1



interface mgmt0ip address 172.23.185.112 255.255.255.0no shutdownexit

vrf context managementip route 0.0.0.0/0 172.23.185.2exittelnet server enablessh key dsa 1024 forcessh server enable

b. Answer the next questions as follows:

Would you like to edit the configuration? (yes/no) [n]:NUse this configuration and save it? (yes/no) [y]: Y

The configuration will be saved.

c. Once the configuration is completed, you will be prompted tolog in as follows:

Nexus-5020-1 login: adminPassword: nbv12345

d. Connect the network management cable.

e. Use the show int brief command to show the interfaces on theswitch.

You will notice between 40 and 52 Ethernet interfaces.

Note: If no FC interfaces are displayed and one or both of the GEMscontain FC ports, ensure that the FCoE feature has been enabledusing the feature FCoE command.

Nexus-5020-1# show int brief

-------------------------------------------------------------------------------Interface Vsan Admin Admin Status SFP Oper Oper Port

Mode Trunk Mode Speed ChannelMode (Gbps)

-------------------------------------------------------------------------------fc2/1 1 auto on down swl -- --fc2/2 1 auto on down swl -- --fc2/3 1 auto on down swl -- --...fc3/4 1 auto on down swl -- --

--------------------------------------------------------------------------------


184


Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth1/1 1 eth access down Link not connected 10G(D) --Eth1/2 1 eth access down Link not connected 10G(D) --Eth1/3 1 eth access down Link not connected 10G(D) --...Eth1/40 1 eth access down Link not connected 10G(D) --

--------------------------------------------------------------------------------Port VRF Status IP Address Speed MTU--------------------------------------------------------------------------------mgmt0 -- up 172.23.185.113 1000 1500Nexus-5020-1#

f. If there are no FC interfaces displayed, enable the FCoEfeature as follows:

Nexus-5020-1# config terminalNexus-5020-1(config)# feature fcoe

Note: It may be necessary to press Enter if the prompt is notdisplayed after entering the feature FCoE command.

g. Save the configuration.

Nexus-5020-1(config)# copy running startup-config

h. Enable NPV mode.

Although NPV mode is supported when connecting to eitherConnectrix MDS or non-Connectrix MDS switches, it onlyneeds to be enabled when connecting a Nexus 5020 to anon-Connectrix MDS switch, as is the case with this example.

If you are connecting the Nexus 5020 to a Connectrix MDSswitch, and do not want to use NPV mode, see “ConfiguringNexus-5020-2” on page 191.

Nexus-5020-1(config)# npv enableVerify that boot variables are set and the changes are saved. Changing to npv mode

erases the current configuration and reboots the switch in npv mode. Do youwant to continue? (y/n): Y

Note: It is recommended that you leave the console connection upwhile the reboot is taking place.

i. Open an SSH session to the Nexus 5020 with an IP address of172.23.185.12.



3. Enable disruptive load-balancing.

Disruptive load-balancing allows the Nexus to evenly distributeCNA logins across all NP ports while running in NPV mode.

To enable disruptive load-balancing, use the npvauto-load-balance disruptive command, as shown next.

Nexus-5020-1# config terminalNexus-5020-1(config)# npv auto-load-balance disruptive

4. If you are configuring a Nexus 5548 or Nexus 5596, you mustmanually enable the QoS class-fcoe. To do this, attach thefollowing QoS policies under system qos:

switch (config-sys-qos)# service-policy type qos input fcoe-default-in-policyswitch (config-sys-qos)# service-policy type queuing input fcoe-default-in-policyswitch (config-sys-qos)# service-policy type queuing output fcoe-default-out-policyswitch (config-sys-qos)# service-policy type network-qos fcoe-default-nq-policy

5. Create the VSANs.

Nexus-5020-1# configNexus-5020-1(config)# vsan databaseNexus-5020-1(config-vsan-db)# vsan 100Nexus-5020-1(config-vsan-db)# vsan 300Nexus-5020-1(config-vsan-db)# exitNexus-5020-1(config)#

6. Create a VLAN for FCoE Traffic and associate with a VSAN.

Note: Due to the one-to-one relationship between a VSAN and VLAN, ifmultiple VSANs will be configured (as is the case in this example) on theNexus 5020, then multiple FCoE VLANs will also need to be configured.

Nexus-5020-1(config)# vlan 100Nexus-5020-1(config-vlan)# fcoe vsan 100Nexus-5020-1(config-vlan)# vlan 300Nexus-5020-1(config-vlan)# fcoe vsan 300

7. Use show FCoE VLAN to verify whether the configuration wassuccessful.

Nexus-5020-1(config-vlan)# show vlan fcoe

VLAN VSAN Status-------- -------- --------100 100 Non-operational300 300 Non-operational


186


8. Create the VFC interfaces.

Nexus-5020-1(config-vlan)# interface vfc 1Nexus-5020-1(config-if)# bind interface ethernet 1/1Nexus-5020-1(config-if)# switchport trunk allowed vsan 100Nexus-5020-1(config-if)# no shutNexus-5020-1(config-if)# interface vfc 2Nexus-5020-1(config-if)# bind interface ethernet 1/2Nexus-5020-1(config-if)# switchport trunk allowed vsan 300Nexus-5020-1(config-if)# no shut

9. Configure the Ethernet Interface settings.

Nexus-5020-1(config-if)# interface ethernet 1/1Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# switchport trunk native vlan 1Nexus-5020-1(config-if)# switchport trunk allowed vlan 1, 100Nexus-5020-1(config-if)# spanning-tree bpduguard enableNexus-5020-1(config-if)# spanning-tree port type edge trunkNexus-5020-1(config-if)# interface ethernet 1/2Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# switchport trunk native vlan 1Nexus-5020-1(config-if)# switchport trunk allowed vlan 1, 300Nexus-5020-1(config-if)# spanning-tree bpduguard enableNexus-5020-1(config-if)# spanning-tree port type edge trunk

10. Add the vfc interfaces to the proper VSAN.

Nexus-5020-1(config-if)# vsan databaseNexus-5020-1(config-vsan-db)# vsan 100 interface vfc1Nexus-5020-1(config-vsan-db)# vsan 300 interface vfc2Nexus-5020-1(config-vsan-db)# exitNexus-5020-1(config-if)#

The output of show int brief displays the following:

Nexus-5020-1 (config-if)# show int brief

-----------------------------------------------------------------------------Interface Vsan Admin Admin Status SFP Oper Oper Port

Mode Trunk Mode Speed ChannelMode (Gb/s)

-----------------------------------------------------------------------------fc2/1 1 NP -- down swl -- --

fc2/2 1 NP -- down swl -- --fc2/3 1 NP -- down swl -- --fc2/4 1 NP -- down swl -- --fc3/1 1 NP -- down swl -- --fc3/2 1 NP -- down swl -- --fc3/3 1 NP -- down swl -- --fc3/4 1 NP -- down swl -- -------------------------------------------------------------------------------



Interface Status IP Address Speed MTU PortChannel

-----------------------------------------------------------------------------Ethernet1/1 notConnect -- -- 1500 --Ethernet1/2 notConnect -- -- 1500 --Ethernet1/3 notConnect -- -- 1500 --..Ethernet1/40 notConnect -- -- 1500 --..



-----------------------------------------------------------------------------vfc1 100 F -- down -- -- --vfc2 300 F -- down -- -- --Nexus-5020-1#

The output from this show int brief command differs from theoutput shown in Step e. Note that the vfc interfaces that werecreated in the previous steps are now displayed.

Note: In this example, the Cisco Nexus-5020-1 is running in NPVmode and will be connected to two different ED-DCX-B Directors.Because of this, we will be using VSANs to statically map one CNAto DCX_1 and the other CNA to DCX_2.

11. Add FC interfaces to the proper VSANs.

As shown in Figure 29 on page 176, Nexus-5020-1 is connected totwo different fabrics. In order to ensure that each CNA attachedto Nexus-5020-1 logs in to the same fabric as its storage port, eachCNA will need to be statically mapped to a particular VSAN. Thisis accomplished by adding the VFC Interface as well as thephysical FC interfaces to the appropriate VSAN. To summarizethe steps:

a. Enter the vsan database configuration mode by using vsandatabase.

b. Add the fc interfaces on 5020_1 to the appropriate VSANusing vsan <VSAN ID> interface vfc <vfc number>.

Nexus-5020-1(config-if)# vsan databaseNexus-5020-1(config-vsan-db)# vsan 100 interface fc 2/1Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 2/3


188


Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 3/1Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 3/3Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 2/2Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 2/4Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 3/2Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 3/4Nexus-5020-1(config-vsan-db)# exitNexus-5020-1(config)#

c. Enable all of the fc interfaces by specifying the interface byusing the interface <interface number> command, and then theno shut command.

Nexus-5020-1(config)# interface fc 2/1-4Nexus-5020-1(config-if)# no shutNexus-5020-1(config)# interface fc 3/1-4Nexus-5020-1(config-if)# no shut

d. Validate the configuration by using the show int briefcommand.

The following is a partial output of this command.

Nexus-5020-1(config)# show int brief



-----------------------------------------------------------------------------fc2/1 100 NP -- down swl -- --fc2/2 300 NP -- down swl -- --fc2/3 100 NP -- down swl -- --fc2/4 300 NP -- down swl -- --fc3/1 100 NP -- down swl -- --fc3/2 300 NP -- down swl -- --fc3/3 100 NP -- down swl -- --fc3/4 300 NP -- down swl -- --...-----------------------------------------------------------------------------Interface Vsan Admin Admin Status SFP Oper Oper Port


-----------------------------------------------------------------------------vfc1 100 F -- down -- -- --vfc2 300 F -- down -- -- --

Nexus-5020-1(config-if)#

12. Connect the cables.



a. Once the interface configuration has been verified, connect thecables between the CNAs and Nexus-5020-1 as well as fromthe FC switches to Nexus-5020-1.

b. Use show int brief to validate the interfaces are initialized.

Note: With Step 12 a and b, assume that the CNA installation stepshave been completed (such as drivers, firmware, and CEE firmware).If they have not been completely installed, the CNA will not beshown in the show int brief output. If this is the case, once the CNAhas been completely installed, re-run the show int brief commandand verify that it is shown as being logged in.

Nexus-5020-1(config-if)# show int brief-----------------------------------------------------------------------------Interface Vsan Admin Admin Status SFP Oper Oper Port


-----------------------------------------------------------------------------fc2/1 100 NP -- up swl NP 2 --fc2/2 100 NP -- up swl NP 2 --fc2/3 300 NP -- up swl NP 2 --fc2/4 300 NP -- up swl NP 2 --fc3/1 100 NP -- up swl NP 2 --fc3/2 100 NP -- up swl NP 2 --fc3/3 300 NP -- up swl NP 2 --fc3/4 300 NP -- up swl NP 2 --

-----------------------------------------------------------------------------Interface Status IP Address Speed MTU Port

Channel-----------------------------------------------------------------------------Ethernet1/1 up -- 10000 1500 --Ethernet1/2 up -- 10000 1500 --Ethernet1/3 notConnect -- -- 1500 --...Ethernet3/4 sfpIsAbsen -- -- 1500 --...-----------------------------------------------------------------------------Interface Vsan Admin Admin Status SFP Oper Oper Port


-----------------------------------------------------------------------------vfc1 100 F -- up -- F auto --vfc2 300 F -- up -- F auto --Nexus-5020-1#


190


c. Use show npv flogi-table vsan <vsan ID> to verify that eachCNA is logged in to the appropriate vsan.

Nexus-5020-1(config-if)# show npv flogi-table vsan 100

-----------------------------------------------------------------------------SERVER EXTERNALINTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE-----------------------------------------------------------------------------vfc1 100 0x680101 10:00:00:00:c9:3c:f8:3c 20:00:00:00:c9:3c:f8:3c fc2/1

Total number of flogi = 1.

Nexus-5020-1(config-if)# show npv flogi table vsan 300

-----------------------------------------------------------------------------SERVER EXTERNALINTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE-----------------------------------------------------------------------------vfc2 300 0x690101 21:01:00:1b:32:2a:c0:b8 20:01:00:1b:32:2a:c0:b8 fc2/3



The fc interfaces are now online and running in NPV mode. Thevfc interfaces come online as soon as one of the fc interfaces in thesame vsan comes online.

13. Save the configuration:

Nexus-5020-1(config-if) # copy running startup-config

14. Zone the CNAs.

Since Nexus-5020-1 is running in NPV mode, all zoning will needto be performed from the ED-DCX-B Director to which it isattached. It is strongly recommended that if the CLI is to be usedfor zoning, then cut and paste should be utilized to minimizetranscription types of errors when manually typing in theWWPNs. The set of steps required to create the appropriate zonesare:

a. Create each zone by using zonecreate "<zone name a>","WWPN1; WWPN2; .... WWPNn".

b. Create the zone config by using cfgcreate "<cfg name>", "<zonename a>; <zone name b>; .... <zone name n>".



c. Enable the zone config by using cfgenable "<cfg name>".

d. Save the configuration by using cfgsave.

The exact steps that would be performed in this example are asfollows:

a. Zone the CNA logged in to 172.23.185.25 to access its storageport.

– Telnet into the switch with an IP address of 172.23.185.24.DCX_1:admin> zonecreate “host1_CNA1”, “10:00:00:00:c9:3c:f8:3c;

50:06:04:82:D5:2E:69:79”DCX_1:admin> cfgcreate “FCoE_1”, “host1_CNA1”DCX_1:admin> cfgenable “FCoE_1”DCX_1:admin> cfgshowDefault Zone: OFFSafe Zone: OFFDefined configuration:Effective configuration:cfg: FCoE_1zone: host1_CNA1

10:00:00:00:c9:3c:f8:3c50:06:04:82:D5:2E:69:79

DCX_1:admin>– Log out of the switch with an IP address of 172.23.185.24.

b. Zone the CNA logged in to 172.23.185.25 to access its Storageports.

– Telnet into the switch with an IP address of 172.23.185.25.DCX_2:admin> zonecreate “host2_CNA1”, “21:01:00:1b:32:2a:c0:b8;


21:01:00:1b:32:2a:c0:b850:06:04:82:D5:2E:69:49


Configuring Nexus-5020-2Nexus-5020-2 has two CNAs connected to it, one on interface 1/1 andthe other on interface 1/2. This example will configure the virtualinterfaces on interface 1/1 first and then interface 1/2.


192


To configure the Nexus-5020-2:






d. Enter a name such as Nexus, and then click OK.







Enter the password for "admin":

Note: You may need to press Enter several times before any text isdisplayed.

h. Enter the customer-supplied password for admin, which inthis case is nbv12345.

Note: It may be helpful to log the output by using the capture textfeature under the Transfer: pull-down menu.

i. Re-enter the password for the admin user.



The Basic System Configuration dialog box displays.

2. Complete the basic configuration as follows:



Note: If the installer does not use install startup script, managementVRF requires manual installation.

Create another login account (yes/no) [n]: NConfigure read-only SNMP community string (yes/no) [n]: NConfigure read-write SNMP community string (yes/no) [n]: NEnter the switch name: Nexus-5020-2Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: YMgmt0 IPv4 address: 10.1.1.113Mgmt0 IPv4 netmask: 255.255.255.0Configure the default gateway? (yes/no) [y]: YIPv4 address of the default gateway: 172.23.185.2Enable the telnet service? (yes/no) [y]: YEnable the ssh service? (yes/no) [n]: YType of ssh key you would like to generate (dsa/rsa): dsaNumber of key bits <768-2048>: 1024Configure the ntp server? (yes/no) [n]: NEnter basic FC configurations (yes/no) [n]: YConfigure default physical FC switchport interface state (shut/noshut) [shut]:



The following configuration will be applied:switchname Nexus-5020-2interface mgmt0ip address 10.1.1.113 255.255.255.0no shutdownexit


b. Answer the questions as follows:



194



c. Once this is completed, you will be prompted to log in asfollows:

Nexus-5020-2 login: adminPassword: nbv12345



You will notice between 40 and 52 Ethernet interfaces.






--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth1/1 1 eth access down Link not connected 10G(D) --Eth1/2 1 eth access down Link not connected 10G(D) --Eth1/3 1 eth access down Link not connected 10G(D) --...Eth1/40 1 eth access down Link not connected 10G(D) --




f. If there are no FC interfaces displayed, enable the FCoEfeature as follows:

Nexus-5020-2# config terminalNexus-5020-2(config)# feature fcoe

Note: It may be necessary to press Enter if the prompt is notdisplayed after entering the feature FCoE command.

g. Save the configuration.


h. Open an SSH session to the Nexus 5020 with an IP address of10.1.1.113.


Nexus-5020-2# configNexus-5020-2(config)# vsan databaseNexus-5020-2(config-vsan-db)# vsan 200Nexus-5020-2(config-vsan-db)# vsan 400Nexus-5020-2(config-vsan-db)# exitNexus-5020-2(config)#


switch (config-sys-qos)# service-policy type qos inputfcoe-default-in-policyswitch (config-sys-qos)# service-policy type queuing inputfcoe-default-in-policyswitch (config-sys-qos)# service-policy type queuing outputfcoe-default-out-policyswitch (config-sys-qos)# service-policy type network-qosfcoe-default-nq-policy





196


6. Use show FCoE VLAN to verify the configuration was successful.


VLAN VSAN Status-------- -------- --------200 200 Non-operational400 400 Non-operational


Nexus-5020-2(config-vlan)# interface vfc 1Nexus-5020-2(config-if)# bind interface ethernet 1/1Nexus-5020-2(config-if)# switchport trunk allowed vsan 200Nexus-5020-2(config-if)# no shutNexus-5020-2(config-if)# interface vfc 2Nexus-5020-2(config-if)# bind interface ethernet 1/2Nexus-5020-2(config-if)# switchport trunk allowed vsan 400Nexus-5020-2(config-if)# no shut

8. Configure the Ethernet Interface settings.

Nexus-5020-2(config-if)# interface ethernet 1/1Nexus-5020-2(config-if)# switchport mode trunkNexus-5020-2(config-if)# switchport trunk native vlan 1Nexus-5020-1(config-if)# spanning-tree bpduguard enableNexus-5020-1(config-if)# spanning-tree port type edge trunkNexus-5020-2(config-if)# switchport trunk allowed vlan 1, 200Nexus-5020-1(config-if)# spanning-tree bpduguard enableNexus-5020-1(config-if)# spanning-tree port type edge trunkNexus-5020-2(config-if)# interface ethernet 1/2Nexus-5020-2(config-if)# switchport mode trunkNexus-5020-2(config-if)# switchport trunk native vlan 1Nexus-5020-1(config-if)# switchport trunk allowed vlan 1, 400

9. Add the vfc interfaces to the proper VSAN.


The output of show int brief displays the following:

Nexus-5020-2(config-if)# show int brief



-------------------------------------------------------------------------------fc2/1 1 auto on trunking swl TE 4 --



fc2/2 1 auto on trunking swl TE 4 --fc2/3 1 auto on trunking swl TE 4 --fc2/4 1 auto on trunking swl TE 4 --fc3/1 1 auto on trunking swl TE 4 --fc3/2 1 auto on trunking swl TE 4 --fc3/3 1 auto on trunking swl TE 4 --fc3/4 1 auto on trunking swl TE 4 --


Channel-----------------------------------------------------------------------------Ethernet1/1 up -- 10000 1500 --Ethernet1/2 up -- 10000 1500 --Ethernet1/3 sfpIsAbsen -- -- 1500 --...Ethernet3/4 sfpIsAbsen -- -- 1500 --...-----------------------------------------------------------------------------Interface Vsan Admin Admin Status SFP Oper Oper Port


-----------------------------------------------------------------------------vfc1 200 F -- up -- F auto --vfc2 400 F -- up -- F auto --Nexus-5020-2(config-if)#

The output from this show int brief command differs from thedisplay shown in Step e. Note that the vfc interfaces that werecreated in the previous steps are now displayed.

Note: In this example, Nexus-5020-2 is connected to two differentMDS switches and the interfaces on each MDS are in differentVSANs. One VSAN will be used for the server using an Emulex CNAand the other VSAN will be used for the server using a QLogic CNA.

Enable all of the fc interfaces.

Nexus-5020-2(config-if)# interface fc2/1-4Nexus-5020-2(config-if)# no shutNexus-5020-2(config-if)# interface fc3/1-4Nexus-5020-2(config-if)# no shut

i. Connect the cables from the CNAs to Nexus-5020-2 as shownin Figure 29 on page 176.


198


j. Validate the configuration using the show int brief command.

The following is a partial output of the show int briefcommand:




-------------------------------------------------------------------------------fc2/1 1 auto on notConnected swl -- --fc2/2 1 auto on notConnected swl -- --fc2/3 1 auto on notConnected swl -- --fc2/4 1 auto on notConnected swl -- --fc3/1 1 auto on notConnected swl -- --fc3/2 1 auto on notConnected swl -- --fc3/3 1 auto on notConnected swl -- --fc3/4 1 auto on notConnected swl -- --


Channel-----------------------------------------------------------------------------Ethernet1/1 up -- 10000 1500 --Ethernet1/2 up -- 10000 1500 --Ethernet1/3 sfpIsAbsen -- -- 1500 --...Ethernet3/4 sfpIsAbsen -- -- 1500 --...




10. Configure the FC interfaces on the Cisco MDS 9513s.



Note: The storage ports attached to each MDS will need to be added tothe proper VSAN in order for the CNAs to discover them. In addition,the Allowed VSANs list for TE-ports (used for Cisco trunking of VSANs)connecting the Nexus 5020 and the MDS will need to be checked toensure that they can carry VSANs 200 and 400.

The basic steps for configuring the FC interfaces on the MDS are:

a. Telnet in to the MDS switch.

b. Create the appropriate vsan on the MDS switch using thecommand vsan <VSAN ID>.

c. Add the appropriate interfaces on the MDS switches to theappropriate VSAN using the vsan <VSAN ID> interface fc<slot/port> command.

The actual commands used for this example are as follows:

• For MDS-9513-1:

MDS-9513-1# config terminalEnter configuration commands, one per line. End with CNTL/Z.MDS-9513-1(config)# vsan databaseMDS-9513-1(config-vsan-db)# vsan 400MDS-9513-1(config-vsan-db)# vsan 400 interface fc 2/14Traffic on fc2/14 may be impacted. Do you want to continue? (y/n) yMDS-9513-1(config-vsan-db)# exitMDS-9513-1(config) # copy running startup-configMDS-9513-1(config)#

• For MDS-9513-2:

MDS-9513-2# config terminalEnter configuration commands, one per line. End with CNTL/Z.MDS-9513-2(config)# vsan databaseMDS-9513-2(config-vsan-db)# vsan 200MDS-9513-2(config-vsan-db)# vsan 200 interface fc 2/13Traffic on fc2/13 may be impacted. Do you want to continue? (y/n) yMDS-9513-2(config-vsan-db)# exitMDS-9513-2(config) # copy running startup-configMDS-9513-2(config)#

11. Connect cables from CNAs to Nexus-5020-2.

a. Once the interface configuration has been verified, connect thecables between the CNAs and Nexus-5020-2 as well as fromthe FC switches to Nexus-5020-2.

b. Use show int brief to validate that the interfaces initialized asexpected.


200





-----------------------------------------------------------------------------fc2/1 1 auto on trunking swl TE 4 --fc2/2 1 auto on trunking swl TE 4 --fc2/3 1 auto on trunking swl TE 4 --fc2/4 1 auto on trunking swl TE 4 --fc3/1 1 auto on trunking swl TE 4 --fc3/2 1 auto on trunking swl TE 4 --fc3/3 1 auto on trunking swl TE 4 --fc3/4 1 auto on trunking swl TE 4 -------------------------------------------------------------------------------Interface Status IP Address Speed MTU Port

Channel-----------------------------------------------------------------------------Ethernet1/1 up -- 10000 1500 --Ethernet1/2 up -- 10000 1500 --Ethernet1/3 sfpIsAbsen -- -- 1500 --...Ethernet3/4 sfpIsAbsen -- -- 1500 --...-----------------------------------------------------------------------------Interface Vsan Admin Admin Status SFP Oper Oper Port



12. Verify that the appropriate VSANs are in the TE ports allowed listand that they are shown as up. To do this, use the show interfacetrunk vsan command, as shown next.

Nexus-5020-2# show interface trunk vsanfc2/1 is trunking

Vsan 1 is up (None), FCID is 0x9a0000Vsan 400 is up (None), FCID is 0x4e0001

fc2/2 is trunkingVsan 1 is up (None), FCID is 0x9a0001Vsan 200 is up (None), FCID is 0xd00001

fc2/3 is trunkingVsan 1 is up (None), FCID is 0x9a0002Vsan 400 is up (None), FCID is 0x4e0002








c. Use show flogi database vsan <vsan ID> to verify that eachCNA is logged in to the appropriate vsan.

Nexus-5020-2(config-if)# show flogi database vsan 200---------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME---------------------------------------------------------------------------vfc1 200 0xd00000 10:00:00:00:c9:3c:f7:c0 20:00:00:00:c9:3c:f7:c0


Nexus-5020-2(config-if)# show flogi database vsan 400

---------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME---------------------------------------------------------------------------vfc2 400 0x4e0000 21:00:00:1b:32:0a:c0:b8 20:00:00:1b:32:0a:c0:b8



13. Zone the CNAs.

It is strongly recommended that if the CLI is to be used forzoning, that cut and paste be utilized to minimize transcriptiontypes of errors when manually typing in the WWPNs. The set ofsteps required to create the appropriate zones are:

a. zone name <ZONE_NAME> vsan <VSAN ID>.

b. member pwwn <PWWN_1>.

c. member pwwn <PWWN_2>.

d. zone set name <ZONE_SET_NAME> vsan <VSAN ID>.


202


e. member <ZONE_NAME>.

f. zoneset activate name <ZONE_SET_NAME> vsan <VSANID>.

g. Copy running startup.

14. Telnet into Nexus-5020-2 by completing the following steps:

Create zones for the VSAN 200:

Nexus-5020-2# config terminalNexus-5020-2(config)# zone name host1_CNA2 vsan 200Nexus-5020-2(config-zone)# member pwwn 10:00:00:00:c9:3c:f7:c0Nexus-5020-2(config-zone)# member pwwn 50:06:04:8A:D5:2E:69:79Nexus-5020-2(config-zone)# exitNexus-5020-2(config)# zoneset name FCoE_200 vsan 200Nexus-5020-2(config-zoneset)# member host1_CNA2Nexus-5020-2(config-zoneset)# zoneset activate name FCoE_200 vsan 200Zoneset activation initiated. check zone statusNexus-5020-2(config)# zoneset distribute full vsan 200Nexus-5020-2(config)# show zoneset active vsan 200zoneset name FCoE_200 vsan 200

zone name host1_CNA2 vsan 200* fcid 0xd00000 [pwwn 10:00:00:00:c9:3c:f7:c0]* fcid 0x6f0000 [pwwn 50:06:04:8a:d5:2e:69:79]

Nexus-5020-2(config)#


Nexus-5020-2(config)# zone name host2_CNA2 vsan 400Nexus-5020-2(config-zone)# member pwwn 21:00:00:1B:32:0A:C0:B8Nexus-5020-2(config-zone)# member pwwn 50:06:04:82:D5:2E:69:59Nexus-5020-2(config-zone)# exitNexus-5020-2(config)# zoneset name FCoE_400 vsan 400Nexus-5020-2(config-zoneset)# member host2_CNA2Nexus-5020-2(config-zoneset)# zoneset activate name FCoE_400 vsan 400Zoneset activation initiated. check zone statusNexus-5020-2(config)# zoneset distribute full vsan 400Nexus-5020-2(config)#




◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel andiSCSI Host Bus Adapters (HBAs) and Converged Network Adapters(CNAs) in the Windows Environment



◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel HostBus Adapters (HBAs) and Converged Network Adapters (CNAs) in theWindows Environment

◆ EMC Host Connectivity Guide for Windows


◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel andiSCSI Host Bus Adapters (HBAs) and Converged Network Adapters(CNAs) for the Linux Environment

◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel HostBus Adapters (HBAs) and Converged Network Adapters (CNAs) for theLinux v2.6.x Kernel Environment and the v8.x-Series Driver

◆ EMC Host Connectivity Guide for Linux

Configuring the VMware hostTo configure the VMware host, refer to the following documentavailable EMC Online Support at https://support.emc.com:

◆ EMC Host Connectivity Guide for VMware ESX Server

Configuring EMC storageOnce the switch, hosts, and adapters have been configured, thestorage will need to be configured to each initiator to access a numberof LUNs.

Configuring network parametersUsing the network topology shown in Figure 31 on page 180, displayconfiguration parameters and then follow the steps needed to deploythis Layer 2 network.

Etherchannel configuration on a Nexus 7000 switchThe example shown next is the creation of the proper Etherchannelsetup on a Cisco Catalyst or Cisco Nexus 7000 switch to connectsuccessfully to the Nexus 5020 switch using Ethernet port channels.An Etherchannel is a port channel for L2 VLAN connectivity. Do notconfuse an Ethernet channel with SAN-port channels, which are usedfor VSAN connectivity across bundled SAN E_Ports.


204


Create the port channel on Core-N7K-1A switchport, when a part of the Ethernet, is a layer-2 switching port.

interface Port-channelswitchportswitchport trunk encapsulation dot1q1

switchport trunk allowed vlan 2002

switchport mode trunkno ip address

10 GigE interfaces 1/1 and 2/1 on Core-N7K-1 are the membersused to make up this port channel:

interface TenGigabitEthernet1/1switchportswitchport trunk encapsulation dot1qswitchport trunk allowed vlan 200switchport mode trunkno ip addresschannel-group 1 mode active3

interface TenGigabitEthernet2/1switchportswitchport trunk encapsulation dot1qswitchport trunk allowed vlan 200switchport mode trunkno ip addresschannel-group 1 mode active

Spanning tree mode for vlan 200 is set to run RPVST versus MST:

spanning-tree mode rapid-pvstspanning-tree vlan 200 priority 4096

Create the port channel on Core-N7K-2Perform the steps from “Create the port channel on Core-N7K-1” onpage 204 on Core-N7K-2.

VLAN and Etherchannel configuration on Nexus 5020This section shows how to configure the Nexus-5020-1 andNexus-5020-2.

1. 802.1q is the only supported type for connection to N5K.2. Only VLAN 200 is allowed across the port channel.3. Interface is assigned to port channel 1, active keyword turns on LACP protocol for this link.



Configure Nexus-5020-1

IMPORTANT

It is important to configure the port channels in this order for "easyon" implementation.

Nexus-5020-1(config)# vlan 2001

Nexus-5020-1(config-vlan)# exitNexus-5020-1(config)# feature lacp ß Link aggregation protocol enabled on

Nexus-5020-1Nexus-5020-1(config)# interface port-channel 1Nexus-5020-1(config-if-range)# exitNexus-5020-1(config)# interface ethernet 2/1, 3/12

Nexus-5020-1(config-if-range)# channel-group 1 mode active3

Message***Ethernet1/3 Ethernet1/4 added to port-channel1***Nexus-5020-1(config-if-range)# interface port-channel 1 ß places you back into

the port-channel interfaceNexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# switchport trunk allowed vlan 2004

Nexus-5020-1(config-if)# no shutNexus-5020-1(config-if)# copy running startup-config

Configure Nexus-5020-2Repeat the steps in “Configure Nexus-5020-1” on page 205.

Common show commands to display configuration results forport-channel operation:

Nexus-5020-1# sh port-channel summaryFlags: D - down U - up in port-channel

I - Individual S - suspendedH - Hot-standby (LACP only)R - Module-removed

---------------------------------------------------------------Group Port- Type Protocol Member Ports Channel---------------------------------------------------------------1 Po1(U) Eth LACP Eth1/3(U) Eth1/4(U)5

1. VLAN 200 is configured on the Nexus 5020, matching the VLAN on the core switch.2. Assigns interfaces 2/1 & 3/1 to the port channel, you can also use hyphens to assign a range of interfaces,

such as 1/3-4.3. Enables LACP.4. Allows only VLAN 200 across this port-channel.5. Up and connected using LACP.


206


Spanning tree configuration and verification on Nexus 5020You need to know what spanning-tree protocol the VLAN willoperate in. Default spanning-tree protocol on the Nexus 5020 isRPVST+. The following example assumes that the core Ethernetswitch is running RSTP.

Nexus-5020-1# show spanning-tree vlan 200VLAN0200

Spanning tree enabled protocol rstpRoot ID Priority 32768

Address 001d.a1e5.ec58Cost 1Port 4097 (port-channel2)Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32984 (priority 32768 sys-id-ext 200)Address 000d.ec6d.7b41Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------Po1 Root FWD 1 128.4097 P2p

Nexus-5020-1# sho spanning-tree active detailVLAN0200 is executing the rstp compatible Spanning Tree protocolBridge Identifier has priority 32768, sysid 211, address 000d.ec6d.7b41Configured hello time 2, max age 20, forward delay 15Current root has priority 32768, address 001d.a1e5.ec58Root port is 4097 (port-channel1), cost of root path is 1Topology change flag not set, detected flag not setNumber of topology changes 1 last change occurred 0:13:54 ago

from port-channel1Times: hold 1, topology change 35, notification 2

hello 2, max age 20, forward delay 15Timers: hello 0, topology change 0, notification 0

Nexus-5020-1# show spanning-tree summarySwitch is in rapid-pvst modeRoot bridge for: nonePort Type Default is disableEdge Port [PortFast] BPDU Guard Default is disabledEdge Port [PortFast] BPDU Filter Default is disabledBridge Assurance is disabledLoopguard Default is disabledPathcost method used is short

Name Blocking Listening Learning Forwarding STP Active---------------------- -------- --------- -------- ---------- ----------VLAN0200 0 0 0 1 1---------------------- -------- --------- -------- ---------- ----------1 vlan 0 0 0 1 1



Configuration of Multiple Spanning Tree for a VLANMultiple Spanning Tree (MST), which is the IEEE 802.1s standard,allows you to assign two or more VLANs to a spanning-tree instance.MST is not the default spanning-tree mode.

This example shows how to enter MST configuration mode, mapVLANs 2 and 4 to MST instance 1, name the region region1, and setthe configuration revision to 1.

Nexus-5020-1(config)# spanning-tree mode mst1

Nexus-5020-1(config)# spanning-tree mst configuration2Nexus-5020-1 (config-mst)# instance 1 vlan 2,4Nexus-5020-1 (config-mst)# name region1Nexus-5020-1 (config-mst)# revision 1Nexus-5020-1 (config-mst)# exit3

Multicast on Nexus5020

By default, the Nexus 5020 switch is enabled for multicast support.IGMP snooping is enabled by default.

Internet Group Management Protocol (IGMP) is used to manage themembership of Internet Protocol multicast groups. IGMP is used byIP hosts and adjacent multicast routers to establish multicast groupmemberships. IGMP Snooping is the process of listening to IGMPtraffic. IGMP snooping, as implied by the name, is a feature thatallows the switch to "listen in" on the IGMP conversation betweenhosts and routers by processing the layer 3 IGMP packets sent in amulticast network. (IGMP Snooping is "on" by default for the Nexus5020.)

To display proper operation of the multicast environment and to seeenabled igmp options, use <show ip igmp snooping>. The output hereshows all defaults. This example shows default VLAN 1 and VLAN5. VLAN 5 is running multicast.

Nexus-5020-1# show ip igmp snoopingGlobal IGMP Snooping Information:IGMP Snooping enabled

IGMP Snooping information for vlan 1IGMP snooping enabledIGMP querier noneSwitch-querier disabledExplicit tracking enabled

1. Enables MST feature.2. Enters config mode for MST.3. Commits all the changes and exits MST configuration.


208


Fast leave disabledReport suppression enabledRouter port detection using PIM Hellos, IGMP QueriesNumber of router-ports: 0Number of groups: 0

IGMP Snooping information for vlan 5IGMP snooping enabledIGMP querier present, address: 172.16.24.1, version: 3Querier interval: 125 secsQuerier last member query interval: 10 secsQuerier robustness: 2Switch-querier enabled, address 172.16.24.1, currently runningExplicit tracking enabledFast leave enabledReport suppression enabledRouter port detection using PIM Hellos, IGMP QueriesNumber of router-ports: 1Number of groups: 1

Other useful commands to verify proper operation on the core switch,in this case a Nexus 7000:

Note: The configuration commands listed below will work for both theNexus 7000 and the Cat6500.

show vlan

show port-channel summary

show port-channel database

show port-channel traffic

show port-channel compatibility-parameters

show port-channel load-balance

show port-channel usage

sho interface port-channel 2 switchport

sho interface port-channel 2 trunk

show lacp neighbor

sho interface port-channel 2 counters

sho interface port-channel 2 counters errors



N7K-1# show spanning-treeVLAN0200Spanning tree enabled protocol rstpRoot ID Priority 4096

Address 0011.bc52.20d3This bridge is the rootHello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 4096Address 0011.bc52.20d3Hello Time 2 sec Max Age 20 sec Forward Delay 15 secAging Time 300

Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------Po1 Desg FWD 1 128.1665 P2p


210


Nexus 7000, Nexus 5000, and MDS 9500 series topologyThis section details the setup steps required to create theconfiguration shown in Figure 32 on page 212, the target topology. Itincludes the following information:




◆ “Configuring the Nexus 7000” on page 214


◆ “Configuring the MDS 9513” on page 238

Summary of configuration steps“Configuring N7K-1,” discussed further on page 214.

1. Create storage VDC.


3. Create vfcs.

4. Configure vPC.

5. Connect cables.

“Configuring N7K-2,” discussed further on page 223.

1. Create storage VDC.


3. Create vfcs.

4. Configure vPC.

5. Connect cables.

“Configuring Nexus-5548-1,” discussed further on page 231.


2. Create vfcs.

3. Configure vPC.

4. Connect cables.



“Configuring Nexus-5548-2,” discussed further on page 234.


2. Create vfcs.

3. Configure vPC.

4. Connect cables.

“Configuring MDS-9513-1,” discussed further on page 238.


2. Create vfcs.

3. Connect cables.

“Configuring MDS-9513-2,” discussed further on page 240.


2. Create vfcs.

3. Connect cables.

The process of configuring vPC is covered in “Virtual PortChannel”on page 293. As a result, the vPC configuration steps will not berepeated in this case study.

Nexus 7000, Nexus 5000, and MDS 9500 series topology 211

212




8 7 6 5 4 3

M1

F12 1

8 7 6 5 4 3 2 1

Host_1

CNA_1 CNA_2

Nexus-5548-1(FC-SW)

Nexus-5548-2(NPV)

10GbE (no FCoE)

N7K-1 N7K-2

1 2 3 4 5 6

M1

F1

1 2 3 4 5 6 7 8 9 10

11

12

12

11

10 9 8 7 6 5 4 3 2 1

7 8

1 2 3 4 5 6 7 8

1 2 3 4

FCoE 1 2 3 4 5

MDS-9513-1

FCoE5 4 3

MDS-9513-2

VDC 1

VDC Storage

VDC 1

VDC Storage

SAN A SAN B

vPC

vPC

10GbE (FCoE only)10GbE (Converged)

VLAN 100

VLAN/ VSAN 200

Po403

Po402

Po401Po200

Po200 Po100Po100

Po301

Po302

Po303

vPC200 vPC100

5 4 3 2 1 FC

Native FC

VMAX/VNX

ICO-IMG-001031



Table 2 lists the Target topology IP address and Domain IDassignments.

Table 3 lists the Target topology host information.

Table 4 lists the Target topology storage information.

Table 2 IP address and Domain ID assignment

Target topology IP address Domain ID assignment

N7K-1 10.1.1.10 N/A

N7K-1-Storage VDC 10.1.1.11 1

MDS-9513-1 10.1.1.12 3

Nexus-5548-1 10.1.1.13 5

N7K-2 10.2.1.10 N/A

N7K-2-Storage VDC 10.2.1.11 2

MDS-9513-2 10.2.1.12 4

Nexus-5548-2 10.2.1.13 6

Table 3 Host 1 information

CNA Vendor CNA Port # WWPN Storage Ports

Emulex 0 10:00:00:00:c9:3c:f8:3c 1 and 2

Emulex 0 10:00:00:00:c9:3c:f7:c0 3 and 4

Table 4 Storage 1 information

Name Storage WWPN Host WWPN

Storage 1 50:06:04:82:D5:2E:69:79 10:00:00:00:c9:3c:f8:3c

Storage 2 50:06:04:82:D5:2E:69:49 10:00:00:00:c9:3c:f8:3c

Storage 3 50:06:04:8A:D5:2E:69:79 10:00:00:00:c9:3c:f7:c0

Storage 4 50:06:04:8A:D5:2E:69:49 10:00:00:00:c9:3c:f7:c0


214


PrerequisitesBefore configuring the Nexus or MDS switches, confirm thefollowing:

◆ All switch chassis are powered up and have had IP Addressesassigned to them.

◆ In the Nexus 7000, slot 1 contains an M1 linecard and slot 2contains an F1 line card.

◆ Nexus-5548-2 has the full storage services license installed.

◆ In MDS-9513-1, slot 1 contains an FCoE line card.

◆ In MDS-9513-2, slot 1 contains an FCoE line card and slot 2contains a line card that supports FC.

Configuring the Nexus 7000This section contains the following information:

◆ “Configuring N7K-1” on page 214

◆ “Configuring N7K-2” on page 223

Configuring N7K-1 Note: This section assumes that the N7K chassis is powered up and has hadan IP Address assigned to it. It also assumes that slot 1 contains an M1 linecard and slot 2 contains an F1 line card.

Nexus 7000 FCoE requirements◆ Minimum release of NX-OS 5.2(1)

◆ FCoE module N7K-F132XP-15

◆ FCoE license FCOE-N7K-F132XP

◆ FCoE will run on a Storage VDC (virtual device context)

Note: The Nexus 7000 does not have any native FC capabilities and does notoperate in NPV mode.

Create the Storage VDCAfter logging in to the Nexus 7000, you will be in the default VDCwhere you will complete the following configuration steps. Thesesteps will install the FCoE license and build the Storage VDC.



Note: Make sure the FCoE module is physically installed in the chassis beforetrying to enable the FCoE license.


N7K-1# config tEnter configuration commands, one per line. End with CNTL/Z.N7K-1(config)# install feature-set fcoe

2. Update QoS settings.

N7K-1(config)# system qosN7K-1(config-sys-qos)# service-policy type network-qos default-nq-7e-policyN7K-1(config-sys-qos)# exitN7K-1(config)#

3. Enable LACP and LLDP.

N7K-1(config)# feature lacpN7K-1(config)# feature lldpN7K-1(config)#

4. Create the Storage VDC.

Note: The vdc name "StorageVDC" is used for example purposes onlyand is not a requirement.

N7K-1(config)# vdc StorageVDC type storageNote: Creating VDC, one moment please ...N7K-1(config-vdc)#

5. Associate the FCoE license to the FCoE module.

N7K-1(config-vdc)# license fcoe module 2N7K-1(config)#

6. Verify the license is installed.

N7K-1 (config)# show license usageFeature Ins Lic Status Expiry Date Comments

Count--------------------------------------------------------------------------------FCOE-N7K-F132XP Yes 1 In use Never -Note: output truncated--------------------------------------------------------------------------------

7. Verify that the FCoE feature is installed.

N7K-1(config)# show feature-set fcoeFeature Set Name ID State-------------------- -------- --------fcoe 1 enabled


216


8. Allocate Ethernet interface to Storage VDC.

In this example we have the FCoE module inserted into slot 2.

When allocating ports to the storage VDC, keep in mind that bothmembers of each port group will need to be assigned to the sameVDC. For the FCoE "F1" blade, the port groups are as shown inTable 5.

N7K-1(config)# vdc StorageVDCN7K-1(config-vdc)# allocate interface ethernet 2/1-8Moving ports will cause all config associated to them in source vdc to be removed.

Are you sure you want to move the ports (y/n)? [yes] yN7K-1(config-vdc)#

9. Allocate the VLANS to be used for FCoE.

Note: By default the range is from 1 - 3967.

N7K-1 (config-vdc)# allocate fcoe-vlan-range 200N7K-1 (config-vdc)#

Table 5 Port groups

Port Group Port Number

Group 1 1 and 2

Group 2 3 and 4

Group 3 5 and 6

Group 4 7 and 8

Group 5 9 and 10

Group 6 11 and 12

Group 7 13 and 14

Group 8 15 and 16

Group 9 17 and 18

Group 10 19 and 20

Group 11 21 and 22

Group 12 23 and 24

Group 13 25 and 26

Group 14 27 and 28

Group 15 29 and 30

Group 16 31 and 32



10. Switch to the newly created VDC and complete the VDC setup.

Note: The answers to the following prompts are provided for the sake ofthis example only and do not necessarily represent best practices.

N7K-1 (config-vdc)# switchto vdc StorageVDC

---- System Admin Account Setup ----

Do you want to enforce secure password standard (yes/no) [y]: n

Enter the password for "admin": <Enter password>Confirm the password for "admin": <Re-enter password>

---- Basic System Configuration Dialog VDC: 4 ----

This setup utility will guide you through the basic configuration ofthe system. Setup configures only enough connectivity for managementof the system.

Please register Cisco Nexus7000 Family devices promptly with yoursupplier. Failure to register may affect response times for initialservice calls. Nexus7000 devices must be registered to receiveentitled support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytimeto skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): y

Create another login account (yes/no) [n]: n

Configure read-only SNMP community string (yes/no) [n]: n

Configure read-write SNMP community string (yes/no) [n]: n

Enter the switch name : StorageVDC

Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: y

Mgmt0 IPv4 address : 10.1.1.11

Mgmt0 IPv4 netmask : 255.255.255.0

Configure the default gateway? (yes/no) [y]: y

IPv4 address of the default gateway : 10.1.1.1

Configure advanced IP options? (yes/no) [n]: n


218


Enable the telnet service? (yes/no) [n]: n

Enable the ssh service? (yes/no) [y]: y

Type of ssh key you would like to generate (dsa/rsa) [rsa]: rsa

Number of rsa key bits <1024-2048> [1024]: 1024

Configure default interface layer (L3/L2) [L3]: L3

Configure default switchport interface state (shut/noshut) [shut]: shut

The following configuration will be applied:no password strength-checkswitchname StorageVDC

interface mgmt0ip address 10.1.1.11 255.255.255.0no shutdownvrf context managementip route 0.0.0.0/0 10.1.1.1exitno feature telnetssh key rsa 1024 forcefeature sshno system default switchportsystem default switchport shutdown

Would you like to edit the configuration? (yes/no) [n]: n

Use this configuration and save it? (yes/no) [y]: y

Configuration update aborted: This vdc has had a global configuration change sincethe last saved config. Please save config in default vdc before proceeding

Cisco Nexus Operating System (NX-OS) SoftwareTAC support: http://www.cisco.com/tacCopyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.The copyrights to certain works contained in this software areowned by other third parties and used and distributed underlicense. Certain components of this software are licensed underthe GNU General Public License (GPL) version 2.0 or the GNULesser General Public License (LGPL) Version 2.1. A copy of eachsuch license is available athttp://www.opensource.org/licenses/gpl-2.0.php andhttp://www.opensource.org/licenses/lgpl-2.1.phpN7K-1-StorageVDC#



11. Ensure that the appropriate interfaces have been allocated to theVDC.

N7K-1-StorageVDC# show interface brief

-------------------------------------------------------------------------------Interface Status Speed

(Gbps)-------------------------------------------------------------------------------sup-fc0 up 1

--------------------------------------------------------------------------------

Port VRF Status IP Address Speed MTU--------------------------------------------------------------------------------mgmt0 -- up 10.1.1.11 1000 1500

--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch#--------------------------------------------------------------------------------Eth2/1 1 eth access down Administratively down 10G(D) --Eth2/2 1 eth access down Administratively down 10G(D) --Eth2/3 1 eth access down Administratively down 10G(D) --Eth2/4 1 eth access down Administratively down 10G(D) --Eth2/5 1 eth access down Administratively down 10G(D) --Eth2/6 1 eth access down Administratively down 10G(D) --Eth2/7 1 eth access down Administratively down 10G(D) --Eth2/8 1 eth access down Administratively down 10G(D) --N7K-1-StorageVDC#

12. Enable needed features within Storage VDC.

Note: Enabling these features within the Storage VDC does not enablethem in the default VDC.

N7K-1-StorageVDC# configEnter configuration commands, one per line. End with CNTL/Z.N7K-1-StorageVDC(config)# feature lldpN7K-1-StorageVDC(config)# feature lacp


N7K-1-StorageVDC(config)# exitN7K-1-StorageVDC# exitN7K-1(config-vdc)# copy running-config startup-config[########################################] 100%Copy complete, now saving to disk (please wait)...N7K-1(config-vdc)#


220


14. Verify the Storage VDC.

N7K-1(config-vdc)# show vdc StorageVDC

vdc_id vdc_name state mac type lc------ -------- ----- ---------- --------- ------------------------------------1 N7K-1 active 00:24:f7:17:05:c1 Ethernet m1 f1 m1xl2 StorageVDC active 00:24:f7:17:05:c2 Storage f1

Configure FCoE settings on the Storage VDC

1. Switch to the Storage VDC.

N7K-1# switchto vdc StorageVDCN7K-1-StorageVDC# configN7K-1-StorageVDC(config)#

2. Create the FCoE VLAN.

Note: VLANs created in the storage VDC must fall within the range ofVLAN IDs specified in the "allocate fcoe-vlan-range <VLAN RANGE>"configuration parameter. See step 8 on page 216 for additionalinformation.

N7K-1-StorageVDC(config)# vlan 100N7K-1-StorageVDC(config-vlan)# fcoe vsan 100N7K-1-StorageVDC(config-vlan)#

3. Create the FCoE VSAN and add the vfc for the storage port.

N7K-1-StorageVDC(config-vlan)# vsan databaseN7K-1-StorageVDC(config-vsan-db)# vsan 100N7K-1-StorageVDC(config-vsan-db)# exitN7K-1-StorageVDC(config)#

4. Configure the PortChannels.

Note: As shown in the target topology diagram, Figure 32 on page 212,in this example there are 5 physical interfaces being used in the storageVDC. Two of these physical interfaces will be added to PortChannel 401,two others will be added to PortChannel 403. Each of these PortChannelswill be configured in this section.



IMPORTANT

One of the PortChannels that will be configured in this sectionwill be attached to an MDS. When configuring a PortChannelinterface on an MDS, the PortChannel ID must be greater than256 to avoid a potential overlap with a SAN-PortChannel IDinstance. This restriction on PortChannel IDs does not exist onthe Nexus 7000. As a result, if a PortChannel ID less than orequal to 256 is used on the Nexus 7000, it will not be possible touse the same ID on the MDS. Although it is not a requirementfor the IDs to match on both sides of the PortChannel, it isconsidered a best practice to have them match to eliminateconfusion in the future should the need to troubleshoot arise.

a. Configure PortChannel 401.

N7K-1-StorageVDC(config)# interface port-channel 401N7K-1-StorageVDC(config-if)# switchportN7K-1-StorageVDC(config-if)# switchport mode trunkN7K-1-StorageVDC(config-if)# switchport trunk allowed vlan 100N7K-1-StorageVDC(config-if)# no shut

b. Configure PortChannel 403.


5. Configure the physical interfaces.

Note: As shown in the target topology diagram, Figure 32 on page 212,there are 5 physical interfaces in this example being used in the storageVDC. Two of these physical interfaces will be added to PortChannel 401,two others will be added to PortChannel 403 and the final interface willbe used to connect to port 2 on the Storage array. Each of these interfaceswill be configured in this section.

a. Configure interface Ethernet 2/1-2.

N7K-1-StorageVDC(config)# interface ethernet 2/1-2N7K-1-StorageVDC(config-if-range)# switchport mode trunkN7K-1-StorageVDC(config-if-range)# switchport trunk allowed vlan 100N7K-1-StorageVDC(config-if-range)# channel-group 403 mode activeN7K-1-StorageVDC(config-if-range)# no shutN7K-1-StorageVDC(config-if-range)# exitN7K-1-StorageVDC(config)#


222


b. Configure interface Ethernet 2/3-4.


c. Configure interface Ethernet 2/8.

N7K-1-StorageVDC(config)# interface ethernet 2/8N7K-1-StorageVDC(config-if)# switchport mode trunkN7K-1-StorageVDC(config-if)# switchport trunk allowed vlan 100N7K-1-StorageVDC(config-if)# no shutN7K-1-StorageVDC(config-if)# exitN7K-1-StorageVDC(config)#


Note: As shown in the target topology diagram, Figure 32 on page 212,there are 5 physical interfaces in this example being used in the storageVDC. Two of these physical interfaces will be added to PortChannel 401,two others will be added to PortChannel 403 and the final interface willbe used to connect to port 2 on the Storage array. Each of these interfaceswill be configured in this section

a. Configure interface vfc-port-channel 401.

N7K-1-StorageVDC(config)# interface vfc-port-channel 401N7K-1-StorageVDC(config-if)# switchport mode eN7K-1-StorageVDC(config-if)# switchport trunk allowed vsan 100N7K-1-StorageVDC(config-if)# no shut

b. Configure interface vfc-port-channel 403.


c. Configure the interface for the storage port on vfc 8.

N7K-1-StorageVDC(config)# interface vfc 8N7K-1-StorageVDC(config-if)# switchport mode fN7K-1-StorageVDC(config-if)# switchport trunk allowed vsan 100N7K-1-StorageVDC(config-if)# bind int e 2/8N7K-1-StorageVDC(config-if)# no shut



7. Add the vfc for the storage port.

N7K-1-StorageVDC(config-if)# vsan databaseN7K-1-StorageVDC(config-vsan-db)# vsan 100 interface vfc 8N7K-1-StorageVDC(config-vsan-db)# exitN7K-1-StorageVDC(config)#

Configuring N7K-2 Note: This section assumes that the N7K chassis is powered up and has hadan IP Address assigned to it. It also assumes that slot 1 contains an M1 linecard and slot 2 contains an F1 line card.

Nexus 7000 FCoE requirements◆ Minimum release of NX-OS 5.2(1)

◆ FCoE module N7K-F132XP-15

◆ FCoE license FCOE-N7K-F132XP

◆ FCoE will run on a Storage VDC (virtual device context)

Note: The Nexus 7000 does not have any native FC capabilities and does notoperate in NPV mode.

Create the Storage VDCAfter logging in to the Nexus 7000, you will be in the default VDCwhere you will complete the following configuration steps. Thesesteps will install the FCoE license and build the Storage VDC.

Note: Make sure the FCoE module is physically installed in the chassis beforetrying to enable the FCoE license.


N7K-2# config tEnter configuration commands, one per line. End with CNTL/Z.N7K-2(config)# install feature-set fcoe

2. Update QoS settings.

N7K-2(config)# system qosN7K-2(config-sys-qos)# service-policy type network-qos default-nq-7e-policyN7K-2(config-sys-qos)# exitN7K-2(config)#

3. Enable LACP and LLDP.

N7K-2(config)# feature lacpN7K-2(config)# feature lldpN7K-2(config)#


224


4. Create the Storage VDC.

Note: The vdc name "StorageVDC" is used for example purposes onlyand is not a requirement.

N7K-2(config)# vdc StorageVDC type storageNote: Creating VDC, one moment please ...N7K-2(config-vdc)#

5. Associate the FCoE license to the FCoE module.

N7K-2(config-vdc)# license fcoe module 2N7K-2(config)#

6. Verify the license is installed.

N7K-2 (config)# show license usageFeature Ins Lic Status Expiry Date Comments

Count--------------------------------------------------------------------------------FCOE-N7K-F132XP Yes 1 In use Never -Note: output truncated--------------------------------------------------------------------------------

7. Verify that the FCoE feature is installed.

N7K-2(config)# show feature-set fcoeFeature Set Name ID State-------------------- -------- --------fcoe 1 enabled

8. Allocate Ethernet interface to Storage VDC.

In this example we have the FCoE module inserted into slot 2.

When allocating ports to the storage VDC, keep in mind that bothmembers of each port group will need to be assigned to the sameVDC. For the FCoE "F1" blade, the port groups are as shown inTable 6.

Table 6 Port groups (page 1 of 2)

Port Group Port Number

Group 1 1 and 2

Group 2 3 and 4

Group 3 5 and 6

Group 4 7 and 8

Group 5 9 and 10

Group 6 11 and 12

Group 7 13 and 14



N7K-2(config)# vdc StorageVDCN7K-2(config-vdc)# allocate interface ethernet 2/1-8Moving ports will cause all config associated to them in source vdc to be removed.

Are you sure you want to move the ports (y/n)? [yes] yN7K-2(config-vdc)#

9. Allocate the VLANS to be used for FCoE.

Note: By default the range is from 1 - 3967.

N7K-2 (config-vdc)# allocate fcoe-vlan-range 200N7K-2 (config-vdc)#

10. .Switch to the newly created VDC and complete the VDC setup.

Note: The answers to the following prompts are provided for the sake ofthis example only and do not necessarily represent best practices.

N7K-2 (config-vdc)# switchto vdc StorageVDC

---- System Admin Account Setup ----

Do you want to enforce secure password standard (yes/no) [y]: n

Enter the password for "admin": <Enter password>Confirm the password for "admin": <Re-enter password>

---- Basic System Configuration Dialog VDC: 4 ----

This setup utility will guide you through the basic configuration ofthe system. Setup configures only enough connectivity for managementof the system.

Please register Cisco Nexus7000 Family devices promptly with yoursupplier. Failure to register may affect response times for initial

Group 8 15 and 16

Group 9 17 and 18

Group 10 19 and 20

Group 11 21 and 22

Group 12 23 and 24

Group 13 25 and 26

Group 14 27 and 28

Group 15 29 and 30

Group 16 31 and 32

Table 6 Port groups (page 2 of 2)


226


service calls. Nexus7000 devices must be registered to receiveentitled support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytimeto skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): y

Create another login account (yes/no) [n]: n

Configure read-only SNMP community string (yes/no) [n]: n

Configure read-write SNMP community string (yes/no) [n]: n

Enter the switch name : StorageVDC

Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: y

Mgmt0 IPv4 address : 10.1.2.11

Mgmt0 IPv4 netmask : 255.255.255.0

Configure the default gateway? (yes/no) [y]: y

IPv4 address of the default gateway : 10.1.2.1

Configure advanced IP options? (yes/no) [n]: n

Enable the telnet service? (yes/no) [n]: n

Enable the ssh service? (yes/no) [y]: y

Type of ssh key you would like to generate (dsa/rsa) [rsa]: rsa

Number of rsa key bits <1024-2048> [1024]: 1024

Configure default interface layer (L3/L2) [L3]: L3

Configure default switchport interface state (shut/noshut) [shut]: shut

The following configuration will be applied:no password strength-checkswitchname StorageVDC

interface mgmt0ip address 10.1.2.11 255.255.255.0no shutdownvrf context managementip route 0.0.0.0/0 10.1.2.1exitno feature telnetssh key rsa 1024 forcefeature ssh



no system default switchportsystem default switchport shutdown

Would you like to edit the configuration? (yes/no) [n]: n

Use this configuration and save it? (yes/no) [y]: y

Configuration update aborted: This vdc has had a global configuration change sincethe last saved config. Please save config in default vdc before proceeding

Cisco Nexus Operating System (NX-OS) SoftwareTAC support: http://www.cisco.com/tacCopyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.The copyrights to certain works contained in this software areowned by other third parties and used and distributed underlicense. Certain components of this software are licensed underthe GNU General Public License (GPL) version 2.0 or the GNULesser General Public License (LGPL) Version 2.1. A copy of eachsuch license is available athttp://www.opensource.org/licenses/gpl-2.0.php andhttp://www.opensource.org/licenses/lgpl-2.1.phpN7K-2-StorageVDC#

11. Ensure that the appropriate interfaces have been allocated to theVDC.

N7K-2-StorageVDC# show interface brief

-------------------------------------------------------------------------------Interface Status Speed

(Gbps)-------------------------------------------------------------------------------sup-fc0 up 1--------------------------------------------------------------------------------

Port VRF Status IP Address Speed MTU--------------------------------------------------------------------------------mgmt0 -- up 10.1.1.11 1000 1500--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch#--------------------------------------------------------------------------------Eth2/1 1 eth access down Administratively down 10G(D) --Eth2/2 1 eth access down Administratively down 10G(D) --Eth2/3 1 eth access down Administratively down 10G(D) --Eth2/4 1 eth access down Administratively down 10G(D) --Eth2/5 1 eth access down Administratively down 10G(D) --Eth2/6 1 eth access down Administratively down 10G(D) --Eth2/7 1 eth access down Administratively down 10G(D) --Eth2/8 1 eth access down Administratively down 10G(D) --N7K-2-StorageVDC#


228


12. Enable needed features within Storage VDC.

Note: Enabling these features within the Storage VDC does not enablethem in the Default VDC.

N7K-2-StorageVDC# configEnter configuration commands, one per line. End with CNTL/Z.N7K-2-StorageVDC(config)# feature lldpN7K-2-StorageVDC(config)# feature lacpN7K-2-StorageVDC(config)# feature npiv


N7K-2-StorageVDC(config)# exitN7K-2-StorageVDC# exitN7K-2(config-vdc)# copy running-config startup-config[########################################] 100%Copy complete, now saving to disk (please wait)...N7K-2(config-vdc)#

14. Verify the Storage VDC.

N7K-2(config-vdc)# show vdc StorageVDC

vdc_id vdc_name state mac type lc------ -------- ----- ---------- --------- ------------------------------------1 N7K-2 active 00:24:f7:17:05:c1 Ethernet m1 f1 m1xl2 StorageVDC active 00:24:f7:17:05:c2 Storage f1

Configure FCoE settings on the Storage VDC1. Switch to the Storage VDC.

N7K-2# switchto vdc StorageVDCN7K-2-StorageVDC# configN7K-2-StorageVDC(config)#

2. Create the FCoE VLAN.

Note: VLANs created in the storage VDC must fall within the range ofVLAN IDs specified in the "allocate fcoe-vlan-range <VLAN RANGE>"configuration parameter. See step 8 on page 224 for additionalinformation.

N7K-2-StorageVDC(config)# vlan 200N7K-2-StorageVDC(config-vlan)# fcoe vsan 200N7K-2-StorageVDC(config-vlan)#

3. Create the FCoE VSAN and add the vfc for the storage port.

N7K-2-StorageVDC(config-vlan)# vsan databaseN7K-2-StorageVDC(config-vsan-db)# vsan 200



N7K-2-StorageVDC(config-vsan-db)# exitN7K-2-StorageVDC(config)#


Note: As shown in the target topology diagram, Figure 32 on page 212,in this example there are 5 physical interfaces being used in the storageVDC. Two of these physical interfaces will be added to PortChannel 401,two others will be added to PortChannel 403. Each of these PortChannelswill be configured in this section.

IMPORTANT








230


Note: As shown in the target topology diagram, Figure 32 on page 212,there are 5 physical interfaces in this example being used in the storageVDC. Two of these physical interfaces will be added to PortChannel 401,two others will be added to PortChannel 403 and the final interface willbe used to connect to port 2 on the Storage array. Each of these interfaceswill be configured in this section.






N7K-2-StorageVDC(config)# interface ethernet 2/8N7K-2-StorageVDC(config-if)# switchport mode trunkN7K-2-StorageVDC(config-if)# switchport trunk allowed vlan 200N7K-2-StorageVDC(config-if)# no shutN7K-2-StorageVDC(config-if)# exitN7K-2-StorageVDC(config)#


Note: As shown in the target topology diagram, Figure 32 on page 212,there are 5 physical interfaces in this example being used in the storageVDC. Two of these physical interfaces will be added to PortChannel 401,two others will be added to PortChannel 403 and the final interface willbe used to connect to port 2 on the Storage array. Each of these interfaceswill be configured in this section


N7K-2-StorageVDC(config)# interface vfc-port-channel 301N7K-2-StorageVDC(config-if)# switchport mode e



N7K-2-StorageVDC(config-if)# switchport trunk allowed vsan 200N7K-2-StorageVDC(config-if)# no shut




N7K-2-StorageVDC(config)# interface vfc 8N7K-2-StorageVDC(config-if)# switchport mode fN7K-2-StorageVDC(config-if)# switchport trunk allowed vsan 200N7K-2-StorageVDC(config-if)# bind int e 2/8N7K-2-StorageVDC(config-if)# no shut

7. Add the vfc for the storage port.

N7K-2-StorageVDC(config-if)# vsan databaseN7K-2-StorageVDC(config-vsan-db)# vsan 200 interface vfc 8N7K-2-StorageVDC(config-vsan-db)# exitN7K-2-StorageVDC(config)#




ConfiguringNexus-5548-1

Note: This section assumes that the Nexus 5548 chassis is powered up andhas had an IP Address assigned to it.

1. Enable the FCoE and LACP features.

Nexus-5548-1# config tEnter configuration commands, one per line. End with CNTL/Z.Nexus-5548-1 (config)# feature fcoeNexus-5548-1 (config)# feature lacp

2. Configure QOS parameters.

Nexus-5548-1 (config)# system qosNexus-5548-1 (config-sys-qos)# service-policy type qos input fcoe

default-in-policyNexus-5548-1 (config-sys-qos)# service-policy type queuing input

fcoe-default-in-policyNexus-5548-1 (config-sys-qos)# service-policy type queuing output

fcoe-default-out-policy


232


Nexus-5548-1 (config-sys-qos)# service-policy type network-qosfcoedefault-nq-policy

3. Configure the VLAN and VSAN.

Nexus-5548-1 (config)# vlan 100Nexus-5548-1 (config)# fcoe vsan 100Nexus-5548-1 (config)# vsan databaseNexus-5548-1 (config-vsan-db)# vsan 100Nexus-5548-1 (config-vsan-db)# vsan 100 interface vfc 10Nexus-5548-1 (config-vsan-db)# exitNexus-5548-1 (config)#


Note: As shown in the target topology diagram, Figure 32 on page 212,there are 5 physical interfaces in this example being used for FCoE onNexus-5548-1; they are 1/1-4 and 1/10. Two of these physical interfaceswill be added to PortChannel 401, two others will be added toPortChannel 402, and the remainder will be used for the host to attach to.Each of the PortChannels will be configured in this section.

IMPORTANT

One of the PortChannels that will be configured in this sectionwill be attached to an MDS. When configuring a PortChannelinterface on an MDS, the PortChannel ID must be greater than256 to avoid a potential overlap with a SAN-PortChannel IDinstance. This restriction on PortChannel IDs does not exist onthe Nexus 5548. As a result, if a PortChannel ID less than orequal to 256 is used on the Nexus 5548, it will not be possible touse the same ID on the MDS. Although it is not a requirementfor the IDs to match on both sides of the PortChannel, it isgenerally considered a best practice to have them match toeliminate confusion in the future should the need totroubleshoot arise.


Note: PortChannel 401 is being connected to a Nexus 7k. As a result,the link between the two will actually be an FCoE ISL.

Nexus-5548-1 (config)# interface port-channel 401Nexus-5548-1 (config-if)# switchportNexus-5548-1 (config-if)# switchport mode trunkNexus-5548-1 (config-if)# switchport trunk allowed vlan 100Nexus-5548-1 (config-if)# no shut




Nexus-5548-1 (config)# interface port-channel 402Nexus-5548-1 (config-if)# switchportNexus-5548-1 (config-if)# switchport mode trunkNexus-5548-1 (config-if)# switchport trunk allowed vlan 100Nexus-5548-1 (config-if)# no shutNexus-5548-1 (config-if)# exitNexus-5548-1 (config)#

5. Configure load balancing.

Note: According to Cisco documentation, on the N5K the default loadbalance mechanism on the LACP port channel is source-dest-ip. If left inthis mode, all FCoE traffic will take the same link in the port channelwhen the N5K is forwarding FCoE frames to the N7K and MDS, as thisprovides SID-DID load balancing for FCoE frames. To enable the N5K toload balance using exchange ID, you must configure the N5K forsource-dest-port load balancing.

Nexus-5548-1 (config)# port-channel load-balance source-dest-portNexus-5548-1 (config)#




Nexus-5548-1 (config)# interface ethernet 1/1-2Nexus-5548-1 (config-if-range)# switchport mode trunkNexus-5548-1 (config-if-range)# switchport trunk allowed vlan 100Nexus-5548-1 (config-if-range)# channel-group 402 mode activeNexus-5548-1 (config-if-range)# no shutNexus-5548-1 (config-if-range)# exitNexus-5548-1 (config)#




234



Nexus-5548-1 (config)# interface ethernet 1/10Nexus-5548-1 (config-if)# switchport mode trunkNexus-5548-1 (config-if)# switchport trunk allowed vlan 100Nexus-5548-1 (config-if)# no shutNexus-5548-1 (config-if)# exitNexus-5548-1 (config)#




Nexus-5548-1 (config)# interface vfc-port-channel 401Nexus-5548-1 (config-if)# switchport mode eNexus-5548-1 (config-if)# switchport trunk allowed vsan 100Nexus-5548-1 (config-if)# no shut


Nexus-5548-1 (config)# interface vfc-port-channel 402Nexus-5548-1 (config-if)# switchport mode eNexus-5548-1 (config-if)# switchport trunk allowed vsan 100Nexus-5548-1 (config-if)# no shut


Nexus-5548-1 (config)# interface vfc 10Nexus-5548-1 (config-if)# switchport mode fNexus-5548-1 (config-if)# switchport trunk allowed vsan 100Nexus-5548-1 (config-if)# no shut


In this section, a Nexus 5548 that is running in NPV mode will beconnected to both a Nexus 7k and an MDS switch at the same time.

This configuration presents a problem since currently only a singleFCoE NPV link can be configured on the Nexus 5548. As a result, theuplink to the Nexus 7k will be an FCoE NPV uplink and the otherwill be an FC NPV uplink to the MDS. If the Nexus 7k was not a partof this configuration, then the uplink to the MDS switch could be anFCoE NPV uplink.



1. Enable the FCoE and NPV features.

Nexus-5548-2# config tEnter configuration commands, one per line. End with CNTL/Z.Nexus-5548-2 (config)# feature fcoeNexus-5548-2 (config)# feature npvVerify that boot variables are set and the changes are saved. Changing to npv mode

erases the current configuration and reboots the switch in npv mode. Do youwant to continue? (y/n):y

Note: The switch will reboot at this point. Once the switch comes backup, continue on with the next steps.

2. Enable the LACP feature.

Nexus-5548-2# config tEnter configuration commands, one per line. End with CNTL/Z.Nexus-5548-2 (config)# feature lacp

3. Configure QOS parameters.

Nexus-5548-2 (config)# system qosNexus-5548-2 (config-sys-qos)# service-policy type qos input fcoe

default-in-policyNexus-5548-2 (config-sys-qos)# service-policy type queuing input

fcoe-default-in-policyNexus-5548-2 (config-sys-qos)# service-policy type queuing output

fcoe-default-out-policyNexus-5548-2 (config-sys-qos)# service-policy type network-qos

fcoedefault-nq-policy

4. Configure the VLAN and VSAN.

Nexus-5548-2 (config)# vlan 200Nexus-5548-2 (config)# fcoe vsan 200Nexus-5548-2 (config)# vsan databaseNexus-5548-2 (config-vsan-db)# vsan 200Nexus-5548-2 (config-vsan-db)# vsan 200 interface vfc 10Nexus-5548-2 (config-vsan-db)# exitNexus-5548-2 (config)#


Note: As shown in the target topology diagram, Figure 32 on page 212,there are 5 physical interfaces in this example being used for FCoE onNexus-5548-2; they are e1/3-4, e1/10 and fc2/1-2. Two of these physicalinterfaces will be added to PortChannel 301, two others will be added toPortChannel 302, and the remainder will be used for the host to attach to.Each of the PortChannels will be configured in this section.


236


IMPORTANT



Note: Since Nexus-5548-2 is running in NPV mode, the link betweenthe two will actually be an FCoE NPV link.

Nexus-5548-2 (config)# interface port-channel 301Nexus-5548-2 (config-if)# switchportNexus-5548-2 (config-if)# switchport mode trunkNexus-5548-2 (config-if)# switchport trunk allowed vlan 200Nexus-5548-2 (config-if)# no shut


Nexus-5548-2 (config)# interface san-port-channel 302Nexus-5548-2 (config-if)# channel mode activeNexus-5548-2 (config-if)# switchport mode npNexus-5548-2 (config-if)# switchport trunk allowed vsan 100Nexus-5548-2 (config-if)# switchport trunk mode onNexus-5548-2 (config-if)# no shutNexus-5548-2 (config-if)# exitNexus-5548-2 (config)#

6. Configure load balancing.

Note: According to Cisco documentation, on the N5K the default loadbalance mechanism on the LACP port channel is source-dest-ip. If left inthis mode, all FCoE traffic will take the same link in the port channelwhen the N5K is forwarding FCoE frames to the N7K and MDS, as thisprovides SID-DID load balancing for FCoE frames. To enable the N5K toload balance using exchange ID, you must configure the N5K forsource-dest-port load balancing.



Nexus-5548-2 (config)# port-channel load-balance source-dest-portNexus-5548-2 (config)#


Note: As shown in the target topology diagram, Figure 32 on page 212,there are 5 physical interfaces in this example being used for FCoE onNexus-5548-2. Two of these physical interfaces will be added toPortChannel 401, two others will be added to PortChannel 402, and theremainder will be used for the host to attach to. Each of the PortChannelswill be configured in this section.


Nexus-5548-2 (config)# interface fc 2/1-2Nexus-5548-2 (config-if-range)# switchport trunk mode onNexus-5548-2 (config-if-range)# channel-group 302 forceNexus-5548-2 (config-if-range)# no shutNexus-5548-2 (config-if-range)# exitNexus-5548-2 (config)#




Nexus-5548-2 (config)# interface ethernet 1/10Nexus-5548-2 (config-if)# switchport mode trunkNexus-5548-2 (config-if)# switchport trunk allowed vlan 200Nexus-5548-2 (config-if)# spanning-tree port type edge trunkNexus-5548-2 (config-if)# spanning-tree bpduguard enableNexus-5548-2 (config-if)# no shutNexus-5548-2 (config-if)# exitNexus-5548-2 (config)#


Note: As shown in the target topology diagram, Figure 32 on page 212,there are 5 physical interfaces in this example being used for FCoE onNexus-5548-2. Two of these physical interfaces will be added toPortChannel 301, two others will be added to PortChannel 302, and theremainder will be used for the host to attach to. Each of the PortChannelswill be configured in this section.


238



Nexus-5548-2 (config)# interface vfc-port-channel 301Nexus-5548-2 (config-if)# switchport mode npNexus-5548-2 (config-if)# switchport trunk allowed vsan 200Nexus-5548-2 (config-if)# no shut


Nexus-5548-2 (config)# interface vfc-port-channel 302Nexus-5548-2 (config-if)# switchport mode npNexus-5548-2 (config-if)# switchport trunk allowed vsan 200Nexus-5548-2 (config-if)# no shut


Nexus-5548-2 (config)# interface vfc 10Nexus-5548-2 (config-if)# switchport mode fNexus-5548-2 (config-if)# switchport trunk allowed vsan 200Nexus-5548-2 (config-if)# no shut

Configuring the MDS 9513This section contains the following information:

◆ “Configuring MDS-9513-1” on page 238

◆ “Configuring MDS-9513-2” on page 240

ConfiguringMDS-9513-1

According to the Cisco documentation, there is no need to enableFCoE explicitly on the MDS switch. The following features will beenabled once an FCoE capable line card is detected:

◆ install feature-set fcoe

◆ feature-set fcoe

◆ feature lldp

◆ feature vlan-vsan-mapping

You do need to enable LACP as you will be building an LACP portchannel.

1. Enable the LACP feature.

MDS-9513-1# config tMDS-9513-1(config) # feature lacp

2. Set the appropriate system QoS settings.

MDS-9513-1(config) # system qosMDS-9513-1(config-sys-qos) # service-policy type network-qos default-nq-7e-policy



3. Create the required VLAN and VSAN.

MDS-9513-1(config) # vsan databaseMDS-9513-1(config-vsan-db) # vsan 100MDS-9513-1(config-vsan-db) # vlan 100MDS-9513-1(config-vlan) # fcoe vsan 100MDS-9513-1(config-vlan) # exitMDS-9513-1(config) #

4. Create PortChannels.

a. Create PortChannel 402.

MDS-9513-1(config) # interface ethernet 1/1-2MDS-9513-1(config-if-range) # switchport mode trunkMDS-9513-1(config-if-range) # switchport trunk allowed vlan 100MDS-9513-1(config-if-range) # channel-group 402 force mode activeMDS-9513-1(config-if-range) # no shutMDS-9513-1(config-if-range) # interface ethernet-port-channel 402MDS-9513-1(config-if) # no shutMDS-9513-1(config-if) # int vfc-port-channel 402MDS-9513-1(config-if) # switchport mode eMDS-9513-1(config-if) # switchport mode trunk allowed vsan 100MDS-9513-1(config-if) # no shutMDS-9513-1(config-if) # endMDS-9513-1 #

b. Create PortChannel 403.

MDS-9513-1(config) # interface ethernet 1/3-4MDS-9513-1(config-if-range) # switchport mode trunkMDS-9513-1(config-if-range) # switchport trunk allowed vlan 100MDS-9513-1(config-if-range) # channel-group 403 force mode activeMDS-9513-1(config-if-range) # no shutMDS-9513-1(config-if-range) # interface ethernet-port-channel 403MDS-9513-1(config-if) # no shutMDS-9513-1(config-if) # int vfc-port-channel 403MDS-9513-1(config-if) # switchport mode eMDS-9513-1(config-if) # switchport mode trunk allowed vsan 100MDS-9513-1(config-if) # no shutMDS-9513-1(config-if) # exitMDS-9513-1 (config) #

5. Create the vfc for the FCoE storage port to connect to on interface1/5.

MDS-9513-1(config) # interface vfc 5MDS-9513-1(config-if) # bind interface ethernet 1/5MDS-9513-1(config-if) # no shut

6. Add the newly created vfc to the vsan database.

MDS-9513-1(config) # vsan databaseMDS-9513-1(config-vsan-db) # vsan 100 interface vfc 5


240


MDS-9513-1(config-vsan-db) # exitMDS-9513-1(config) #

7. Configure the physical interface for the storage port vfc.

MDS-9513-1(config) # interface e1/5MDS-9513-1(config-if) # switchport mode trunkMDS-9513-1(config-if) # switchport trunk allowed vlan 100MDS-9513-1(config-if) # no shut

ConfiguringMDS-9513-2

According to the Cisco documentation, there is no need to enableFCoE explicitly on the MDS switch. The following features will beenabled once an FCoE capable line card is detected:

◆ install feature-set fcoe

◆ feature-set fcoe

◆ feature lldp

◆ feature vlan-vsan-mapping

You do need to enable LACP as you will be building an LACPPortChannel.

1. Enable the LACP and NPIV features.

MDS-9513-2# config tMDS-9513-2(config) # feature lacpMDS-9513-2(config) # feature npiv

2. Set the appropriate system QoS settings.

MDS-9513-2(config) # system qosMDS-9513-2(config-sys-qos) # service-policy type network-qos default-nq-7e-policy

3. Create the required VLAN and VSAN.

MDS-9513-2(config) # vsan databaseMDS-9513-2(config-vsan-db) # vsan 200MDS-9513-2(config-vsan-db) # vlan 200MDS-9513-2(config-vlan) # fcoe vsan 200MDS-9513-2(config-vlan) # exitMDS-9513-2(config) #

4. Create PortChannels.

a. Create PortChannel 302.

Note: PortChannel 302 will make use of FC ports 2/1-2.

MDS-9513-2(config) # interface ethernet 2/1-2MDS-9513-2(config-if-range) # switchport mode f



MDS-9513-2(config-if-range) # switchport rate-mode dedicatedMDS-9513-2(config-if-range) # channel-group 302 forceMDS-9513-2(config-if-range) # no shutMDS-9513-2(config-if-range) # interface ethernet-port-channel 302MDS-9513-2(config-if) # no shutMDS-9513-2(config-if) # int vfc-port-channel 302MDS-9513-2(config-if) # switchport mode fMDS-9513-2(config-if) # switchport mode trunk allowed vsan 200MDS-9513-2(config-if) # no shutMDS-9513-2(config-if) # endMDS-9513-2 #

b. Create PortChannel 303.

MDS-9513-2(config) # interface ethernet 1/3-4MDS-9513-2(config-if-range) # switchport mode trunkMDS-9513-2(config-if-range) # switchport trunk allowed vlan 200MDS-9513-2(config-if-range) # channel-group 303 force mode activeMDS-9513-2(config-if-range) # no shutMDS-9513-2(config-if-range) # interface ethernet-port-channel 303MDS-9513-2(config-if) # no shutMDS-9513-2(config-if) # int vfc-port-channel 303MDS-9513-2(config-if) # switchport mode eMDS-9513-2(config-if) # switchport mode trunk allowed vsan 200MDS-9513-2(config-if) # no shutMDS-9513-2(config-if) # exitMDS-9513-2 (config) #

5. Create the vfc for the FCoE storage port to connect to on interface1/5.

MDS-9513-2(config) # interface vfc 5MDS-9513-2(config-if) # bind interface ethernet 1/5MDS-9513-2(config-if) # no shut

6. Add the newly created vfc to the vsan database.

MDS-9513-2(config) # vsan databaseMDS-9513-2(config-vsan-db) # vsan 200 interface vfc 5MDS-9513-2(config-vsan-db) # exitMDS-9513-2(config) #

7. Configure the physical interface for the storage port vfc.

MDS-9513-2(config) # interface e1/5MDS-9513-2(config-if) # switchport mode trunkMDS-9513-2(config-if) # switchport trunk allowed vlan 200MDS-9513-2(config-if) # no shut


242


Nexus 5000/Nexus 2232PP topologyThis section details the setup steps required to create theconfiguration shown in Figure 34 on page 244, the target topology. Itincludes the following information:


◆ “Existing topology example” on page 243




Summary of configuration steps

Nexus-5020-1 This section provides a summary and examples of the steps to createthe configuration shown in the target topology, as shown in Figure 34on page 244.

1. Configure the switch management interface, basic settings andpassword.

2. Enable the FCoE, LACP, FEX, LLDP and NPV (if applicable)features.

3. Create the appropriate VSANs.

4. Create the appropriate VLANs.

5. Create and configure a FEX instance.

6. Create and configure the FEX EtherChannel.

7. Create VFC interfaces and assign to the appropriate VSAN.

8. Configure Ethernet uplink interface settings.

9. Add FC interfaces to the appropriate VSAN.

10. Connect cables and verify login.


12. Perform fabric zoning.

13. Provision storage.



Existing topology exampleIt is assumed that the customer has an existing LAN and mirroredSAN as shown in Figure 33 and is adding the Nexus 5000/Nexus2232 to both.


Core-N7K-1

CNA_1

Host_1 (Windows)

CNA_1:EmulexVSAN = 100WWPN= 10:00:00:00:c9:3c:f8:3cStorage = Storage port 1

CNA_2:EmulexVSAN = 200WWPN = 10:00:00:00:c9:3c:f7:c0Storage = Storage port 3



Storage port 2:50:06:04:82:Da5:2E:69:49

Storage port 1:50:06:04:82:D5:2E:69:79

CNA_2

Core-N7K-2

1/1 2/1 1/1 2/1

1/2

1/3

1/4

1/5

StorageFabric BFabric A

1 2 3 4

1/2

1/2

1/3

1/4

1/5

1/2

1/2

1/1

1/3

1/4

1/5

1/2

1/1

1/3

1/4

1/5

CNA_1

Host_1 (Linux)

VSAN 100

VSAN 300

VSAN 200

VSAN 400

4Gb/s FC

CNA_1:QLogicVSAN = 300WWPN = 21:01:00:1b:32:2a:c0:b8 Storage = Storage port 2

CNA_2:QLogicVSAN = 400WWPN = 21:00:00:1b:32:0a:c0:b8Storage = Storage port 4

CNA_2

DCX_1Domain 1

172.23.185.24

DCX_2Domain 2

172.23.185.25


172.23.185.22


172.23.185.23

Nexus 5000/Nexus 2232PP topology 243

244




Host_1 (Windows)

CNA_1 CNA_2

DCX_1Domain 1

172.23.185.24

DCX_2Domain 2

172.23.185.25


172.23.185.22


172.23.185.23

Storage SAN BSAN ANexus-5020-1IP = 172.23.185.112Domain = N/A (NPVmode)

Nexus-5020-2IP = 172.23.185.113Domain = 3

1/21/31/41/5

1/21/31/41/5

1/21/31/41/5

1/21/31/41/5

1/11/1

1/11/1

Host_2(Linux)

CNA_1 CNA_2

CNA_1:QLogicVSAN = 300WWPN = 21:01:00:1b:32:2a:c0:b8ENode MAC = 00:1b:32:2a:c0:b8IP Address = 10.246.53.107Storage = Storage port 2


1 2 3 4

Storage port 1:50:06:04:82:D5:2E:69:79

Storage port 2:50:06:04:82:D5:2E:69:49



Nexus-5020-1

Nexus-5020-2

CNA_1:EmulexVSAN = 100WWPN= 10:00:00:00:c9:3c:f8:3cENode MAC = 00:00:c9:3c:f8:3cIP Address = 10.246.53.106Storage = Storage port 1

CNA_2:EmulexVSAN = 200WWPN = 10:00:00:00:c9:3c:f7:c0ENode MAC = 00:00:c9:3c:f7:c0IP Address = 10.246.54.106Storage = Storage port 3

VLAN/VSAN 100

VLAN/VSAN 200

VLAN/VSAN 300

VLAN/VSAN 400

10GbE

4Gb/s FC

Core-N7K-1

Port-channel 1

Port-channel 2

1/1 2/1 2/11/1

Core-N7K-2

SAN PortChannels


1/1

1/2

1/3

1/4

1/5

1/6

1/1

1/2

1/3

1/4

1/5

1/6

Port-Channel 3

Port-Channel 4

Host 1

Host 2

PO3PO4

FEX instance 102FEX instance 101

ICO-IMG-000850_Straight Thru



PrerequisitesFCoE requires the use of specialized host adapters, referred to as aConverged Network Adapter (CNA), and specialized switchhardware known as a Fibre Channel Forwarder (FCF). CNAs areinstalled in the host and then attached to an Ethernet port thatprovides the FCF functionality. The connection between the CNA andEthernet switch port can be either optical or copper based.

FCF prerequisites The Nexus 5020 requires:

◆ 2 U of rack space, preferably at the top of the rack where theservers that will be connecting to the Nexus 5020 are located

◆ Two power drops of either 110 V or 220 V◆ An IP address for management purposes and a network port to

which to connect the Nexus 5020 management port◆ A customer-supplied password for the admin user◆ An available Domain ID in the fabric to which the Nexus 5020

will be connected

Note: If the Nexus 5020 will be carved into multiple VSANs, a DomainID will need to be reserved for each VSAN.

◆ For each Nexus 5020, at least one expansion module containing 4G FC ports will need to be installed

◆ An FCoE license◆ Fabric Extender Transceivers (FET)

Note: FETs are optional, but are used in this example to connect eachNexus 5020 to its Nexus 2232PP.

Laptop prerequisites Laptop prerequisites include:

Note: The laptop used in this example is needed only to initially configurethe Nexus 5020 and will be disconnected after the installation is complete.

◆ Com port with a DB-9 connector◆ Download and install PuTTY




246


Note: The operating system used for this example was Windows 2000Professional and HyperTerminal was installed by default. Some of thelater Windows-based operating systems do not have HyperTerminalinstalled. This will need to be done before attempting to configure theNexus 5020s. See the help feature for the particular version of Windowsyou are using for more information on installing HyperTerminal.








◆ “Configuring network parameters” on page 271



◆ Nexus-5020-1 will be attached to a Brocade fabric consisting oftwo Brocade ED-DCX-B switches. Since the Nexus 5020 does notcurrently support FC-SW interop to an EMC ED-DCX-B,Nexus-5020-1 will need to run in NPV mode.

◆ NPV mode allows the Nexus 5020 to act as an N_Port and passlogin requests directly from the CNA to the Fabric by using theNPIV protocol. For more information on NPV and NPIV, refer tothe “NPIV” section in the Networked Storage Concepts and ProtocolsTechBook at http://elabnavigator.EMC.com, under the TopologyResource Center tab. Further information on how to configure NPVmode and the supported features with NPV is available athttp://www.cisco.com.




◆ Since Nexus-5020-1 will be connected to two differentED-DCX-Bs, VSAN assignments to the Nexus 5020 VFC and FCinterfaces will be used to allow the CNA to attach to the properfabric.






The new connection dialog box is opened with the entry namedialog box highlighted.








Do you want to enforce secure password standard(yes/no): Y

Note: The password should contain characters from at least three ofthe following classes: lowercase letters, uppercase letters, digits, andspecial characters.


248


h. Enter and then confirm the customer-supplied password foradmin, which in this case is Seabiscuit!.


Enter the password for "admin":Confirm the password for "admin":





Create another login account (yes/no) [n]: NConfigure read-only SNMP community string (yes/no) [n]: NConfigure read-write SNMP community string (yes/no) [n]: NEnter the switch name: Nexus-5020-1Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: YMgmt0 IPv4 address: 172.23.185.112Mgmt0 IPv4 netmask: 255.255.255.0Configure the default gateway? (yes/no) [y]: YIPv4 address of the default gateway: 172.23.185.2Enable the telnet service? (yes/no) [y]: YEnable the http-server? (yes/no) [y]: YEnable the ssh service? (yes/no) [n]: YType of ssh key you would like to generate (dsa/rsa): dsaNumber of key bits <768-2048>: 1024Configure the ntp server? (yes/no) [n]: NEnter basic FC configurations (yes/no) [n]: YConfigure default physical FC switchport interface state (shut/noshut) [shut]:




vrf context management



ip route 0.0.0.0/0 172.23.185.2exittelnet server enablessh key dsa 1024 forcessh server enable





Nexus-5020-1 login: adminPassword: Seabiscuit!



You will notice between 40 and 52 Ethernet interfaces as wellas between 4 and 16 FC interfaces if the GEM module containsFC ports.






--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth1/1 1 eth access down Link not connected 10G(D) --Eth1/2 1 eth access down Link not connected 10G(D) --


250


Eth1/3 1 eth access down Link not connected 10G(D) --...Eth1/40 1 eth access down Link not connected 10G(D) --


f. Enable NPV mode.

Although NPV mode is supported when connecting to eitherConnectrix MDS or non-Connectrix MDS switches, it onlyneeds to be enabled when connecting a Nexus 5020 to anon-Connectrix MDS switch, as is the case with this example.

If you are connecting the Nexus 5020 to a Connectrix MDSswitch, and do not want to use NPV mode, see “ConfiguringNexus-5020-2” on page 191.

Nexus-5020-1(config)# npv enableVerify that boot variables are set and the changes are saved. Changing to npv mode

erases the current configuration and reboots the switch in npv mode. Do youwant to continue? (y/n): Y

Note: It is recommended that you leave the console connection upwhile the reboot is taking place.

g. Open an SSH session to the Nexus 5020 with an IP address of172.23.185.112.

3. Enable disruptive load-balancing.

Disruptive load-balancing allows the Nexus to evenly distributeCNA logins across all NP ports while running in NPV mode.

To enable disruptive load-balancing, use the npvauto-load-balance disruptive command, as shown next.

Nexus-5020-1# npv auto-load-balance disruptiveEnabling this feature may flap the server interfaces whenever load is not in a

balanced state. This process may result in traffic disruption. Do you want toproceed? (y/n): Y

Nexus-5020-1(config)#




switch (config-sys-qos)# service-policy type qos inputfcoe-default-in-policyswitch (config-sys-qos)# service-policy type queuing inputfcoe-default-in-policyswitch (config-sys-qos)# service-policy type queuing outputfcoe-default-out-policyswitch (config-sys-qos)# service-policy type network-qosfcoe-default-nq-policy


Nexus-5020-1(config)# vsan databaseNexus-5020-1(config-vsan-db)# vsan 100Nexus-5020-1(config-vsan-db)# vsan 300Nexus-5020-1(config-vsan-db)# exitNexus-5020-1(config)#




7. Create the VLAN that will be used as the default VLAN on theFCoE interfaces.

Nexus-5020-1(config-vlan)# vlan 700

Note: This step is only required if you are changing the default VLAN onethernet interface. In later examples, VLAN 700 will be used for iSCSI.

8. Use show FCoE VLAN to verify whether the configuration wassuccessful.


VLAN VSAN Status-------- -------- --------100 100 Operational300 300 Operational


252


9. Enable the Fabric Extender feature.

Nexus-5020-1(config-vlan)# feature fex


Note: The pinning max-links parameter does not need to be set since theNexus-2232PP will be connected to Nexus-5020-1 via an EtherChannel.

Nexus-5020-1(config)# fex 101Nexus-5020-1(config-fex)# description FEX0101

11. Create an EtherChannel that will be used to connect toNexus-2232-1.

Nexus-5020-1(config-fex)# interface port-channel 3

12. Configure the EtherChannel as a FEX fabric interface.

Nexus-5020-1(config-if)# switchport mode fex-fabric

13. Associate the FEX instance with the EtherChannel.

Nexus-5020-1(config-if)# fex associate 101

14. Configure and associate the PortChannel with the appropriateEthernet interfaces.

Nexus-5020-1(config-if)# interface ethernet 1/1-4Nexus-5020-1(config-if)# switchport mode fex-fabricNexus-5020-1(config-if)# fex associate 101Nexus-5020-1(config-if-range)# channel-group 3

15. Using any of the supported media types, connect Nexus-5020-1 toNexus 2232-1 as shown in the target topology diagram shown inFigure 34 on page 244.

16. Using any of the supported media types, connect Host_1 /CNA_1 and Host_2 / CNA_1 to Nexus 2232-1 as shown in thetarget topology diagram shown in Figure 34 on page 244.

17. Display the switch interfaces using show interface brief:

Nexus-5020-1(config-if-range)# show int brief-------------------------------------------------------------------------------Interface Vsan Admin Admin Status SFP Oper Oper Port


-------------------------------------------------------------------------------fc2/1 1 NP off down swl -- --fc2/2 1 NP off down swl -- --



fc2/3 1 NP off down swl -- --...fc3/4 1 NP off down swl -- --

--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth1/1 1 eth fabric up none 10G(D) 3Eth1/2 1 eth fabric up none 10G(D) 3Eth1/3 1 eth fabric up none 10G(D) 3Eth1/4 1 eth fabric up none 10G(D) 3Eth1/5 1 eth access down Link not connected 10G(D) --...Eth1/40 1 eth access down Link not connected 10G(D) --

--------------------------------------------------------------------------------Port-channel VLAN Type Mode Status Reason Speed ProtocolInterface--------------------------------------------------------------------------------Po3 1 eth fabric up none a-10G(D) none

--------------------------------------------------------------------------------Port VRF Status IP Address Speed MTU--------------------------------------------------------------------------------mgmt0 -- up 172.23.185.112 1000 1500

--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth101/1/1 1 eth access up none 10G(D) --Eth101/1/2 1 eth access up none 10G(D) --...Eth101/1/32 1 eth access down SFP not inserted 10G(D) --Nexus-5020-1(config-if-range)#

18. Display the EtherChannel interface.

Nexus-5020-1(config-if)# show interface port-channel 3 fex-intfFabric FEXInterface Interfaces---------------------------------------------------------------------Po3 Eth101/1/1 Eth101/1/2 Eth101/1/3 Eth101/1/4


254


19. Create the VFC interfaces and bind them to an interface on the2232PP.

Nexus-5020-1(config-if-range)# interface vfc 1011Nexus-5020-1(config-if)# bind interface ethernet 101/1/1Nexus-5020-1(config-if)# switchport trunk allowed vsan 100Nexus-5020-1(config-if)# no shutNexus-5020-1(config-if)# interface vfc 1012Nexus-5020-1(config-if)# bind interface ethernet 101/1/2Nexus-5020-1(config-if)# switchport trunk allowed vsan 300Nexus-5020-1(config-if)# no shut

20. Assign each VFC to the appropriate VSAN as shown in the targettopology diagram, Figure 34 on page 244.

Nexus-5020-1(config-if)# vsan databaseNexus-5020-1(config-vsan-db)# vsan 100 interface vfc1011Nexus-5020-1(config-vsan-db)# vsan 300 interface vfc1012Nexus-5020-1(config-vsan-db)# exitNexus-5020-1(config)#

The output of the show interface brief shows the following:




-------------------------------------------------------------------------------fc2/1 1 NP off down swl -- --fc2/2 1 NP off down swl -- --...fc3/4 1 NP off down swl -- --

--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth1/1 1 eth fabric up none 10G(D) 3Eth1/2 1 eth fabric up none 10G(D) 3Eth1/3 1 eth fabric up none 10G(D) 3Eth1/4 1 eth fabric up none 10G(D) 3...Eth1/40 1 eth access down Link not connected 10G(D) ----------------------------------------------------------------------------------



Port-channel VLAN Type Mode Status Reason Speed ProtocolInterface--------------------------------------------------------------------------------Po3 1 eth fabric up none a-10G(D) none




-------------------------------------------------------------------------------vfc1011 100 F off down -- -- --vfc1012 300 F off down -- -- --

--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth101/1/1 1 eth access up none 10G(D) --Eth101/1/2 1 eth access up none 10G(D) --Eth101/1/3 1 eth access up none 10G(D) --Eth101/1/4 1 eth access up none 10G(D) --Eth101/1/5 1 eth access down SFP not inserted 10G(D) --...Eth101/1/32 1 eth access down SFP not inserted 10G(D) --Nexus-5020-1(config)#


Each interface will be set to trunk mode to allow it to handlemultiple VLANs and 802.1q tagged frames. The native VLAN isset to 700 in this case to allow for all non-FCoE untagged framesto be forwarded onto VLAN 700.

The allowed VLAN list is set to allow the default VLAN as well asthe FCoE VLAN.

Nexus-5020-1(config-if)# interface ethernet 101/1/1Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# switchport trunk native vlan 700Nexus-5020-1(config-if)# switchport trunk allowed vlan 100,700Nexus-5020-1(config-if)# spanning-tree port type edge trunkNexus-5020-1(config-if)# interface ethernet 101/1/2Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# switchport trunk native vlan 700Nexus-5020-1(config-if)# switchport trunk allowed vlan 300,700Nexus-5020-1(config-if)# spanning-tree port type edge trunk


256



As shown in Figure 29 on page 176, Nexus-5020-1 is connected totwo different fabrics. In order to ensure that each CNA attachedto Nexus-5020-1 logs in to the same fabric as its storage port, eachCNA will need to be statically mapped to a particular VSAN. Thisis accomplished by adding the VFC Interface as well as thephysical FC interfaces to the appropriate VSAN. To summarizethe steps:



Nexus-5020-1(config-if)# vsan databaseNexus-5020-1(config-vsan-db)# vsan 100 interface fc 2/1Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 2/3Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 2/2Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 2/4Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 3/1Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 3/3Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 3/2Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 3/4Nexus-5020-1(config-vsan-db)# exitNexus-5020-1(config)#



23. Attach the fiber cables between Nexus-5020-1 and the BrocadeDCX Directors as shown in the target topology diagram inFigure 34 on page 244.

24. Validate the configuration by using the show int brief command.







-----------------------------------------------------------------------------fc2/1 100 NP -- down swl -- --fc2/2 300 NP -- down swl -- --fc2/3 100 NP -- down swl -- --fc2/4 300 NP -- down swl -- --fc3/1 100 NP -- down swl -- --fc3/2 300 NP -- down swl -- --fc3/3 100 NP -- down swl -- --fc3/4 300 NP -- down swl -- --...-----------------------------------------------------------------------------Interface Vsan Admin Admin Status SFP Oper Oper Port


-----------------------------------------------------------------------------vfc1 100 F -- down -- -- --vfc2 300 F -- down -- -- --


25. Use show npv flogi-table vsan <vsan ID> to verify that eachCNA is logged in to the appropriate vsan.

Nexus-5020-1(config-if)# show npv flogi-table vsan 100-----------------------------------------------------------------------------SERVER EXTERNALINTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE-----------------------------------------------------------------------------vfc1 100 0x680101 10:00:00:00:c9:3c:f8:3c 20:00:00:00:c9:3c:f8:3c fc2/1


Nexus-5020-1(config-if)# show npv flogi table vsan 300

-----------------------------------------------------------------------------SERVER EXTERNALINTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE-----------------------------------------------------------------------------vfc2 300 0x690101 21:01:00:1b:32:2a:c0:b8 20:01:00:1b:32:2a:c0:b8 fc2/3




258


The fc interfaces are now online and running in NPV mode. Thevfc interfaces come online as soon as one of the fc interfaces in thesame vsan comes online.



27. Zone the CNAs.

Since Nexus-5020-1 is running in NPV mode, all zoning will needto be performed from the ED-DCX-B Director to which it isattached. It is strongly recommended that if the CLI is to be usedfor zoning, then cut and paste be utilized to minimizetranscription types of errors when manually typing the WWPNs.The set of steps required to create the appropriate zones are:

a. Create each zone by using zonecreate "<zone name a>","WWPN1; WWPN2; .... WWPNn".

b. Create the zone config by using cfgcreate "<cfg name>", "<zonename a>; <zone name b>; .... <zone name n>".

c. Enable the zone config by using cfgenable "<cfg name>".

d. Save the configuration by using cfgsave.

The exact steps that would be performed in this example are asfollows:

a. Zone the CNA logged in to 172.23.185.25 to access its storageport.

– Telnet into the switch with an IP address of 172.23.185.24.DCX_1:admin> zonecreate “host1_CNA1”, “10:00:00:00:c9:3c:f8:3c;


10:00:00:00:c9:3c:f8:3c50:06:04:82:D5:2E:69:79




b. Zone the CNA logged in to 172.23.185.25 to access its Storageports.

– Telnet into the switch with an IP address of 172.23.185.25.DCX_2:admin> zonecreate “host2_CNA1”, “21:01:00:1b:32:2a:c0:b8;


21:01:00:1b:32:2a:c0:b850:06:04:82:D5:2E:69:49




◆ Nexus-5020-2 will be attached to a Cisco fabric consisting of twoMDS-9513 switches. Since the Nexus 5020 is being connected toanother Cisco switch, FC-SW mode is supported andNexus-5020-2 will not need to run in NPV mode.

◆ Since Nexus-5020-2 will be connected to two different MDS-9513svia TE_Ports, VSAN assignments are only required on the Nexus5020 VFC interfaces.








260









Do you want to enforce secure password standard(yes/no): Y

Note: The password should contain characters from at least three ofthe following classes: lowercase letters, uppercase letters, digits, andspecial characters.

h. Enter and then confirm the customer-supplied password foradmin, which in this case is Seabiscuit!.


Enter the password for "admin":Confirm the password for "admin":





Create another login account (yes/no) [n]: NConfigure read-only SNMP community string (yes/no) [n]: NConfigure read-write SNMP community string (yes/no) [n]: N



Enter the switch name: Nexus-5020-2Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: YMgmt0 IPv4 address: 172.23.185.113Mgmt0 IPv4 netmask: 255.255.255.0Configure the default gateway? (yes/no) [y]: YIPv4 address of the default gateway: 172.23.185.2Enable the telnet service? (yes/no) [y]: YEnable the http-server? (yes/no) [y]: YEnable the ssh service? (yes/no) [n]: YType of ssh key you would like to generate (dsa/rsa): dsaNumber of key bits <768-2048>: 1024Configure the ntp server? (yes/no) [n]: N 172.23.185.113Enter basic FC configurations (yes/no) [n]: YConfigure default physical FC switchport interface state (shut/noshut) [shut]:









Nexus-5020-2 login: adminPassword: Seabiscuit!




262


You will notice between 40 and 52 Ethernet interfaces as wellas between 4 and 16 FC interfaces if the GEM module containsFC ports.






--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth1/1 1 eth access down Link not connected 10G(D) --Eth1/2 1 eth access down Link not connected 10G(D) --Eth1/3 1 eth access down Link not connected 10G(D) --...Eth1/40 1 eth access down Link not connected 10G(D) --


f. Open an SSH session to the Nexus 5020 with an IP address of172.23.185.113.


switch (config-sys-qos)# service-policy type qos inputfcoe-default-in-policyswitch (config-sys-qos)# service-policy type queuing input



fcoe-default-in-policyswitch (config-sys-qos)# service-policy type queuing outputfcoe-default-out-policyswitch (config-sys-qos)# service-policy type network-qosfcoe-default-nq-policy


Nexus-5020-2(config)# vsan databaseNexus-5020-2(config-vsan-db)# vsan 200Nexus-5020-2(config-vsan-db)# vsan 400Nexus-5020-2(config-vsan-db)# exitNexus-5020-2(config)#




6. Create the VLAN that will be used as the default VLAN on theFCoE interfaces.

Nexus-5020-2(config-vlan)# vlan 701

Note: This step is only required if you are changing the default VLAN onethernet interface. In later examples, VLAN 700 will be used for iSCSI.

7. Use show FCoE VLAN to verify the configuration was successful.


VLAN VSAN Status-------- -------- --------200 200 Operational400 400 Operational

8. Enable the Fabric Extender feature.

Nexus-5020-2(config-vlan)# feature fex



264


Note: The pinning max-links parameter does not need to be set since theNexus-2232PP will be connected to Nexus-5020-2 via an EtherChannel.

Nexus-5020-2(config)# fex 102Nexus-5020-2(config-fex)# description FEX0102

10. Create an EtherChannel that will be used to connect toNexus-2232-2.

Nexus-5020-2(config-fex)# interface port-channnel 4

11. Configure the EtherChannel as a FEX fabric interface.

Nexus-5020-2(config-if)# switchport mode fex-fabric

12. Associate the FEX instance with the EtherChannel.

Nexus-5020-2(config-if)# fex associate 102

13. Configure and associate the PortChannel with the appropriateethernet interfaces.

Nexus-5020-2(config-if)# interface ethernet 1/1-4Nexus-5020-2(config-if)# switchport mode fex-fabricNexus-5020-2(config-if)# fex associate 102Nexus-5020-2(config-if-range)# channel-group 4

14. Using any of the supported media types, connect Nexus-5020-2 toNexus 2232-2 as shown in the target topology diagram shown inFigure 34 on page 244.

15. Using any of the supported media types, connect Host_1 /CNA_2 and Host_2 / CNA_2 to Nexus 2232-2 as shown in thetarget topology diagram shown in Figure 34 on page 244.

16. Display the switch interfaces using show interface brief:

Nexus-5020-2(config-if-range)# show int brief



-------------------------------------------------------------------------------fc2/1 1 auto on down swl -- --fc2/2 1 auto on down swl -- --fc2/3 1 auto on swl -- --...fc3/4 1 auto on down swl -- --



--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth1/1 1 eth fabric up none 10G(D) 4Eth1/2 1 eth fabric up none 10G(D) 4Eth1/3 1 eth fabric up none 10G(D) 4Eth1/4 1 eth fabric up none 10G(D) 4Eth1/5 1 eth access down Link not connected 10G(D) --...Eth1/40 1 eth access down Link not connected 10G(D) --



--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth102/1/1 1 eth access up none 10G(D) --Eth102/1/2 1 eth access up none 10G(D) --...Eth102/1/32 1 eth access down SFP not inserted 10G(D) --Nexus-5020-2(config-if-range)#

17. Display the EtherChannel interface.

Nexus-5020-2(config-if)# show interface port-channel 4 fex-intfFabric FEXInterface Interfaces---------------------------------------------------------------------Po4 Eth102/1/1 Eth102/1/2 Eth102/1/3 Eth102/1/4

18. Create the VFC interfaces and bind them to an interface on the2232PP.

Nexus-5020-2(config-if-range)# interface vfc 1021Nexus-5020-2(config-if)# bind interface ethernet 102/1/1Nexus-5020-2(config-if)# switchport trunk allowed vsan 200


266


Nexus-5020-2(config-if)# no shutNexus-5020-2(config-if)# interface vfc 1022Nexus-5020-2(config-if)# bind interface ethernet 102/1/2Nexus-5020-2(config-if)# switchport trunk allowed vsan 400Nexus-5020-2(config-if)# no shut

19. Assign each VFC to the appropriate VSAN as shown in the targettopology diagram, Figure 34 on page 244.

Nexus-5020-2(config-if)# vsan databaseNexus-5020-2(config-vsan-db)# vsan 200 interface vfc1021Nexus-5020-2(config-vsan-db)# vsan 400 interface vfc1022Nexus-5020-2(config-vsan-db)# exitNexus-5020-2(config)#

The output of the show interface brief shows the following:




-------------------------------------------------------------------------------fc2/1 1 auto on down swl -- --fc2/2 1 auto on down swl -- --...fc3/4 1 auto on down swl -- --

--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth1/1 1 eth fabric up none 10G(D) 3Eth1/2 1 eth fabric up none 10G(D) 3Eth1/3 1 eth fabric up none 10G(D) 3Eth1/4 1 eth fabric up none 10G(D) 3...Eth1/40 1 eth access down Link not connected 10G(D) --


--------------------------------------------------------------------------------Port VRF Status IP Address Speed MTU



--------------------------------------------------------------------------------mgmt0 -- up 172.22.185.113 1000 1500



-------------------------------------------------------------------------------vfc1021 200 F off down -- -- --vfc1022 400 F off down -- -- --

--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth102/1/1 1 eth access up none 10G(D) --Eth102/1/2 1 eth access up none 10G(D) --Eth102/1/3 1 eth access up none 10G(D) --Eth102/1/4 1 eth access up none 10G(D) --Eth102/1/5 1 eth access down SFP not inserted 10G(D) --...Eth102/1/32 1 eth access down SFP not inserted 10G(D) --Nexus-5020-2(config)#


Each interface will be set to trunk mode to allow it to handlemultiple VLANs and 802.1q tagged frames. The native VLAN isset to 700 in this case to allow for all non-FCoE untagged framesto be forwarded onto VLAN 700.

The allowed VLAN list is set to allow the default VLAN as well asthe FCoE VLAN.

Nexus-5020-2(config-if)# interface ethernet 102/1/1Nexus-5020-2(config-if)# switchport mode trunkNexus-5020-2(config-if)# switchport trunk native vlan 700Nexus-5020-2(config-if)# switchport trunk allowed vlan 100, 700Nexus-5020-2(config-if)# spanning-tree port type edge trunkNexus-5020-2(config-if)# interface ethernet 102/1/2Nexus-5020-2(config-if)# switchport mode trunkNexus-5020-2(config-if)# switchport trunk native vlan 700Nexus-5020-2(config-if)# switchport trunk allowed vlan 300, 700Nexus-5020-2(config-if)# spanning-tree port type edge trunk



268


As shown in Figure 29 on page 176, Nexus-5020-2 is connected totwo different fabrics. In order to ensure that each CNA attachedto Nexus-5020-2 logs in to the same fabric as its storage port, eachCNA will need to be statically mapped to a particular VSAN. Thisis accomplished by adding the VFC Interface to the appropriateVSAN. To summarize the steps:



Nexus-5020-2(config-if)# vsan databaseNexus-5020-2(config-vsan-db)# vsan 200 interface fc 2/1Nexus-5020-2(config-vsan-db)# vsan 200 interface fc 2/3Nexus-5020-2(config-vsan-db)# vsan 400 interface fc 2/2Nexus-5020-2(config-vsan-db)# vsan 400 interface fc 2/4Nexus-5020-2(config-vsan-db)# vsan 200 interface fc 3/1Nexus-5020-2(config-vsan-db)# vsan 200 interface fc 3/3Nexus-5020-2(config-vsan-db)# vsan 400 interface fc 3/2Nexus-5020-2(config-vsan-db)# vsan 400 interface fc 3/4Nexus-5020-2(config-vsan-db)# exitNexus-5020-2(config)#



22. Attach the fiber cables between Nexus-5020-2 and the BrocadeDCX Directors as shown in the target topology diagram inFigure 34 on page 244.

23. Validate the configuration by using the show int brief command.





-------------------------------------------------------------------------------fc2/1 200 auto on trunking swl TE 4 --fc2/2 400 auto on trunking swl TE 4 --



fc2/3 200 auto on trunking swl TE 4 --fc2/4 400 auto on trunking swl TE 4 --fc3/1 200 auto on trunking swl TE 4 --fc3/2 400 auto on trunking swl TE 4 --fc3/3 200 auto on trunking swl TE 4 --fc3/4 400 auto on trunking swl TE 4 --...



-------------------------------------------------------------------------------vfc1021 200 F on trunking -- TF auto --vfc1022 400 F on trunking -- TF auto --

Nexus-5020-2 (config-if)#

24. Use show npv flogi-table vsan <vsan ID> to verify that eachCNA is logged in to the appropriate vsan.

Nexus-5020-2 (config-if)# show flogi da--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------vfc1021 500 0x930033 10:00:00:00:c9:3c:f7:c0 20:00:00:00:c9:3c:f7:c0vfc1022 500 0x930134 21:00:00:1b:32:0a:c0:b8 21:00:00:1b:32:0a:c0:b8


Nexus-5020-2#

The fc interfaces are now online and running in FC-SW mode.The vfc interfaces come online as soon as one of the fc interfacesin the same vsan comes online.



26. Zone the CNAs.

Since Nexus-5020-2 is running in FC-SW mode, zoning can beperformed from either the MDS-9513 Director to which it isattached or from the Nexus 5000 itself. It is stronglyrecommended that if the CLI is to be used for zoning, then cutand paste be utilized to minimize transcription types of errorswhen manually typing in the WWPNs.


270


The set of steps required to create the appropriate zones are:

a. zone name <ZONE_NAME> vsan <VSAN ID>.b. member pwwn <PWWN_1>.c. member pwwn <PWWN_2>.d. zone set name <ZONE_SET_NAME> vsan <VSAN ID>.e. member <ZONE_NAME>.f. zoneset activate name <ZONE_SET_NAME> vsan <VSANID>.g. Copy running startup.

27. Telnet into Nexus-5020-2 by completing the following steps:


Nexus-5020-2# config terminalNexus-5020-2(config)# zone name host1_CNA2 vsan 200Nexus-5020-2(config-zone)# member pwwn 10:00:00:00:c9:3c:f7:c0Nexus-5020-2(config-zone)# member pwwn 50:06:04:8A:D5:2E:69:79Nexus-5020-2(config-zone)# exitNexus-5020-2(config)# zoneset name FCoE_200 vsan 200Nexus-5020-2(config-zoneset)# member host1_CNA2Nexus-5020-2(config-zoneset)# zoneset activate name FCoE_200 vsan 200Zoneset activation initiated. check zone statusNexus-5020-2(config)# zoneset distribute full vsan 200Nexus-5020-2(config)# show zoneset active vsan 200zoneset name FCoE_200 vsan 200zone name host1_CNA2 vsan 200* fcid 0xd00000 [pwwn 10:00:00:00:c9:3c:f7:c0]* fcid 0x6f0000 [pwwn 50:06:04:8a:d5:2e:69:79]Nexus-5020-2(config)#


Nexus-5020-2(config)# zone name host2_CNA2 vsan 400Nexus-5020-2(config-zone)# member pwwn 21:00:00:1B:32:0A:C0:B8Nexus-5020-2(config-zone)# member pwwn 50:06:04:82:D5:2E:69:59Nexus-5020-2(config-zone)# exitNexus-5020-2(config)# zoneset name FCoE_400 vsan 400Nexus-5020-2(config-zoneset)# member host2_CNA2Nexus-5020-2(config-zoneset)# zoneset activate name FCoE_400 vsan 400Zoneset activation initiated. check zone statusNexus-5020-2(config)# zoneset distribute full vsan 400Nexus-5020-2(config)#

Log out of the switch with an IP address of 172.23.185.25.




◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel andiSCSI Host Bus Adapters (HBAs) and Converged Network Adapters(CNAs) in the Windows Environment

◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel HostBus Adapters (HBAs) and Converged Network Adapters (CNAs) in theWindows Environment

◆ EMC Host Connectivity Guide for Windows


◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel andiSCSI Host Bus Adapters (HBAs) and Converged Network Adapters(CNAs) for the Linux Environment

◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel HostBus Adapters (HBAs) and Converged Network Adapters (CNAs) for theLinux v2.6.x Kernel Environment and the v8.x-Series Driver

◆ EMC Host Connectivity Guide for Linux

Configuring the VMware hostTo configure the VMware host, refer to the following documentavailable at EMC Online Support at https://support.emc.com:

◆ EMC Host Connectivity Guide for VMware ESX Server

Configuring EMC storageOnce the switch, hosts, and adapters have been configured, thestorage will need to be configured to each initiator to access a numberof LUNs.

Configuring network parametersUsing the network topology shown in Figure 31 on page 180, displayconfiguration parameters and then follow the steps needed to deploythis Layer 2 network.


272


Configuring Cisco VE_PortsVE_Ports allow for two Cisco Nexus 5000 products to be connectedtogether and merged into the same fabric using FCoE links. VE_Portscan be created as either individual links or as shown in Figure 35,configured to use a PortChannel. The topology used in this exampleis shown in Figure 35.

Figure 35 VE_Ports topology

Note: To complete the following steps, feature lacp must be enabled.

Setup steps On Nexus 5000 A:

1. Create the appropriate VSANs.

NEXUS5000A# config tEnter configuration commands, one per line. End with CNTL/Z.IOP054071(config)# vsan databaseIOP054071(config-vsan-db)# vsan 200IOP054071(config-vsan-db)# vsan 300IOP054071(config-vsan-db)# vsan 400IOP054071(config-vsan-db)# exitIOP054071(config)#

2. Create the appropriate VLAN and associate them with theappropriate VSAN.

IOP054071(config)# vlan 200IOP054071(config-vlan)# fcoe vsan 200IOP054071(config-vlan)# vlan 300IOP054071(config-vlan)# fcoe vsan 300IOP054071(config-vlan)# vlan 400IOP054071(config-vlan)# fcoe vsan 400IOP054071(config-vlan)# exitIOP054071(config)#

VLAN/VSAN 100

VLAN/VSAN 200

VLAN/VSAN 300

VLAN/VSAN 100

VLAN/VSAN 200

VLAN/VSAN 300

Nexus 5000 A Nexus 5000 Be1/2

DCB Cloud

Po 777

e1/1

e1/2

e1/1

ICO-IMGo000974



3. Create the port-channel that will be used to connect the twoswitches.

IOP054071(config)# interface port-channel 777IOP054071(config-if)# switchport mode trunkIOP054071(config-if)# exitIOP054071(config)#

4. Create the vfc interface that will be associated with theport-channel, specify it as an E_Port and allow the appropriateVSANs.

IOP054071(config)# interface vfc777IOP054071(config-if)# bind interface port-channel777IOP054071(config-if)# switchport mode EIOP054071(config-if)# no shutdownIOP054071(config-if)# exitIOP054071(config)#

5. Add the Ethernet interfaces to the port-channel.

IOP054071(config)# int e1/1-2IOP054071(config-if)# switchport mode trunkIOP054071(config-if-range)# channel-group 777 mode activeIOP054071(config-if-range)# exitIOP054071(config)#

At this point the VE_Port should initialize and can be viewed usingthe show fcoe da command.

IOP054071(config)# show fcoe database

-------------------------------------------------------------------------------INTERFACE FCID PORT NAME MAC ADDRESS-------------------------------------------------------------------------------

VE Ports:-------------------------------------------------------------------------------INTERFACE MAC ADDRESS VSAN-------------------------------------------------------------------------------vfc777 00:05:73:af:13:68 200vfc777 00:05:73:af:13:68 300vfc777 00:05:73:af:13:68 400IOP054071(config)#

Configuring Cisco VE_Ports 273

274


Configuring Cisco FCoE NPV portsFCoE NPV allows for traditional Fibre Channel NPV functionality torun over FCoE links.

FCoE NPV ports can be created as either individual links orconfigured to use a PortChannel.

The topology used in this example is shown in Figure 36.

Figure 36 VN_Ports topology

Note: In this configuration, the upstream switch cannot be in NPV modeunless it is connected to another upstream switch.

Setup steps On Nexus 5596-1:

1. Configure the needed features for this configuration.

Note: If the environment is a strictly FCoE environment with no FCinterfaces, then the FCoE-NPV feature can be used to replace the separateFCoE and NPV features. This does not require a switch reboot.

Nexus-5596-1# config tEnter configuration commands, one per line. End with CNTL/Z.Nexus-5596-1(config)# feature lacpNexus-5596-1(config)# feature fcoeNexus-5596-1(config)# feature npvVerify that boot variables are set and the changes are saved. Changing to npv mode

erases the current configuration and reboots the switch in npv mode. Do youwant to continue? (y/n):y

2. Configure needed QoS policies.

Nexus-5596-1# config tEnter configuration commands, one per line. End with CNTL/Z.



Nexus-5596-1(config)# service-policy type qos input fcoe-default-in-policyNexus-5596-1(config)# service-policy type queuing input fcoe-default-in-policyNexus-5596-1(config)# service-policy type queuing output fcoe-default-out-policyNexus-5596-1(config)# service-policy type network-qos fcoe-default-nq-policy

3. Create the appropriate VLAN and associate them with thecorresponding VSAN.

Nexus-5596-1 (config)# vlan 200Nexus-5596-1 (config-vlan)# fcoe vsan 200Nexus-5596-1 (config-vlan)# vlan 300Nexus-5596-1 (config-vlan)# fcoe vsan 300Nexus-5596-1 (config-vlan)# vlan 400Nexus-5596-1 (config-vlan)# fcoe vsan 400Nexus-5596-1 (config-vlan)# exitNexus-5596-1 (config)#

4. Create the port-channel that will be used to connect the twoswitches.

Nexus-5596-1 (config)# interface port-channel 300Nexus-5596-1 (config-if)# switchport mode trunkNexus-5596-1 (config-if)# exitNexus-5596-1 (config)#

5. Create the vfc interface that will be associated with theport-channel, specify it as an NP_Port. By default the vfc interfacewill trunk all created VSAN'S.

Nexus-5596-1 (config)# interface vfc300Nexus-5596-1 (config-if)# bind interface port-channel300Nexus-5596-1 (config-if)# switchport mode NPNexus-5596-1 (config-if)# no shutdownNexus-5596-1 (config-if)# exitNexus-5596-1 (config)#

6. Add the Ethernet interfaces to the port-channel.

Nexus-5596-1(config-if)# switchport mode trunkNexus-5596-1(config-if-range)# channel-group 300 mode activeNexus-5596-1(config-if-range)# exitNexus-5596-1(config)#

7. Verify the vfc interface is working properly by looking at theinterface.

Nexus-5596-1(config)# sho int vfc300vfc300 is trunking (Not all VSANs UP on the trunk)

Bound interface is port-channel300Hardware is Virtual Fibre ChannelPort WWN is 21:2b:00:05:73:b1:7a:3fAdmin port mode is NP, trunk mode is on

Configuring Cisco FCoE NPV ports 275

276


snmp link state traps are enabledPort mode is TNPPort vsan is 1Trunk vsans (admin allowed and active) (1,100,200,300,600,800,1002)Trunk vsans (up) (300)Trunk vsans (isolated) ()Trunk vsans (initializing) (1,100,200,600,800,1002)1 minute input rate 15773704 bits/sec, 1971713 bytes/sec, 2825 frames/sec1 minute output rate 13929328 bits/sec, 1741166 bytes/sec, 2111 frames/sec

22280246234 frames input, 36599930008008 bytes0 discards, 0 errors

9395517636 frames output, 12524663141656 bytes3 discards, 0 errors

last clearing of "show interface" counters neverInterface last changed at Tue Aug 23 14:01:23 2011



Complex FCoE topologiesPrevious section in this chapter discuss the process of configuring theNexus switches in various basic topologies. These basic topologiesare provided for example purposes and can be combined to createmore complex topologies. This section describes two complextopologies based on the following:

◆ “Fixed port chassis” on page 277

◆ “Modular chassis” on page 278

Note: The Nexus 7000 used in the modular case study is not currentlysupported by EMC for FCoE connectivity.

Fixed port chassisThe fixed port chassis configuration is based on the combination ofthe Nexus 5596 and the Nexus 2232PF (FEX) module. The basicbuilding block of this configuration is a pair of racks (pod).

Each rack in the pod has a single Nexus 2232 at the top of the rackand contains 16 servers. The servers in each rack are cross-connectedso that all SAN A connections from both racks go to the top of onecabinet; the SAN B connection from both cabinets are all connected tothe top of the other rack.

Figure 37 on page 278 shows an example of a fixed port chassisconfiguration.

Complex FCoE topologies 277

278


Figure 37 Fixed port chassis configuration example

In this example, these racks are part of a row that contains 30 racksfor a total of 480 servers or 960 ports.

Also shown are also a total of 96 storage ports evenly distributedamong the 5596s. Since there are 960 server ports, this will equate to a10:1 server to storage port ratio.

Modular chassisThe modular chassis configuration is based on the Nexus 7000. In thisexample, there are 24 racks and 384 servers. There are also two Nexus7018s and each Nexus contains (14) 32 port 10G "F1" line cards for atotal of 448 ports.

Figure 38 on page 279 shows an example of a modular chassisconfiguration.



Figure 38 Modular chassis configuration example

Currently, this configuration is slightly oversubscribed since theserver facing side allows for 320 Gb/s of connectivity while thebackplane can only handle 230 Gb/s per slot. However, a fullynon-blocking solution is planned for a future release.

The configuration also has 768 OM3 fiber cables to allow forconnectivity from the servers to the switch. There is a total of 78storage ports connected to the Nexus 7000s.

Complex FCoE topologies 279

280


Nexus Series switches firmware upgrade procedureThis procedure transfers files from an FTP server to the bootflash:directory on the Nexus Series switches and then uses install all toactually install the new version of firmware. It requires that the newfirmware files already be uploaded to an FTP server that is accessiblefrom the Management port of the Nexus Series switches.

IMPORTANT

This procedure does require the switch to be reloaded, duringwhich time all traffic will be disrupted for 10-15 minutes.


◆ “Assumptions” on page 280

◆ “Upgrading firmware” on page 280

AssumptionsThis procedure assumes the following:

◆ The firmware files are stored in the outgoing/firmware directoryof an FTP server at the IP address of 168.159.216.19

◆ The name of the kickstart image isn5000-uk9-kickstart.4.0.1a.N1.1a.bin

◆ The name of the system image is n5000-uk9.4.0.1a.N1.1a.bin

Upgrading firmware

Note: The text in bold is input by the user.

To upgrade firmware:

1. Create a backup of your existing configuration file (make sure therunning configuration is correct).

Nexus-5020-2# copy running-config startup-config

2. Back up the existing configuration file. You can do this throughthe ftp server using the following format:



copy startup-config ftp://<server IP address>/startup-config_<switch name>_<date>vrf management

Nexus-5020-2# copy startup-configftp://168.159.216.19/startup-config_Nexus5020-2_102308 vrf management

Enter username: anonymousPassword: [email protected]***** Transfer of file Completed Successfully *****

3. Verify the status of the Supervisor is "active" and each module is"ok".

Nexus-5020-2# show moduleMod Ports Model-type Model Status--- ----- ------------------------------- ------------------ ---------1 40 40x10GE/Supervisor N5K-C5020P-BF active *2 8 4x10GE + 4x1/2/4G FC Module N5K-M1404 ok3 8 4x10GE + 4x1/2/4G FC Module N5K-M1404 ok

Mod Sw Hw World-Wide-Name(s) (WWN)--- ---------------------------------------------------------------------1 4.1(3)N1(1) 1.0 --2 4.1(3)N1(1) 1.0 20:41:00:0d:ec:a3:28:80 to 20:44:00:0d:ec:a3:28:803 4.1(3)N1(1) 0.100 20:81:00:0d:ec:a3:28:80 to 20:84:00:0d:ec:a3:28:80

Mod Mac Address(es) Serial-Num--- ---------------------------------- ---------------1 000d.eca3.2888 to 000d.eca3.28af JAB122800VP2 000d.eca3.28b0 to 000d.eca3.28b7 JAB1231022S3 000d.eca3.28b8 to 000d.eca3.28bf JAB1231023U

4. Transfer the kickstart image to the bootflash: directory on theNexus 5020.

Nexus-5020-2# copyftp://168.159.216.19/outgoing/firmware/n5000-uk9-kickstart.4.1.3.N2.1.binbootflash:/n5000-uk9-kickstart.4.1.3.N2.1.bin vrf management


5. Transfer the system image to the bootflash: directory on theNexus 5020.

Nexus-5020-2# copyftp://168.159.216.19/outgoing/firmware/n5000-uk9.4.1.3.N2.1.binbootflash:/n5000-uk9.4.1.3.N2.1.bin vrf management


Nexus Series switches firmware upgrade procedure 281

282


6. Verify that the file was copied in the required directory.

Nexus-5020-2# cd bootflash:Nexus-5020-2# dir

20217856 Apr 17 10:05:59 2009 n5000-uk9-kickstart.4.0.1a.N1.1a.bin85704356 Apr 17 10:06:40 2008 n5000-uk9.4.0.1a.N1.1a.bin

7. Ensure that the software images are not damaged or corrupted inthe saved bootflash:

When copying a new image to your switch, confirm that the image was not corruptedduring the copy process. Use the show version image command to verify that therequired image was copied successfully.

Nexus-5020-2# show version image bootflash:n5000-uk9.4.1.3.N2.1.binimage name: n5000-uk9.4.1.3.N2.1.binsystem: version 4.1(3)N2(1)compiled: 11/6/2009 11:00:00 [11/06/2009 20:40:29]

8. Use the install all command to install the firmware.

Nexus-5020-2# install all kickstart bootflash:/n5000-uk9-kickstart.4.1.3.N2.1.binsystem bootflash:/n5000-uk9.4.1.3.N2.1.bin

Verifying image bootflash:/n5000-uk9-kickstart.4.0.1a.N1.1a.bin for boot variable"kickstart".

[# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS

Verifying image bootflash:/n5000-uk9.4.0.1a.N1.1a.bin for boot variable "system".[# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS

Verifying image package type.[# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS

Extracting "system" version from image bootflash:/n5000-uk9.4.0.1a.N1.1a.bin.[# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS

Extracting "kickstart" version from imagebootflash:/n5000-uk9-kickstart.4.0.1a.N1.1a.bin.

[# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS

Extracting "bios" version from image bootflash:/n5000-uk9.4.0.1a.N1.1a.bin.[# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS

Collecting "running" plugin(s) information.[# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS

Collecting plugin(s) information from "new" image.[# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS

Compatibility check is done:



Module bootable Impact Install-type Reason------ -------- --------- ---------------- ------

1 yes disruptive reset Reset due to single supervisor

Images will be upgraded according to following table:

Module Image Running-Version New-Version Upg-Required------ ---------- ---------------------- ----------------------1 system 4.1(3)N1(1) 4.1(3)N2(1) yes1 kickstart 4.1(3)N1(1) 4.1(3)N2(1) yes1 bios v1.3.0(09/08/09) no

Do you want to continue with the installation (y/n)? [n] Y

Install is in progress, please wait.

Setting boot variables.[# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS

Performing configuration copy.[# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS

Install has been successful.

9. Reconnect to the switch after the reload has completed, ifapplicable. Verify that the status of the Supervisor is "active" andeach module is "ok".

Nexus-5020-2# show moduleMod Ports Model-type Model Status--- ----- ------------------------------- ------------------ -----------1 40 40x10GE/Supervisor N5K-C5020P-BF active *2 8 4x10GE + 4x1/2/4G FC Module N5K-M1404 ok3 8 4x10GE + 4x1/2/4G FC Module N5K-M1404 ok

Mod Sw Hw World-Wide-Name(s) (WWN)--- -------------------------- --------------------------------------1 4.1(3)N2(1) 1.0 --2 4.1(3)N2(1) 1.0 20:41:00:0d:ec:a3:28:80 to 20:44:00:0d:ec:a3:28:803 4.1(3)N2(1) 0.100 20:81:00:0d:ec:a3:28:80 to 20:84:00:0d:ec:a3:28:80

Mod Mac Address(es) Serial-Num--- ---------------------------------- ------------1 000d.eca3.2888 to 000d.eca3.28af JAB122800VP2 000d.eca3.28b0 to 000d.eca3.28b7 JAB1231022S3 000d.eca3.28b8 to 000d.eca3.28bf JAB1231023U

Nexus Series switches firmware upgrade procedure 283

284


Troubleshooting the Nexus Series switchesThis section explains which troubleshooting outputs to capture fromNexus Series switches, including:

◆ “Show tech-support selections” on page 284

◆ “General connectivity troubleshooting” on page 285

◆ “Fibre Channel troubleshooting” on page 286

◆ “Hardware-related errors” on page 288

◆ “High CPU issue troubleshooting” on page 288

◆ “Gatos Port ASIC troubleshooting” on page 289

◆ “Altos Fabric ASIC troubleshooting” on page 289

◆ “FCoE troubleshooting” on page 289

◆ “DCBX troubleshooting” on page 290

◆ “Verify zoning configuration and that CNA WWPN is zonedactive” on page 290

◆ “Verify CNA WWPN is registered in nameserver in your VSAN”on page 290

◆ “Ethernet PortChannel troubleshooting” on page 290

◆ “Verify PortChannels” on page 291

◆ “Spanning-tree troubleshooting” on page 291

◆ “Multicast troubleshooting” on page 292

Show tech-support selections

Capturing show tech-support information is useful when gatheringdata for troubleshooting and escalations.

◆ show tech-support details is used to collect everything on theswitch.

◆ sh tech-support commands lists out what will be displayed.

To dump output of a show tech-support details to bootflash: on theswitch for retrieval with FTP use tac-pac bootflash:///

dir bootflash:73482 Oct 06 18:26:34 2008 show_tech_out.gz output is zipped



> Redirect it to a fileaaa Display aaa informationaclmgr ACL commandsadjmgr Display Adjmgr informationarp Display ARP informationbootvar Gather detailed information for bootvar troubleshootingbrief Display the switch summarycdp Gather information for CDP trouble shootingcfs Gather detailed information for cfs troubleshootingcommands Show commands executed as part of show tech-support commandsdetails Gather detailed information for troubleshootingdevice-alias Show device-alias technical support informationethpm Gather detailed information for ETHPM troubleshootingfc Get fibre channel related informationfcdomain Gather detailed information for fcdomain troubleshootingfcns Show information for fcns technical supportflogi Gather detailed information for flogi troubleshootingfspf Show information for fspf technical supportim Gather detailed information for IM troubleshootinginterface-vlan Gather detailed information for interface-vlan troubleshootinginternal Gather internal info for troubleshootingip Display IP informationl3vm Display VRF informationlicense Gather detailed information for license troubleshootinglogging Show information on logging for technical support staffmodule Gather info related to a modulemonitor Gather detailed information for monitor troubleshootingplatform Get platform related informationport-channel Gather detailed information for port channel troubleshootingprivate-vlan Gather detailed information for private-vlan troubleshootingradius Display radius informationrouting Display routing informationrscn Show information for rscn technical supportsecurity show tech support information for securitysnmp Gather info related to snmpstp Gather detailed information for STP troubleshootingsysmgr Gather detailed information for sysmgr troubleshootingvdc Gather detailed information for VDC troubleshootingvlan Gather detailed information for VLAN troubleshootingvsan Gather info related to an vsanzone Show information for zoneserver technical support| Pipe command output to filter

General connectivity troubleshootingshow loggingSFP issues - show system internal ethpm event-history errorsshow interface ethx/xsh tech-support ethpm (this can be large output, need to capture)show cdp neighborsshow mac

Troubleshooting the Nexus Series switches 285

286


ethernet:

show interface ethernet {x/y} counters brief ---> 5min in/out rateshow interface ethernet {x/y} priority-flow-controlshow interface ethernet {x/y} statusshow interface ethernet {x/y} status err-disableshow policy-map interface [ethernet {x/y}] ---> qos queue drops

vfc:

show interface vfc {x} counters briefshow interface vfc {x} brief

Access-lists:

show platform afm info attachment brief ---> look for the interface with theacl in question and note group# & ASIC#show platform afm info group-config id {group# } asic {asic# } ---> group# andasic# are from previous command

Fibre Channel troubleshootingshow tech fc

This output will capture the following shows when the switch is setin switch mode:

`show interface brief``show interface``show port internal info all``show port internal event-history lock``show port internal event-history msgs``show port internal event-history errors``show port internal mem-stats detail``show san-port-channel internal event-history all``show san-port-channel internal event-history errors``show san-port-channel internal event-history msgs``show san-port-channel internal event-history lock``show san-port-channel internal mem-stats detail``show san-port-channel usage``show san-port-channel summary``show san-port-channel consistency detail``show tech-support device-alias``show device-alias database``show device-alias pending``show device-alias pending-diff``show device-alias status``show device-alias session status``show device-alias merge status``show device-alias statistics``show device-alias internal info`



`show device-alias internal peer-info``show device-alias internal validation-info``show device-alias internal event-history session-fsm``show device-alias internal errors``show device-alias internal mem-stats detail``show device-alias internal msgs``show fcdomain domain-list``show tech-support fcns``show fcns database vsan 1-4093``show fcns database detail vsan 1-4093``show fcns database local vsan 1-4093``show fcns database local detail vsan 1-4093``show fcns statistics vsan 1-4093``show fcns statistics detail vsan 1-4093``show fcns internal info vsan 1-4093``show fcns internal event-history``show fcns internal event-log``show fcroute unicast``show fcs database``show fcs ie``show fctimer``show flogi database``show flogi internal info``show fspf``show fspf database``show tech-support rscn``show rscn internal vsan 1-4093``show rscn internal event-history``show rscn internal mem-stats detail``show rscn internal session-history vsan 1-4093``show rscn internal merge-history vsan 1-4093``show rscn statistics vsan 1-4093``show rscn scr-table vsan 1-4093``show rscn session status vsan 1-4093``show vsan``show vsan membership``show tech-support zone``show zone status vsan 1-4093``show zoneset active vsan 1-4093``show zoneset vsan 1-4093``show zone vsan 1-4093``show fcalias vsan 1-4093``show zone policy vsan 1-4093``show zoneset pending active vsan 1-4093``show zoneset pending vsan 1-4093``show zone pending vsan 1-4093``show zone pending active vsan 1-4093``show fcalias pending vsan 1-4093``show zone policy pending vsan 1-4093``show zone pending-diff vsan 1-4093``show zone analysis active vsan 1-4093``show zone analysis vsan 1-4093`


288


`show zone ess vsan 1-4093``show zone internal vsan 1-4093``show zone internal change event-history vsan 1-4093``show zone internal ifindex-table vsan 1-4093``show zone internal merge event-history vsan 1-4093``show zone internal event-history``show zone internal event-history errors``show zone internal tcam event-history vsan 1-4093``show zone statistics vsan 1-4093``show system default zone``show zone internal ddas-table``show zone internal sdv-table vsan 1-4093``show zone internal mem-stats``show zone internal mem-stats detail``show zone internal transit-table received vsan 1-4093``show zone internal transit-table forwarded vsan 1-4093``show zone internal transit-table rejected vsan 1-4093`

This output will capture the following shows when switch is set innpv mode:

`show interface brief``show interface``show port internal info all``show port internal event-history lock``show port internal event-history msgs``show port internal event-history errors``show port internal mem-stats detail`

`show vsan``show vsan membership`

Hardware-related errorsshow hardware internal fc-mac all-ports <<< Link Issues - ports flappingshow hardware internal plog errors

Gatos port statistics for FC : If there is any one port having issues

show hardware internal gatos port fc x/yshow hardware internal gatos port fc x/y {counters|error|rx|tx|interrupt}

High CPU issue troubleshootingshow platform fwm info pif all ---> gives physical interface (pif) detailsshow platform fwm info lif all ---> gives logical interface (lif) detailsshow ip traffic



Gatos Port ASIC troubleshootingshow hardware internal gatos port ethernet {x/y}show hardware internal gatos port ethernet {x/y} counters errorshow hardware internal gatos port ethernet {x/y} counters rxshow hardware internal gatos port ethernet {x/y} counters txshow hardware internal gatos port ethernet {x/y} interruptshow platform fwm info pif ethernet {x/y} ---> note the gatos number for the port

for next command

show platform fwm info gatos-errors {gatos# } ---> gives reason for drops inprevious command

show hardware internal gatos event-history errors | grep error_interrupt

Altos Fabric ASIC troubleshootingshow hardware internal altos counters interrupts all ---> shows non-zero interupts

FCoE troubleshootingshow tech-support flogi

show int ethernet x/x priority-flow-control

LLDP Transmit and Receive should be enabled on the interface for DCBX negotiationto be successful

switch# show lldp interface ethernet 2/1 | grep tx_entx_enabled: TRUEswitch# show lldp interface ethernet 2/1 | grep rx_enrx_enabled: TRUE

Verify ethernet interface is UP - sh int ethernet 28/1

Verify vfc interface is connected - show int vfc 1


290


DCBX troubleshootingshow lldp neighborsshow lldp interface ethernet {x/y} | grep tx_en ---> LLDP Tx should be enabled for

DCBX negotiation to be successful

show lldp interface ethernet {x/y} | grep rx_en ---> LLDP Rx should be enabled forDCBX negotiation to be successful

show lldp interface ethernet {x/y} | grep "No remote peers exist" ---> check thatpeer supports LLDP

show platform software dcbx internal errors | grep "Received bad LLDPPacket" ---> check LLDP rxed by peer is good

show platform software dcbx internal info interface ethernet {x/y} | grep"DCX TLV" ---> If DCBX TLV's are not shown, peer doesn't support DCBX TLV

show platform software dcbx internal info interface ethernet {x/y} ---> forevery feature negotiation result: "error=0" and an "Operating Config" should exist

Can use ethanalyzer to see packets going from/to CNA Typical Menlo based CNA trace

ethanalyzer local interface inbound-hi brief display-filtereth.addr==00:c0:dd:0a:ad:1d

(find the eth addr from show int ethernet x/y and show int vfc x/y )

Verify zoning configuration and that CNA WWPN is zoned activesh zoneset active

zoneset name ZS_V11 vsan 11zone name Z_V11 vsan 11* fcid 0x2e0001 [pwwn 21:01:00:1b:32:36:a2:ed]* fcid 0x5c0400 [pwwn 50:06:0b:00:00:1e:8a:a9]

Verify CNA WWPN is registered in nameserver in your VSANshow fcns database vsan 11

Ethernet PortChannel troubleshootingWhen you add an interface to a channel group, the Cisco NX-OSchecks certain interface attributes to ensure that the interface iscompatible with the channel group. The Cisco NX-OS also checks anumber of operational attributes for an interface before allowing thatinterface to participate in the port-channel aggregation.



The compatibility check includes the following operational attributes:

◆ Port mode

◆ Access VLAN

◆ Trunk native VLAN

◆ Allowed VLAN list

◆ Flow control

Use the show port-channel compatibility-parameters command tosee the full list of compatibility checks that the Cisco NX-OS uses.

Verify PortChannelssh spanning-tree

show port-channel summary

show port-channel databaseshow port-channel trafficshow port-channel internal info allshow platform fwm info pc port-channel {# }show lacp neighbor

Spanning-tree troubleshootingshow tech-support stp

show spanning-tree <option>

<CR>> Redirect it to a fileactive Report on active interfaces onlyblockedports Show blocked portsbridge Status and configuration of this bridgebrief Brief summary of interface informationdetail Detailed informationinconsistentports Show inconsistent portsinterface Spanning Tree interface status and configurationinternal Show internal STP informationmst Multiple spanning treespathcost Show Spanning pathcost optionsroot Status and configuration of the root bridgesummary Summary of port statesvlan VLAN Switch Spanning Trees| Pipe command output to filter


292


Multicast troubleshootingshow ip igmp snooping <option>

explicit-tracking Display explicit-tracking database for IGMPv3groups Display snooping information for group addressmrouter Display multicast routers detectedquerier Display snooping querier informationstatistics Display packet/error counter statisticsvlan Display Vlan IGMP snooping membership information| Pipe command output to filter

sh mac-address-table multicast <option>

count Display only the count of MAC entriesigmp-snooping Display Entries learned from IGMP Snoopinguser Display User Entered Entrieslan VLAN

sh interface ethernet x/x counters storm-control



Virtual PortChannelThis section provides information on Cisco virtual PortChannel(vPC) and includes topology examples and Nexus case studies.

◆ “Introduction” on page 293

◆ “vPC overview” on page 299

◆ “vPC topologies overview” on page 306

◆ “vPC topology considerations” on page 309

◆ “vPC case studies” on page 324

IntroductionA virtual PortChannel (vPC) allows PortChannel links that arephysically connected to two different Cisco Nexus products (7000,6000, or 5000) to appear as a single virtual PortChannel to a thirddevice. This technology allows for Layer 2 multipathing whilemaintaining a loop free topology.

The purpose of this chapter is to explain the advent of vPC andprovide detailed case studies that describe how to configure severalof the supported vPC topologies. The case studies are based onconfigurations that have been set up within the EMC E-Lab™ andfocus primarily on the storage protocols FCoE, iSCSI, and NAS.

This section explains the following:

◆ “vPC and the storage protocols” on page 293

◆ “Underlying protocols” on page 294

vPC and the storage protocolsThis section briefly describes how the new vPC protocol interactswith the three major storage protocols, each discussed further in thissection.

◆ “FCoE” on page 293

◆ “iSCSI” on page 294

◆ “NAS” on page 294

FCoE Like Fibre Channel, FCoE gains its path failover and load balancingfeatures from host-based multipathing products, such as EMCPowerPath or MPIO. FCoE does not operate over vPC because a host

Virtual PortChannel 293

294


must use NIC teaming/bonding to operate with vPC, whichinherently breaks the native Fibre Channel multipathing software.However, FCoE runs successfully with CNAs participating in vPCs,but the FCoE traffic will not take advantage of the vPC protocol itself.

iSCSI iSCSI is like FCoE in that it gains its path failover and load-balancingfeatures from host-based multipathing products, such as EMCPowerPath and MPIO. iSCSI will operate in a vPC environment, butdoes not take advantage of the vPC protocol. iSCSI relies on theupper layer applications to provide failover capabilities.

NAS NAS, along with any other type of Ethernet traffic capable of runningover a NIC teaming/bonding configuration within a server or host,will function over the vPC protocol.

Underlying protocolsTo better understand how vPC works, it is important to have aworking knowledge of the two underlying protocols that have had animpact on its development, each discussed further in this section:

◆ “Spanning Tree Protocol (STP)” on page 294

◆ “Link Aggregation” on page 296

Spanning TreeProtocol (STP)

The Spanning Tree Protocol (STP) is a Link Layer network protocolthat was developed to guarantee a loop-free topology within aswitched or bridged (Layer 2) network topology. STP prevents loopsin a Layer 2 network by placing redundant links into a blocking, ornon-forwarding, state. While this allows for large-scale Layer 2networks to be built, it does have its drawbacks, such as:

◆ In any type of redundant topology, only one link can beforwarding from one Layer 2 device to another Layer 2 device at atime.

◆ In the event of a failover from an active link to a redundant link,the STP protocol will cause the interfaces to stop forwarding alltraffic for a period of time. The length of time depends on the STPversion in use.

Both of these limitations are necessary, do work well, and allow forlarge and reliable Layer 2 networks. However, in today's data centersthere is a new trend to merge traditional Ethernet traffic with newerstorage protocols, which has driven the need to re-think the way inwhich a Layer 2 network should behave.



For more detailed information on the Spanning Tree Protocol, refer tothe Spanning Tree Protocol (STP)” section in the "Ethernet Basics"chapter in the Fibre Channel over Ethernet (FCoE), Data Center Bridging(DCB) Concepts and Protocols TechBook athttp://elabnavigator.EMC.com, under the Topology ResourceCenter tab. Figure 39 shows a high-level example of how STP works.

Figure 39 STP example

In Figure 39, STP puts the redundant links to the Root Bridge into ablocking state to eliminate any loops created by the topology.Although this is the way STP is designed to work, the downside isthat the redundant link sits in an idle state until the primary link isbrought down. This effectively cuts the available bandwidth to theupstream switches in half. Also, when a failover occurs, the switcheswill stop forwarding all traffic until the topology is stable again.Disruption times vary depending on the size of the Layer 2 network,the complexity of the network, the version of STP in use, and where afailure occurs. The benefits and limitations of Spanning Tree are asfollows:

◆ Benefits include:

• Scalable

• Resilient

X X

Root bridge

SW 1 SW 2

SW 3 SW 4

X = Blocking port

ICO-IMG-000877



296


◆ Limitations include:

• Limits bandwidth

• Convergence times

Link Aggregation This section provides information on the following:

◆ “Link Aggregation” on page 296

◆ “Link Aggregation Control Protocol (LACP)” on page 298

Link AggregationLink Aggregation is a method of bundling multiple physicalinterfaces into a single virtual interface within a single piece ofhardware, such as a switch or router. Servers with multiple NetworkInterface Cards (NICs) can also take advantage of this through theuse of NIC Teaming (for Windows Operating Systems) or NIC Bonding(for Linux Operating Systems).

Link Aggregation, generically known in Cisco terminology as aPortChannel, can also be called an EtherChannel or LAG, dependingon which vendor is being referenced.

Link aggregation groups (LAG) appear as one interface to a switchand therefore are treated as one instance to Spanning-Tree. Thisallows all ports within a LAG to be active and forward trafficsimultaneously.



As shown in Figure 40, PortChannels are created between switches1–3 and 2–4. The total theoretical bandwidth of each PortChannel is40 Gb/s.

Figure 40 PortChannel example

When a LAG is used, two things happen:

◆ Overall bandwidth increases

◆ STP sees the virtual interface as a single entity

As a result, STP will forward through the virtual link rather than putone or more of the redundant links into a blocking state. If one of thephysical links within a PortChannel fails, then any traffic flowscurrently using that link would failover to one of the other linkswithin the bundle, bypassing STP. This failover time is withinmilliseconds and is transparent to most applications.

A PortChannel can be configured manually or by using the standardsbased Link Aggregation Control Protocol (LACP). If LACP is notused, the configurations on both sides of a PortChannel must beconfigured identically or the link will not come up. For LACPcompatibility, refer to Table 7 on page 299.

The benefits and limitations of Link Aggregation are as follows:

◆ Benefits include:

• Bypass STP• Increased bandwidth• Quick failover times

10 Gig

10 Gig 10 Gig

Port Channel4 x 10 Gig = 40 Gig

Port Channel4 x 10 Gig = 40 Gig

SW 1 SW 2

SW 3 SW 4

ICO-IMG-000878


298


◆ Limitations include:

• Point-to-point only• Not scalable

For more detailed information on Link Aggregation, refer to “LinkAggregation” section in the "Ethernet Basics" chapter in the FibreChannel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts andProtocols TechBook at http://elabnavigator.EMC.com, under theTopology Resource Center tab.

IMPORTANT

It is recommended that LACP be used to configure the PortChannelinterfaces.

Link Aggregation Control Protocol (LACP)LACP is part of the IEEE specification (802.3ad) and was developedto allow for the dynamic formation of Link Aggregations betweentwo participating peer devices. This is accomplished by devices onboth ends of a link participating in the protocol by transmitting LinkAggregation Control Protocol Data Units (LACPDUs). If a sendingdevice finds a LACP-enabled device on the other end, then acommunication process begins in which both ends of the link willforward configuration information. All links participating in LACPwill be able to determine if they belong to the same Link Aggregationand will form the bundle as configured.

For example, if you were to build a four-link bundle using LACPbetween two end devices (switches, in this case) you would first needto configure the switches to tell the participating interfaces that theyall belong to the same PortChannel with LACP enabled. Once thedevices are connected, the interfaces will communicate and form thePortChannel dynamically, based on the pre-configuration of theinterfaces.

LACP can be configured in two ways, active or passive.

◆ Active will continually send out frames hoping to reach anotherLACP-enabled interface

◆ Passive will not send out any LACP frames but will listen forthem and form a link only after it has received frames from anActive link.

If you configure a PortChannel as ON, then the LACP protocol willnot be invoked and manual configurations will be used. Interfaces




configured as ON will not participate in the LACP protocol andtherefore will not work with interfaces running LACP. For this link toform, both ends of the link need to be ON and the configurationsmust match.

Table 7 lists LACP compatibility.

vPC overviewCisco's virtual PortChannel (vPC) feature was created to overcomeboth Spanning Tree and Link Aggregation limitations. vPCovercomes these limitations by allowing a multi-chassis PortChannelto be created while bypassing STP.

More details are provided in the following Cisco documentation,located at www.cisco.com:

◆ Virtual PortChannel Quick Configuration Guide

◆ Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide

◆ Cisco Nexus 5000 Series Layer 2 Switching Configuration Guide

A vPC allows links that are physically connected to two differentCisco Nexus products (7000 or 5000) to appear as a singlePortChannel to a third device. The third device can be a switch,server, or any other networking device capable of running LinkAggregation protocol. For an example, refer to Figure 41 on page 300.The links from the server are physically connected to SW 3 and SW 4.These links appear as a single PortChannel on the server.

Table 7 LACP compatibility

Switch Switch/Host Comments

active active Recommended.

active passive Link aggregation occurs if negotiation is successful.

on on Link aggregation occurs without LACP.

Note: Although this works, it is not recommended.

on active Will not work.

on passive Will not work.


300


A vPC provides multipathing, which allows you to createredundancy by enabling multiple parallel paths between devices andalso allows for traffic to be load balanced where alternative pathsexist. The third device sees the multi-chassis vPC as a single entity.This allows for all paths to be actively forwarding because STP is nolonger putting the redundant links into a blocking state.

The following sections provide more information on vPC:

◆ “vPC example” on page 300

◆ “Benefits” on page 301

◆ “Concepts” on page 301

vPC exampleFigure 41 shows an example of vPC.

Figure 41 vPC example

Note: Figure 41 shows that a vPC can be created to a host. This is notapplicable in all topologies.

vPC x

vPC z

vPC y

vPCDevice

vPCDevice

Nexus 5000 Nexus 5000

SW 1 SW 2

SW 3 SW 4

ICO-IMG-000879

vPC Domain

Peer-keepalivelink

vPC Peer link



BenefitsThe benefits of vPC include:

◆ Allows a single device to create a PortChannel across twoupstream devices

◆ Eliminates Spanning Tree Protocol blocked ports

◆ Provides a loop-free topology

◆ Uses all available uplink bandwidth

◆ Provides fast convergence if either the link or a device fails

◆ Provides link-level resiliency

◆ Helps ensure high availability

◆ Hashing algorithm by default is the same as used in traditionalload-balancing.

• Layer 3 – Source-destination ip addresses

• Layer 2 – Source-destination mac addresses

ConceptsTo better understand vPC, consider the following vPC-specificterminology, as described in the Virtual PortChannel QuickConfiguration Guide, located at www.cisco.com.

vPC The combined EtherChannel between the vPC peer devices and thedownstream device.

vPC domain This domain includes both vPC peer devices, the vPC peer-keepalivelink, and all of the port channels in the vPC connected to thedownstream devices. It is also associated to the configuration modethat you must use to assign vPC global parameters. The vPC DomainID must be the same on both switches. Refer to “vPC domain” onpage 302 for more information.

vPC Domain ID A unique numeric value in the range of 1-1000 assigned to a vPCDomain.

vPC peer switch One of a pair of devices that are connected with the specialEtherChannel known as the vPC peer link.


302


vPC peer link The link used to synchronize states between the vPC peer devices.The vPC peer link carries control traffic between two vPC switches aswell as multicast, broadcast data traffic. In some link failurescenarios, it also carries unicast traffic. You should have at least two10 Gigabit Ethernet interfaces for peer links. Refer to “vPC peer link”on page 303 for more information.

vPC peer-keepalivelink

The peer-keepalive link monitors the vitality of a vPC peer CiscoNexus 5000 Series device. The peer-keepalive link sends configurable,periodic keepalive messages between vPC peer devices. The vPCpeer-keepalive link can be a management interface or switchedvirtual interface (SVI)

No data or synchronization traffic moves over the vPCpeer-keepalive link; the only traffic on this link is a message thatindicates that the originating switch is operating and running vPCs.Refer to “peer-keepalive link and messages” on page 305 for moreinformation.

vPC member port Interfaces that belong to the vPCs.

Host vPC port vPC between a host running NIC teaming or bonding to a set ofupstream switches.

Switch vPC vPC running between two sets of switches. This is accomplished byconfiguring vPCs within a domain to connect to a set of downstreamswitches.

vPC domain A vPC domain is used to identify the vPC switches that willparticipate together within a vPC configuration. Any two switchesneeding to share vPC configurations will need to share the same vPCDomain ID, a value from 1 to 1000. A Nexus 5000 can be part of onlyone vPC domain at a time. Cisco switches capable of running VirtualDevice Context (VDC) can have a different vPC Domain ID percontext. To configure the vPC Domain ID, assign it through GlobalConfiguration Mode.

Note: The vPC Domain ID is completely separate and has nothing to do withthe FC Domain ID.

After the ID is assigned, you will be placed into the vPC domainconfiguration mode where you will finish any configurations neededwithin the domain. Switches that are members of the same vPCdomain are referred to as vPC peer devices.



Some of the configuration options within the vPC domainconfiguration are:

◆ Peer-keepalive—This is a mandatory configuration parameterthat is needed to identify the vPC destination peer switch by IPaddress. The peer switches use this link to communicate bysending periodic keepalive messages. You must have Layer 3connectivity between the peer switches to transmit thesemessages. The system cannot bring up the vPC peer link unless apeer-keepalive link is already up and running.

Note: It is recommend that you configure the vPC peer-keepalive link onthe Cisco Nexus 5000 Series switches to run in the management VRFusing the mgmt 0 interfaces.

◆ Role priority—Optional configuration to explicitly set vPCpriority (1 to 65636; default value is 32677). The peer device withthe lower priority becomes the vPC primary switch.

◆ System priority—Optional configuration to explicitly set theLACP priority within a vPC domain (1 to 65636; default value is32677).

Note: It is recommended that the vPC Peer switches be set as the primarydevices. This priority will need to be the same on both vPC switches orthe vPC will not come up.

◆ System mac—Optional configuration. The switch willautomatically assign a virtual mac address to the vPC Domain.

The Cisco Nexus 5000 Series Switches Virtual PortChannel QuickConfiguration Guide, located at www.cisco.com, explains vPC peerlink in more detail.

vPC peer link vPC peer links are the physical links connecting two vPC switches.These are used to carry the synchronization and control data betweenpeer switches. In the event of some link failure scenarios, they canalso carry normal user traffic. These links do not carry thepeer-keepalive messages.

Note: For redundancy, it is recommended that at least two 10 Gb links beused to form an EtherChannel between two vPC switches. If one of theinterfaces in the vPC peer link fails, the switch will automatically failover tothe other interface.


304


You can have only two switches as vPC peers. Each switch can serveas a vPC peer to only one other vPC peer. The vPC peer switches canalso have non-vPC links to other switches.

To make a valid configuration, you configure an EtherChannel oneach switch and then configure the vPC domain. You assign theEtherChannel on each switch as a peer link.

Since the vPC configuration involves two separate switches, there arecertain configurations that must match on both switches or the vPCwill not come up.

Table 8 provides a list of the parameters that must match using thesho vpc consistency-parameters global command.

Table 8 Parameters (page 1 of 2)

Name Type Local value Peer value

QoS 2 ([], [3], [], [], [], []) ([], [3], [], [], [], [])

Network QoS (MTU) 2 (1538, 2240, 0, 0, 0, 0) (1538, 2240, 0, 0, 0, 0)

Network Qos (Pause) 2 (F, T, F, F, F, F) (F, T, F, F, F, F)

Input Queuing (Bandwidth) 2 (50, 50, 0, 0, 0, 0) (50, 50, 0, 0, 0, 0)

Input Queuing (Absolute 2 (F, F, F, F, F, F) (F, F, F, F, F, F)

Priority)

Output Queuing (Bandwidth) 2 (50, 50, 0, 0, 0, 0) (50, 50, 0, 0, 0, 0)

Output Queuing (AbsolutePriority)

2 (F, F, F, F, F, F) (F, F, F, F, F, F)

STP Mode 1 Rapid-PVST Rapid-PVST

STP Disabled 1 None None

STP MST Region Name 1 "" ""

STP MST Region Revision 1 0 0

STP MST Region Instance to 1

VLAN Mapping

STP Loopguard 1 Disabled Disabled

STP Bridge Assurance 1 Enabled Enabled



The Cisco Nexus 5000 Series Switches Virtual PortChannel QuickConfiguration Guide, located at www.cisco.com, explains vPC peerlink in more detail.

peer-keepalive linkand messages

The peer-keepalive link between vPC peer switches is actually aLayer 3 connection used to transmit periodic keepalive messages.This peer-keepalive link does not use the vPC peer link connectionsused to carry the vPC traffic between vPC peers.

Note: On Nexus 5000 switches, it is recommended that you configure themgmt 0 interface to carry the keepalive Hello packets.

You must have Layer 3 connectivity between the peer switches totransmit these messages. The system cannot bring up the vPC peerlink unless a peer-keepalive link is already up and running.

A keepalive message is nothing more than a frame being transmittedby peer switches indicating that they are alive and running vPC. Nodata or synchronization information is sent over the keepalive link.

STP Port Type, Edge 1 Normal, Disabled, Normal, Disabled,

BPDUFilter, Edge BPDUGuard Disabled Disabled

STP MST Simulate PVST 1 Enabled Enabled

Allowed VLANs - 1,100, 300, 402, 700-701,800, 900-910

1, 400-402, 700-701, 800,900-910

Local suspended VLANs - 100, 200, 300, 400-401, 500,1000

-

Local suspended VLANs - 100, 200, 300, 400-401 -

Table 8 Parameters (page 2 of 2)

Name Type Local value Peer value


306


The keepalive messages are configurable through the vPC Domainconfiguration. Table 9 lists the default values used for the keepalivemessages.

The Cisco Nexus 5000 Series Switches Virtual PortChannel QuickConfiguration Guide, located at www.cisco.com, explainspeer-keepalive link and messages in more detail.

vPC topologies overviewThis section provides basic examples of the following vPC topologies:

◆ “Switch vPC/Straight-through topology” on page 306

◆ “Switch vPC/cross-connect topology” on page 307

◆ “Host vPC/direct-connect topology” on page 309

For more details on deploying the different topologies mentioned inthis section, refer to “vPC topology considerations” on page 309.

For case studies of each of the topologies, refer to “vPC case studies”on page 324.

Switch vPC/Straight-through topologyIn a switch vPC/straight through topology, the vPCs are created on thevPC Peer switches (SW 1 and SW 2), but the downstream switches donot physically connect to both of the vPC switches.

Table 9 Default values for keepalive messages

keepalive message Default value

--Keepalive interval 1000 msec

--Keepalive timeout 5 seconds

--Keepalive hold timeout 3 seconds

--Keepalive vrf management

--Keepalive udp port 3200

--Keepalive tos 192



Figure 42 shows the vPCs connected directly to a set of downstreamswitches.

Figure 42 Switch vPC/straight-through topology example

An example of when to deploy this topology is when you areinterested in added bandwidth and active failover to a single switch,in this case SW 3 or SW 4. Failover capabilities between an edgeswitch and its upstream switch is limited to the number of physicallinks within the bundle between the two.

For considerations when deploying vPC/straight-throughtopologies, refer to:

◆ “Switch vPC/straight-through considerations” on page 310

For a vPC straight-through topology case study, refer to:

◆ “Nexus 5000/2232/vPC (TOR/FEX) — Straight-throughtopology” on page 325

Switch vPC/cross-connect topologyIn a switch vPC/cross-connect topology, the vPCs are created on thevPC devices (SW 1 and SW 2) and are physically connected to both ofthe downstream devices (SW 3 and SW 4). In this example, thedownstream switches have PortChannels configured and arephysically connected to both of the upstream vPC switches, as shownin Figure 43 on page 308.


308


Figure 43 Switch vPC/cross-connect topology example

An example of when to deploy this topology is when a networksolution is needed to provide redundancy from an edge switch toboth upstream switches. This topology will allow for networkfailover from an edge switch to either of the vPC switches.

For considerations when deploying vPC/cross-connect topologies,refer to:

◆ “Switch vPC/cross-connect considerations” on page 317

For a vPC cross-connect topology case study, refer to:

◆ “Nexus 5000/4000/vPC (TOR/FCF or FEX) — Cross-connecttopology” on page 357



Host vPC/direct-connect topologyA Host vPC topology represents a vPC configuration that connectsdirectly to a server or a set of servers. In Figure 44, the Host vPC is asingle server connected physically to both SW 3 and SW 4.

Figure 44 Host vPC direct-connect topology example

An example of when to deploy this topology is when you areconnecting a host directly to a set of vPC devices. In this example, thehost is able to take advantage of the vPC by running NICTeaming/Bonding to the upstream Nexus 5000 switches.

For considerations when deploying vPC/direct-connect topologies,refer to:


For a vPC direct-connect topology case study, refer to:

◆ “Host vPC— Direct-connect topology” on page 369

vPC topology considerationsThe following considerations are focused on elements that areapplicable across all of the different vPC topologies:

◆ Generation 2 CNAs that support FIP are required for FCoE in avPC environment.


310


◆ NX-OS must be version 4.1(3)N1(1) or later.

◆ iSCSI is not supported with NIC Teaming for Windows.

◆ iSCSI is supported with NIC Bonding for Linux.

◆ NAS, like most other Ethernet Protocols, will function over any ofthe topologies mentioned in this section.

This section further discusses topology, failover, bandwidth, andprotocol considerations needed when deploying the followingtopologies:

◆ “Switch vPC/straight-through considerations” on page 310

◆ “Switch vPC/cross-connect considerations” on page 317


These considerations are based on the three storage protocols capableof running over Ethernet: FCoE, iSCSI, and NAS.

Switch vPC/straight-through considerationsConsiderations are discussed for the following two switchvPC/straight-through configurations:

◆ “Nexus 5000 to Nexus 2000 configuration” on page 310


Nexus 5000 to Nexus2000 configuration

The Nexus 2232s fabric extenders provide data plane functionalityonly and must utilize the control plane of the Nexus 5000 switchesthey are attached to for configuration. As a result, the interfaces onthe Nexus 2232 will appear as interfaces on the same switch throughthe configuration interface of the Nexus 5000 switch.



Figure 45 shows an example of the topology connecting the Nexus5000 switches to the Nexus 2000 switches.

Figure 45 Switch vPC/straight-through Nexus 5000 to Nexus 2000 configuration

An example of when to deploy this topology is when you areinterested in aggregating bandwidth and active failover to a singleupstream switch from an edge switch, in this case SW 3 to SW 1 orSW 4 to SW 2.

Failover considerations◆ Connectivity is from a single edge switch to a single upstream

switch only.

◆ There is no active failover to the second vPC switch in thisconfiguration.


312


◆ Failover between an edge switch and its upstream switch islimited to the number of physical links within the bundle. Forexample, in Figure 45 on page 311, there are four active linksbetween SW 3 and SW 1, which represent all failover capabilitiesbetween the two switches.

◆ This topology will provide a redundant configuration to the host.

Bandwidth considerations◆ Bandwidth is limited to the aggregate amount between the edge

switch and its upstream switch. For example, in Figure 45 thatwould be the four links (10g x 4 = 40g) between SW 3 and SW 1.

◆ Failover within a bundle will decrease the amount of availablebandwidth. For example, if two of the four physical links betweenSW 3 and SW 1 were to fail, the amount of available bandwidthwould decrease to the remaining two links to 20g.

◆ Up to eight physical links can be used to connect between aNexus 5000 and Nexus 2000, for a total of 80g.

◆ All three storage protocols are capable of taking advantage ofaggregated bandwidth.

Protocol considerations◆ FCoE

• FCoE is supported.

• Dual connections from a CNA to a single switch are supportedif two separate vPCs are used for each CNA port.

– See Figure 46 on page 313 for an example.



Figure 46 Supported FCoE topology example

• Dual connections from a single host to a single switch is notsupported on the same vPC.

– See Figure 47 on page 314 for an example.


314


Figure 47 Non-supported topology for FCoE example

• A host's CNA port ENode MAC address must be bound to acorresponding vfc interface on the upstream Nexus 5000.

• NIC Teaming for Windows

– active/active is supported– active/passive is supported

• Multipathing for block storage is enabled through the use of ahigher-level application, such as PowerPath or MPIO.

◆ iSCSI

• iSCSI is supported.

• NIC Teaming for Windows with iSCSI is not supported.

• NIC Bonding for Linux with iSCSI is supported.

• Multipathing is enabled through the use of a higher-levelapplication, such as PowerPath or MPIO.



◆ NAS

• NAS is supported.

• NIC Teaming for Windows is supported.

• NIC Bonding for Linux is supported.


The Nexus 4001I is an FIP Snooping Bridge (FSB) that resides withinan IBM blade server. The functionality, performance, and behavior isidentical to that of an external FSB.

Figure 48 shows an example of the topology connecting the Nexus5000s to the Nexus 4000s.

Figure 48 Switch vPC/straight-through Nexus 5000 to Nexus 4000 configuration

This topology is only applicable if an IBM Blade server is being used.In this example the Blade Server has two separate Nexus 4001Isinstalled to act as two separate edge switches. Like the Nexus 5000 toNexus 2000 topology, this will aggregate bandwidth between a singleNexus 4000 and its upstream Nexus 5000 switch.


316


Note: vPCs are not configurable between the Nexus 4000s and the internalCNAs on the Blade Servers.

Failover considerations◆ Connectivity is from a single edge switch (Nexus 4000) to a single

upstream switch only.

◆ There is no active failover to the second vPC switch in thisconfiguration.

◆ Failover between a Nexus 4000 and its upstream switch is limitedto the number of physical links within the bundle. For example,in Figure 48 on page 315, there are four active links between SW 3and SW 1, which represent all failover capabilities between thetwo switches.

◆ This topology provides a redundant configuration to the host.

Bandwidth considerations◆ Bandwidth is limited to the aggregate amount between the edge

switch and its upstream switch. For example, in Figure 48 thatwould be the four links (10g x 4 = 40g) between SW 3 and SW 1.

◆ Failover within a bundle will decrease the amount of availablebandwidth. For example, if two of the four physical links betweenSW3 and SW1 were to fail, the amount of available bandwidthwould decrease to the remaining two links to 20g.

◆ The Nexus 4001I has six external ports for 10 GbE connectivity.

◆ The Nexus 4001I has 14 internal 10 GbE ports for connectivity tothe Blade servers.

◆ Each Blade server has two internal connections to the Nexus4001Is.

• One connection to each of the Nexus 4000s.

◆ Up to six physical links can be used to connect between a Nexus5000 and Nexus 4000, for a total of 60g.

◆ All three storage protocols are capable of taking advantage ofaggregated bandwidth.


• FCoE is supported.



• Dual connections from a CNA to a single switch is notpossible within the Blade Server.

• Connections to each CNA within a Blade Server can be to thesame FCoE VLAN/VSAN.

• Connections to each CNA within a Blade Server can be fromseparate FCoE VLAN/VSANs.

• Connections from a Nexus 4000 to the internal CNA are notcapable of vPC configurations. vPC is not configurablebetween the Nexus 4000 and an internal CNA.

• A host's CNA Encode MAC address must be bound to acorresponding vfc interface on the upstream Nexus 5000.


– active/active is not supported.– active/passive is supported.

• An upper-layer application, such as EMC PowerPath orMPIO, is needed for active failover between CNAs for blockI/O.

◆ iSCSI


• NIC Teaming for Windows is not supported.


• An upper-layer application, such as EMC PowerPath orMPIO, is needed for active failover between CNAs for blockI/O.

◆ NAS




Switch vPC/cross-connect considerationsConsiderations are discussed for the following two switchvPC/cross-connect configurations:




318



The Nexus 2232 Fabric Extenders provide data plane functionalityonly and must utilize the control plane of the Nexus 5000 switchesthey are attached to for configuration. As a result, the interfaces onthe Nexus 2232 will appear as interfaces on the same switch throughthe configuration interface of the Nexus 5000.

Figure 49 shows an example of the topology connecting Nexus 5000switches to Nexus 2000 switches.

Figure 49 Switch vPC/cross-connect Nexus 5000 to Nexus 2000 configuration

An example of when to deploy this topology is when you areinterested in aggregating bandwidth and active failover to bothupstream switches from each edge switch, in this case SW 3 to SW 1and SW 2 or SW 4 to SW 1 and SW 2.

Failover considerations◆ Connectivity from a Nexus 2000 is to each of the Nexus 5000

switches.



◆ There is active failover from a Nexus 2000 to either of theupstream Nexus 5000 switches.

◆ This topology provides redundant failover paths from the host tothe upstream Nexus 5000 switch.

◆ Failover can happen within a bundle between a Nexus 2000 andNexus 5000 switch.

◆ From a networking standpoint, this is considered to be a "HighlyAvailable" configuration.

Bandwidth considerations◆ Bandwidth is aggregated from a 2000 to each of the upstream

Nexus 5000 switches. For example, in Figure 49 on page 318, SW 3has two physical connections to SW 1 for a total of 20g, and twophysical connections to SW 2 for a total of 20g.

◆ In this example, the overall amount of bandwidth available to theupstream Nexus 5000 switches from a Nexus 2000 switch is 40g.

◆ Failover within a bundle will decrease the amount of availablebandwidth for that bundle. For example, if one of the physicallinks between SW 3 and SW 1 were to fail, the amount of availablebandwidth would decrease to the remaining link to 10g. Thiswould not decrease the amount of bandwidth to SW 2.

◆ Up to eight physical links can be used to connect between aNexus 5000 switch and Nexus 2000 switch, for a total of 80g. Thiscan be divided between each upstream Nexus 5000 switch.

◆ iSCSI and NAS are capable of taking advantage of aggregatedbandwidth in this topology.


• FCoE is not supported.

◆ iSCSI





◆ NAS



320





The Nexus 4001I is a FIP Snooping Bridge (FSB) that resides within anIBM Blade server. This model will only function within a Bladeserver. The functionality, performance, and behavior is identical tothat of an external FSB.

Figure 50 shows an example of this topology connecting the Nexus5000 switches to the Nexus 4000 switches.

Figure 50 Switch vPC/cross-connect Nexus 5000 to Nexus 4000 configuration

This topology is only applicable if an IBM Blade server is being used.In this example the Blade Server has two separate Nexus 4001Iswitches installed to act as two separate edge switches. Similar to theNexus 5000 to Nexus 2000 topology, this will aggregate bandwidthbetween the Nexus 4000 switches and both upstream Nexus 5000switches.



Failover considerations◆ Connectivity from a Nexus 4000 is to each of the Nexus 5000

switches.

◆ There is active failover from a Nexus 4000I to either of theupstream Nexus 5000 switches.

◆ This topology provides redundant failover paths from the Bladeserver to the upstream Nexus 5000 switches.

◆ Failover can happen within a bundle between a Nexus 4000 andNexus 5000 switch.

◆ From a networking standpoint, this is considered to be a "HighlyAvailable" configuration.

◆ Nexus 4000 switches are internal to the Blade server, therefore ifthe Blade server were to lose power, both internal Nexus 4000switches would also power down.

Bandwidth considerations◆ Bandwidth is aggregated from a Nexus 4k to each of the

upstream Nexus 5000 switches. For example; in Figure 50 onpage 320, SW 3 has two physical connections to SW 1 for a total of20g, and two physical connections to SW 2 for a total of 20g.

◆ In this example, the overall amount of bandwidth available to theupstream Nexus 5000 switches from a Nexus 4000 switch is 40g.

◆ Failover within a bundle will decrease the amount of availablebandwidth for that bundle. For example, if one of the physicallinks between SW 3 and SW 1 were to fail, the amount of availablebandwidth would decrease to the remaining link, to 10g. Thiswould not decrease the amount of bandwidth to SW 2.

◆ Up to six physical links can be used to connect between a Nexus5000 switch and Nexus 4000 switch, for a total of 60g. This can bedivided between each upstream Nexus 5000 switch.

◆ iSCSI and NAS are capable of taking advantage of aggregatedbandwidth in this topology.

◆ FCoE is supported but additional links are needed.

• The bandwidth for FCoE will be limited to the single 10 G linkbetween the Nexus 4000 and the upstream Nexus 5000. Sincethe additional links are for FCoE only, none of the otherprotocols will be able to take advantage of these links.

• Refer to the “Protocol considerations” section, next, for moreinformation.


322



• FCoE is supported, but additional links are needed betweenthe Nexus 4000 switches and the Nexus 5000 switches.Figure 51 shows an example of this topology with separateFCoE links to allow for FCoE traffic between the Nexus 4000switches and the Nexus 5000 switches.

Figure 51 Supported FCoE topology

• Additional configuration steps will also be needed, which canbe found in the“Nexus 5000/4000/vPC (TOR/FCF or FEX) —Cross-connect topology” on page 357 case study.

• A host's CNA Enode MAC address must be bound to acorresponding vfc interface on the upstream Nexus 5000switch.




– active/active is not supported.– active/passive is supported.


• Spanning-tree needs to be considered more closely in thistopology to make sure that Layer 2 traffic flows correctly.

• The FCoE links need to be configured as the primary pathback to root for VLAN 1 to make sure that the FIP frames onVLAN 1 make it to the correct VFC interfaces on the Nexus5000 switches.

◆ iSCSI





◆ NAS




Host vPC/direct-connect topologyConsiderations are discussed for the following hostvPC/direct-connect configuration:

◆ “Nexus 5000 to the host configuration” on page 323

Nexus 5000 to the hostconfiguration

In a direct-connect topology, such as that shown in Figure 44 onpage 309, the host is connected directly to the Fibre ChannelForwarders (FCFs) for FCoE traffic. This example uses the Nexus5000 switches.

Failover considerations◆ For failover capabilities, multipathing for FCoE and iSCSI must

rely on a higher-level application, such as PowerPath or MPIO.

◆ Since this topology is focusing on the direct connections betweenthe host and the switches, the failover is limited to the physicalconnection between them.


324


Bandwidth considerations◆ A host CNA port can only be connected to a single 10 GbE switch

port for FCoE.◆ Since this topology focuses just on the connections from the host

to the switches, the bandwidth is limited to the 10 GbEconnections used for FCoE.

◆ Dual connections between a single switch and a host are notsupported for FCoE.

◆ The behavior of the direct-connect topology is similar to thestraight-through and cross-connect topologies. The bandwidthrules that apply to the host are still applicable.

Protocol considerations◆ FCoE is supported in a NIC Teaming/Bonding scenario.◆ Generation 2 CNAs that support FIP are required for FCoE in a

vPC environment.◆ The PortChannel used to connect to the host must be bound to the

vfc interface created on the Nexus 5000s. An example of this canbe found in “Host vPC/direct-connect topology” on page 309.

◆ iSCSI is not supported with NIC Teaming for Windows.◆ iSCSI is supported with NIC Bonding for Linux.◆ NAS and all other Ethernet traffic will operate and function in the

same traditional manner.◆ The host must be connected to the FCF (Nexus 5000s in this

example) for FCoE to function.

vPC case studiesThis section offers various topologies built and documented withE-Lab. The following topologies are described:

◆ “Nexus 5000/2232/vPC (TOR/FEX) — Straight-throughtopology” on page 325

◆ “Nexus 5000/4000/vPC (TOR/FCF or FEX) — Cross-connecttopology” on page 357

◆ “Host vPC— Direct-connect topology” on page 369Following these case studies, steps are provided for:

◆ “Configuring the iSCSI initiator and target” on page 388◆ “Configuring the FCoE initiator and target” on page 405



Nexus 5000/2232/vPC (TOR/FEX) — Straight-through topologyThis section details the setup steps required to create theconfiguration shown in the target topology example illustrated inFigure 54 on page 329. This section includes the followinginformation:

◆ “Configuration overview” on page 325



◆ “Target topology example” on page 328


Configurationoverview

Supported protocols, a high-level diagram of straight-throughtopology, and key points to consider are provided in this section.

Supported protocolsTable 10 describes the protocols that have been set up by E-Lab andare supported in this topology.

Topology overviewA high-level diagram of the straight-through topology that will beconfigured in this section is shown in Figure 52 on page 326. The termstraight-through is derived from the fact that each Nexus 2232PP isonly connected to a single Nexus 5000 parent switch.

Table 10 Supported protocols

Protocol Topology Teaming / Bonding PowerPath / MPIO

FCoE Y Y Y

iSCSI Y N Y

NAS Y Y N/A


326


Figure 52 Straight-through topology example

Key pointsConsider the following:

◆ For any CNA that will be connected to the 2232PP, the vfc that isassociated with each CNA port will need to be bound to theENode MAC Address of the CNA port and not the physicalinterface on the switch where the CNA port will attach.

◆ EMC recommends the use of PortChannels between the Nexus5000 and Nexus 2232PP instead of using static pinning.

◆ Adding the PortChannel between the Nexus 5000 and the Nexus2232 to a VPC will cause FCoE to stop functioning properly. E-Labhas observed that one of the logins from each host will be loggedout and not be allowed to reinitialize.

◆ Bouncing the vPC Peer link is highly discouraged and is verydisruptive to FCoE.

◆ Each Nexus 2232PP may only be connected to a single parentNexus 5000. FCoE is not supported, and will not functionproperly, in Active /Active or Cross-connected topologies wherethe Nexus 2232PP is connected to more than one parent Nexus5000.

◆ Generation 2 CNAs that support FIP are required for aFCoE-over-vPC environment.


vPC x vPC y

vPCDevice

vPCDevice


SW 1 SW 2

SW 3 SW 4

ICO-IMG-000880

vPC Domain

Peer - keepalivelink

vPC Peer link



Summary ofconfiguration steps

This section provides a summary and examples of the steps to createthe configuration shown in the target topology example illustrated inFigure 54 on page 329.

1. Enable the vPC and LACP features.

2. Set the vPC Domain ID.

3. Create the peer keep alive link.

4. Set the switch priority.

5. Create a PortChannel for the vPC peer link and configure it as thevPC peer link.

6. Add vPC peer link interfaces to the PortChannel created in Step 5.

7. Attach cables for the vPC peer link.

8. Create the PortChannels for the host-facing interfaces and addthem to the appropriate vPC.

9. Add the host-facing interfaces to the PortChannel created inStep 8.

10. Ensure the vfc interfaces for any of the CNAs attached to the2232PP are bound to the ENode MAC Address and not thephysical interface.

11. Ensure each vfc to the appropriate VSAN as shown in the targettopology diagram, Figure 54 on page 329.


328


Existing topologyexample

It is assumed that the customer has an existing LAN and mirroredSAN and is enabling vPC on both.


Target topologyexample

The Ethernet uplinks, aggregation switches, FC uplinks, FC SANs,and storage ports have been removed from the target topology in thisexample for the sake of clarity. The target topology in this use case,shown in Figure 54 on page 329, is assumed to include all of theelements in the existing topology with slight modifications made toinclude the 2232 into the configuration. The steps in this section willonly describe the steps needed to add vPC to this topology.

Host_1 (Windows)

CNA_1 CNA_2

DCX_1Domain 1

172.23.185.24

DCX_2Domain 2

172.23.185.25


172.23.185.22


172.23.185.23


Nexus-5020-2IP = 172.23.185.113Domain = 3

1/21/31/41/5

1/21/31/41/5

1/21/31/41/5

1/21/31/41/5

1/11/1

1/11/1

Host_2(Linux)

CNA_1 CNA_2



1 2 3 4

Storage port 1:50:06:04:82:D5:2E:69:79

Storage port 2:50:06:04:82:D5:2E:69:49



Nexus-5020-1

Nexus-5020-2



VLAN/VSAN 100

VLAN/VSAN 200

VLAN/VSAN 300

VLAN/VSAN 400

10GbE

4Gb/s FC

Core-N7K-1

Port-channel 1

Port-channel 2

1/1 2/1 2/11/1

Core-N7K-2

SAN PortChannels


1/1

1/2

1/3

1/4

1/5

1/6

1/1

1/2

1/3

1/4

1/5

1/6

Port-Channel 3

Port-Channel 4

Host 1

Host 2

PO3PO4






CNA_1 CNA_2

Nexus-5020-1IP = 172.23.185.112Domain = N/A (NPVmode)

Nexus-5020-2IP = 172.23.185.113Domain = 3

CNA_1 CNA_2





VLAN/VSAN 100

VLAN/VSAN 200

VLAN/VSAN 300

VLAN/VSAN 400

1/1

1/2

1/3

1/4

1/5

1/6

1/1

1/2

1/3

1/4

1/5

1/6

Port-Channel 3

Port-Channel 4

Host 1 Host 2

PO3PO4

Nexus-2232-2FEX instance 102

Nexus-2232-1FEX instance 101

ICO-IMG-000885


mgmt 0 mgmt 0

PO5

vPC keepalivelink

ManagementLAN

vPC 100 vPC 100PO12 PO12

PO11

PO11

Host_2 (Linux)Host_1 (Windows)

10GbE4Gb/s FC

GbE

vPC peer link

vPC Domain 58


330


Configuring the Nexus5000

To configure the Nexus 5000, complete the following steps:


Before configuring vPC, the vPC feature must be enabled on theNexus 5000. The LACP feature must be enabled since LACP willbe enabled on the PortChannels.

Note: LACP should already be enabled on the switch if the steps in theNexus 5000 /Nexus 2232 straight through topology example werefollowed.

Nexus-5020-1# config terminalNexus-5020-1(config)# feature vpcNexus-5020-1(config)# feature lacp


Nexus-5020-1(config)# vpc domain 58Nexus-5020-1(config-vpc-domain)#


Nexus-5020-1(config-vpc-domain)# peer-keepalivedestination 172.23.185.113


Nexus-5020-1(config-vpc-domain)# system-priority 4000Nexus-5020-1(config-vpc-domain)# exitNexus-5020-1(config)#

5. Create a PortChannel and configure it as the vPC peer link.

Nexus-5020-1(config)# interface port-channel 5Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# vpc peer-link

6. Add vPC peer link interfaces to the PortChannel.

Nexus-5020-1(config)# interface ethernet 1/5-6Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# channel-group 5 mode active

7. Repeat Step 1 through Step 6 on the Nexus-5020-2.


9. Verify the vPC state.

Nexus-5020-1# show vpc briefLegend:

(*) - local vPC is down, forwarding via vPC peer-link



vPC Domain ID : 58Peer status : peer adjacency formed okvPC keep-alive status : peer is aliveConfiguration consistency status: successType-2 consistency status : successvPC role : secondaryNumber of vPCs configured : 0Peer Gateway : DisabledDual-active excluded VLANs : -

vPC Peer-link status---------------------------------------------------------------------id Port Status Active vlans-- ---- ------ --------------------------------------------------1 Po5 up -Nexus-5020-1#



vPC Domain ID : 58Peer status : peer adjacency formed okvPC keep-alive status : peer is aliveConfiguration consistency status: successType-2 consistency status : successvPC role : primaryNumber of vPCs configured : 2Peer Gateway : DisabledDual-active excluded VLANs : -

vPC Peer-link status---------------------------------------------------------------------id Port Status Active vlans-- ---- ------ --------------------------------------------------1 Po5 up 1,700

vPC status----------------------------------------------------------------------------id Port Status Consistency Reason Active vlans------ ----------- ------ ----------- -------------------------- -----------100 Po11 up success success 1,500,700200 Po12 up success success 1,500,700

Nexus-5020-2#

10. On the Nexus-5020-1, create the PortChannels for the host-facinginterfaces and add them to the appropriate vPC.

Nexus-5020-1(config-if)# interface port-channel 11Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# switchport trunk native vlan 700


332


Nexus-5020-1(config-if)# switchport trunk allowed vlan 1, 500, 700Nexus-5020-1(config-if)# vpc 100Nexus-5020-1(config-if)# interface port-channel 12Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# switchport trunk native vlan 700Nexus-5020-1(config-if)# switchport trunk allowed vlan 1, 500, 700Nexus-5020-1(config-if)# vpc 200

11. On the Nexus-5020-1, add the host-facing interfaces to thePortChannel.

Nexus-5020-1(config-if)# interface ethernet 101/1/1Nexus-5020-1(config-if)# channel-group 11 mode activeNexus-5020-1(config-if)# interface ethernet 101/1/2Nexus-5020-1(config-if)# channel-group 12 mode active

12. On the Nexus-5020-2, create the PortChannels for the host-facinginterfaces and add them to the appropriate vPC.

Nexus-5020-2(config-if)# interface port-channel 11Nexus-5020-2(config-if)# switchport mode trunkNexus-5020-2(config-if)# switchport trunk native vlan 700Nexus-5020-2(config-if)# switchport trunk allowed vlan 1, 500, 700Nexus-5020-2(config-if)# vpc 100Nexus-5020-2(config-if)# interface port-channel 12Nexus-5020-2(config-if)# switchport mode trunkNexus-5020-2(config-if)# switchport trunk native vlan 700Nexus-5020-2(config-if)# switchport trunk allowed vlan 1, 500, 700Nexus-5020-2(config-if)# vpc 200

13. On the Nexus-5020-2, add the host-facing interfaces to thePortChannel.

Nexus-5020-2(config-if)# interface ethernet 102/1/1Nexus-5020-2(config-if)# channel-group 11 mode activeNexus-5020-2(config-if)# interface ethernet 102/1/2Nexus-5020-2(config-if)# channel-group 12 mode active

14. View the vPC configuration.



vPC Domain ID : 58Peer status : peer adjacency formed okvPC keep-alive status : peer is aliveConfiguration consistency status: successType-2 consistency status : successvPC role : secondaryNumber of vPCs configured : 2Peer Gateway : Disabled



Dual-active excluded VLANs : -



Nexus-5020-1#






Nexus-5020-2#15. Ensure the vfc interfaces for any of the CNAs attached to the

Nexus 2232PP are bound to the physical interface on the 2232PP.This configuration step should have been performed when the5000/2232PP straight-through topology was originallyconfigured. It is included below for convenience.


334


• On Nexus-5020-1:

Nexus-5020-1(config-if)# interface vfc 1011Nexus-5020-1(config-if)# bind interface ethernet 101/1/1Nexus-5020-1(config-if)# no shutNexus-5020-1(config-if)# interface vfc 1012Nexus-5020-1(config-if)# bind interface ethernet 101/1/2Nexus-5020-1(config-if)# no shut

• On Nexus-5020-2:

Nexus-5020-2(config-if)# interface vfc 1021Nexus-5020-2(config-if)# bind interface ethernet 102/1/1Nexus-5020-2(config-if)# no shutNexus-5020-2(config-if)# interface vfc 1022Nexus-5020-2(config-if)# bind interface ethernet 102/1/2Nexus-5020-2(config-if)# no shut

16. Ensure each VFC to the appropriate VSAN as shown in the targettopology diagram.

• On Nexus-5020-1:


• On Nexus-5020-2:


UCS C240M3 Network Adapter Virtualization with Cisco Nexus 5500/2232/vPC (TOR/FEX)Straight-through topology

This section contains information on the following:

◆ “Ethernet connectivity with vPC and FEX Straight-Through” onpage 334

◆ “SAN connectivity with vPC and FEX Straight-Through” onpage 346

Ethernet connectivitywith vPC and FEXStraight-Through

This section details the setup required to create the vNICconfigurations. This section includes the following information:


◆ “Target topology ” on page 336



◆ “Configuring the Nexus 5596T-1” on page 337


◆ “Configuring the UCS C-240 M3 server” on page 340

◆ “Configuration verification on Nexus 5596T-1” on page 344

◆ “Configuration verification on Nexus 5596T-2” on page 345

◆ “Best practices” on page 346

Summary of configuration stepsComplete steps are discussed in “Configuring the Nexus 5596T-1” onpage 337 and “Configuring the Nexus 5596T-2” on page 339. Thefollowing is a summary of these steps.

1. Enable feature-set virtualization.

2. Enable feature FEX.

3. Configure FEX associate fabric.

4. Configure vPC domain.

5. Configure vPC peer-link.

6. Configure the port to be VN-Tag mode.

7. Enable vEthernet auto-create.

8. Configure the port-profile.

Complete steps are discussed in “Configuring the UCS C-240 M3server” on page 340. The following is a summary of these steps.

1. Enable VN-Tag mode.

2. Create vNIC.

3. Power cycle the server to have the configuration take effect.

Complete steps are discussed in “Configuration verification onNexus 5596T-1” on page 344 and “Configuration verification onNexus 5596T-2” on page 345. The following is a summary of thesesteps.

1. Check dynamically created vEthernet interfaces on Nexus.

2. Check vNIC staus on Nexus 5596T switches.


336


Target topologyFigure 55 shows the target topology used in this section.

Figure 55 Ethernet connectivity with vPC and FEX Straight-Through topologyexample

The following table lists the target topology IP address and DomainID assignment.


Nexus 5596T-1 10.246.54.10 12

Nexus 5596T-2 10.246.54.11 12

Nexus2 232-1

Nexus 2232-2



Configuring the Nexus 5596T-11. Enable feature-set virtualization.

Before you can configure the VN-Tag mode and vEthernetinterface, you must enable feature-set virtualization.

IOP054010# conf tIOP054010(config)# feature-set virtualization


IOP054010(config)#Feature fex

3. Configure 2232-1 as FEX 100 and associate with Ethernet ports.

IOP054010(config)# fex 100IOP054010(config-fex)# pinning max-links 1IOP054010(config-fex)# description "FEX0100"

a. Connect the Nexus 2232-1 to Nexus 5596T port Ethernet 1/46and 1/48 and bound to port-channel 1.

IOP054010(config)# interface Ethernet1/46IOP054010(config-if)# channel-group 1


b. Configure port-channel 1 to be FEX associated.

IOP054010(config)# int port-channel 1IOP054010(config-if)# switchport mode fex-fabricIOP054010(config-if)# fex associate 100

c. Configure port-channel members to be FEX-fabric mode.

IOP054010(config)# int Ethernet1/46IOP054010(config-if)# switchport mode fex-fabricIOP054010(config-if)# fex associate 100


4. Configure the vPC domain.

a. Globally configure VPC domain and peer-keepaliveconnectivity. Normally the vPC peer-keepalive destinationsare reachable via management network.

IOP054010(config)# vpc domain 12IOP054010(config-vpc-domain)# peer-keepalive

destination 10.246.54.11


338


IOP054010(config-vpc-domain)# peer-gatewayIOP054010(config-vpc-domain)# auto-recovery

b. Connect Ethernet port 1/7 and 1/8 between Nexus 5596T-1and Nexus5596T-2 and bound them into port-channel 7.

IOP054010(config)# interface Ethernet1/7IOP054010(config-if)# switchport mode trunkIOP054010(config-if)# channel-group 7


5. Configure vPC peer-link on the port-channel.

IOP054010(config)# interface port-channel7IOP054010(config-if)# switchport mode trunkIOP054010(config-if)# spanning-tree port type

networkIOP054010(config-if)# vpc peer-link

6. Configure VN-Tag mode for interfaces connecting host.

IOP054010(config)# int ethernet 100/1/32IOP054010(config-if)# switchport mode vntag


IOP054010(config)# vethernet auto-create

8. Configure port-profile for vEthernet for vNIC.

You need to manually create port-profile for vEthernet, which iscorresponding to the vNIC on host. Since vNIC Uplink failovermode will be configured, both port-profiles should be configuredon each Nexus 5596T switches.

IOP054010(config)# port-profile type vethernetNIC-VLAN50

IOP054010(config-port-prof)# switchport access vlan 50IOP054010(config-port-prof)# state enabled









IOP054011(config)# Feature fex



a. Connect the Nexus 2232-2 to Nexus 5596T-2 port Ethernet1/47 and 1/48 and bound to port-channel 1.









a. Globally configure vPC domain and peer-keepaliveconnectivity. Normally, the vPC peer-keepalive destinationsare reachable via management network.

IOP054011(config)# vpc domain 12IOP054011(config-vpc-domain)# peer-keepalivedestination 10.246.54.10


340







IOP054011(config)# interface port-channel7IOP054011(config-if)# switchport mode trunkIOP054011(config-if)# spanning-tree port type networkIOP054011(config-if)# vpc peer-link




IOP054011(config)# vethernet auto-create

8. Configure port-profile for vEthernet for vNIC.

You need to manually create port-profile for vEthernet, which iscorresponding to the vNIC on host. Since vNIC Uplink failovermode will be configured, both port-profiles should be configuredon each Nexus 5596T switches.





Configuring the UCS C-240 M3 server1. Enable the VN-Tag mode for UCS.

Before you can create new vNIC on Cisco UCS with VIC card, youmust enable VN-Tag mode.



a. In the CIMC, go to Server > Inventory and choose the CiscoVIC Adapters tab.

b. In the Adapter Card 2 section, choose the General tab andselect Modify Adapter Properties under the Actions menu.

c. The Modify Adapter Properties dialog box displays. CheckEnable VNTAG Mode, and click Save Changes.

2. Create vNIC.

a. In the CIMC, go to Server > Inventory and choose the CiscoVIC Adapters tab. Then, select the vNICs tab.

b. Click Add and fill in the fields.


342


It is mandatory to fill in the Uplink Port and ChannelNumber fields. In Port Profile field, select the appropriateprofile from the drop-down menu.

c. In order to enable vNIC failover, you need check the EnableUplink Failover box and fill in Failback Timeout field.

The following two graphics show examples of two differentvirtual interfaces created with two different parameters.



3. Power cycle the server.

You must power cycle the UCS server in order for the createdconfiguration to take effect.


344


Go to CIMC> Server > Summary. In the Server Summarywindow, choose Power off Server and then Power on Server.

Configuration verification on Nexus 5596T-11. Check the dynamically created vEthernet interface.

Since vPC and vNIC failover mode is enabled for eachport-profile, the vEthernet interface configurations on Nexus5596T-1 and Nexus 5596T-2 are same.

IOP054010# sh running-config

interface Vethernet32769inherit port-profile NIC-VLAN50bind interface Ethernet100/1/32 channel 50


2. Check the vNIC status.

IOP054010# show interface vethernet 32769 detailvif_index: 16--------------------------veth is bound to interface Ethernet100/1/32

(0x1f6307c0)priority: 0vntag: 7status: activechannel id: 50registered mac info:

vlan 0 - mac 01:00:5e:00:00:01vlan 0 - mac 01:00:5e:00:00:fcvlan 0 - mac 33:33:00:00:00:01



vlan 0 - mac 33:33:00:01:00:03vlan 0 - mac 33:33:ff:d3:0b:14vlan 0 - mac 6c:20:56:a4:75:37vlan 0 - mac ff:ff:ff:ff:ff:ff

IOP054010# show interface vethernet 32770 detailvif_index: 18

--------------------------veth is bound to interface Ethernet100/1/32

(0x1f6307c0)priority: 0vntag: 8status: standbychannel id: 60registered mac info:

vlan 0 - mac 01:00:5e:00:00:01vlan 0 - mac 01:00:5e:00:00:fcvlan 0 - mac 33:33:00:00:00:01vlan 0 - mac 33:33:00:01:00:03vlan 0 - mac 33:33:ff:89:f2:79vlan 0 - mac 6c:20:56:a4:75:38vlan 0 - mac ff:ff:ff:ff:ff:ff

Configuration verification on Nexus 5596T-21. Check the dynamically created vEthernet interface.

Since vPC and vNIC failover mode is enabled, for eachport-profile, the vEthernet interface configurations on Nexus5596T-1 and Nexus 5596T-2 are same.

IOP054010# sh running-config



2. Check the vNIC status.

IOP054011# sh interface vethernet 32769 detailvif_index: 19--------------------------veth is bound to interface Ethernet100/1/32

(0x1f6307c0)priority: 0vntag: 7 status: standbychannel id: 50registered mac info:


346


vlan 0 - mac 01:00:5e:00:00:01vlan 0 - mac 01:00:5e:00:00:fcvlan 0 - mac 33:33:00:00:00:01vlan 0 - mac 33:33:00:01:00:03vlan 0 - mac 33:33:ff:d3:0b:14vlan 0 - mac 6c:20:56:a4:75:37vlan 0 - mac ff:ff:ff:ff:ff:ff

IOP054011# sh interface vethernet 32770 detailvif_index: 20--------------------------veth is bound to interface Ethernet100/1/32

(0x1f6307c0)priority: 0vntag: 8status: activechannel id: 60registered mac info:

vlan 0 - mac 01:00:5e:00:00:01vlan 0 - mac 01:00:5e:00:00:fcvlan 0 - mac 33:33:00:00:00:01vlan 0 - mac 33:33:00:01:00:03vlan 0 - mac 33:33:ff:89:f2:79vlan 0 - mac 6c:20:56:a4:75:38vlan 0 - mac ff:ff:ff:ff:ff:ff

Best practicesBefore configuring the VN-Tag mode on the Ethernet interfaces, youneed to first remove port-channel membership if present; otherwise,the configuration will not be accepted.

SAN connectivity withvPC and FEX

Straight-Through

This section details the setup required to create the vNICconfigurations. This section includes the following information:





◆ “Configuring the UCS C240 M3 Rack server” on page 353

◆ “Configuration verification on Nexus 5596T switch” on page 356

◆ “Best practices” on page 356

Summary of configuration stepsComplete steps are discussed in “Configuring the Nexus 5596T-1” onpage 349 and “Configuring the Nexus 5596T-2” on page 351.



The following is a summary of these steps.

1. Enable feature-set virtualization.


3. Configure FEX associate fabric.


5. Configure vPC peer-link.

6. Configure the port to be VN-Tag mode.

7. Configure vEthernet port and bound to Ethernet port.

8. Associate VSAN to VLAN.

9. Create VFC interface and bound to vEthernet port.

10. Add VFC into VSAN database.

Complete steps are discussed in “Configuring the UCS C240 M3 Rackserver” on page 353. The following is a summary of these steps.

1. Enable VN-Tag mode.

2. Create vHBA.

3. Power cycle the server to have the configuration take effect.

Complete steps are discussed in “Configuration verification onNexus 5596T switch” on page 356. The following is a summary of thisstep.

1. Check vHBA status on Nexus 5596T switches.


348


Target topologyFigure 56 shows the target topology used in this section.

Figure 56 SAN connectivity with vPC and FEX Straight-Through topology example

The following table lists the target topology IP address and DomainID assignment.


Nexus5 596T-1 10.246.54.10 12

Nexus 5596T-2 10.246.54.11 12

Nexus 2232-1

Nexus 2232-2



















a. Globally configure VPC domain ad peer-keepaliveconnectivity. Normally the vPC peer-keepalive destinationsare reachable via management network.

IOP054010(config)# vpc domain 12


350


IOP054010(config-vpc-domain)# peer-keepalivedestination 10.246.54.11


b. Connect Ethernet port 1/7 and 1/8 between Nexus 5596T-1and Nexus 5596T-2 and bound them into port-channel 7








7. Configure vEthernet interface.

IOP054010(config)# int vethernet 201IOP054010(config-if)# switchport mode trunkIOP054010(config-if)# switchport trunk allowed vlan

1,100IOP054010(config-if)# bind interface ethernet

100/1/32 channel 201



100/1/32 channel 202


IOP054010(config)# vlan 100IOP054010(config-vlan)# fcoe vsan 100

9. Configure VFC and bind to vEthernet interface.

IOP054010(config)# int vfc 201



IOP054010(config-if)# no shutIOP054010(config-if)# bind interface vethernet 201


IOP054010(config)# vsan databaseIOP054010(config-vsan-db)# vsan 100 interface vfc 201













c. Configure port-channel members to be FEX-fabric mode





352


a. Globally configure VPC domain ad peer-keepaliveconnectivity. Normally the vPC peer-keepalive destinationsare reachable via management network.

IOP054011(config)# vpc domain 12IOP054011(config-vpc-domain)# peer-keepalive

destination 10.246.54.10IOP054011(config-vpc-domain)# peer-gatewayIOP054011(config-vpc-domain)# auto-recovery









7. Configure vEthernet interface.



100/1/32 channel 202


IOP054011(config)# vlan 200IOP054011(config-vlan)# fcoe vsan 200

9. Configure VFC and bind to vEthernet interface.

IOP054011(config)# int vfc 202IOP054011(config-if)# no shutIOP054011(config-if)# bind interface vethernet 202




IOP054011(config)# vsan databaseIOP054011(config-vsan-db)# vsan 200 interface vfc 202

Configuring the UCS C240 M3 Rack server1. Enable the VN-Tag mode for UCS

Before you can create new vNIC on Cisco UCS with VIC card, youmust enable VN-Tag mode.

a. In the CIMC, go to Server > Inventory and choose the CiscoVIC Adapters tab.

b. In the Adapter Card 2 section, choose the General tab andselect Modify Adapter Properties under the Actions menu.

The Modify Adapter Properties dialog box displays. CheckEnable VNTAG Mode, and click Save Change

2. Create vNIC.

a. In the CIMC, go to Server > Inventory and choose the CiscoVIC Adapters tab, and select vNICs tab.

b. Click Add and fill in the fields. It is mandatory to fill inUplink Port and Channel Number field.

The Channel number value must be same as the switchvEthernet interface configuration.

The following two graphics show examples of two differentvirtual interfaces created with two different parameters.


354




3. Power cycle the server.

You must power cycle the UCS server in order to have the createdconfiguration take effect.


356


Go to CIMC> Server > Summary. In the Server Summarywindow, choose Power off Server and then Power on Server.

Configuration verification on Nexus 5596T switchTo confirm the configuration, enter the following command.

IOP054010# show flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------vfc201 100 0xec0140 20:00:6c:20:56:a4:75:3a 10:00:6c:20:56:a4:75:3a

Configuration verification on Nexus5596T-2

IOP054011# show flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------vfc202 200 0xd10002 20:00:6c:20:56:a4:75:39 10:00:6c:20:56:a4:75:39

Best practicesConsider the following best practices:

◆ The peer-keepalive traffic should never be carried in a VLANover the peer link; such a configuration would make the peerkeepalive useless.

◆ Before configuring VN-Tag mode on the Ethernet interfaces, youneed to remove port-channel membership first if present,otherwise the configuration will not be accepted.

◆ When you need to manually delete the vEthernet interface, youmust use the following sequence, otherwise the configuration willnot be accepted:



1. Remove the VFC bound to vEthernet.

2. Shutdown the vEthernet interface.

3. Delete binding between physical Ethernet and vEthernetinterface.

4. Delete the vEthernet interface.

Nexus 5000/4000/vPC (TOR/FCF or FEX) — Cross-connect topologyThis section details the setup steps required to create theconfiguration shown in the target topology example illustrated inFigure 59 on page 362. This section includes the followinginformation:

◆ “Configuration overview” on page 357◆ “Summary of configuration steps” on page 359◆ “Existing topology example” on page 361◆ “Target topology example” on page 361◆ “Configuring the Nexus 5000” on page 363◆ “Configuring the Nexus 4001I” on page 367


Supported protocols, a high-level diagram of the cross-connecttopology, and key points to consider are provided in this section.

Supported protocolsTable 11 describes the protocols have been set up by E-Lab andsupported in this topology.

Topology overviewA high-level diagram of the cross-connect topology that will beconfigured in this section is shown in Figure 57 on page 358. The termcross-connect is derived from the fact that each Nexus 4001I isconnected to both of the Nexus 5000 parent switches. For thistopology, the FCoE links were configured as separate links,straight-through, between the Nexus 5000s and 4001Is.



FCoE Y N/A Y

iSCSI Y N Y

NAS Y N/A N/A


358


The connections from the Nexus 4001I to the internal CNAs withinthe IBM Blade server were configured as traditional CNAs.

Figure 57 Cross-connect topology example


◆ For any CNA that will be connected to Nexus 5000, the VFC thatit is associated with each CNA port will need to be bound to theENode MAC Address of the CNA port and not the physicalinterface on the switch where the CNA port will attach.

◆ EMC recommends the use of LACP for PortChannels between theNexus 5000 and Nexus 4001I instead of using static PortChannels.

Windows host

vPC Domain

10 GbE

Legend

FCoE

IBM Blade Server

GbE

Po500 Po600

Po5

ICO-IMG-000900

Management LAN



◆ Add a separate FCoE link between the Nexus 5000s and Nexus4001I to carry FCoE traffic. FCoE will not function properly whenrunning over cross-connected vPCs where the FCoE traffic isforwarding to more than one of the parent Nexus 5000s.

◆ Bouncing the vPC Peer link is highly discouraged and is verydisruptive to FCoE.

◆ E-Lab testing found that to get the IBM internal CNAs tosuccessfully log in to the Nexus 5000s with vPC configured in across-connect method, the spanning-tree path cost needed to beadjusted to allow for VLAN 1 to use the FCoE links between theNexus 5000 and Nexus 4000. This may differ depending on eachtopology, but be aware that if VLAN1 does not traverse the FCoEPortChannels then the FIP frames will not make it to the VFCinterfaces on the Nexus 5000, which prohibits the CNA to log into the fabric.

◆ Generation 2 CNAs that support FIP are required for a FCoE overvPC environment.




Note: The Nexus 4001I configuration steps included within this section arefor the vPC and FCoE interswitch connections only.

Nexus 5000 configuration steps1. Enable the vPC and LACP features.




5. Create a PortChannel for the vPC peer link and configure it as thevPC peer link.

6. Add vPC peer link interfaces to the PortChannel created in Step 5.

7. Attach cables for vPC Peer link.

8. Repeat Step 1 through Step 6 on Nexus-5020-2.



360


10. Nexus-5020-1: Create the PortChannels for the 4001I-facinginterfaces and add them to the appropriate vPCs.

11. Nexus-5020-1: Create the FCoE PortChannels for the Nexus4001I-facing interfaces.

12. Nexus-5020-2: Create the PortChannels for the Nexus4001I-facing interfaces and add them to the appropriate vPCs.

13. Nexus-5020-2: Create the FCoE PortChannels for the Nexus4001I-facing interfaces.

14. View the vPC configurations.

15. Ensure each VFC interface for any of the CNAs attached to the4001I is bound to the Enode MAC Address and not the physicalinterface.

16. Ensure each VFC is assigned to the appropriate VSAN as shownin the target topology diagram.

Nexus 4000 configuration steps1. Create the PortChannel for the Nexus 5000 facing vPC interfaces.

2. Create the PortChannel for the Nexus 5000 facing FCoE interface.

3. Create the server-side converged interface.

Note: E-Lab testing found that to get the IBM internal CNAs tosuccessfully log in to the Nexus 5000s with vPC configured in across-connect method, the spanning-tree path cost needed to be adjustedto allow for VLAN 1 to use the FCoE links between the Nexus 5000 andNexus 4000. This may differ depending on each topology, but be awarethat if VLAN1 does not traverse the FCoE PortChannels then the FIPframes will not make it to the VFC interfaces on the Nexus 5000, whichprohibits the CNA to log in to the fabric.

4. Change the spanning-tree path cost for the PortChannelconnecting to the Nexus 5000 vPC.




It is assumed that the customer has an existing LAN and mirroredSAN as shown in Figure 58 and is enabling vPC on both.


Target topologyexample

Note: The Ethernet uplinks, aggregation switches, FC uplinks, FC SANs, andstorage ports have been removed from the target topology in this example forthe sake of clarity. The target topology in this use case is assumed to includeall of the elements in the existing topology with slight modifications made toinclude the 2232 into the configuration. The steps in this section will onlydescribe the steps needed to add vPC to this topology.

Host_1 (Windows)

CNA_1 CNA_2

DCX_1Domain 1

172.23.185.24

DCX_2Domain 2

172.23.185.25


172.23.185.22


172.23.185.23


Nexus-5020-2IP = 172.23.185.113Domain = 3

1/21/31/41/5

1/21/31/41/5

1/21/31/41/5

1/21/31/41/5

1/11/1

1/11/1

Host_2(Linux)

CNA_1 CNA_2



1 2 3 4

Storage port 1:50:06:04:82:D5:2E:69:79

Storage port 2:50:06:04:82:D5:2E:69:49



Nexus-5020-1

Nexus-5020-2



VLAN/VSAN 100

VLAN/VSAN 200

VLAN/VSAN 300

VLAN/VSAN 400

10GbE

4Gb/s FC

Core-N7K-1

Port-channel 1

Port-channel 2

1/1 2/1 2/11/1

Core-N7K-2

SAN PortChannels


1/1

1/2

1/3

1/4

1/5

1/6

1/1

1/2

1/3

1/4

1/5

1/6

Port-Channel 3

Port-Channel 4

Host 1

Host 2

PO3PO4




362



Host_1 (Windows)

Nexus 4001I Nexus 4001I

1/20

1/5

1/6

1/3 1/30

1/5

1/6

VPC 500 VPC 600

PO5

mgmt 0 mgmt 0

vPC peer link

1/15

1/16

1/29 1/19

1/15 1/16 1/191/19

1/4

IBM Blade Server

CNA_1 CNA_2

VLAN 401

VLAN 300

VLAN 401

VLAN 300

CNA_1:QLogicVSAN = 401WWPN= 21:00:00:c0:dd:10:12:1dENode MAC = 00:c0:dd:10:12:1dIP Address = 10.246.53.140Storage = Storage port 1

CNA_2:EmulexVSAN = 200WWPN = 21:00:00:c0:dd:10:12:1fENode MAC = 00:c0:dd:10:12:1fIP Address = 10.246.54.140Storage = Storage port 2

Windows Host

FCoE 10g EthernetConverged

1g Ethernet

21:00:00:c0:dd:10:12:1f21:00:00:c0:dd:10:12:1d

vPC Domain 58

Management LAN

Legend

ICO-IMG-000901

vPC peer-keepalive link




To configure the Nexus 5000, complete the following steps.


Before configuring vPC, the vPC feature must be enabled on theNexus 5000. The LACP feature must be enabled since LACP willbe enabled on the PortChannels.

Note: LACP should already be enabled on the switch if the steps in theNexus 5000 /Nexus 4001I straight through topology example werefollowed.

Nexus-5020-1# config terminalNexus-5020-1(config)# feature vpcNexus-5020-1(config)# feature lacp


Nexus-5020-1(config)# vpc domain 58Nexus-5020-1(config-vpc-domain)#




Nexus-5020-1(config-vpc-domain)# system-priority 4000Nexus-5020-1(config-vpc-domain)# exitNexus-5020-1(config)#

5. Create a PortChannel and configure it as the vPC peer link.

Nexus-5020-1(config)# interface port-channel5Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# vpc peer-link

6. Add vPC peer link interfaces to the PortChannel.

Nexus-5020-1(config)# interface ethernet 1/5-6Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# channel-group 5 mode active

7. Repeat Step 1 through Step 6 on the Nexus-5020-2.




364





vPC Peer-link status---------------------------------------------------------------------id Port Status Active vlans-- ---- ------ --------------------------------------------------1 Po5 up -Nexus-5020-1#






Nexus-5020-2#



10. On the Nexus-5020-1, create the PortChannels for the Nexus4001I-facing interfaces and add them to the appropriate vPC.

Nexus-5020-1(config-if)# interface e1/29Nexus-5020-1(config-if)# channel-group 500 mode activeNexus-5020-1(config-if)# interface port-channel 500Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# switchport trunk allowed vlan remove 401,500Nexus-5020-1(config-if)# vpc 500Nexus-5020-1(config-if)# interface e1/30Nexus-5020-1(config-if)# channel-group 600 mode activeNexus-5020-1(config-if)# interface port-channel 600Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# switchport trunk allowed vlan remove 401,500Nexus-5020-1(config-if)# vpc 600

11. On the Nexus-5020-1, create the PortChannels and apply toNexus 4001I-facing interfaces.

Nexus-5020-1(config-if)# int e1/3Nexus-5020-1(config-if)# channel-group 401 mode activeNexus-5020-1(config-if)# interface port-channel 401Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# switchport trunk allowed vlan 1,401

12. On the Nexus-5020-2, create the PortChannels for the 4001I-facinginterfaces and add them to the appropriate vPC.

Nexus-5020-2(config-if)# interface e1/19Nexus-5020-2(config-if)# channel-group 500 mode activeNexus-5020-2(config-if)# interface port-channel 300Nexus-5020-2(config-if)# switchport mode trunkNexus-5020-2(config-if)# switchport trunk allowed vlan remove 401,300Nexus-5020-2(config-if)# vpc 500Nexus-5020-2(config-if)# interface e1/20Nexus-5020-2(config-if)# channel-group 600 mode activeNexus-5020-2(config-if)# interface port-channel 600Nexus-5020-2(config-if)# switchport mode trunkNexus-5020-2(config-if)# switchport trunk allowed vlan remove 401,300Nexus-5020-2(config-if)# vpc 600

13. On the Nexus-5020-2, create the FCoE PortChannels and apply tothe Nexus 4001I-facing interfaces.

Nexus-5020-2(config-if)# int e1/4Nexus-5020-2(config-if)# channel-group 300 mode activeNexus-5020-2(config-if)# interface port-channel 300Nexus-5020-2(config-if)# switchport mode trunkNexus-5020-2(config-if)# switchport trunk allowed vlan 1,401


366


14. View the vPC configuration.






Nexus-5020-1#








Nexus-5020-2#15. Ensure the VFC interfaces for any of the CNAs attached to the

4001I are bound to the ENode MAC Address and not the physicalinterface.

• On Nexus-5020-1:

Nexus-5020-1(config-if)# interface vfc 1224Nexus-5020-1(config-if)# bind mac-address 00:c0:dd:10:12:1dNexus-5020-1(config-if)# no shut

• On Nexus-5020-2:

Nexus-5020-2(config-if)# interface vfc 1225Nexus-5020-2(config-if)# bind mac-address 00:c0:dd:10:12:1fNexus-5020-2(config-if)# no shut

16. Ensure each VFC to the appropriate VSAN as shown in the targettopology diagram.

• On Nexus-5020-1:


• On Nexus-5020-2:

Nexus-5020-2(config-if)# vsan databaseNexus-5020-2(config-vsan-db)# vsan 300 interface vfc1225Nexus-5020-2(config-vsan-db)# exitNexus-5020-2(config-if)#

Configuring the Nexus4001I

To configure the Nexus 4001I, complete the following steps.

On the Nexus-4001I-1

1. Create the PortChannel for the Nexus 5000-facing vPC interfaces.

Nexus-4000-1(config-if)# interface e1/15-16Nexus-4000-1(config-if)# channel-group 1 mode activeNexus-4000-1(config-if)# interface port-channel 1Nexus-4000-1(config-if)# switchport mode trunkNexus-4000-1(config-if)# switchport trunk allowed vlan remove 401,300


368


2. Create the PortChannel for the Nexus 5000-facing FCoE interface.

Nexus-4000-1(config-if)# interface e1/19Nexus-4000-1(config-if)# channel-group 401 mode activeNexus-4000-1(config-if)# interface port-channel 401Nexus-4000-1(config-if)# switchport mode trunkNexus-4000-1(config-if)# switchport trunk allowed vlan allowed 1,401Nexus-4000-1(config-if)# fip-snooping port-mode fcf

3. Create the server-side CNA interface.

Nexus-4000-1(config-if)# interface e1/12Nexus-4000-1(config-if)# switchport mode trunkNexus-4000-1(config-if)# switchport trunk allowed vlan 1,401Nexus-4000-1(config-if)# spanning-tree port type edge trunkNexus-4000-1(config-if)# spanning-tree bpduguard enable

4. Change the spanning-tree path cost.

Nexus-4000-1(config-if)# interface port-channel 1Nexus-4000-1(config-if)# spanning-tree cost 20

On the Nexus-4001I-2

1. Create the PortChannel for the Nexus 5000-facing vPC interfaces.

Nexus-4000-2(config-if)# interface e1/15-16Nexus-4000-2(config-if)# channel-group 1 mode activeNexus-4000-2(config-if)# interface port-channel 1Nexus-4000-2(config-if)# switchport mode trunkNexus-4000-2(config-if)# switchport trunk allowed vlan remove 401,300

2. Create the PortChannel for the Nexus 5000-facing FCoE interface.

Nexus-4000-2(config-if)# interface e1/19Nexus-4000-2(config-if)# channel-group 300 mode activeNexus-4000-2(config-if)# interface port-channel 300Nexus-4000-2(config-if)# switchport mode trunkNexus-4000-2(config-if)# switchport trunk allowed vlan allowed 1,300Nexus-4000-2(config-if)# fip-snooping port-mode fcf

3. Create the Server side CNA interface.

Nexus-4000-2(config-if)# interface e1/12Nexus-4000-2(config-if)# switchport mode trunkNexus-4000-2(config-if)# switchport trunk allowed vlan 1,300Nexus-4000-2(config-if)# spanning-tree port type edge trunkNexus-4000-2(config-if)# spanning-tree bpduguard enable

4. Change the spanning-tree path cost.

Nexus-4000-2(config-if)# interface port-channel 1Nexus-4000-2(config-if)# spanning-tree cost 20



Host vPC— Direct-connect topologyThis section details the setup steps required to create theconfiguration shown in the target topology example illustrated inFigure 60 on page 370. This section includes the followinginformation:

◆ “Configuration overview” on page 369




◆ “vPC verification commands” on page 385


Supported protocols, a high-level diagram of a vPC direct-connecttopology, and key points to consider are provided in this section.

Supported protocolsTable 12 describes the protocols have been set up by E-Lab and aresupported in this topology.

Topology overviewA high-level diagram of vPC direct-connect with host topology thatwill be configured in this section is shown in Figure 60 on page 370.In this topology, the host's CNA ports are directly connected to a pairof Nexus 5000 switches with the use of vPC. On the host-side NICteaming software is required to be able to work with vPC.



FCoE Y Y Y

iSCSI Y N Y

NAS Y Y N/A


370


Figure 60 Direct-connect topology example


◆ For any CNA that will be connected to Nexus 5000, the VFC thatit is associated with each CNA port will need to be bound to thePortChannel or vPC of the switch where the CNA port willattach.

◆ Generation 2 CNAs that support FIP are required for aFCoE-over-vPC environment.

◆ VFC interfaces will not be online if the PortChannel on a singleNexus 5000 contains more than one member port in a vPC.


◆ FC Features Package (FC_FEATURES_PKG) is necessary forrunning FCoE. If this is not installed, a temporary 90-day licenseis issued.

◆ FCoE and IP traffic (such as iSCSI) can traverse simultaneouslyover one vPC link.

vPC vPC


ICO-IMG-000890

vPC DomainPeer-keepalive

link

vPC Peer link

vPC Peer Device vPC Peer Device

vPC MemberPort

vPC MemberPort

Mgmt0 Mgmt0

Management





Note: The Nexus 5000 configuration steps included within this section are forthe vPC direct-connect topology only.

Nexus 5000 configuration steps1. Enable the vPC and LACP features.

2. Create a vPC domain and enter vpc-domain mode.

3. Configure the vPC peer-keepalive link.

4. Configure system priority (optional).

5. Configure vPC role priority (optional).

6. Create a PortChannel and make it a vPC peer link.

7. Create another PortChannel and make it a vPC.

The following additional steps are needed if FCoE is going to beimplemented in vPC environment.

1. Enable FCoE on the Nexus 5000.

2. Create a VSAN and map it to a VLAN that has been designated tocarry FCoE traffic.

3. Configure the VLANs that are allowed to transverse the virtualPortChannel links.

4. Create a virtual Fibre Channel interface (vfc) and add it to theVSAN that was created in Step 2.

5. Bind the virtual Fibre Channel interface to the PortChannel usedin the vPC configuration.

6. Verify that the vfc is up and operational.

7. Verify that the virtual Fibre Channel interface has logged in to thefabric.

8. Verify that the vPC is up and operational.


372



Figure 61 shows an example of the existing topology.


2/11/1 1/1 2/1


Port-channel 1

Port-channel 2

1/21/31/41/5

1/21/31/41/5


1/21/1 1/21/1

SAN A SAN BStorage

1 2 3 4

1/1

1/1

DCX_1Domain 1

172.23.185.24

DCX_2Domain 2

172.23.185.25

SAN PortChannels

1/11/21/31/41/5

1/11/21/31/41/5


172.23.185.22


172.23.185.23


Host_1(Windows)

Host_2(Linux)

VLAN / VSAN 100

VLAN / VSAN 200

VLAN / VSAN 300

VLAN / VSAN 400

10 GbE

4 Gb/s FC




CNA_2:

Nexus-5020-1

IP = 172.23.185.113 Domain = 3



Storage port 1: 50:06:04:82:D5:2E:69:79

Storage port 2: 50:06:04:82:D5:2E:69:49






This section contains information on configuring the Nexus 5000 inthe following environments:

◆ “iSCSI in a vPC environment example” on page 373

◆ “FCoE in a vPC environment example” on page 377

In the first configuration example shown in Figure 62 on page 374,the connection from LAN aggregation switches to the LAN core hasbeen removed and replaced by a simple direct connection to an iSCSItarget for the sake of clarity.

In the second configuration example, showing FCoE in a vPCenvironment, Figure 63 on page 378, the dual SAN connections toSAN A and SAN B are simplified to direct connections to FCoEstorage ports.

iSCSI in a vPC environment exampleFigure 62 on page 374 is an example of vPC deployment. Here, theiSCSI initiator has dual CNA ports configured as a NIC team andeach switch in the vPC domain contains one of the vPC memberports. A storage device is connected on one of the switches to serve asan iSCSI target device.


374


Figure 62 iSCSI target topology

The following are the exact steps to configure vPC on Nexus-5010-1and Nexus-5010-2 switches.

Note: VLAN 200 is used in the configuration example. This example assumesthe following configuration parameters are defined for port e1/18 onNexus-5010-1 and the management IP addresses of both switches.

Host_1 (Windows)

CNA_1 CNA_2 Nexus-5020-1IP = 10.32.139.16

Nexus-5020-2IP = 10.32.139.136


PO 2

mgmt0 mgmt0

vPC keep alive link

vPC peer link

vPC 10

vPC Domain 10

Storage

1

VLAN 200 Trunk Port

10GbEGbE

Management LAN

1/11/21/3 1/18

1/111/121/13

CNA_1:QlogicPhysical MAC Address = 00c0.dd10.28baNIC Team MAC Address = 00c0.dd10.28b8NIC Team IP Address = 100.100.100.246Team Type = Load Balancing

CNA_2:QlogicPhysical MAC Address = 00c0.dd10.28b8NIC Team MAC Address = 00c0.dd10.28b8NIC Team IP Address = 100.100.100.246Team Type = Load Balancing

Host 1

SP B4-0 (B10):Physical MAC Address = 0060.1632.161eIP Address = 100.100.100.66

Storage Port 1

ICO-IMG-000902



Nexus-5010-1:

vrf context managementip route 0.0.0.0/0 10.32.136.1

interface mgmt0ip address 10.32.139.16/24

interface Ethernet1/18switchport access vlan 200

Nexus-5010-2:



Configuring Nexus-5010-1

To configure the Nexus-5010-1, complete the following steps.

1. Enable the vPC and LACP feature.

Nexus-5010-1# config terminalEnter configurations commands, one per line. End with

CNTL/Z.Nexus-5010-1(config)# feature lacpNexus-5010-1(config)# feature vpc

2. Create a vPC domain and enter vpc domain mode.

Nexus-5010-1(config)# vpc domain 10

3. Configure the vPC peer keep alive link.


4. Configure the system priority (optional).

System priority is not needed since the vPC of switch peerswitches is not configured with LACP.

5. Configure the vPC role priority (optional).

In this example default switch values are used. If role priority isnot configured, the vPC local system MAC address will be the tiebreaker; the lowest MAC address will win the primary role.

6. Create the vPC peer link.

Nexus-5010-1(config)# interface ethernet 1/1-2Nexus-5010-1(config-if-range)# channel-group 2 mode activeNexus-5010-1(config-if-range)# interface port-channel 2Nexus-5010-1(config-if)# vpc peer-link


376


Nexus-5010-1(config-if)# switchport mode trunkNexus-5010-1(config-if)# switchport trunk allowed vlan 1,200

7. Move the PortChannel to vPC.

Nexus-5010-1(config)# interface ethernet 1/3Nexus-5010-1(config-if)# channel-group 10Nexus-5010-1(config-if)# interface port-channel 10Nexus-5010-1(config-if)# vpc 10Nexus-5010-1(config-if)# switchport access vlan 200


To configure the Nexus-5010-2, complete the following steps.













Nexus-5010-2(config)# interface ethernet 1/11-12Nexus-5010-2(config-if-range)# channel-group 2 mode activeNexus-5010-2(config-if-range)# interface port-channel 2Nexus-5010-2(config-if)# vpc peer-linkNexus-5010-2(config-if)# switchport mode trunkNexus-5010-2(config-if)# switchport trunk allowed vlan 1,200




Nexus-5010-2(config)# interface ethernet 1/13Nexus-5010-2(config-if)# channel-group 10 mode onNexus-5010-2(config-if)# int port-channel 10Nexus-5010-2(config-if)# vpc 10Nexus-5010-2(config-if)# switchport access vlan 200

After completing the necessary configuration, IP connectivity will beestablished from 100.100.100.246 to 100.100.100.66 and the iSCSIinitiator and target can be configured. For more information on howto configure the iSCSI initiator and target refer to “Configuring theiSCSI initiator and target” on page 388.

FCoE in a vPC environment exampleFigure 63 on page 378 is another example of vPC deployment. Here,the FCoE initiator has a dual-port CNA and each port is connected toa vPC member port on both switches in the vPC domain. It isimportant to take note that in order to run FCoE over a vPC topology,a port-channel can only have a single member interface.


378


Figure 63 FCoE vPC direct connect target topology

The following are the exact steps to configure vPC on Nexus-5010-1and Nexus-5010-2 switches.

Note: VSAN 80(VLAN 800) is used in Nexus-5010-1 while VSAN 60 (VLAN600) is used in Nexus-5010-2. The following are the pre-configuredmanagement IP addresses s of both switches.

Nexus-5010-1:



Host_1 (Windows)

CNA_1 CNA_2


PO2

Management LAN

vPC keep alive link

vPC peer link

vPC 10

vPC Domain 10

Storage

1 2

Nexus-5020-1IP = 10.32.139.16Domain = 167

Nexus-5020-2IP = 10.32.139.136Domain = 216

10GbEGbE

VLAN 800 /VSAN 80

VLAN 600 /VSAN 60

Trunk Port

mgmt0 mgmt01/11/21/3

1/18 1/111/121/13

1/18

CNA_2:QlogicVSAN = 60WWPN = 21:00:00:c0:dd:10:28:b9ENode MAC = 00c0.dd10.28b9Storage = Storage port 2

CNA_1:QlogicVSAN = 80WWPN= 21:00:00:c0:dd:10:28:bbENode MAC = 00c0.dd10.28bbStorage = Storage port 1

Host 1

SP A3-1 (A13):WWPN= 50:06:01:65:3b:64:03:c4

Storage Port 1

SP B3-1 (B13):WWPN= 50:06:01:6d:3b:64:03:c4

Storage Port 2

ICO-IMG-000903



Nexus-5010-2:



Configuring Nexus-5010-1Two types of configuration are described in this section:

◆ “vPC configuration” on page 379

◆ “FCoE configuration” on page 380

vPC configuration

To configure the Nexus 5010-1 in a vPC, complete the following steps.















380



Nexus-5010-1(config)# interface ethernet 1/3Nexus-5010-1(config-if)# channel-group 10Nexus-5010-1(config-if)# interface port-channel 10Nexus-5010-1(config-if)# vpc 10

FCoE configuration

To configure the Nexus 5010-1 in an FCoE environment, complete thefollowing steps.


Nexus-5010-1 (config)# feature fcoeFC license checked out successfullyfc_plugin extracted successfullyFC plugin loaded successfullyFCoE manager enabled successfullyFC enabled on all modules successfully


Nexus-5010-1(config)# vsan databaseNexus-5010-1(config-vsan-db)# vsan 80Nexus-5010-1(config-vsan-db)# exitNexus-5010-1(config)# vlan 800Nexus-5010-1(config-vlan)# fcoe vsan 80


Nexus-5010-1(config)# int po 10Nexus-5010-1(config-if)# switchport trunk allowed vlan 1, 800


Nexus-5010-1(config)# int vfc 3Nexus-5010-1(config)# vsan databaseNexus-5010-1(config-vsan-db)# vsan 80 interface vfc 3

5. Bind the virtual Fibre Channel interface to PortChannel used inthe vPC configuration.

Nexus-5010-1(config)# int vfc 3Nexus-5010-1(config-if)# bind interface port-channel 10Warning: VFC will not come up for pre-FIP CNANexus-5010-1(config-if)# no shutdown




Nexus-5010-1# show interface vfc 3vfc3 is up

Bound interface is port-channel10FCF priority is 128Hardware is Virtual Fibre ChannelPort WWN is 20:02:00:0d:ec:b1:58:ffAdmin port mode is F, trunk mode is onsnmp link state traps are enabledPort mode is F, FCID is 0xad0002Port vsan is 805 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec5 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec



last clearing of "show interface" counters neverInterface last changed at Wed Oct 20 16:44:23 2010


Nexus-5010-1# show flogi database interface vfc3--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------vfc3 80 0xa70000 21:00:00:c0:dd:10:28:bb 20:00:00:c0:dd:10:28:bb



Nexus-5010-1# show vpc statistics vpc 10port-channel10 is upvPC Status: Up, vPC number: 10Hardware: Port-Channel, address: 000d.ecb1.58ca (bia 000d.ecb1.58ca)MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255Encapsulation ARPAPort mode is trunkfull-duplex, 10 Gb/sBeacon is turned offInput flow-control is off, output flow-control is offSwitchport monitor is offMembers in this channel: Eth1/3Last clearing of "show interface" counters never30 seconds input rate 24 bits/sec, 0 packets/sec30 seconds output rate 20592 bits/sec, 26 packets/secLoad-Interval #2: 5 minute (300 seconds)

input rate 48 bps, 0 pps; output rate 26.93 Kbps, 19 pps


382


RX1730715 unicast packets 6369922 multicast packets 185340 broadcast packets8285977 input packets 572001240 bytes10 jumbo packets 0 storm suppression packets0 runts 0 giants 0 CRC 0 no buffer0 input error 0 short frame 0 overrun 0 underrun 0 ignored0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop0 input with dribble 0 input discard0 Rx pause

TX3925711 unicast packets 2969727 multicast packets 5110579 broadcast packets12006017 output packets 3776402534 bytes10 jumbo packets0 output errors 0 collision 0 deferred 0 late collision0 lost carrier 0 no carrier 0 babble0 Tx pause

28 interface resets


Two types of configuration are described in this section:

◆ “vPC configuration” on page 382

◆ “FCoE configuration” on page 383

vPC configuration

To configure the Nexus 5010-2 in a vPC, complete the following steps.

















Nexus-5010-2(config)# interface ethernet 1/13Nexus-5010-2(config-if)# channel-group 10 mode onNexus-5010-2(config-if)# interface port-channel 10Nexus-5010-2(config-if)# vpc 10

FCoE configuration

To configure the Nexus 5010-2 in an FCoE environment, complete thefollowing steps.


Nexus-5010-2 (config)# feature fcoeFC license checked out successfullyfc_plugin extracted successfullyFC plugin loaded successfullyFCoE manager enabled successfullyFC enabled on all modules successfully


Nexus-5010-2(config)# vsan databaseNexus-5010-2(config-vsan-db)# vsan 60Nexus-5010-2(config-vsan-db)# exitNexus-5010-2(config)# vlan 600Nexus-5010-2(config-vlan)# fcoe vsan 60


Nexus-5010-2(config)# interface port-channel 10Nexus-5010-2(config-if)# switchport trunk allowed vlan 1, 600



384


Nexus-5010-2(config)# int vfc 13Nexus-5010-2(config)# vsan databaseNexus-5010-2(config-vsan-db)# vsan 60 interface vfc 13

5. Bind the virtual Fibre Channel interface to PortChannel used inthe vPC configuration.

Nexus-5010-2(config)# int vfc 13Nexus-5010-2(config-if)# bind interface port-channel 10Warning: VFC will not come up for pre-FIP CNANexus-5010-2(config-if)# no shutdown


Nexus-5010-2# show interface vfc 13vfc13 is up

Bound interface is port-channel10FCF priority is 128Hardware is Virtual Fibre ChannelPort WWN is 20:0c:00:0d:ec:b1:5f:7fAdmin port mode is F, trunk mode is onsnmp link state traps are enabledPort mode is F, FCID is 0xdf0000Port vsan is 605 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec5 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec



last clearing of "show interface" counters neverInterface last changed at Wed Oct 20 08:10:26 2010


Nexus-5010-2# show flogi database interface vfc13--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------vfc13 60 0xd80000 21:00:00:c0:dd:10:28:b9 20:00:00:c0:dd:10:28:b9



Nexus-5010-2# show vpc statistics vpc 10port-channel10 is upvPC Status: Up, vPC number: 10Hardware: Port-Channel, address: 000d.ecb1.5f54 (bia 000d.ecb1.5f54)MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255



Encapsulation ARPAPort mode is trunkfull-duplex, 10 Gb/sBeacon is turned offInput flow-control is off, output flow-control is offSwitchport monitor is offMembers in this channel: Eth1/13Last clearing of "show interface" counters never30 seconds input rate 24 bits/sec, 0 packets/sec30 seconds output rate 248 bits/sec, 0 packets/secLoad-Interval #2: 5 minute (300 seconds)input rate 56 bps, 0 pps; output rate 296 bps, 0 pps

RX474335 unicast packets 7161571 multicast packets 91614 broadcast packets7727520 input packets 520807278 bytes3 jumbo packets 0 storm suppression packets0 runts 0 giants 0 CRC 0 no buffer0 input error 0 short frame 0 overrun 0 underrun 0 ignored0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop0 input with dribble 0 input discard0 Rx pause

TX77429 unicast packets 150109 multicast packets 383119 broadcast packets610657 output packets 65548518 bytes4 jumbo packets0 output errors 0 collision 0 deferred 0 late collision0 lost carrier 0 no carrier 0 babble0 Tx pause

22 interface resets

After completing the necessary configuration, each PWWN of theCNAs will be able to perform FLOGI to both switches, which can bezoned to the PWWN of the FCoE storage ports. Finally, the FCoEinitiator and target can be configured. For more information on howto configure the FCoE initiator and target, refer to “Configuring theFCoE initiator and target” on page 405.

vPC verificationcommands

The following table lists vPC verification commands.

Command Purpose

show feature Displays whether vPC is enabled or not.

show port-channel capacity Displays how many EtherChannels are configuredand how many are still available on the switch.

show running-config vpc Displays running configuration information for vPCs.

show vpc brief Displays brief information on the vPCs.


386


Verification command examples

The following are examples of verification commands.

Nexus-5010-1 # show vpc briefLegend:


vPC Domain ID : 10Peer status : peer adjacency formed okvPC keep-alive status : peer is aliveConfiguration consistency status: successvPC role : primaryNumber of vPCs configured : 1Peer Gateway : DisabledDual-active excluded VLANs : -

vPC Peer-link status---------------------------------------------------------------------id Port Status Active vlans-- ---- ------ --------------------------------------------------1 Po2 up -

vPC status----------------------------------------------------------------------------id Port Status Consistency Reason Active vlans------ ----------- ------ ----------- -------------------------- -----------10 Po10 up success success 400

Nexus-5010-1 # show vpc peer-keepalive

vPC keep-alive status : peer is alive--Peer is alive for : (2416906) seconds, (42) msec--Send status : Success--Last send at : 2010.10.20 08:39:32 13 ms--Sent on interface : mgmt0

show vpcconsistency-parameters

Displays the status of those parameters that must beconsistent across all vPC interfaces.

show vpc peer-keepalive Displays information on the peer-keepalive messages.

show vpc role Displays the peer status, the role of the local switch,the vPC system MAC address and system priority,and the MAC address and priority for the local vPCswitch.

show vpc statistics Displays statistics on the vPCs.

Command Purpose



--Receive status : Success--Last receive at : 2010.10.20 08:39:32 149 ms--Received on interface : mgmt0--Last update from peer : (0) seconds, (448) msec

vPC Keep-alive parameters--Destination : 10.32.139.16--Keepalive interval : 1000 msec--Keepalive timeout : 5 seconds--Keepalive hold timeout : 3 seconds--Keepalive vrf : management--Keepalive udp port : 3200--Keepalive tos : 192

Nexus-5010-2 # show port-channel capacityPort-channel resources

768 total 2 used 766 free 0% used

Nexus-5010-1 # show featureFeature Name Instance State-------------------- -------- --------cimserver 1 disabledfabric-binding 1 disabledfc-port-security 1 disabledfcoe 1 enabledfcsp 1 disabledfex 1 disabledfport-channel-trunk 1 disabledhttp-server 1 enabledinterface-vlan 1 disabledlacp 1 enabledlldp 1 enablednpiv 1 disablednpv 1 disabledport_track 1 disabledprivate-vlan 1 disabledsshServer 1 enabledtacacs 1 disabledtelnetServer 1 enabledudld 1 disabledvpc 1 enabledvtp 1 disabled

Nexus-5010-2 # show running-config vpc

!Command: show running-config vpc!Time: Tue Aug 24 18:04:34 2010

version 4.2(1)N2(1)feature vpc


388


vpc domain 10peer-keepalive destination 10.32.139.136

interface port-channel2vpc peer-link

interface port-channel10vpc 10

Configuring the iSCSI initiator and targetBefore proceeding with steps in this section, the followingassumptions are made:

◆ The CNA, device driver, firmware, and its software suite (such asSAN Surfer CNA Networking CLI and SAN Surfer FC HBAManager) have been installed on the server.

◆ Fiber cables have been connected from the CNA ports to the pairof switches participating in the vPC domain.


◆ “Configuring NIC Teaming” on page 388

◆ “Configuring the iSCSI initiator” on page 390

◆ “Configuring the iSCSI target” on page 393

◆ “Connecting to iSCSI target and verification” on page 402

Configuring NICTeaming

To configure NIC Teaming, complete the following steps:

1. Ensure that there is no configuration (IP address, subnet mask,default gateway, etc.) on both CNA ports that you are intendingto configure as part of NIC Teaming.

2. Start the NIC Teaming software. Usually, a shortcut is placed ondesktop or you could run it under Start > Programs > SoftwareName. In this example, QLogic CNAs are used, thus SAN SurferCNA Networking CLI software is used in NIC teamingconfiguration.



3. Once opened, the NIC Teaming main menu will show the optionslike shown next:

4. Select Configure New Team by typing option 3 from the mainmenu.

5. You will be asked to select a Team Type, as shown in the nextscreen. Type the chosen Team Type after the prompt. In thisexample, Team Type 2 is used (Load Balanced).


390


6. Once the NIC team has been created, you will notice the newvirtual port created under Network Connections, as shown in thenext screen. You can change the interface name to a moremeaningful name, such as "NIC team (vPC)," as shown.

7. Configure the newly created interface with IP address, subnetmask, and default gateway (if needed).

IMPORTANT

In this example, there is no need to assign a default gatewaysince the connection from the iSCSI initiator to the iSCSI targetis within one network segment.

Configuring the iSCSIinitiator

To configure the iSCSI initiator, complete the following steps.

1. Install the Microsoft iSCSI Initiator software. This can bedownloaded from http://www.microsoft.com.

Note: Some operating systems, such as Microsoft Windows 2008, havebundled iSCSI initiator. If you are using one of these operating systems,you can skip the installation part and proceed with Step 2.



2. Open the iSCSI initiator software. Typically a shortcut is placedon your desktop. The Initiator Node Name or IQN that you see inthe General tab is needed to register the iSCSI initiator asexplained in “Configuring the iSCSI target” on page 393.

3. Configure the host to discover the iSCSI target. Discovery of iSCSItargets can be accomplished in one of the two following ways.

• Explicitly add the iSCSI target IP address.


392


In the Discovery tab, click Add.

Or,

• Enter the iSNS IP address.

For more information about configuring iSNS, refer to theEMC Host Connectivity Guide for Windows or the EMCCLARiiON iSCSI Server Setup Guide for Windows at EMCOnline Support at https://support.emc.com.

4. Type the IP address of the iSCSI target, then click OK. In thisexample, 100.100.100.66 is used as the iSCSI target IP address.



The iSCSI target will not be discovered until the IP address hasbeen configured in iSCSI 10 GE port. Refer to the “Configuringthe iSCSI target” section, next, for information on configuringiSCSI target IP address.

Configuring the iSCSItarget

This example uses an EMC CLARiiON storage array with EMCUnisphere™ software as an iSCSI target device. Unisphere is amanagement interface for managing VNX series, CLARiiON, andCelerra systems. Unisphere provides customizable dashboard viewsand reporting capabilities that present users with valuableinformation in the context of how they manage storage. An exampleof the Unisphere interface is shown in Figure 64.

Figure 64 Unisphere navigation and user interface

More information about Unisphere can be found at EMC OnlineSupport at https://support.emc.com.

To configure an iSCSI target, complete the following steps.

1. Assign an IP address on the iSCSI 10 GE target port. In thisexample, the SP B4-0 (B-10) port is used. To assign an IP address:


394


a. Click System > Hardware in the Unisphere main pane, thennavigate to the SP B4-0 port in the SP tree.

b. Right-click the port, then click Properties, as shown next.



The iSCSI Port Properties dialog box displays, as shown next.

c. Click Add, then assign the iSCSI target IP address and subnetmask, as shown in the next example.


396


IMPORTANT

In this example, there is no need to assign a default gatewaysince the connection from the iSCSI initiator to the iSCSI targetis within one network segment.

Note: After an IP address has been assigned to the iSCSI target,performing a simple connectivity test, like ping, is helpful to ensure thatthere is IP connectivity between the iSCSI initiator and the iSCSI target. Ifa firewall is used in the network, contact your network administrator andensure iSCSI TCP ports (typically ports 860 and 3260) are allowed in thesecurity policy.

d. Click OK.

2. Access the Create Initiator Record dialog box to register the iSCSIinitiator’s IQN. To access this dialog box:

a. Click System from the Unisphere main menu.



b. Click the Storage System Connectivity Status in the left-handpane of the System Management menu.

The Create Initiator Record dialog box, shown next, displays.

3. In the Create Initiator Record dialog box, complete the following:

a. Type the iSCSI initiator's IQN in the WWN/IWN text box. Tofind the host's IQN, refer to Step 2 in “Configuring the iSCSIinitiator” on page 390.

b. Select the iSCSI target port to which the iSCSI initiator isconnecting from the Sp-Port drop-down menu. In this case,port B-10 or SP B4-0 is used.

c. Indicate the Initiator Type and Failover Mode.

d. If the iSCSI initiator does not have any existing host record onthe iSCSI target, click New Host and type the Host Name andIP Address.

e. Click OK once you have completed the information.


398


f. Verify the IQN is registered and logged in by clicking theStorage System Connectivity Status in the left-hand pane ofthe System Management menu, as shown next.

4. Create the Storage Group in the iSCSI target.

a. Go to the Storage tab on the main Unisphere pane.

b. In the Storage Groups pane, select the Storage Group from thedrop-down menu.

c. Click Create.

d. Type the Storage Group name you want, then click OK.



5. Assign the host to the newly created Storage Group by clickingConnect Hosts.


400


The Storage Group Properties window displays, as shown next.

a. Use the right-arrow button to select the host.

b. Click OK.



Once the host has been assigned, the host name will appear in theHost tab in the Details pane, as shown next.

6. To perform LUN Masking, select the Connect LUNs tab from theStorage Group screen.


402



7. In the LUNs tab, Available LUNs pane, select the desired LUNsor devices you want to add and click Add. Then, click OK.

For more information about iSCSI Target Configuration, refer to theEMC Host Connectivity Guide for Windows or EMC CLARiiON iSCSIServer Setup Guide for Windows, located at EMC Online Support athttps://support.emc.com.

Connecting to iSCSItarget and verification

After completing the steps outlined in “Configuring NIC Teaming”on page 388, “Configuring the iSCSI initiator” on page 390, and“Configuring the iSCSI target” on page 393, it is time to connect toiSCSI target and perform verification and validation. Complete thefollowing steps to ensure that iSCSI configuration has beensuccessfully completed.

1. Using Unisphere, locate the iSCSI target's IQN name.



a. The iSCSI target IQN name can be found in the InitiatorInformation window, which can be opened by clicking theSystem tab from the main menu.

b. Click the Storage System Connectivity Status on theleft-hand menu, System Management.

c. Select your iSCSI initiator, then click Detail.

The iSCSI target IQN name is listed in the SP-Port IQN/WWNtext field, as shown in the next example. In this example, thetarget IQN name iqn.1992-04.com.emc:cx.fcntr080500013.b10 isselected.

2. On your host, using the iSCSI initiator software, connect to theiSCSI target's IQN name.

a. In the iSCSI Initiator Properties window, Target tab, selectthe iSCSI target IQN name.

b. Click Log On.


404


c. Click OK.

3. From your host, verify the list of LUNs or devices from theDevices tab. Access the Devices tab by clicking Details on theTargets tab from the iSCSI Initiator Properties window. Asshown in the next example, nine LUNs or devices were presentedby the iSCSI target to the host.



4. From your host, access the Computer Management window, andchoose Storage > Disk Management in the task pane, as shownnext. Use the Disk Management tool to initialize, partition, andformat the disk.

Configuring the FCoE initiator and targetBefore proceeding with the steps in this section, the followingassumptions are made:

◆ The CNA card, device driver, firmware, and its software suite(such as SAN Surfer CNA Networking CLI and SAN Surfer FCHBA Manager) have been installed on the server.

◆ Fiber cables have been connected from the CNA ports to the pairof switches participating in the vPC domain.


406



◆ “Configuring NIC Teaming” on page 406

◆ “Zoning” on page 408

◆ “Configuring the FCoE target” on page 410

◆ “Verifying FCoE LUNs” on page 416

Configuring NICTeaming

To configure NIC Teaming, complete the following steps:

1. Ensure that there is no configuration (IP address, subnet mask,default gateway, etc.) on both CNA ports that you are intendingto configure as part of NIC Teaming.

2. Start the NIC Teaming software. Usually, a shortcut is placed ondesktop or you could run it under Start > Programs > SoftwareName. In this example, QLogic CNAs are used, thus SAN SurferCNA Networking CLI software is used in NIC teamingconfiguration.

3. Once opened, the NIC Teaming main menu will show the optionslike shown next:

4. Select Configure New Team by typing option 3 from the mainmenu.



5. You will be asked to select a Team Type, as shown in the nextscreen. Type the chosen Team Type after the prompt. In thisexample, Team Type 2 is used (Load Balanced).

6. Once the NIC team has been created, you will notice the newvirtual port created under Network Connections, as shown in thenext screen. You can change the interface name to a moremeaningful name, such as "NIC team (vPC)," as shown.

7. (Optional) Configure the newly created interface with IP address,subnet mask, and default gateway.


408


8. Verify that the VN_Ports are showing online in the SAN Surfer FCHBA Manager. The next screenshot shows an example of theVN_Ports showing online in the SANsurfer FC HBA Manager.

Zoning Before zoning, ensure that both the FCoE initiator and target havesuccessfully performed FLOGI to the switch.

Use the show flogi database switch command to check the fabriclogin information. See the command outputs from both switchesshown in the following examples.

In this example, the vfc3 port on Nexus-5010-1 is connected to theCNA1 port while vfc13 port on Nexus-5010-2 is connected to theCNA2 port. Vfc19 is used on both switches to connect FCoE targetdevices. Refer to Figure 63 on page 378 for more information aboutthe topology used in this example.

Nexus-5010-1# show flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------vfc3 80 0xa70000 21:00:00:c0:dd:10:28:bb 20:00:00:c0:dd:10:28:bbvfc19 80 0xa70001 50:06:01:65:3b:64:03:c4 50:06:01:60:bb:60:03:c4


Nexus-5010-2# show flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------vfc13 60 0xd80000 21:00:00:c0:dd:10:28:b9 20:00:00:c0:dd:10:28:b9vfc19 60 0xd80001 50:06:01:6d:3b:64:03:c4 50:06:01:60:bb:60:03:c4




To perform zoning on both switches, complete the following steps.

1. Use the following CLI commands to perform zoning on Nexus5000 switches.

• On the Nexus-5020-1:

Nexus-5010-1# config terminalEnter configuration commands, one per line. End with CNTL/Z.Nexus-5010-1(config)# zone name nx1vpc_cna1_spa3-1 vsan 80Nexus-5010-1(config-zone)# member pwwn 21:00:00:c0:dd:10:28:bbNexus-5010-1(config-zone)# member pwwn 50:06:01:65:3b:64:03:c4Nexus-5010-1(config-zone)# exitNexus-5010-1(config)# zoneset name zoneset_80 vsan 80Nexus-5010-1(config-zoneset)# member nx1vpc_cna1_spa3-1Nexus-5010-1(config-zoneset)# exitNexus-5010-1(config)# zoneset activate name zoneset_80 vsan 80Zoneset activation initiated. check zone statusNexus-5010-1(config)# exitNexus-5010-1# show zoneset active vsan 80zoneset name zoneset_80 vsan 80zone name nx1vpc_cna1_spa3-1 vsan 80* fcid 0xa70000 [pwwn 21:00:00:c0:dd:10:28:bb]* fcid 0xa70001 [pwwn 50:06:01:65:3b:64:03:c4]

• On the Nexus-5020-2:

Nexus-5010-2# config terminalEnter configuration commands, one per line. End with CNTL/Z.Nexus-5010-2(config)# zone name nx2vpc_cna2_spb3-1 vsan 60Nexus-5010-2(config-zone)# member pwwn 21:00:00:c0:dd:10:28:b9Nexus-5010-2(config-zone)# member pwwn 50:06:01:6d:3b:64:03:c4Nexus-5010-2(config-zone)# exitNexus-5010-2(config)# zoneset name zoneset_60 vsan 60Nexus-5010-2(config-zoneset)# member nx2vpc_cna2_spb3-1Nexus-5010-2(config-zoneset)# exitNexus-5010-2(config)# zoneset activate name zoneset_60 vsan 60Zoneset activation initiated. check zone statusNexus-5010-2(config)# exitNexus-5010-2# show zoneset active vsan 60zoneset name zoneset_60 vsan 60zone name nx2vpc_cna2_spb3-1 vsan 60* fcid 0xd80000 [pwwn 21:00:00:c0:dd:10:28:b9]* fcid 0xd80001 [pwwn 50:06:01:6d:3b:64:03:c4]

2. Verify the zoning using the show zoneset active vsan <vsannumber> CLI command.


410


Note: If you encounter any FCoE connectivity issues, refer to the”Troubleshooting Basic FCoE and CEE Problems and Case Studies" chapterin the Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Conceptsand Protocols TechBook at http://elabnavigator.EMC.com, under the TopologyResource Center tab.

Configuring the FCoEtarget

This example uses an EMC CLARiiON storage array with EMCUnisphere software as an FCoE target device. Unisphere is amanagement interface for managing VNX series, CLARiiON, andCelerra systems. Unisphere provides customizable dashboard viewsand reporting capabilities that present users with valuableinformation in the context of how they manage storage. An exampleof the Unisphere interface is shown in Figure 65.

Figure 65 Unisphere navigation and user interface

More information about Unisphere can be found at EMC OnlineSupport at https://support.emc.com.

To configure an FCoE target, complete the following steps.




1. Ensure that the PWWNs of CNAs from the vPC host are loggedin. If the host's PWWNs are not logged in, refer to the”Troubleshooting Basic FCoE and CEE Problems and CaseStudies" chapter in the Fibre Channel over Ethernet (FCoE), DataCenter Bridging (DCB) Concepts and Protocols TechBook athttp://elabnavigator.EMC.com, under the Topology ResourceCenter tab to troubleshoot this problem.

2. After confirming that the PWWNs of the CNAs are logged in, useUnisphere to register the PWWN of CNA1 and CNA2 from thehost.

a. Click the System tab on the main Unisphere pane.

b. From the left-hand Task Pane, click System Management >Storage System Connectivity Status.

The Connectivity Status window displays.

c. Select the CNA's PWWN from the list.

d. Click Register.



412


The Register Initiator Record dialog box displays.

e. Enter the Initiator Type and Failover Mode.

f. Type the Host Name and IP address of the FCoE initiator.

g. Click OK once you complete the information.

3. Repeat Step 1 and Step 2 for the CNA2 PWWN. Register it underthe same Host Name.

Once the registration of the FCoE initiator is completed, you willsee them "registered" under one host in the Connectivity Statuswindow, as shown below.

4. Create a Storage Group in the FCoE target.



a. Go to the Storage tab on the main Unisphere pane.

b. In the Storage Groups pane, select the Storage Group from thedrop-down menu.

c. Type the Storage Group name you want, then click Create.

5. Assign the host to the newly created Storage Group by clickingConnect Hosts.


414



a. Use the right-arrow button to select the host.

b. Click OK.



Once the host has been assigned, the host name will appear in theHost tab in the Details pane, as shown next.

6. To perform LUN Masking, select the Connect LUNs tab from theStorage Group screen.


416



a. In the Available LUNs pane, select the desired LUNs ordevices you want to add.

b. Click Add.

c. Click OK.

Verifying FCoE LUNs After completing the steps from “Configuring NIC Teaming” onpage 406, “Zoning” on page 408, and “Configuring the FCoE target”on page 410, it is time to verify and validate. Disk managementutilities can be used to verify that FCoE LUNs have been assigned tothe host. In this example, EMC PowerPath is used to validate theavailable paths from the FCoE initiator to FCoE target.

Complete the following steps to ensure that FCoE configuration hasbeen successfully completed. This example uses a CLARiiON system.



1. From the host, verify the list of FCoE LUNs or devices using thediskpart or inq tool. As shown below, two LUNs are presented bythe FCoE target to the host.

2. Verify the alternate paths to the LUNS.

Multipathing software, such as EMC PowerPath, can be used toverify the alternate paths to your LUNs. In the next screenshotyou will see that there are two paths from the host to theCLARiiON storage array. Paths from CNA1 to CLARiiON portSP-A13 and CNA2 to CLARiiON port SP-B13 are shown in thisfigure.

Note: For more information about EMC PowerPath, refer to EMCPowerPath and PowerPath/VE for Microsoft Windows located at EMC OnlineSupport at https://support.emc.com.


418


3. From your host, access the Computer Management window,choose Storage > Disk Management in the task pane, as shownnext. Use the Disk Management tool to initialize, partition, andformat the disk.


4

This chapter provides basic information, supported features,topologies, and detailed setup steps for Blade Server solutions.

◆ Introduction ...................................................................................... 420◆ Cisco UCS (B-Series) ........................................................................ 426◆ IBM BladeCenter .............................................................................. 599◆ HP Virtual Connect FlexFabric and Flex-10 ................................. 654

Note: More detailed Fibre Channel over Ethernet information can be found inthe Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts andProtocols TechBook at http://elabnavigator.EMC.com. Information in thisTechBook includes FCoE and Ethernet basics, EMC storage, RecoverPointand Celerra MPFS as solutions in an FCoE environment, and troubleshootingbasic FCoE and DCB problems.

Blade Server SolutionsSetup Examples

Blade Server Solutions Setup Examples 419


420

Blade Server Solutions Setup Examples

IntroductionThis chapter provides an introduction to the Converged 10 GbE BladeServer as well as the various Blade Server FCoE solutions currentlyavailable and gives details on prerequisites, supported interfacetypes, and management capabilities for each. Detailed information onintegrating these FCoE modules into existing infrastructure is alsoprovided.

A Blade Server is a server chassis housing a number of individualminimally-packaged computer motherboard blades, each including:one or more processors, memory, storage (HDD), integrated networkcontrollers, an optional Fibre Channel host bus adapter (HBA), andother input/output (I/O) ports. All blades share a common powersupply and air-cooling resources.

The motivation behind the evolution of blade servers is to allow moreprocessing power in less rack space, simplifying cabling andreducing power consumption.

For more information about Blade servers, refer to the "Blade Servers"chapter in the Non-EMC SAN Products Data Reference Manual,available through the E-Lab Interoperability Navigator, TopologyResource Center tab, at http://elabnavigator.EMC.com.

This section contains basic information on the following:

◆ “Cisco UCS” on page 421

◆ “IBM BladeCenter” on page 423




Cisco UCSCisco unified computing system (UCS), shown in Figure 66, is a datacenter platform offering that unites computing, lossless 10 GbEthernet network, storage access, and virtualization into one system.

Figure 66 Cisco Unified Computer System (UCS)

Cisco UCS offers two server types:

◆ B-Series (Blade Servers)

◆ C-Series (Chassis/Rack-Mount Servers)

This section provides information on the Cisco UCS B-Series, referredto as UCS in this section.

UCS setup includes a pair of Fabric Interconnects and one or moreserver chassis. Each server chassis contains Blade Servers, fabricextender modules, power modules, and fan modules. Two 6x00 UCSswitches form a cluster and provide high availability for data path, aswell as management of the complete UCS system. Unified computingsystem manager (UCSM) provides single management interface tomanage each component of this system.

UCS offers hardware virtualization by creating server profiles, whichallows the virtual servers to be moved easily to a different physicalserver within the same UCS system.

UCS switches can operate in two different FC modes:

◆ End host mode ((N_Port virtualization)

This mode allows the switch to be connected to any NPIV-capableFibre channel switch.

Half slot PROCESSING_NODE

SFF disk drive

Physical KVM access

Power supply

ICO-IMG-000962

Fabric Interconnect

Fabric Interconnect

Introduction 421

422


◆ Switching Mode

Switching mode allows FC and FCoE storage to be directlyconnected to the switch. FC and FCoE storage may only beconnected to the UCS FI module in the following situations:

• Using UCS firmware version 2.1.

• Using pre-UCS firmware version 2.1 and the FI module isconnected to an MDS switch via an uplink port. The MDSswitch is needed in order to provide the ability to create andmodify FC zones.

Table 13 shows the connections supported in each mode of operation.Ethernet and FC modes are independent of each other.

Note the following recommendations and restriction:

◆ Clustered configuration of UCS Fabric Interconnects isrecommended for high availability.

Note: UCS switches must be in the same mode (NPV or switching) toform a cluster.

◆ Configurations that require the storage to be directly connected tothe UCS FI module must be either running firmware version 2.1,or the FI module must be connected to an MDS switch via anuplink port. The MDS switch is needed to provide the ability tocreate and modify FC zones.

◆ Default zoning is not supported.

Table 13 Supported switch modes and port types

FI operational mode Serverport

Ethernet uplink FC uplink FC storage FCoE storage Appliance

Ethernet: End Host ModeFC: End Host Mode

Yes Yes Yes No No Yes

Ethernet: End Host ModeFC: Switching

Yes Yes Yes Yes Yes Yes

Ethernet: SwitchingFC: End Host Mode

Yes Yes Yes No No Yes

Ethernet: SwitchingFC: Switching

Yes Yes Yes Yes Yes Yes



IBM BladeCenterThis section provides basic information on the following:

◆ “Brocade Converged 10 GbE Switch Module” on page 423

◆ “Cisco Nexus 4000” on page 424

Brocade Converged 10 GbE Switch ModuleThe Converged Ethernet solution by Brocade for IBM BladeCenterprovides fourteen internal 10 Gb CEE ports for blade serverconnectivity, eight external 10 Gb Ethernet CEE ports for LANconnectivity, and eight external 8 Gb Fibre Channel ports for storageand SAN connectivity.

Figure 67 shows an example of this switch.

Figure 67 Brocade Converged 10 GbE Switch Module (Brocade 8470)

With the base model Converged 10 GbE Switch Module, you canenable 16 of the 30 ports on the switch (Eight 10 Gb CEE externalports, eight 8 Gb FC external ports, and 14x 10 Gb CEE internalports). If you purchase the Port Upgrade Key, you can enable all 30ports on the Switch Module and use the full switch bandwidth.

Table 14 lists the part numbers for ordering these modules:

Table 14 Brocade and IBM Converged Module part numbers

Product name IBM part number Brocade part number

Brocade Converged 10GbE SwitchModule for IBM BladeCenter

69Y19098470

Brocade Converged 10GbE Switch PortUpgrade for IBM BladeCenter

69Y1917

Introduction 423

424


Figure 68 shows the rear view of IBM H series chassis. Each Brocade8470 module occupies two slots. One module occupies slots 7 and 8and the other module occupies slots 9 and 10.

Figure 68 IBM H series chassis, rear view

Cisco Nexus 4000


The Cisco Nexus 4000 is a 20-port, 10 G FCoE aware Ethernet switchmodule intended for use in IBM BladeCenter and is fully compliantwith the IEEE 802.1 Data Center Bridging (DCB) specification. Thisblade switch is also referred to as the Nexus 4001I Switch Module.

The term FCoE aware indicates that the switch is capable ofperforming FIP snooping and supports the creation of dynamicACLs. It also indicates that the Nexus 4000 does not contain an FCF(Fibre Channel Forwarder) and as a result FCoE frames received from

High-Speed Switch Module Bay 7




Mgm

t Module B

ay 1

Sw

itch Module B

ay 3B

ridge Module B

ay 3

Mgm

t Module B

ay 2

Bridge M

odule Bay 1

Bridge M

odule Bay 2

Sw

itch Module B

ay 1S

witch M

odule Bay 2

Sw

itch Module B

ay 4B

ridge Module B

ay 4

Power 2 Power 1

Serial Ports

Blower 1

Blower 2

ICO-IMG-000948



CNAs connected to the Nexus 4000 that are destined to a native FCdevice must be forwarded to a switch that does contain an FCF beforethe FC frame can be de-encapsulated and forwarded onto the SAN.Because the CNAs are not directly connected to a switch that containsan FCF, the resulting topology is actually referred to as a DCB Cloud.

Due to the nature of a DCB Cloud topology, some of the setup stepswill differ from steps used in a direct connect environment.

Note: For more information on direct connect and DCB Cloud, refer to the“FCoE Initialization Protocol (FIP)” section in the Fibre Channel over Ethernet(FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook athttp://elabnavigator.EMC.com, under the Topology Resource Center tab.

Fourteen of the 10 G ports are used to connect to the server bladesand six of the ports are available to connect to the Nexus 5020.

The six external interfaces can all be bundled into a port channel.

The Nexus 4000 can be managed via Telnet.

Introduction 425


426


Cisco UCS (B-Series)This section contains the following information on the Cisco UCS(B-Series):

◆ “Cisco UCS supported features and topology” on page 426

◆ “UCS fabric interconnect initial configuration” on page 432

◆ “Cisco UCS Fibre Channel Switch Mode configuration example”on page 434

◆ “Cisco UCS Fibre Channel End Host mode configurationexample” on page 452

◆ “Storage provisioning” on page 470

◆ “Directly attaching EMC storage to Cisco UCS FabricInterconnect” on page 477

◆ “Storage provisioning” on page 470

◆ “Setting up and implementing VE_Port on Cisco UCS” onpage 519

◆ “Setting up and implementing VF_Port on Cisco UCS” onpage 545

◆ “Configuring UCS 10G iSCSI using VNX-2 arrays” on page 574

Cisco UCS supported features and topologyThe supported features and topologies have been broken down intofour areas: Management, FCoE interface, Ethernet uplink interface,and FC interface, as shown in Figure 69 on page 427, and arediscussed further in this section.

◆ “Prerequisite” on page 427





◆ “Ethernet end-device interface ” on page 431

Figure 69 is for clarification purposes only and shows all the interfacetypes possible from the Fabric Interconnect, independent of the modeof operation.



Actual reference topologies are provided in the following sections.


Prerequisite FCoE requires the use of specialized host adapters, referred to as aConverged Network Adapter (CNA), and specialized switchhardware known as a Fibre Channel Forwarder (FCF). CNAs areinstalled in the host and then attached to an Ethernet port thatprovides FCF functionality. In UCS, the connection between CNAports and the UCS switch port is through the fabric extenders, asshown in Figure 70 on page 428, which shows the front of the FabricInterconnects and the rear of the UCS chassis.

Cisco UCS (B-Series) 427

428


Figure 70 Connection between Fabric Extender and Fabric Interconnect

Currently supported CNAs are Cisco (Virtual interface cards, or VIC),Emulex, and QLogic.

CNA prerequisitesAll the UCS server blades support PCIe slots. Refer to the EMCSupport Matrix for information on the supported host platforms andCNAs.

FCF prerequisites

The UCS 6120, 6140, and 6248UP switches require:

◆ 1U, 2U, and 1U of rack space respectively. EMC recommendsthese be placed at the top of the rack in which the UCS chassis isinstalled.

◆ Two or four power drops, respectively, of either 110 V or 220 V.

◆ An IP address for management purposes; two for fabricinterconnects and one for cluster manager.

◆ Two network ports to which to connect the switch managementports.

◆ A customer-supplied password for the admin user.

ICO-IMG-000971





Laptop prerequisites

Note: The laptop used in this example is needed only to initially configurethe UCS and will be disconnected after the installation is complete.

◆ Communication port with a DB-9 connector.

◆ Download and install PuTTY.

Note: Installing PuTTY is recommended, but not mandatory.

◆ Ensure that HyperTerminal is installed.

Note: The operating system used for this example was Windows 2000Professional and HyperTerminal was installed by default. Some of the laterWindows-based operating systems do not have HyperTerminal installed.This will need to be done before attempting to configure the UCS switches.See the help feature for the particular version of Windows you are using formore information on installing HyperTerminal.

The following console port parameters are to be used for connection:

◆ 9600 baud

◆ 8 data bits

◆ No parity

◆ 1 stop bit

Management UCS system can be managed with following:

◆ UCS Manager (Recommended)

◆ Telnet and SSH

FCoE interface The following interfaces are supported:

◆ FCoE storage directly connected to the UCS 10 Gb Ethernet portsis supported starting with 1.4 (1m) firmware release when UCSFabric Interconnect is in FC switch mode and connected to aCisco FC switch (MDS or Nexus) for pre-UCS 2.1 software.

Note: Beginning with UCS 2.1 software, direct-attach FC/FCoE storageto Fabric InterConnect (FI) is supported.

◆ Hardware-based FCoE CNAs from Cisco, Emulex, and QLogic


430


Note: Software-based FCoE CNAs from Intel and Broadcom are not yetsupported by EMC with UCS. Refer to the EMC Support Matrix for the mostup-to-date support information.

Ethernet uplinkinterface


Layer 2

◆ Layer 2 interconnect ports and VLAN trunks


◆ Support for up to 1024 VLANs and virtual SANs (VSANs) perinterconnect

◆ Rapid Per-VLAN Spanning Tree Plus (PVRST+)

◆ Internet Group Management Protocol (IGMP) Versions 1, 2, and 3snooping

◆ Cisco EtherChannel technology


◆ Advanced EtherChannel hashing based on Layer 2, 3, and 4information

◆ Jumbo frames on all ports (up to 9216 bytes)


QoS

◆ Layer 2 IEEE 802.1p (class of service [CoS])

◆ Eight hardware queues per port

◆ Per-port QoS configuration

◆ CoS trust

◆ Per-port virtual output queuing

◆ CoS-based egress queuing

◆ Egress strict-priority queuing

◆ Egress port-based scheduling: Weighted Round-Robin (WRR)




FC interface The following are supported:

IMPORTANT

The UCS switches can be configured to run in NPV mode toconnect to either Brocade (FOS or EOS) or MDS products, but not atthe same time in the same VSAN.

◆ NPV — Brocade and Cisco

• Minimum number of NP_Ports per UCS switch is 1• Maximum distance is shortwave• F_Port trunking and channeling when connected to Cisco

NPIV capable switches

Note: Longwave is currently not supported

◆ FC-SW — Cisco MDS and Nexus only

• Maximum number of ISLs per UCS 6x00 is 8• Max domains is per the EMC Support Matrix• Maximum distance is shortwave

Note: Longwave is currently not supported

• Max hops is per the EMC Support Matrix• Interopmode is native• ISL port channels (link aggregation) is supported• ISL Trunking (multiple VSANs) is supported• Scalability

– Same as MDS (except VSAN up to 32)• HBA and FC storage directly connect ed to the UCS FC ports

are supported starting with 1.4 (1m) firmware release when inswitch mode


Ethernet end-deviceinterface



◆ Celerra 10 GbE (iSCSI)

◆ Symmetrix and CLARiiON iSCSI interfaces





432


UCS fabric interconnect initial configurationSet the IP address for fabric interconnects and cluster manager.

Before connecting the system to the network, configure IP address formanagement interface of each fabric interconnects and clustermanager.

1. To create and configure the UCS fabric interconnect switches as acluster, make sure the L1 ports on both the switches are directlyconnected to each other as shown in the following figure. Do thesame for L2 ports.

2. Connect one end of the console cable to the laptop and the otherto one of the UCS switches.

3. Power on the switch.

4. When the unconfigured system boots, it prompts you for thesetup method to be used. Enter Console to continue the initialsetup using the console CLI.

Enter the installation method (console/gui)? Console

5. Enter setup to continue as an initial system setup and then press yto continue.

Enter the setup mode (restore from backup or initial setup) [restore/setup]? SetupYou have chosen to setup a new switch. Continue? (y/n): yEnforce strong password? (y/n): y

Fabric Interconnect A

Fabric Interconnect B

M1

M2

M1

M2ICO-IMG-000972

ManagementNetwork

L1

L2



6. Enter the password for the admin account. Re-enter the passwordfor confirmation. Type yes and press Enter to continue the clusterconfiguration.

Enter the password for "admin": Admin12345Confirm the password for "admin": Admin12345Do you want to create a new cluster on this switch (select 'no' for standalone

setup or if you want this switch to be added to an existing cluster)? (yes/no)[n]: yes

7. Select the fabric interconnect to configure and then enter the IPaddress, gateway, etc., as shown below.

Enter the switch fabric (A/B): AEnter the system name: eLabUCSMgmt0 IPv4 address: 10.246.46.61Mgmt0 IPv4 netmask: 255.255.255.0IPv4 address of the default gateway: 10.246.46.1Cluster IPv4 address : 10.246.46.63Configure the DNS Server IPv4 address? (yes/no) [n]: yesDNS IPv4 address: 10.254.66.23

Configure the default domain name? (yes/no) [n]: yesDefault domain name: lss.emc.com

8. Verify the information entered and to be configured and thenenter yes and press Enter.

Following configurations will be applied:Switch Fabric=ASystem Name=eLabUCSManagement IP Address=10.246.46.61Management IP Netmask=255.255.255.0Default Gateway=10.246.46.1Cluster Enabled=yesCluster Ip Address=10.246.46.63DNS Server=10.254.66.23Domain Name=lss.emc.com

Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no):yes

9. Move the console connections to the second UCS switch andpower it on.

10. When the unconfigured system boots, it prompts you for thesetup method to be used. Enter Console to continue the initialsetup using the console CLI. Then, enter yes and press Enter toadd the second switch to cluster.

Enter the installation method (console/gui)? consoleInstaller has detected the presence of a peer switch. This switch will be added

to the cluster. Continue?[y/n] y


434


11. Enter the password set for the primary switch and then set the IPaddress and save the configuration.

Enter the admin password of the peer Fabric interconnect: Admin12345Retrieving config from peer Fabric interconect... doneApply and save configuration (select 'no' if you want to re-enter)? (yes/no): yesPeer Fabric interconnect Mgmt0 IP Address: 10.246.46.61Peer Fabric interconnect Mgmt0 IP Netmask: 255.255.255.0Cluster IP address : 10.246.46.63Physical Switch Mgmt0 IPv4 address: 10.246.46.62

Cisco UCS Fibre Channel Switch Mode configuration exampleThis section describes how to configure the Cisco UCS Fibre ChannelSwitching Mode, as shown in Figure 72 on page 437, and contains thefollowing information:




◆ “Configuring UCS B-Series” on page 437

Summary of configuration stepsThe configuration steps are summarized as follows:

◆ “On the UCS” on page 434

◆ “On the MDS switch” on page 435

◆ “On the Fabric” on page 435

Configuration for both UCS Fabric interconnects is done using theUCS Manager GUI. For Cisco MDS switches, this configuration isdone using CLI in this example. It can also be done using DeviceManager.

On the UCS 1. Set the IP address for Fabric Interconnects and the clustermanager.

2. Change the mode on Fabric interconnects to Switching Mode.

3. Enable port channelling and Trunking.

4. Create VSAN in SAN cloud.

5. Create VSAN in storage cloud.

6. Create SAN port channel and add interfaces to it.



7. Enable port channel.

8. Assign VSAN to vHBAs of each server to be used.

9. Assign VSAN to storage FC and FCoE interfaces.

On the MDS switch 1. Telnet and log in to the MDS switch.

2. Create VSAN.

3. Configure FC interfaces.

4. Create port channel.

5. Create channel group.

6. Save configuration changes to startup configuration file.

7. Connect cables between UCS switch port and FC switch port.

8. Enable interfaces.

9. Verify the port channel connection.

On the Fabric 1. Perform zoning.

2. Provision storage from the CLARiiON array.

3. Provision storage using initiator groups (ACLX) on SymmetrixVMAX.

Existing topologyThe existing topology is shown inFigure 71 on page 436. Since thiscase study adds connections to the SAN, FCoE, and FC direct-attachstorage to an existing Ethernet environment, only those changesrequired to add the Unified computing system will be shown. For thecomplete configuration process for FC fabric, refer to the FibreChannel SAN Topologies TechBook, are available through the E-LabInteroperability Navigator, Topology Resource Center tab, athttp://elabnavigator.EMC.com.

For Cisco Nexus 7000 configuration information, refer tohttp://www.cisco.com/cisco/web/psa/configure.html.

More details are provided in the Cisco Nexus 7000 Series NX-OSInterfaces Configuration Guide, located at www.cisco.com.



436


Figure 71 Existing topology

Target topologyThe example shown in Figure 72 on page 437, uses CLARiiON VNXFCoE ports and Symmetrix VMAX FC ports as an example. All of thefollowing EMC storage arrays and storage types are supported:

◆ CLARiiON CX/VNX series arrays with FC and FCoE ports

◆ Symmetrix DMX series arrays with FC Ports

◆ Symmetrix VMAX series arrays with FC and FCoE ports

MDS-9148Domain 1

10.246.54.111

MDS-9148Domain 2

10.246.54.111

SAN-A

6140-AIP = 10.246.46.61

6140-BIP = 10.246.46.62

1/21/1

1/271/2

1/3

1/4

SERVER 3:IP: 10.246.59.21

CNA_1:WWPN = 20:00:00:25:B5:C2:B3:00

CNA_2:WWPN = 20:00:00:25:B5:C2:B3:01

VNX FC storage ports

Port3: 50:06:01:60:46:e0:01:49Port4: 50:06:01:68:46:e0:01:49

VMAX FC storage ports

Port5: 50:00:09:72:08:2f:b9:18Port6: 50:00:09:72:08:2f:b9:19

UCS Fabric interconnect - A

SERVER 1:IP: 10.246.59.20

CNA_1:WWPN= 20:00:00:25:B5:C2:B1:00

CNA_2:WWPN = 20:00:00:25:B5:C2:B1:01

10GbE

4 or 8 Gb/s FC

Core-N7K-1

Port-channel 1

Port-channel 2

1/1 2/1 2/11/1

Core-N7K-2

Ser

ver

1

Ser

ver

3

T1/3

T1/4

T1/5

T1/6

T1/3

T1/4

T1/5

T1/6

T T TT T T TT

Cisco UCS blade server chassis

1 2 1 2

Fabric extender-1

Fabric extender-2

UCS Fabric Interconnect - B

1/1 1/1 1/21/2T T T T

FC StorageFC Storage

SAN-B

5 63 4

1/4

1/3

1/37 1/37

ICO-IMG-000982



Figure 72 shows the target topology.


Configuring UCS B-SeriesTo configure the UCS B-Series, complete the following steps:

Note: Step 1 is required for unconfigured Fabric Interconnects, if the switchesare already configured go directly to Step 1. Setting up Fabric Interconnectsalso includes configuring the server interfaces, creating service profiles andvHBAs, which is out of scope for this document. Refer to the Cisco UCSConfiguration Guide for the details.

MDS-9148Domain 1

10.246.54.111

MDS-9148Domain 2

10.246.54.113

FCoE Storage SAN-A

6140-AIP = 10.246.46.61

6140-BIP = 10.246.46.62

1/21/1

1/271/2

1/3

1/4

SERVER 3 (VSAN = 99):IP: 10.246.59.21

CNA_1:WWPN = 20:00:00:25:B5:C2:B3:00 Storage = Storage port 5 and 8

CNA_2:WWPN = 20:00:00:25:B5:C2:B3:01Storage = Storage port 6 and 7

1 2 5 6

VNX-1 FCoE storage ports in VSAN 98 (SAN-APort1: 50:06:01:60:3e:a4:05:24Port2: 50:06:01:68:3e:a4:05:24

VNX-2 FC storage ports in VSAN 98 (SAN-B)Port3: 50:06:01:60:46:e0:01:49Port4: 50:06:01:68:46:e0:01:49

Port5: 50:00:09:72:08:2f:b9:18Port6: 50:00:09:72:08:2f:b9:19

Port7: 50:00:09:72:08:2f:b9:58Port8: 50:00:09:72:08:2f:b9:59

UCS Fabric interconnect - A

SERVER 1 (VSAN = 98):IP: 10.246.59.20

CNA_1:WWPN= 20:00:00:25:B5:C2:B1:00Storage = Storage port 1 and 3

CNA_2:WWPN = 20:00:00:25:B5:C2:B1:01Storage = Storage port 2 and 4

VLAN/VSAN 99VLAN/VSAN 98

10GbE

4 or 8 Gb/s FC

Core-N7K-1

Port-channel 1

Port-channel 2

1/1 2/1 2/11/1

Core-N7K-2

Ser

ver

1

Ser

ver

3

T1/3

T1/4

T1/5

T1/6

T1/3

T1/4

T1/5

T1/6

T T TT T T TT


1 2 1 2

Fabric extender-1

Fabric extender-2

UCS Fabric Interconnect - B

1/1 1/1 1/21/2T T T T

Trunk and channel

FC Storage

SAN-B

7 83 4

1/4

1/3

1/37 1/37

FC Storage FC Storage

ICO-IMG-00098

2/1 2/2 2/1 2/2

3/5 3/5

VMAX-1 FC storage ports in VSAN 99 (SAN-A

VMAX-2 FC storage ports in VSAN 99 (SAN-A


438


1. Change the FC switching mode on the fabric interconnects toSwitching Mode.

In the Equipment tab, choose Equipment > Fabric Interconnects> Fabric Interconnect A, as shown in Figure 73 on page 438.

Note: This will cause both of the Fabric Interconnects to reboot,sequentially resulting in system wide downtime.

Figure 73 Change the mode

This mode allows storage to be directly attached to the UCSswitches. The default mode of the UCS switches is NPV mode.

a. Select the fabric interconnect A.



b. In the General tab, click Set FC Switching Mode on the leftside panel. A pop-up menu displays (not shown). Click OK.The main window displays again and the Set FC SwitchingMode button fades.

c. Click Save Changes to save these new settings.

2. Enable port channelling and Trunking features on FI.

Note: Enabling VSAN Trunking is a global setting. All VSANs will betrunked on all FC uplinks.

a. In the SAN tab, choose SAN > SAN Cloud > Fabric A, asshown in Figure 74.

Figure 74 Enable port channeling and Trunking


440


b. In the right pane, click Enable All Port Channels and EnableFC Uplink Trunking.

c. Click Save Changes.

d. Repeat Step a and Step b for Fabric B.


In the SAN tab, choose SAN > SAN Cloud > VSAN, as shown inFigure 75.

Figure 75 Create VSAN

a. Right-click VSAN and click Create VSAN on the pop-upmenu (not shown), which then opens a windows as shown inFigure 75.

b. In this window, fill in name, VSAN ID, and FCoE VLAN IDfields. In this setup, the VSAN and VLAN ID of 98 is used.

c. Similarly, create another VSAN 99 by following Step a andStep b above.



Default zoning is disabled by default for a new VSAN. If youare using existing VSANs, make sure 'default zoning' is turnedoff, as shown in Figure 76. Grayed out text means it isdisabled.

Note: EMC does not support default zoning.

Figure 76 Disable default zoning

4. Create VSAN in storage cloud.

In the SAN tab, choose SAN > Storage Cloud > VSAN, as shownin Figure 77 on page 442.


442


Figure 77 Create Storage VSAN

Navigate to Storage Cloud VSAN and do the following:

a. Right-click VSAN and click Create VSAN on the pop-upmenu (not shown), which then opens a windows as shown inFigure 77.

b. In this window, fill in name, VSAN ID, and FCoE VLAN IDfields. In this setup, the VSAN and VLAN ID of 98 is used.

c. Similarly, create another VSAN 99 by following Step a andStep b above.

If you are using existing VSANs, make sure 'default zoning' isturned off.

5. Create SAN port channel and add FC interfaces to it.



Note: Make sure FC interfaces to be connected to FC uplink switch areconfigured as Uplink Port (the default configuration).

a. In the SAN tab, choose SAN > SAN Cloud > Fabric A > FCPort channel, as show in Figure 78.

Figure 78 SAN tab

b. Right-click FC Port Channel and select Create Port Channel.

The Create Port Channel pop-up window displays, as shownin Figure 78.

c. Enter the channel ID and name. For this example, port channelnumber 11 is used.


444


d. Click Next, which opens the screen as shown in Figure 79.

Figure 79 Add Ports screen

e. In this new window, select the ports/interfaces from the leftpanel and move into the right panel. In this example, interfaceslot 2 interface 1 and slot 2 interface 2 is used to create achannel.

f. Click Finish and then save changes from the parent window.After this is saved, the port channel will display under the FCPort Channels tree.

g. Repeat Step b to Step e for Fabric B. In Step c, use port channelnumber 21. And in Step e, select interfaces fc2/2 and fc2/27.

6. Enable port channel.

a. In the SAN tab, choose SAN > SAN Cloud > Fabric A > FCPort-channel, as shown in Figure 80 on page 445.



Figure 80 Enable Port-Channel

b. Select port channel 11 and enable it by clicking Enable PortChannel in the right panel. Do the same for Fabric B to enableport channel 21.

Note: Links will remain in the down state until port channeling isconfigured on the uplink FC switch.

7. Assign VSAN to vHBAs of each server to be used.

a. In the Server tab, choose Servers > Service Profiles >Sub-Organizations > IOP > IOP_Blade1 > vHBAs > vHBA0,as shown in Figure 81 on page 446.


446


Figure 81 Assign VSAN to vHBAs

b. Click each vHBA under each server and assign it a VSAN byselecting it from the VSAN drop-down menu on the right sidepanel under Properties.

For this setup, VSAN 98 to vHBAs under IOP_Blade1 andVSAN 99 to vHBAs under IOP_Blade3 is assigned.

8. Assign VSAN to storage FC and FCoE interfaces.

a. In the SAN tab, choose SAN > Storage Cloud > Fabric A >Storage FCoE interfaces, as shown in Figure 82 on page 447.



Figure 82 Assign VSAN to storage FC and FCoE interfaces

b. Assign VSAN to each of the storage ports by selecting fromthe VSAN drop-down list. Do this for all the FCoE storageattached to UCS switches.

c. Do the same for FC storage by selecting Storage FC Interfaces.

In this example, interface 1/37 on both Fabric Interconnects toconnect FCoE storage is used. Similarly, use interfaces 2/3 onboth for FC storage.

On MDS switch Note: Setting up the switch from scratch is out of scope for this documentand not detailed here. Refer to the Cisco UCS Configuration Guide onCisco.com for setup instructions.

1. Telnet into the Cisco MDS switch 10.246.54.111 and log in.

2. Create VSANs.

Once logged in, enter the configuration mode and create VSANs98 and 99 in the VSAN database.


448


IOP054111-9134# conf tEnter configuration commands, one per line. End with CNTL/Z.IOP054111-9134(config)# vsan databaseIOP054111-9134(config-vsan-db)# vsan 98 name UCS_VSAN98IOP054111-9134(config-vsan-db)# vsan 99 name UCS_VSAN99IOP054111-9134(config-vsan-db)# exit


Configure switch interfaces fc1/1 and fc1/2 to have the sameparameters. Speed, mode (e.g., E, TE), trunk mode, port VSANand allowed VSAN must match for these interfaces to form achannel.

Note: Allow only the VSANs that you want to uplink to a switch. Thisprevents any undesired VSANs from UCS switches to merge into theSAN, and vice-versa.

IOP054111-9134(config)# interface fc1/1-2IOP054111-9134(config-if)# switchport speed autoIOP054111-9134(config-if)# switchport mode autoIOP054111-9134(config-if)# switchport trunk mode autoIOP054111-9134(config-if)# switchport trunk allowed vsan 1IOP054111-9134(config-if)# switchport trunk allowed vsan add 98IOP054111-9134(config-if)# switchport trunk allowed vsan add 99IOP054111-9134(config-if)# exit

4. Create a port channel.

Create port channel 10 and configure the parameter to match thatof interfaces fc1/1 and fc1/2. Allow VSAN 98, 99, and defaultVSAN 1.

IOP054111-9134(config)# interface port-channel 10IOP054111-9134(config-if)# switchport trunk mode autoIOP054111-9134(config-if)# switchport mode autoIOP054111-9134(config-if)# switchport speed autoIOP054111-9134(config-if)# switchport trunk allowed vsan 1IOP054111-9134(config-if)# switchport trunk allowed vsan add 98IOP054111-9134(config-if)# switchport trunk allowed vsan add 99IOP054111-9134(config-if)# channel mode activeIOP054111-9134(config-if)# exit

5. Create channel group,

Add interfaces fc1/1 and fc1/2 to channel group 10.

IOP054111-9134(config-if)# interface fc1/1,fc1/2IOP054111-9134(config-if)# channel-group 10fc1/1 fc1/2 added to port-channel 10 and disabledplease do the same operation on the switch at the other end of the port-channel,



then do "no shutdown" at both ends to bring it up6. Save configuration changes to a startup configuration file, and

then save the configuration to a startup file.

IMPORTANT

If this is not done, all changes made here will be lost after aswitch reboot.

IOP054111-9134# copy running-config startup-config[########################################] 100%

7. Connect cables between the UCS switch port and FC switch port.

Connect ports fc1/1 and fc1/2 of Fabric Interconnect A(10.246.46.61) to MDS switch (10.246.54.111) ports fc1/1 andfc1/2, respectively.

8. Enable interfaces on the MDS switch.

IOP054111-9134(config-if)# interface fc1/1-2IOP054111-9134(config-if)# no shut

9. Verify the port channel connection.

Once interfaces are up the port channel should come up. This canbe verified with the following command.

IOP054111-9134# show interface port-channel 10port-channel 10 is trunking (Not all VSANs UP on the trunk)

Hardware is Fibre ChannelPort WWN is 24:0a:00:05:73:50:1f:c0Admin port mode is auto, trunk mode is autosnmp link state traps are enabledPort mode is TEPort vsan is 98Speed is 4 GbpsTrunk vsans (admin allowed and active) (1,98-99)Trunk vsans (up) ()Trunk vsans (isolated) ()Trunk vsans (initializing) (1,98-99)5 minutes input rate 208 bits/sec, 26 bytes/sec, 0 frames/sec5 minutes output rate 296 bits/sec, 37 bytes/sec, 0 frames/sec

297796427116 frames input, 453117894225752 bytes0 discards, 1316 errors1316 CRC, 0 unknown class0 too long, 0 too short


275 input OLS, 180 LRR, 62 NOS, 0 loop inits231 output OLS, 24 LRR, 185 NOS, 208 loop inits


450


Member[1] : fc1/1Member[2] : fc1/2Interface last changed at Mon May 16 22:03:20 2011

IOP054111-9134#

10. Set up MDS switch 10.246.54.113 for connection to UCS FabricInterconnect B.

11. Follow Step 1 to Step 8 to setup the connection and create portchannel. Use IP address of this switch. And use interfaces fc1/2and fc1/27 to form a port channel.

On the Fabric 1. Create zoning on SAN-A for VSAN 98 and 99.

Note: Zoning can also be done using Fabric Manager.

a. Log in to the switch (10.246.54.111) using Telnet or SSH.

b. Enter configuration command and add zones for VSAN 98.

IOP054111-9134# conf tEnter configuration commands, one per line. End with CNTL/Z.IOP054111-9134(config)# zone name UCS_C2B100_VNX1_SPA vsan 98IOP054111-9134(config-zone)# member pwwn 20:00:00:25:B5:C2:B1:00IOP054111-9134(config-zone)# member pwwn 50:06:01:60:3e:a4:05:24IOP054111-9134(config-zone)# exitIOP054111-9134(config)# zone name UCS_C2B100_VNX2_SPA vsan 98IOP054111-9134(config-zone)# member pwwn 20:00:00:25:B5:C2:B1:00IOP054111-9134(config-zone)# member pwwn 50:06:01:60:46:e0:01:49IOP054111-9134(config-zone)# exit

c. Create zoneset for VSAN 98 and add zones created in Step b toit.

IOP054111-9134(config)# zoneset name SAN_A_UCS_VNX_C2B1 vsan 98IOP054111-9134(config-zoneset)# member UCS_C2B100_VNX1_SPAIOP054111-9134(config-zoneset)# member UCS_C2B100_VNX2_SPAIOP054111-9134(config-zoneset)# exit

d. Activate zoneset ‘SAN_A_UCS_VNX_C2B1’ on VSAN 98.

IOP054111-9134(config)# zoneset activate name SAN_A_UCS_VNX_C2B1 VSAN 98Zoneset activation initiated. check zone status

e. At the same configuration prompt, add zones for VSAN 99.

IOP054111-9134# conf tEnter configuration commands, one per line. End with CNTL/Z.IOP054111-9134(config)# zone name UCS_C2B300_VMAX1 _SPA vsan 99IOP054111-9134(config-zone)# member pwwn 20:00:00:25:B5:C2:B3:00IOP054111-9134(config-zone)# member pwwn 50:00:09:72:08:2f:b9:18



IOP054111-9134(config-zone)# exitIOP054111-9134(config)# zone name UCS_C2B300_VMAX2 vsan 99IOP054111-9134(config-zone)# member pwwn 20:00:00:25:B5:C2:B3:00IOP054111-9134(config-zone)# member pwwn 50:00:09:72:08:2f:b9:18IOP054111-9134(config-zone)# exit

f. Create zoneset for VSAN 99 and add zones created in Step e toit.

IOP054111-9134(config)# zoneset name SAN_A_UCS_VMAX_C2B3 vsan 99IOP054111-9134(config-zoneset)# member UCS_C2B300_VMAX1IOP054111-9134(config-zoneset)# member UCS_C2B300_VMAX2IOP054111-9134(config-zoneset)# exit

g. Activate zoneset ‘SAN_A_UCS_VMAX_C2B3’ on VSAN 99.

IOP054111-9134(config)# zoneset activate name SAN_A_UCS_VMAX_C2B3 VSAN 99Zoneset activation initiated. check zone status

2. Create zoning on SAN-B for VSAN 98 and 99.

Note: Zoning can also be done using Fabric Manager.

a. Log in to the switch (10.246.54.111) using Telnet or SSH.

b. Enter configuration command and add zones for VSAN 98.

IOP054111-9134# conf tEnter configuration commands, one per line. End with CNTL/Z.IOP054111-9134(config)# zone name UCS_C2B101_VNX1_SPB vsan 98IOP054111-9134(config-zone)# member pwwn 20:00:00:25:B5:C2:B1:01IOP054111-9134(config-zone)# member pwwn 50:06:01:68:3e:a4:05:24IOP054111-9134(config-zone)# exitIOP054111-9134(config)# zone name UCS_C2B101_VNX2_SPB vsan 98IOP054111-9134(config-zone)# member pwwn 20:00:00:25:B5:C2:B1:01IOP054111-9134(config-zone)# member pwwn 50:06:01:68:46:e0:01:49IOP054111-9134(config-zone)# exit

c. Create zoneset for VSAN 98 and add zones created in Step b toit.

IOP054111-9134(config)# zoneset name UCS_VNX_C2B1 vsan 98IOP054111-9134(config-zoneset)# member UCS_C2B101_VNX1_SPBIOP054111-9134(config-zoneset)# member UCS_C2B101_VNX2_SPBIOP054111-9134(config-zoneset)# exit

d. Activate zoneset 'SAN_B_UCS_VNX_C2B1' on VSAN 98.

IOP054111-9134(config)# zoneset activate name SAN_B_UCS_VNX_C2B1 VSAN 98Zoneset activation initiated. check zone status


452


e. At the same configuration prompt, add zones for VSAN 99.

IOP054111-9134# conf tEnter configuration commands, one per line. End with CNTL/Z.IOP054111-9134(config)# zone name UCS_C2B301_VMAX1 vsan 99IOP054111-9134(config-zone)# member pwwn 20:00:00:25:B5:C2:B3:01IOP054111-9134(config-zone)# member pwwn 50:00:09:72:08:2f:b9:19IOP054111-9134(config-zone)# exitIOP054111-9134(config)# zone name UCS_C2B301_VMAX2 vsan 99IOP054111-9134(config-zone)# member pwwn 20:00:00:25:B5:C2:B3:01IOP054111-9134(config-zone)# member pwwn 50:00:09:72:08:2f:b9:58IOP054111-9134(config-zone)# exit

f. Create zoneset for VSAN 99 and add zones created in Step e toit.

IOP054111-9134(config)# zoneset name SAN_B_UCS_VMAX_C2B3 vsan 99IOP054111-9134(config-zoneset)# member UCS_C2B300_VMAX1IOP054111-9134(config-zoneset)# member UCS_C2B300_VMAX2IOP054111-9134(config-zoneset)# exit

g. Activate zoneset 'SAN_B_UCS_VMAX_C2B3' on VSAN 99.

IOP054111-9134(config)# zoneset activate name SAN_B_UCS_VMAX_C2B3 VSAN 99Zoneset activation initiated. check zone status

3. Continue with the steps listed in the “Storage provisioning”section, “On a CLARiiON/VNX array” on page 470.

Cisco UCS Fibre Channel End Host mode configuration exampleThis section describes how to configure the Cisco UCS Fibre ChannelEnd Host Mode and contains the following information:




◆ “Configuring UCS B-Series” on page 456

Summary of configuration stepsThe configuration steps are summarized as follows:

◆ “On the UCS” on page 453

◆ “On the Brocade SAN (SAN-A)” on page 453

◆ “On the Cisco MDS (SAN-B)” on page 453

◆ “On the Cisco SAN (SAN-B)” on page 453



Configuration for both UCS Fabric interconnects is done using theUCS Manager GUI. For Cisco MDS switches, this configuration isdone using CLI in this example. It can also be done using DeviceManager and DCNM.

On the UCS 1. Set the IP address for Fabric Interconnects and the clustermanager.

2. Set Chassis discovery policy.

3. Change connectivity policy.

4. Create VSAN.

5. Assign VSAN to vHBA.

6. Assign VSAN to FC interfaces.

7. Enable F- Port channeling and Trunking.


9. Assign VSAN to port channel.

On the Brocade SAN(SAN-A)

1. Connect cables between the Brocade switch and UCS.

2. Perform zoning.

3. Provision storage using initiator groups (ACLX) on theSymmetrix VMAX.

On the Cisco MDS(SAN-B)

1. Telnet and log in to the MDS switch.

2. Create VSAN.

3. Enable feature fport-channel-trunk.



6. Assign VSAN to interfaces.

7. Create channel group.

8. Save configuration.

9. Connect cables between MDS switch and UCS.

On the Cisco SAN(SAN-B)

1. Perform zoning.

2. Provision storage using initiator groups (ACLX) on SymmetrixVMAX.


454


Existing topologyThe existing topology is shown inFigure 83 on page 455. Since thiscase study adds connections to the SAN, FCoE, and FC direct-attachstorage to an existing Ethernet environment, only those changesrequired to add the Unified computing system will be shown. For thecomplete configuration process for FC fabric, refer to the FibreChannel SAN Topologies TechBook, are available through the E-LabInteroperability Navigator, Topology Resource Center tab, athttp://elabnavigator.EMC.com.

For Cisco Nexus 7000 configuration information, refer tohttp://www.cisco.com/cisco/web/psa/configure.html.

More details are provided in the Cisco Nexus 7000 Series NX-OSInterfaces Configuration Guide, located at www.cisco.com.





Target topologyThe example shown in Figure 84 on page 456, uses Symmetrix VMAXFC ports as an example.

Note: Direct Connect is not supported when UCS switches are in End Hostmode (/EHM).

BrocadeDomain 1

10.246.54.192

MDS-9148Domain 2

10.246.54.113(VSAN 99)

FC Storage

SAN-A

1/21/1

1/21/1

1/3

10GbE

4 or 8 Gb/s FC

Core-N7K-1

Port-channel 1

Port-channel 2

1/1 2/1 2/11/1

Core-N7K-2

SAN-B

3 4

1/3

FI-A (6248UP)

SERVER 1:IP: 10.246.59.81

CNA_1:WWPN= 20:00:00:03:02:C2:B1:10

CNA_2:WWPN = 20:00:00:03:02:C2:B1:20

Ser

ver

1

Ser

ver

7

T T TT T T TT

1 2 1 2

IOM (2208XP) - 1

FI-B (6248UP)1/1 1/1 1/21/2

T T TT T T TT

T T TTT T TTT T TTT T TT

IOM (2208XP) - 1

8 links

1/3 1/11 1/3 1/11

2/3 2/9 2/3 2/9………... ………...


SERVER 7:IP: 10.246.59.82

CNA_1: WWPN = 20:00:00:03:02:C2:B7:08

CNA_2:WWPN = 20:00:00:03:02:C2:B7:18

6248-AIP = 10.246.79.31

6248-BIP = 10.246.79.32

Virtual (/cluster IP); 10.246.79.33

8 links

VMAX FC storage ports:Port1: 50:00:09:73:00:07:19:20Port2: 50:00:09:73:00:07:19:61

Cluster link


456


Figure 84 shows the target topology.


Configuring UCS B-SeriesTo configure the UCS B-Series, complete the following steps:

1. Set up the IP address for Fabric Interconnect and cluster.

Note: Refer to “UCS fabric interconnect initial configuration” onpage 432 for details.

2. Set the chassis discovery policy.

BrocadeDomain 1

10.246.54.192

MDS-9148Domain 2

10.246.54.113(VSAN = 99)

SAN-A

6248-AIP = 10.246.79.31

6248-BIP = 10.246.79.32

Virtual (/cluster IP); 10.246.79.33

1/11/2

1/11/2

SERVER 7:IP: 10.246.59.82

CNA_1 (VSAN = 99): WWPN = 20:00:00:03:02:C2:B7:08 Storage = Storage port 1

CNA_2 (VSAN = 99):WWPN = 20:00:00:03:02:C2:B7:18Storage = Storage port 2

VMAX FC storage ports:Port1: 50:00:09:73:00:07:19:20Port2: 50:00:09:73:00:07:19:61

FI-A (6248UP)

SERVER 1:IP: 10.246.59.81

CNA_1(VSAN = 99):WWPN= 20:00:00:03:02:C2:B1:10Storage = Storage port 1

CNA_2 (VSAN = 99):WWPN = 20:00:00:03:02:C2:B1:20Storage = Storage port 2

VLAN/VSAN = 99

10GbE

4 or 8 Gb/s FC

Core-N7K-1

Port-channel 1

Port-channel 2

1/1 2/1 2/11/1

Core-N7K-2

Ser

ver

1

Ser

ver

7

T T TT T T TT


1 2 1 2

IOM (2208XP) - 1

FI-B (6248UP)1/1 1/1 1/21/2

F-port Trunk and channel

SAN-B1 2

1/3

FC Storage

T T TT T T TT

T T TTT T TTT T TTT T TT

IOM (2208XP) - 1

8 links channel

1/3 1/11 1/3 1/11

2/3 2/9 2/3 2/9

Channel ………... ………...

1/3

Cluster link



a. In the Equipment tab, choose equipment and then in theright-hand panel select the Policies > Global Policies tab asshown in Figure 85.

Figure 85 Set chassis discovery policy

b. Change Chassis Discovery Policy changed Action: toPlatform max and Link Grouping Preference to PortChannel.

Note: Chassis Discovery Policy > Action default is 4. To be able touse all eight (8) links available on a new IOM, EMC recommends thisvalue to be set at Platform Max.

c. Save these settings.


458


3. Change connectivity policy, as shown in Figure 86.

Figure 86 Change connectivity policy

a. In the Equipment tab, choose Equipment > Chassis >(appropriate Chassis), as shown in Figure 86.

b. In the Connectivity Policy tab, change Admin State inchassis-conn-policy-chassis-2-fabric-A andchassis-conn-policy-chassis-2-fabric-B to Global. This willinherit the port-channel settings set in Step 2.

c. After saving this change, acknowledge the chassis by right-clicking the chassis in the left panel and then selectingAcknowledge chassis from the pop-up-menu (not shown).

d. Repeat Step a and Step b for each chassis to be used. (Only onechassis is used in this example.)




a. In the SAN tab, select SAN > VSANs, as shown in Figure 87.

Figure 87 Create VSAN in SAN cloud

b. Right-click VSAN and click Create VSAN on the pop-upmenu (not shown).

The Create VSAN window displays, as shown in Figure 88.

Figure 88 Create VSAN


460


c. In the Create VSAN window, fill in the Name, VSAN ID, andFCoE VLAN fields. In this setup, 99 is used in the VSAN andVLAN ID fields.

Note: Default zoning is disabled for a new VSAN. If you are usingexisting VSANs, make sure Default Zoning is Disabled.

Note: Ensure that the FCoE VLAN IDs are not the same as any of theVLAN IDs used for carrying Ethernet traffic.

5. Assign VSAN to vHBA.

a. Assign VSAN to vHBAs to be used in this setup. In theServers tab, choose Servers > Service Profiles >Sub-Organizations > IOP > server profile, as shown inFigure 89.

Figure 89 Assign VSAN to vHBA

b. In the left-hand panel, select the vHBA.

c. In the right-hand panel, Fabric ID field, select which FI youwant to connect to. In this case, FI-A.

d. Change VSAN to IOP_SAN99.

e. Save the settings.

f. Repeat Step a to Step dfor each vHBA.

6. Assign uplink FC interface to VSAN 99.



a. In the SAN tab select SAN > SAN Cloud > Fabric A > FCInterface 2/3 as shown in Figure 90.

Figure 90 Assign uplink FC interface

b. In the left-hand panel select the uplink interface to beconnected to uplink FC switch. This setup uses 2/3 and 2/9.

c. In the right- hand panel, General tab, change VSAN: field toIOP_VSAN(99).

d. Save the settings.

7. Enable F_Port channeling and trunking.

Note: Do not enable channels and trunks on FI-A since Cisco's F_Porttrunking and channeling are not supported with Brocade switches.

Note: Enabling VSAN Trunking is a global setting. All VSANs will betrunked on all FC uplinks.


462


a. In the SAN tab select SAN >SAN Cloud > Fabric x (B in thiscase), to enable port channeling and uplink trunking on eachFabric Interconnect, as shown in Figure 91.

Figure 91 Enable port channel and trunking

b. In the left-hand panel, General tab, select FabricInterconnect-B.

c. In the right-hand panel, click Enable All Port Channels.

d. Also click Enable FC Uplink Trunking.

e. Save the settings.

8. Create port channel on FI-B.



a. In the SAN tab select SAN >SAN Cloud > Fabric x >FC PortChannels, as shown in Figure 92.

Figure 92 Create FC port channels

b. Right-click FC Port Channels and select Create port channel.The Add Ports screen displays, as shown in Figure 93.

Figure 93 Add Ports


464


c. In the Ports column, select the ports you want to channel andmove them to the Ports in the port channel column.


9. Assign VSAN to port channel.

a. In the SAN tab assign VSAN to port channel, by selectingSAN > SAN Cloud > Fabric x > FC Port Channels > FCPort-Channel x, as shown in Figure 94.

Figure 94 Assign VSAN to port channel

b. In the left-hand panel, select the port channel just created (inthis case, FC Port-Channel 1).

c. In the right-hand panel, change VSAN: field to IOP_VSAN99.


e. In the left-hand pane, right-click the port channel again andclick Enable port channel.

On the Brocade SAN(SAN A)

This setup connects FI-A to the Brocade SAN. Only a one switchsetup is used in this example.

Note: Setting up the switch from scratch is out of scope for this document.Refer to the Brocade switch configuration guides on Brocade.com for setupinstructions.

1. Connect cables between UCS FI-A and the Brocade switch andverify the connectivity and server logins using followingcommand:

Brcd300_192:admin> switchshowswitchName: Brcd300_192switchType: 71.2



switchState: OnlineswitchMode: NativeswitchRole: PrincipalswitchDomain: 1switchId: fffc01switchWwn: 10:00:00:27:f8:50:eb:1azoning: OFFswitchBeacon: OFF

Index Port Address Media Speed State Proto==============================================

0 0 010000 -- N8 No_Module FC1 1 010100 -- N8 No_Module FC2 2 010200 -- N8 No_Module FC3 3 010300 -- N8 No_Module FC4 4 010400 -- N8 No_Module FC5 5 010500 -- N8 No_Module FC6 6 010600 -- N8 No_Module FC7 7 010700 -- N8 No_Module FC8 8 010800 id N8 No_Light FC9 9 010900 id N8 No_Light FC

10 10 010a00 id N8 No_Light FC11 11 010b00 id N2 Online FC F-Port 50:00:09:73:00:07:19:2012 12 010c00 id N2 Online FC F-Port 1 N Port + 1 NPIV public13 13 010d00 id N2 Online FC F-Port 1 N Port + 1 NPIV public14 14 010e00 id N8 No_Light FC15 15 010f00 id N8 No_Light FC16 16 011000 -- N8 No_Module FC17 17 011100 -- N8 No_Module FC18 18 011200 -- N8 No_Module FC19 19 011300 -- N8 No_Module FC20 20 011400 -- N8 No_Module FC21 21 011500 -- N8 No_Module FC22 22 011600 -- N8 No_Module FC23 23 011700 -- N8 No_Module FC

Brcd300_192:admin>

2. Create zones.

Create a zone for each initiator to give servers access to storage,using the following command:

Brcd300_192:admin> zonecreate "UCS_C2B110_Symm9E1","20:00:00:03:02:C2:B1:10;50:00:09:73:00:07:19:20"

Brcd300_192:admin> zonecreate "UCS_C2B708_Symm9E1","20:00:00:03:02:C2:B7:08;50:00:09:73:00:07:19:20"


466


3. Create and activate the zoneset.

Create a zoneset and add the two zones created in Step 2, usingthe following command:

Brcd300_192:admin> cfgcreate "UCS2x_FOS_setup", "UCS_C2B110_Symm9E1;UCS_C2B708_Symm9E1"

Brcd300_192:admin> cfgenable "UCS2x_FOS_setup"Do you want to enable 'UCS2x_FOS_setup' configuration (yes, y, no, n): [no] y


Save the zoning configurations to memory, using the followingcommand:

Brcd300_192:admin> cfgsaveDo you want to save Defined zoning configuration only? (yes, y, no, n): [no] y

On the Brocade SAN(SAN B)

This setup connects FI-B to Cisco MDS SAN. Only a one switch setupis used in this example.

Note: Zoning can also be done using Cisco DCNM.

1. Create zones on VSAN 99;

Create zones to give each server access to storage port, using thefollowing command:

IOP054113-9148# conf tEnter configuration commands, one per line. End with CNTL/Z.

IOP054113-9148(config)# zone name UCS_C2B120_Symm9f1 vsan 99IOP054113-9148(config-zone)# member pwwn 20:00:00:03:02:c2:b1:20IOP054113-9148(config-zone)# member pwwn 50:00:09:73:00:07:19:61IOP054113-9148(config-zone)# exit

IOP054113-9148(config)# zone name UCS_C2B718_Symm9f1 vsan 99IOP054113-9148(config-zone)# member pwwn 20:00:00:03:02:c2:b7:18IOP054113-9148(config-zone)# member pwwn 50:00:09:73:00:07:19:61IOP054113-9148(config-zone)# exit

2. Create zone set.

Create a zone set and add zone members to it, using the followingcommand:

IOP054113-9148(config)# zoneset name UCS2x_MDS_setup vsan 99IOP054113-9148(config-zoneset)# member UCS_C2B120_Symm9f1IOP054113-9148(config-zoneset)# member UCS_C2B718_Symm9f1IOP054113-9148(config-zoneset)# exit



3. Activate the zoneset.

Activate the zone set created in Step 2 using the followingcommand:

IOP054113-9148(config)# zoneset activate name UCS2x_MDS_setup vsan 99Zoneset activation initiated. check zone statusIOP054113-9148(config)# exit

4. Verify zoning.

Verify all the zoning information in active zoneset, usingfollowing command:

IOP054113-9148# show zoneset active vsan 99zoneset name UCS2x_MDS_setup vsan 99zone name UCS_C2B120_Symm9f1 vsan 99* fcid 0x4f0101 [pwwn 20:00:00:03:02:c2:b1:20]* fcid 0x4f0000 [pwwn 50:00:09:73:00:07:19:61]

zone name UCS_C2B718_Symm9f1 vsan 99* fcid 0x4f0102 [pwwn 20:00:00:03:02:c2:b7:18]* fcid 0x4f0000 [pwwn 50:00:09:73:00:07:19:61]

On the Cisco MDS(SAN B)

Note: Setting up the switch from scratch is out of scope for this document.Refer to the Cisco UCS configuration guide on Cisco.com for setupinstructions.

1. Telnet into the Cisco MDS switch 10.246.54.113 and log in.

2. Create VSAN.

Once logged in, enter the configuration mode and create VSAN99 in the VSAN database, using the following command:

IOP054113-9148# conf tEnter configuration commands, one per line. End with CNTL/Z.IOP054113-9148(config)# vsan databaseIOP054113-9148(config-vsan-db)# vsan 99 name UCS_VSAN99IOP054113-9148(config-vsan-db)# exit

3. Enable the fport-channel-trunk feature.

This feature is needed for the switch to recognize F_Port trunksand channels. Use the following command:

IOP054113-9148(config)# feature fport-channel-trunk



468


Configure switch interfaces fc1/1 and fc1/2 to have the sameparameters. Speed, mode, trunk mode, port VSAN, and allowedVSANs must match for these interfaces to form a channel.

IMPORTANT

Allow only the VSANs you want to uplink to a switch. Thisprevents any undesired VSANs from UCS switches frommerging into the SAN, and vice-versa.

IOP054113-9148(config)# interface fc1/1-2IOP054113-9148(config-if)# switchport speed autoIOP054113-9148(config-if)# switchport mode autoIOP054113-9148(config-if)# switchport trunk mode autoIOP054113-9148(config-if)# switchport trunk allowed vsan 1IOP054113-9148(config-if)# switchport trunk allowed vsan add 99IOP054113-9148(config-if)# exit

5. Create a port channel.

Create port channel 10 and configure the parameters to matchthat of interfaces fc1/1 and fc/2, using the following command.

IOP054113-9148(config)# interface port-channel 10IOP054113-9148(config-if)# switchport speed autoIOP054113-9148(config-if)# switchport mode autoIOP054113-9148(config-if)# switchport trunk mode autoIOP054113-9148(config-if)# switchport trunk allowed vsan 1IOP054113-9148(config-if)# switchport trunk allowed vsan add 99IOP054113-9148(config-if)# channel mode activeIOP054113-9148(config-if)# exit

6. Assign VSAN to interfaces.

Assign VSAN 99 to port connected to UCS and storage port. Alsoassign it to port channel, using the following command:

IOP054113-9148(config)# vsan databaseIOP054113-9148(config-vsan-db)# vsan 99 interface fc1/1-3IOP054113-9148(config-vsan-db)# vsan 99 interface port-channel 10IOP054113-9148(config-vsan-db)# exit

7. Channel group creation.

Add interfaces fc1/1 and fc1/2 to channel group 10 and bring upthe interfaces and port channel, using the following command:

IOP054113-9148(config)# interface fc1/1-2IOP054113-9148(config-if)# channel-group 10IOP054113-9148(config-if)# no shutIOP054113-9148(config-if)# interface port-channel 10



IOP054113-9148(config-if)# no shutIOP054113-9148(config-if)# exit

8. Save configuration changes to a startup configuration file, andthen save the configuration to a startup file.

IMPORTANT

This step must be followed. Otherwise, all changes made herewill be lost after a switch reboot.

IOP054113-9148# copy running-config startup-config

9. Connect cables between the UCS switch port and FC switch port.

Connect ports fc1/1 and fc1/2 of Fabric Interconnect B(10.246.79.32) to MDS switch (10.246.54.113) ports fc1/1 andfc1/2, respectively.

Also connect storage port to Cisco MDS port fc1/3.

10. Verify port channel connection.

Once interfaces are up, the port channel should come up. This canbe verified using following command:

IOP054113-9148# show interface port-channel 10port-channel 10 is up

Hardware is Fibre ChannelPort WWN is 24:0a:00:05:9b:e6:8a:40Admin port mode is auto, trunk mode is autosnmp link state traps are enabledPort mode is FPort vsan is 99Speed is 4 Gbps5 minutes input rate 224 bits/sec, 28 bytes/sec, 0 frames/sec5 minutes output rate 216 bits/sec, 27 bytes/sec, 0 frames/sec

80 frames input, 10484 bytes0 discards, 0 errors0 CRC, 0 unknown class0 too long, 0 too short


2 input OLS, 1 LRR, 2 NOS, 0 loop inits17 output OLS, 12 LRR, 2 NOS, 7 loop inits

Member[1] : fc1/1Member[2] : fc1/2Interface last changed at Thu Dec 8 14:19:51 2011


470


11. Verify all the devices log in correctly into the name server, usingthe following command:

IOP054113-9148# show fcns database vsan 99VSAN 99:--------------------------------------------------------------------------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE--------------------------------------------------------------------------0x4f0000 N 50:00:09:73:00:07:19:61 (EMC) scsi-fcp:target 2530x4f0100 N 24:01:54:7f:ee:0b:21:40 (Cisco) npv0x4f0101 N 20:00:00:03:02:c2:b1:20 scsi-fcp:init fc-gs0x4f0102 N 20:00:00:03:02:c2:b7:18 scsi-fcp:init fc-gs

Total number of entries = 4

Storage provisioningThis section contains the following storage provisioning information:

◆ “On a CLARiiON/VNX array” on page 470

◆ “On a Symmetrix VMAX array using initiator groups (ACLX)” onpage 472

On a CLARiiON/VNX arrayTo provision storage from the CLARiiON array, complete thefollowing steps.

1. Log in to Unisphere and select the CLARiiON system to whichUCS is connected.

a. Click the Hosts icon from the top menu bar and selectConnectivity status.

This will open new window showing all the connectivity tothe CLARiiON, as shown in Figure 95 on page 471.



Figure 95 CLARiiON Connectivity Status window

b. Click on Register.

The Register Initiator Record dialog box displays allowingyou to register the initiator record, as shown in Figure 96.

Figure 96 Register Initiator Record dialog box

c. Provision storage to the connected host by using the Storageassignment wizard.


472


Note: LUN provisioning is out of scope of this document. Refer to EMCCLARiiON Unisphere documentation for instructions.

2. Continue with the steps listed in “On a Symmetrix VMAX arrayusing initiator groups (ACLX)” on page 472.

On a Symmetrix VMAX array using initiator groups (ACLX)

Note: The ACLX must be enabled on the port to do LUN masking. Thisprocess will differ depending on the EMC Enginuity™ version running on thearray. For versions earlier than Enginuity v5874, these processes includecreating and adding to existing masking records. For v5874 or later, theprocess includes creating and modifying masking views.

Access Logix (ACLX) will allow the user to create independentgroups for initiators, director ports, and devices. It will then allow theuser to associate and dissolve these groups using a single transaction.

For details information on provisioning storage and the CLI methodrefer to Symmetrix documentation, available EMC Online Support athttps://support.emc.com.

1. Log in to the Symmetrix Management Console (SMC) by pointingthe web browser to the Symmetrix serial number or IP addressfollowed by port number as shown next:

https://hk1xxxxxxx3:8443/or

https://xx.xx.xx.xxx:8443/

After a successful login from the web, the Symmetrixmanagement Console will be displayed, as shown in Figure 97 onpage 473.



Figure 97 EMC Symmetrix Manager Console, Tasks

2. In the Tasks SMC window, select Operations > Provisioning >Provision Storage (Masking Wizard).

This wizard will guide you in assigning and masking LUNs.

The following is explanation of each step in the masking wizard:

• Select Symmetrix ID.

This screen allows selection of Symmetrix ID. A new maskingview can be created from here or the current view can bemodified.

In this example, create a new masking view with name'UCS_Temp'.

• Select initiators.

This step involves the creation of an initiator group andadding initiators to it.


474


In the Select initiator screen, click Create to create a newinitiator group. A new window, Initiator GroupManagement, will display. In this window, select the initiatorsfrom the Available initiators list or create a new initiator andadd it to the Selected Initiators list. Click OK.

For this example, create an initiator group name'UCS_Ch2_Blade3' and adding both the server initiators fromBlade 3 to this group.

• Select ports.

This step creates a port group. This allows the creation of anew port group or allows you to select from the existing portgroup.

Select the Symmetrix ports through which the server shouldget access to the LUNs.

For this example, create a new group named'UCS_CH2_Blade3'. Though each HBA has access to separateSymmetrix ports through Fabric-A and Fabric-B, putting theSymmetrix ports together in a group will allow the LUNs to bemanaged together in a view. Select 2F0 and 3F0 and add themto the group.

– Select devices and assign devices using storage groupmanagement.

This creates a storage group.

Click Create' in the Select devices panel. The Storage GroupManagement window displays, which allows you to selectdevices.

Create the storage group 'UCS_SG' and select the devicesource type from the drop-down menu ('Symmetrix'). Thenselect devices from the Available Devices list and add it to listof Group Members. Click OK.

• Apply the FAST policy (optional).

This is an optional step allowing the creation of a FAST policy.For this example, click Next, which will be used as the defaultpolicy.

The Summary page displays, as shown in Figure 98 onpage 475, showing all the completed selections.



• Click Finish to create the masking view.

Figure 98 Masking Wizard Summary window


476


The following definitions are provided for ACLX:

Notes:

• Initiator Group (IG)– Up to 64 characters for group name– Up to 32 initiators per IG– An initiator can belong to only one IG– An initiator can be a mixture of types of initiators– IGs can be cascaded, but currently it’s only supported one

level deep– Maximum of 1024 initiators to be addressed by an IG– Maximum of 8192 IGs

• Masking View (MV)

– Maximum of 8192 MVs• Director/Port Group (PG)

– All possible 128 director/ports can be in the PG– Maximum of 8192 PGs

• Port Masking Record (PMR)

– Limited number of slots– 8192 PMRs if all SG's are <2048 devices– 4096 PMRs if all SGs are 4096 devices

• Storage Group (SG)

– Up to 4096 devices per SG– Maximum number of 8192 SGs

IG Initiator Group. A grouping of all initiators thatshare the same director, ports, and storage

MV Masking View. The association between the IG,SG, and PG tat make up the actual masking as seenfrom the host.

PG Director/Port Group. A list of director/ports thatwill have the desired storage masked to them andavailable to the desired I/O.

PMR Port Masking Record. This is Enginuity’s internalassociation of the IG, SG, and DG.

SG Storage Group. A list of devices that will bemasked to a desired PG and available to thedesired IG.



Directly attaching EMC storage to Cisco UCS Fabric InterconnectCisco UCS introduced local zoning feature with the UCS 2.1 releasedsoftware. The FC or FCoE targets can be directly attached to theFabric Interconnect (FI) switches without an upstream switch, such asCisco MDS/N5K/N7K. The example shown in Figure 100 onpage 479 is an example of FC and FCoE target direct connection to theFI switches.

This section discusses FCoE local zoning. However, FC local zoning issimilar to FCoE for direct connect configuration.


◆ “Firmware functionalities” on page 477

◆ “Creating local zoning with a new service profile” on page 478

◆ “Creating local zoning with an existing service profile” onpage 501

◆ “Viewing new service profile for the local zoning from cli” onpage 505

◆ “Connecting newly zoned FCoE targets to a Windows host” onpage 507

Firmware functionalitiesThe following lists new Cisco UCS 2.1 functionalities beginning withthe 2.1 (1a) firmware software release:

◆ Better fabric consolidation

• End-to-end FCoE• LAN/SAN consolidation upstream from UCS Fabric

Interconnect switches (FIs)• Fabric Interconnect in NPV mode

◆ Directly connect FC/FCoE storage to FI

• No need for upstream SAN switch• Zoning services through UCSM• Implicit zoning

◆ File and block data over a single cable

• FCoE, iSCSI, NFS, CIFS• Port and cable consolidation• Unified Uplink Ports capabilities


478


Creating local zoning with a new service profileThis section describes the steps required to create zoning with a newservice profile.

◆ “Existing topology” on page 478◆ “Target topology” on page 479◆ “Prerequisites” on page 479◆ “Setting UCS to FC Switching Mode” on page 479◆ “Removing existing zones” on page 482◆ “Creating VSANs in Storage Cloud” on page 484◆ “Configuring FCoE storage port” on page 487◆ “Assigning VSAN to FCoE storage port” on page 487◆ “Creating FCoE zoning using new service profile” on page 489◆ “Viewing new service profile for the local zoning” on page 499

Existing topology Figure 100 shows the existing topology.


GEN-002057

1/1 1/2 1/37 1/38

1/5 1/61/3 1/4

Core-N7K-1

1/1 2/1

UCS Fabricinterconnect - A

UCS Fabricinterconnect - B

Port-channel 1

1/1 1/2 1/37 1/38

1/5 1/61/3 1/4

Core-N7K-2

1/1 2/1Port-channel 2

Fabricextender-1

Fabricextender-2

1 2

Ser

ver

1

1 2

Ser

ver

3


SERVER 1:IP: 10.246.59.20

CNA_1:WWPN = 20:00:00:25:B5:C2:B1:00

CNA_2:WWPN = 20:00:00:25:B5:C2:B1:01

SERVER 3:IP:10.246.59.21

CNA_1:WWPN = 20:00:00:25:B5:C2:B3:00

CNA_2:WWPN = 20:00:00:25:B5:C2:B3:01

6140-AIP = 10.246.46.61

6140-BIP = 10.246.46.62

VNX FCoE storage portsPort3: 50:06:01:60:46:e0:01:49Port4: 50:06:01:68:46:e0:01:49

VMAX FC storage portsPort5: 50:00:09:72:08:2f:b9:18Port6: 50:00:09:72:08:2f:b9:19

10GbE4 or 8 Gb/s FC

FCoE storage

3 4

FC storage

5 6



Target topology Figure 100 shows the target topology.


Prerequisites To take a full advantage of UCS 2.1 software capabilities, the CiscoUCS FI switches should be model 6248 or later. The 6100 switches donot support all of the UCS 2.1 functionalities, such as ConfiguringUnified Ports. All other UCS prerequisites are explained in“Prerequisite” on page 427.

Setting UCS to FCSwitching Mode

This section describes setting the UCS to FC Switching Mode. Theswitching mode can be set on either a fresh UCS setup or on anexisting one. In either case, UCS local zoning requires FabricInterconnect to be configured in FC Switching Mode. Each arediscussed further in this section:

◆ “FC Switching Mode on a fresh UCS setup” on page 480◆ “FC Switching Mode on an existing UCS setup” on page 481

GEN-002058

1/1 1/2 1/37 1/38

1/5 1/61/3 1/4

Core-N7K-1

1/1 2/1

UCS Fabricinterconnect - A

UCS Fabricinterconnect - B

Port-channel 1

1/1 1/2 1/37 1/38

1/5 1/61/3 1/4

Core-N7K-2

1/1 2/1Port-channel 2

Fabricextender-1

Fabricextender-2

1 2

Ser

ver

1

1 2

Ser

ver

3


SERVER 1:IP: 10.246.59.20

CNA_1:WWPN = 20:00:00:25:B5:C2:B1:00

CNA_2:WWPN = 20:00:00:25:B5:C2:B1:01

SERVER 3:IP:10.246.59.21

CNA_1:WWPN = 20:00:00:25:B5:C2:B3:00

CNA_2:WWPN = 20:00:00:25:B5:C2:B3:01

6140-AIP = 10.246.46.61

6140-BIP = 10.246.46.62

VNX FCoE storage portsPort3: 50:06:01:60:46:e0:01:49Port4: 50:06:01:68:46:e0:01:49

VMAX FC storage portsPort5: 50:00:09:72:08:2f:b9:18Port6: 50:00:09:72:08:2f:b9:19

10GbE4 or 8 Gb/s FC

FCoE storage

3 4

FC storage

5 6


480


Note: The UCS Local Zoning feature is not supported with the current UCSfirmware version 2.1(1a) when Uplinks exist with either FC or FCoEswitches.

FC Switching Mode on a fresh UCS setupTo set up FC Switching Mode on a fresh UCS setup, complete thefollowing steps.

1. In the Cisco Unified Computing System Manager window,Equipment tab, select Equipment > Fabric Interconnects >Fabric Interconnect A.

2. In the General tab on the left side panel, click Set FC SwitchingMode.

3. Click Save Changes to save these new settings.

Note: This mode allows storage directly connected to the UCS switchesto create local zoning on UCSM.



FC Switching Mode on an existing UCS setupTo set up FC Switching Mode on an existing UCS setup, complete thefollowing steps.

1. In the Cisco Unified Computing System Manager window, SANtab, SAN Uplinks tab, which displays a split window for Portand Port Channels and SAN Pin Groups.

2. Click the SAN Uplink Manager link on the bottom of the Portsand Port Channels window.

The SAN Uplink Manager displays.


482


3. For the Uplink Mode: switch, click Set FC Switching Mode. Apop-up window displays (not shown). Click OK.

4. The main window displays and the Set FC Switching Mode isfaded.

5. Click OK to exit.

Removing existingzones

This section describes removal of existing zones from FabricInterconnect (FI). When UCSM is set to switching mode, EMC FC orFCoE storage (with UCS 2.1 code) is directly connected to both FIsand zoning is performed using the UCS local zoning.



IMPORTANT

Any external zoning on the UCS must be removed. Currently, UCSdoes not support the existence of local and upstream switch zoning.Remove all existing zoning from the FI before attempting localzoning.

Prerequisites◆ Connect the cable to the UCS FI module.

◆ Communication port with a DB-9 connector.

◆ Download and install PuTTY.

Note: Installing PuTTY is recommended, but not mandatory.

◆ Ensure that HyperTerminal is installed.

Note: The operating system used for this example was Windows 2000Professional and HyperTerminal was installed by default. Some of the laterWindows-based operating systems do not have HyperTerminal installed.This will need to be done before attempting to configure the UCS switches.See the help feature for the particular version of Windows you are using formore information on installing HyperTerminal.

The following console port parameters are to be used for connection:

◆ 9600 baud

◆ 8 data bits

◆ No parity

◆ 1 stop bit

To remove the existing zone using CLI, complete the following steps:

1. Remove the existing zone on Fabric Interconnect A since the areno longer valid and are not needed by running the followingcommands:

ucs-i2-FI-A# connect nxos ! Will connect to Fabric Interconnect Aucs-i2-FI-A(nxos)# scope fc-uplink ! Will enable you to see the unwanted zonesucs-i2-FI-A(nxos)# show vsan ! Show all the VSANsucs-i2-FI-A(nxos)# scope vsan default ! Will enable you access to default VSANucs-i2-FI-A(nxos)# clear-unmanaged-fc-zones-all ! Clears all unwanted zonesucs-i2-FI-A(nxos)# commit-bufferucs-i2-FI-A(nxos)# exitucs-i2-FI-A(nxos)# exitucs-i2-FI-A(nxos)# show zone ! Make sure no zone exists


484


2. Remove the existing zone on Fabric Interconnect B by moving theHyperTerminal cable from Fabric Interconnect A to FabricInterconnect B and then run the commands shown in Step 1.

Creating VSANs inStorage Cloud

To create VSANs in storage cloud, complete the following steps:

1. In the Cisco Unified Computing System Manager window, SANtab, select SAN > Storage Cloud.

2. In the main window select the VSANs tab.



3. Right-click Storage Cloud and Create Storage VSAN on thepop-up menu (not shown). The Create Storage VSAN windowdisplays.

4. In this window, fill out the following fields:

a. Name: FCoE_zoning_1_A

b. FC Zoning: Enabled

c. Select Fabric A

d. VSAN ID: 250 (in this setup example)

e. FCoE VLAN ID: 250 (in this setup example)


486


5. Similarly, create another VSAN 251 by following Step 3 andStep 4.

a. Name: FCoE_zoning_1_B

b. FC Zoning: Enabled

c. Select Fabric B

d. VSAN ID: 251(in this setup example)

e. FCoE VLAN ID: 251 (in this setup example)



Configuring FCoEstorage port

To prepare an FCoE storage port, complete the following steps.

1. In the Cisco Unified Computing System Manager window,Equipment tab, select Fabric Interconnects > Fabric InterconnectA.

2. Select Fixed Module > Ethernet Ports.

3. Right-click on an Ethernet port, (Port 23, in this example).

4. In the General tab, right-click on Reconfigure and selectConfigure as FCoE Storage Port from the drop-down menu (notshown).

5. Use the above steps to create another FCoE Storage Port on FabricInterconnect B.

Assigning VSAN toFCoE storage port

This section describes assigning a previously created VSAN to anFCoE storage port. If the VSAN was not previously created in thestorage container, then the VSAN will not display in the drop-downmenu.


488


To assign the VSAN to an FCoE storage port, complete the followingsteps:

1. In the Cisco Unified Computing System Manager window,Equipment tab, select Fabric Interconnects > Fabric InterconnectA.

2. Select Fixed Module > Ethernet Ports.

3. Select the Ethernet port, (Port 23 in this example).

4. In the General tab, Properties box, right-click on the VSANdrop-down menu and select VSAN #, (VSAN 250, in thisexample).

5. Click Save Changes.

6. Similarly, create another Ethernet port on Fabric Interconnect B251 by following Step 2, selecting the VSAN # (VSAN 251, in thisexample), and then saving the changes.



Creating FCoE zoningusing new service

profile

To create local zoning from a new service profile, complete thefollowing steps:

1. In the Cisco Unified Computing System Manager window,Servers tab, select Service Profile > root.

2. Right-click root and select Create Service Profile (Expert) on thepop-up menu (not shown).


490


The Identify Service Profile windows displays.

3. In the Identify Service Profile window, fill in the followingfields:

a. Name: FCoE_Zoning

b. UUID Assignment: Select from the drop-down menu.

c. Click Next.



The Networking window displays.

4. In the Networking window, enter the vNIC information for bothFabric A and Fabric B and click Next.

The Storage window displays.

The actual local zoning setup starts in the Storage window.


492


5. In the Storage window, complete the following fields:

a. Answer "How would you like to configure your SANconnectivity?" by selecting Expert.

b. Select the WWNN from the drop-down menu.

c. Click + Add.



The Create vHBA window displays.

6. In the Create vHBA window, complete the followinginformation:

a. Name: Enter a name for the vHBA.

b. WWPN Assignment: Select from the drop-down menu.

c. Fabric ID: Select A.

d. Select VSAN: Select the name given to VSAN from "“CreatingVSANs in Storage Cloud” on page 484."

7. Repeat Step 6 to create another vHBA for Fabric B.

The following figure shows both vHBAs when they were createdusing Step 6 and Step 7.


494


8. Click Next.

The Zoning window displays.



The actual zoning is done in the Zoning window.

9. In the Select vHBA Initiator Groups box, click + Add.


496


The Create vHBA Initiator Group window displays.

10. In the Create vHBA Initiator Group window, complete thefollowing information:

a. Name: Enter a name for the vHBA Initiator group.

b. Storage connection Policy: This must be Specific StorageConnection Policy.

c. Select + in FC Target Endpoints box.

The Create FC Target Endpoint window displays.



11. In the Create FC Target Endpoint window, complete thefollowing information:

a. WWPN: Enter WWPN for the FCoE target.

b. Path: A or B (Path B, in this example).

c. Select VSAN: Select the VSAN for this target.

d. Click OK.

e. This brings you back to the Create vHBA Initiator Groupwindow. Click OK.

12. After both FC targets are created, return to the Zoning window.

13. Select FCoE0 in the Select vHBA Initiators box andFCoE_Zone_A in the Select vHBA Initator Groups box and thenclick >> Add To >>.


498


14. Select FCoE1 in the Select vHBA Initiators box andFCoE_Zone_B in the Select vHBA Initator Groups box and thenclick >> Add To >>.

The FCoE0 and FCoE1 Storage vHBA Initiators are added.

15. Click Finish.



Viewing new serviceprofile for the local

zoning

To view the newly created local zoning, complete the following steps.

1. In the Cisco Unified Computing System Manager window,Equipment tab, select Equipment > Chassis > Chassis 1 >Servers > Server 3.

2. Click Associate Service Profile, (Server 3, in this example), on thepop-up menu (not shown). This associates the new Service Profileto the server.

Note: You must associate the service profile to the server to view localzoning.


500


The newly created zoning can be viewed in two different ways, asfollows:

◆ In the Cisco Unified Computing System Manager window,Servers tab, select Service Profiles > root and select the serviceprofile (FCoE_Zoning, in this example), and then click the FCZones tab in the main window. The newly created local zoningare displayed.



◆ In the Cisco Unified Computing System Manager window, SANtab, select SAN and then select the VSANs tab, then FC Zonesfrom the main window. The newly created local zoning aredisplayed.

Creating local zoning with an existing service profileThis section describes the steps required to create zoning with anexisting service profile. In this scenario, an existing profile namedESX5_Sowji is selected.

The zoning steps are similar to the steps discussed in “Creating localzoning with a new service profile” on page 478 with the exception ofzone creation itself.

To create local zoning using an existing service profile, complete thefollowing steps.


502


1. In the Cisco Unified Computing System Manager window,Servers tab, select Service Profiles > root > ESX5_Sowji (for thisexample).

2. In the main window, select the vHBA Initiator Groups tab andthen click +Add on the right side of the window.

The Create vHBA Initiator Group window displays.



3. In the Create vHBA Initiator Group window enter:

a. Name: Enter a name.

b. Storage connection Policy: This must be Specific StorageConnection Policy.

c. Select + on the right side of the FC Target Endpoints box.

The Create FC Target Endpoint windows displays.


504


4. In the Create FC Target Endpoint window, complete thefollowing information:

a. WWPN: Enter WWPN for the FCoE target.

b. Path: A or B (Path A, in this example).

c. Select VSAN: Select the VSAN for this target.

d. Click OK.

The information displays.



The FC Zones for this setup (zones with VSAN 350 and 351) is shownin the following figure.

Viewing new service profile for the local zoning from cliThe newly created local zoning also can be viewed from cli. Thefollowing example is from serial connection to one of the Fabricinterconnects. You are also able to connect to both Fabric Interconnectswitches by using telnet.

Serial connect to the Fabric Interconnect A switch and log in.

The following command display shows most of the local zoningcreated using UCS manager.

ucs-i2-FI-A# connect nxos aucs-i2-FI-A(nxos)# show flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------vfc711 250 0x8e0000 50:06:01:64:3d:e0:05:e5 50:06:01:60:bd:e0:05:e5vfc728 1 0x840001 20:00:00:25:b5:c2:b2:00 20:00:00:25:b5:c2:b2:bbvfc737 350 0xc40001 20:00:00:25:b5:c2:b2:0f 20:00:00:25:b5:00:00:00vfc742 4 0x120001 20:00:00:23:04:c6:a3:32 20:00:00:23:04:c6:a3:32vfc749 51 0x800000 20:00:00:25:b5:01:c2:03 20:00:00:25:b5:00:c2:04vfc751 4 0x120000 20:00:00:25:b5:c2:b4:05 20:00:00:25:b5:00:c2:04vfc755 4 0x120002 20:00:00:25:b5:1c:1b:f0 20:00:00:25:b5:1c:1b:ff


506


vfc760 4 0x120003 20:00:00:25:b5:c1:b2:00 20:00:00:25:b5:c1:b2:eevfc765 350 0xc40000 50:00:09:73:00:0e:5d:e8 50:00:09:73:00:0e:5c:00vfc769 250 0x8e0001 20:00:00:25:b5:c4:b2:0e 20:00:00:25:b5:c3:b2:0f


ucs-i2-FI-A(nxos)# show zoneset active vsan 350zoneset name ucs-ucs-i2-FI-vsan-350-zoneset vsan 350zone name ucs_ucs-i2-FI_A_2_ESX5_Sowji_VHBA_1 vsan 350

* fcid 0xc40001 [pwwn 20:00:00:25:b5:c2:b2:0f]* fcid 0xc40000 [pwwn 50:00:09:73:00:0e:5d:e8]

ucs-i2-FI-A(nxos)# show zoneset active vsan 250zoneset name ucs-ucs-i2-FI-vsan-250-zoneset vsan 250zone name ucs_ucs-i2-FI_A_1_FCoE_Zoning_FCoE0 vsan 250

* fcid 0x8e0001 [pwwn 20:00:00:25:b5:c4:b2:0e]* fcid 0x8e0000 [pwwn 50:06:01:64:3d:e0:05:e5]

ucs-i2-FI-B# connect nxos bucs-i2-FI-A(nxos)# show zoneset active vsan 251zoneset name ucs-ucs-i2-FI-vsan-251-zoneset vsan 251

zone name ucs_ucs-i2-FI_B_1_FCoE_Zoning_FCoE1 vsan 251* fcid 0xeb0001 [pwwn 20:00:00:25:b5:c4:b2:0f]* fcid 0xeb0000 [pwwn 50:06:01:65:3d:e0:05:e5]

ucs-i2-FI-B(nxos)# show zoneset active vsan 351zoneset name ucs-ucs-i2-FI-vsan-351-zoneset vsan 351

zone name ucs_ucs-i2-FI_B_2_ESX5_Sowji_VHBA2 vsan 351* fcid 0x2c0001 [pwwn 20:00:00:25:b5:c2:b2:1f]* fcid 0x2c0000 [pwwn 50:00:09:73:00:0e:5d:a8]

ucs-i2-FI-B(nxos)# show flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------vfc712 251 0xeb0000 50:06:01:65:3d:e0:05:e5 50:06:01:60:bd:e0:05:e5vfc729 1 0x560001 20:00:00:25:b5:c2:b2:01 20:00:00:25:b5:c2:b2:bbvfc738 351 0x2c0001 20:00:00:25:b5:c2:b2:1f 20:00:00:25:b5:00:00:00vfc741 4 0x300001 21:00:00:23:04:c6:a3:33 20:00:00:23:04:c6:a3:33vfc750 51 0xe10000 20:00:00:25:b5:01:c2:04 20:00:00:25:b5:00:c2:04vfc752 4 0x300000 20:00:00:25:b5:c1:b2:06 20:00:00:25:b5:00:c2:04vfc756 4 0x300002 20:00:00:25:b5:1c:1b:f1 20:00:00:25:b5:1c:1b:ffvfc759 4 0x300003 20:00:00:25:b5:c1:b2:01 20:00:00:25:b5:c1:b2:eevfc766 351 0x2c0000 50:00:09:73:00:0e:5d:a8 50:00:09:73:00:0e:5c:00vfc770 251 0xeb0001 20:00:00:25:b5:c4:b2:0f 20:00:00:25:b5:c3:b2:0f




Connecting newly zoned FCoE targets to a Windows hostThe FCoE target LUNs behind the FCoE targets can be accessed by aserver. In this example, we are displaying FCoE LUNs created in“Creating local zoning with an existing service profile” on page 501.

The host used to display the target was a Windows 2008. In order toshow the target LUNs, a failover tool such as EMC PowerPath mustbe installed, which is beyond the scope of this document.

The FCoE disk devices are displayed in the following figure.


508


Cisco UCS Fibre Channel Server Mode: Boot from SAN exampleThis section describes how to enable Cisco UCS server to boot fromSAN, as shown in Figure 101 on page 509 and contains the followinginformation:


◆ “Case topology” on page 509

◆ “Configuring UCS B-Series boot from SAN” on page 509


Configuration for UCS Fabric Interconnect local zoning is done usingthe UCS Manager GUI.

The configuration steps are summarized as following:

1. Create a Service profile.

2. Create a Boot Policy.

a. Create boot option from DVD.

b. Create boot option from SAN - Configure vHBA.

c. Create boot option from SAN - Configure SAN target.

d. Verify current boot policy configuration.

3. Use the created boot policy for the service profile.

4. Associate the service profile to a UCS blade server.

5. Launch the KVM console for Server 4 and boot it with ESXi serverISO.

a. Map the ESXi ISO to virtual media.

b. Select a LUN from SAN to install the operating system.

6. Boot the server from SAN once installation is completed on thestorage device.



Case topology The existing topology is shown in Figure 101.

Figure 101 Existing case topology example

Configuring UCSB-Series boot from

SAN

To configure UCS B-Series boot from SAN, complete the followingsteps:

1. Create the appropriate Service profile, which needs to configuredwith SAN boot. In this case, test_boot.

a. In Servers > New tab (top, right) create a new Service profiletest_boot.


510


The Create Service Profile dialog box displays, as shown next.

Fill in the following fields:

– In the Naming section, complete the Name andDescription fields. In this example: test_boot

– In the Connections section, complete the following fields:vNICs:

Primary vNIC: eth0Secondary vNIC: eth1

vHBAs:Primary vHBA: fc0Secondary vHBA: fc1

– In the Boot Order section, options can be filled in at a latertime.

2. Create a Boot Policy to enable UCS blade server boot from SANdevice. You need to configure both vHBA and SAN target.

• To create a new boot policy:

a. In the Server tab, select Policies > root > Boot Policy.



b. Right-click Boot Policy and select Create Boot Policy.

The Create Boot Policy dialog box displays.

c. Fill in the Name field, in this case, Demo.

d. Add a description in the Description field.

• To create a boot option from SAN, configure the vHBA:


512


a. In the Create Boot Policy window, select vHBAs > AddSAN Boot.

The Add SAN Boot dialog box displays.

b. Complete the Add SAN Boot dialog box for both the primaryand secondary vHBA, as shown below:

For the Primary, complete the following fields:

– vHBA: fc1– Type: Primary– Click OK.For the Secondary, complete the following fields:

– vHBA: fc0– Type: Secondary– Click OK.

• To create boot option from SAN, configure the SAN target.



a. In the Create Boot Policy window, select vHBAs > AddSan Boot Target, as shown in the following figure.

The Add SAN Boot Target dialog box displays, as shown inthe following figure.

b. Complete the following fields:

– Boot Target LUN and Boot Target WWPN fields.– Select Primary.– Click OK.


514


Now complete these steps for the SAN secondary target. In theCreate Boot Policy window, select vHBAs > Add San BootTarget, as shown below.

The Add SAN Boot Target dialog box displays, as shown inthe following figure.

Complete the following fields:

– Boot Target LUN and Boot Target WWPN fields.– Select Secondary.– Click OK.

c. Verify the current boot policy configuration.



In the Create Boot Policy window in the right plane, the BootOrder window should display a total of four (4) boot policyconfigurations, as shown in following figure.

3. Using the created boot policy for the service profile, in theServers tab select Service Profile > root, as shown in followingfigure.

a. Select the correct service profile, test_boot in this example.

b. Select the Boot Order tab.


516


c. The Modify Boot Policy dialog box displays.

d. Select the correct boot policy name, Demo in this example,from the drop-down menu.

4. Associate the service profile to a UCS blade server.

a. In the Equipment tab, expand Chassis and select theparticular server you want to perform a SAN boot. In thisexample, select Chassis 2 > Server 4, as shown in thefollowing figure.



b. Right-click Server 4 and select Associate Service Profile andselect the correct profile you configured for SAN boot, asshown in the above figure.

5. Launch the KVM Console for Server 4 and boot it with ESXiserver ISO.

a. Map the ESXi ISO to virtual media from the blade server 2-4'sKVM window, KVM Console tab and click Boot Server.

b. Be sure to use F2 to enter setup configuration to ensure theserver's first boot option is from EFI: Cisco vKVM-MappedvDVD1.22 CDROM File1.


518


c. Monitor ESXi server boot from the blade server 2-4's KVMwindow KVM Console tab.

d. Select a LUN from SAN to install the operating system. In thisexample, 60G LUN 514f0c52b3e00109 is used.



– Choose SAN storage to install the ESXi server, as shown inabove figure.The 514f0c52b3e00109 shown with the target LUN is notthe WWPN of target port, but the LUN NAA identifier.

– Confirm the installation as shown in following window,which automatically displays.

6. Boot the server from SAN once the installation is completed onstorage device.

Setting up and implementing VE_Port on Cisco UCSCisco UCS supports the following FCoE modes of operation:

◆ Fibre Channel switch mode, which allows the UCS to operate in astand-alone configuration (e.g., without the need for uplinks toanother FC switch) and to make use of VE_Ports should anuplink be needed

◆ Fibre Channel end-host mode, which allows the uplinks from theUCS to act as VF_Ports

The setup and steps for Fibre Channel end-host mode are discussedin “Setting up and implementing VF_Port on Cisco UCS” onpage 545.

Configuring local zoning, which eliminates the need for externalswitches, is discussed in “Directly attaching EMC storage to CiscoUCS Fabric Interconnect” on page 477.

This section describes required setup steps if there are FCoE-capableNexus 5K switches that may be connected between a UCS and EMCFCoE storage in VE_Port mode.


520


Summary of configuration steps to set up UCS VSAN and Port ChannelThe following summarizes the steps explained in this section to setup the UCS VSAN and Port Channel.

1. Create FCoE VSANs on UCS Fabric Interconnects.

2. Create a physical link between UCS and the Nexus 5K switches.

3. Create Port Channel between UCS and the Nexus 5K switches.

4. Bind Port Channels to VSANs.

Target topologyFigure 102 on page 520 shows the target topology for this case study.


FCoE Storage

Cisco Nexus 55xx - A

Cisco UCS 62xxFabric Interconnect - A

Cisco UCS 62xxFabric Interconnect - B

Cisco Nexus 55xx - B

GEN-002067Cisco UCS 5108 Chassis

FCoE 1/25FCoE 1/26

Cisco UCS 5108 Chassis

FCoE 1/25FCoE 1/26

Eth 1/21Eth 1/22

Eth 1/21Eth 1/22

FCoE

Ethernet

Coverged

Key:

FCoE Port Channel 20VSAN 600, VLAN 600

VFC 20


VFC 21

VSAN 600, VLAN 600VFC 125


Eth 1/25 Eth 1/25



Create FCoE VSANs on UCS Fabric InterconnectsThe first step is to create VSANs in UCS SAN Cloud. To create FCoEVSANs to carry FCoE I/O traffic on UCS, complete the followingsteps.

1. In the Cisco Unified Computing System Manager window, SANtab, select SAN > SAN Cloud.

2. In the right panel select All and click on the VSANs tab.


522


3. Right-click on SAN Cloud and select Create VSAN.

The Create VSAN window displays to create the required VSAN.



4. In the Create VSAN window, fill out the following fields:

a. Name: FCoE-VSAN600

b. FC Zoning: Disabled

c. Select: Fabric A

d. VSAN ID: 600

e. FCoE VLAN: 600

f. Click OK to save.

It is recommended to match the VSAN ID with VLAN ID for the easeof use and troubleshooting.

Similarly, create another VSAN 601 by following Step 3 and Step 4, asshown in the following figure.


524


Verify both VSANs were successfully created, as shown in thefollowing figure.

Create a physical link between UCS and the Nexus 5K switches1. Connect two UCS Fabric Interconnect (FI-A) 10 GbE ports to two

N5K 10 GbE ports on Switch A.

2. Repeat Step 1 by adding two more cables between FI-B and N5KSwitch B.

Create Port Channel between UCS and the Nexus 5K switchesA single FCoE link is the minimum required. It is necessary to createa Port Channel between the UCS and upstream switch if more thanone FCoE link is required.

This section provides steps to create a Port Channel between a UCSand two Nexus 5K switches. Port Channel creates multipleconnections and, as a result, more bandwidth between the UCS andthe Nexus switches.

It is a best practice to create a Port Channel.

To configure FCoE Port Channels, complete the following steps.



1. In the Cisco Unified Computing System Manager window, SANtab, select SAN > SAN Cloud > Fabric A > FCoE Port Channels.

2. Right-click on FCoE Port Channel and select Create FCoE PortChannel from the drop-down menu.


526


A Set Port Channel Name window displays.

3. In the Set Port Channel Name window, fill out the followingfields:

a. ID: 20

b. Name: FCoE-PC20

c. Click Next.



An Add Ports window displays.

4. Use the >> button to move ports 25 and 26 from the Ports columnto Ports in the port channel column.


528


5. Confirm that FCoE Port Channels have been created.

6. Click Finish.

Repeat Step 1 through Step 6 to create Port Channels for Fabric B.



Bind Port Channels to VSANsTo bind VSANs to the Port Channels, complete the following steps.

1. In the Cisco Unified Computing System Manager window, SANtab, select SAN > SAN Cloud > Fabric A > FCoE Port Channels> FCoE Port-Channel 20 (Fabric A).

2. In the General tab, fill out the following field:

In the VSAN drop-down menu, select VSAN FCoE-VSAN600(600) and click Save Changes.


530


The Port Channel is now bound to VSAN 600, as shown in thefollowing figure.

Repeat Step 1 and Step 2 to bind Port Channel for Fabric B, as shownin the following three figures.



Preparing and setting up for VE_Port connections to UCSThere are numerous configuration steps required on the Nexus 5Kswitches in order to make these switches able to communicate withUCS FI modules that are running in FC Switch mode.

There are also many features on the Nexus 5K switches that must beenabled in order for the N5Ks to communicate properly with the UCSsystems. Examples of such features include:

◆ Fibre Channel over Ethernet (FCoE)



532


◆ NX-OS Quality of Service (QoS)

◆ Link Level Discovery Protocol (LLDP)

It is recommended that the user of N5K switches to installFC_FEATURE_PKG licenses.

The following summarizes the steps explained in this section toprepare and set up for VE_Port connections to UCS.

1. Enabling Nexus 5K features.

2. Setting up Nexus 5K switches to communicate with UCS.

3. Configuring storage ports on the Nexus 5K switches.

Enabling Nexus 5K featuresTo enable the Nexus 5K features, complete the following steps.

1. Enable the FCoE feature on the Nexus 5K switch.

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)# feature fcoeIOP054050(config)#IOP054050(config)# show feature

Feature Name Instance State-------------------- -------- --------...fcoe 1 enabled

2. Ensure that LACP is enabled on the Nexus 5k switch.

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)#IOP054050(config)# feature lacpIOP054050(config)#IOP054050(config)# show feature | include lacplacp 1 enabled

3. Configure Cisco NX-OS quality of service (Qos):

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)# system qosIOP054050(config-sys-qos)# service-policy type qos input

fcoe-default-in-policyIOP054050(config-sys-qos)# service-policy type queuing input

fcoe-default-in-policy



IOP054050(config-sys-qos)# service-policy type queuing outputfcoe-default-out-policy

IOP054050(config-sys-qos)# service-policy type network-qosfcoe-default-nq-policy

IOP054050(config-sys-qos)# exitIOP054050(config)# exitIOP054050#IOP054050# show startup-config ipqos

!Command: show startup-config ipqos...system qos

service-policy type queuing input fcoe-default-in-policyservice-policy type queuing output fcoe-default-out-policyservice-policy type qos input fcoe-default-in-policyservice-policy type network-qos fcoe-default-nq-policy

Repeat Step 1 through Step 3 on the other Nexus 5K switch.

Setting up Nexus 5K switches to communicate with UCSTo set up the Nexus 5K switches to communicate with UCS, completethe following steps.

1. Create a VLAN to be used for FCoE.

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)# vlan 600IOP054050(config-vlan)# name FCoE-VLAN600IOP054050(config-vlan)# exitIOP054050(config)# exitIOP054050# show vlan brief

VLAN Name Status Ports---- ------------------------------ -------------------------------1 default active Po10, Po200, Eth1/1, Eth1/2

Eth1/5, Eth1/6, Eth1/7,...600 FCoE-VLAN600 active

2. Create a Nexus 5K VSAN to be used for FCoE.

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)# vsan databaseIOP054050(config-vsan-db)# vsan 600IOP054050(config-vsan-db)# vsan 600 name FCoE-VSAN600IOP054050(config-vsan-db)# exitIOP054050(config)# exit


534


IOP054050#IOP054050# show vsan 600vsan 600 information

name:FCoE-VSAN600 state:activeinteroperability mode:defaultloadbalancing:src-id/dst-id/oxidoperational state:down

3. Map the Nexus 5K VLAN to the corresponding VSAN.

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)# vlan 600IOP054050(config-vlan)# fcoe vsan 600IOP054050(config-vlan)# exitIOP054050(config)# exitIOP054050#IOP054050# show vlan fcoe

Original VLAN ID Translated VSAN ID Association State---------------- ------------------ -----------------

600 600 Operational

4. Create Nexus 5K FCoE Uplink Port and Port Channel.

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)# interface ethernet 1/21-22IOP054050(config-if-range)# switchport mode trunkIOP054050(config-if-range)# switchport trunk allowed vlan 600IOP054050(config-if-range)# channel-group 20 mode activeIOP054050(config-if-range)# no shutdownIOP054050(config-if-range)# exitIOP054050(config)# exit

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)# interface port-channel 20IOP054050(config-if)# no shutdownIOP054050(config-if)# exitIOP054050(config)# exit

IOP054050# show port-channel databaseport-channel20

Last membership update is successful2 ports in total, 2 ports upFirst operational port is Ethernet1/21Age of the port-channel is 0d:00h:01m:12sTime since last bundle is 0d:00h:01m:12sLast bundled member is Ethernet1/22



Ports: Ethernet1/21 [active ] [up] *Ethernet1/22 [active ] [up]

5. Create Nexus 5K Virtual Fibre Channel (vfc).

Cisco Nexus 5K requires Virtual Fibre Channel (vfc) creation andthe vfc must be bound to the Port Channel interface for FCoEtraffic. The Port Channel 20 previously created is bound to vfc 20.

It is recommended to match the VSAN ID with VLAN ID for theease of use and troubleshooting.

In the following example, the UCS used was in FC Switch Mode,therefore the Nexus configuration switch was in E mode.

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)# interface vfc 20IOP054050(config-if)# bind interface port-channel 20IOP054050(config-if)# switchport trunk allowed vsan 600IOP054050(config-if)# switchport mode EIOP054050(config-if)# no shutdownIOP054050(config-if)# exitIOP054050(config)# exit

IOP054050# show interface vfc 20vfc20 is trunking

Bound interface is port-channel20Hardware is EthernetPort WWN is 20:13:54:7f:ee:a6:68:bfAdmin port mode is E, trunk mode is onsnmp link state traps are enabledPort mode is TEPort vsan is 1Trunk vsans (admin allowed and active) (600)Trunk vsans (up) (600)Trunk vsans (isolated) ()Trunk vsans (initializing) ()1 minute input rate 72 bits/sec, 9 bytes/sec, 0 frames/sec1 minute output rate 112 bits/sec, 14 bytes/sec, 0 frames/sec



last clearing of "show interface" counters Mon Apr 22 19:16:07 2013

Interface last changed at Mon Apr 22 19:17:06 2013

6. Repeat Step 1 through Step 5 on the other Nexus 5K switch.


536


Configuring storage ports on the Nexus 5K switchesTo configure storage ports on the Nexus 5K switches, complete thefollowing steps.

1. Configure Nexus 5K storage port Ethernet interface.

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)# interface ethernet 1/25IOP054050(config-if)# switchport mode trunkIOP054050(config-if)# switchport trunk allowed vlan 600IOP054050(config-if)# spanning-tree port type edge trunkWarning: Edge port type (portfast) should only be enabled on ports

connected to a single host. Connecting hubs, concentrators,switches, bridges, etc... to this interface when edge port type(portfast) is enabled, can cause temporary bridging loops. Use withCAUTION

IOP054050(config-if)# no shutdownIOP054050(config-if)# exitIOP054050(config)# exit

2. Create vfc and bind the storage port to it.

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)# interface vfc 125IOP054050(config-if)# switchport trunk allowed vsan 600IOP054050(config-if)# bind interface ethernet 1/25IOP054050(config-if)# no shutdownIOP054050(config-if)# exitIOP054050(config)# exit

3. Map the storage port vfc to the FCoE VSAN.

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP054050(config)# vsan databaseIOP054050(config-vsan-db)# vsan 600 interface vfc 125IOP054050(config-vsan-db)# exitIOP054050(config)# exit

4. Confirm vfc is created and bound to correct Ethernet Interfaceand VSAN.

IOP054050# show interface vfc 125vfc125 is trunking

Bound interface is Ethernet1/25Hardware is EthernetPort WWN is 20:7c:54:7f:ee:a6:68:bfAdmin port mode is F, trunk mode is onsnmp link state traps are enabledPort mode is TF



Port vsan is 600Trunk vsans (admin allowed and active) (600)Trunk vsans (up) (600)Trunk vsans (isolated) ()Trunk vsans (initializing) ()1 minute input rate 456 bits/sec, 57 bytes/sec, 0 frames/sec1 minute output rate 576 bits/sec, 72 bytes/sec, 0 frames/sec



last clearing of "show interface" counters Mon Apr 22 19:36:13 2013

Interface last changed at Mon Apr 22 19:36:17 2013


Configuring the storage devices to a Blade Server in a Windows 2008 environmentThe following summarizes the steps explained in this section toconfigure the storage devices to a a Blade server Windows 2008 hostenvironment.

1. Creating a SAN Pin Group.

2. Assigning VSANs to vHBAs.

3. Verifying Virtual Interface (VIF) Path on UCS.

4. Creating zoning for vHBAs and storage system.

5. Verifying EMC storage devices and vHBA WWPN from Nexusswitches.

6. Verifying EMC storage devices on Windows 2008.

Creating a SAN Pin GroupTo assign the VSAN created on the UCS to a server, you must create aPIN group. To create a SAN Pin Group, complete the following steps.


538


1. In the Cisco Unified Computing System Manager window, SANtab, select SAN > SAN Cloud > SAN Pin Groups > Create SANPin Group.



The Create SAN Pin Group window displays.

2. In the Create SAN Pin Group window, fill out the followingfields:

a. Name: FCoE_EPort.

b. Target: Enable both Fabric A and Fabric B.

c. Interface: From the drop-down menus select the Port Channelcreated previously.

d. Click OK.

The Pin Group is created, as shown in the following figure.


540



Assigning VSANs to vHBAsTo assign the previously created VSANs to the blade server vHBAs,complete the following steps.

1. In the Cisco Unified Computing System Manager window,Server tab, select Servers > Service Profiles > root >FCoE_Zoning > vHBAs > vHBA FCoE0.



2. In the General tab, fill out the following fields:

a. Fabric ID: Select A.

b. VSAN: Select FCoE-VSAN600 from drop down menu.

c. Pin Group: Select SAN Pin Group FCoE_EPort from thedrop-down menu.

d. Click Save Changes.


542


3. In the Cisco Unified Computing System Manager window,Server tab, select Severs > Service Profiles > root >FCoE_Zoning > vHBAs > vHBA FCoE1.

4. In this General tab, fill out the following fields:

a. Fabric ID: Select B.

b. VSAN: Select FCoE-VSAN601 from the drop-down menu.

c. Pin Group: Select SAN Pin Group FCoE_EPort fromdrop-down menu.




Verifying Virtual Interface (VIF) Path on UCSIn the Cisco Unified Computing System Manager window, Servertab, select Servers > Service Profiles > root > FCoE_Zoning and clickon the VIF Path tab to verify the VIF paths on UCS created byassigning of VSANs to the vHBAs. The following figure shows theVIF for vHBA0 and vHBA1.

Creating zoning for vHBAs and storage systemUse either Cisco DCNM or CLI to create zoning for the vHBAs andFCoE storage ports.

Verifying EMC storage devices and vHBA WWPN from NexusswitchesTo verify zoned ports from storage ports and host ports are visiblefrom both Nexus switches, complete the following steps.

1. Check the vfc from Nexus switches.

IOP054050# show flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------...vfc125 600 0x740000 50:06:01:64:3d:e0:05:e5 50:06:01:60:bd:e0:05:e5



544


iop054051# show flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------...vfc125 601 0x6c0000 50:06:01:65:3d:e0:05:e5 50:06:01:60:bd:e0:05:e5


2. Check the WWPNs of ports from Nexus switches.

IOP054050# show zoneset active vsan 600zoneset name Zoneset1 vsan 600zone name Zone1 vsan 600* fcid 0x6b0000 [pwwn 20:00:00:25:b5:c4:b2:0e]* fcid 0x740000 [pwwn 50:06:01:64:3d:e0:05:e5]

IOP054050#

iop054051# show zoneset active vsan 601zoneset name Zoneset1 vsan 601

zone name Zone1 vsan 601* fcid 0x730000 [pwwn 20:00:00:25:b5:c4:b2:0f]* fcid 0x6c0000 [pwwn 50:06:01:65:3d:e0:05:e5]

iop054051#



Verifying EMC storage devices on Windows 2008From Windows 2008, verify that EMC storage devices are visible arevisible from Server Manager.

Setting up and implementing VF_Port on Cisco UCSThis section describes required setup steps if there are FCoE-capableNexus 5K switches that may be connected between a UCS and EMCFCoE storage in VF_Port mode.

Summary of configuration steps to set up UCS VSAN and Port ChannelThe following summarizes the steps explained in this section to setup the UCS VSAN and Port Channel.

1. Create FCoE VSANs on UCS Fabric Interconnects.

2. Create a physical link between UCS and the Nexus 5K switches.

3. Create Port Channel between UCS and the Nexus 5K switches.

4. Bind Port Channels to VSANs.


546


Target topologyFigure 103 shows the target topology for this case study.


Create FCoE VSANs on UCS Fabric InterconnectsThe first step is to create VSANs in UCS SAN Cloud. To create FCoEVSANs to carry FCoE I/O traffic on UCS, complete the followingsteps.

FCoE Storage

Cisco Nexus 55xx - A

Cisco UCS 62xxFabric Interconnect - A

Cisco UCS 62xxFabric Interconnect - B

Cisco Nexus 55xx - B

GEN-002124Cisco UCS 5108 Chassis

FCoE 1/25FCoE 1/26

Cisco UCS 5108 Chassis

FCoE 1/25FCoE 1/26

Eth 1/21Eth 1/22

Eth 1/21Eth 1/22

FCoE

Ethernet

Coverged

Key:


VFC 10


VFC 11



Eth 1/25 Eth 1/25



1. In the Cisco Unified Computing System Manager window, SANtab, select SAN > SAN Cloud.

2. In the right panel select All and click on the VSANs tab.


548


3. Right-click on SAN Cloud and select Create VSAN.

The Create VSAN window displays to create the required VSAN.



4. In the Create VSAN window, fill out the following fields:

a. Name: FCoE-VSAN500

b. FC Zoning: Disabled

c. Select: Fabric A

d. VSAN ID: 500

e. FCoE VLAN: 500

f. Click OK to save.

It is recommended to match the VSAN ID with VLAN ID for the easeof use and troubleshooting.

Similarly, create another VSAN 501 by following Step 3 and Step 4, asshown in the following figure.


550


Verify both VSANs were successfully created, as shown in thefollowing figure.

Create a physical link between UCS and the Nexus 5K switches1. Connect two UCS Fabric Interconnect (FI-A) 10 GbE ports to two

N5K 10 GbE ports on Switch A.

2. Repeat Step 1 by adding two more cables between FI-B and N5KSwitch B.

Create Port Channel between UCS and the Nexus 5K switchesA single FCoE link is the minimum required. It is necessary to createa Port Channel between the UCS and upstream switch if more thanone FCoE link is required.

This section provides steps to create a Port Channel between a UCSand two Nexus 5K switches. Port Channel creates multipleconnections and, as a result, more bandwidth between the UCS andthe Nexus switches.

It is a best practice to create a Port Channel.

To configure FCoE Port Channels, complete the following steps.



1. In the Cisco Unified Computing System Manager window, SANtab, select SAN > SAN Cloud > Fabric A > FCoE Port Channels.

2. Right-click on FCoE Port Channel and select Create FCoE PortChannel from the drop-down menu.


552


A Set Port Channel Name window displays.

3. In the Set Port Channel Name window, fill out the followingfields:

a. ID: 10

b. Name: FCoE-PC10

c. Click Next.



An Add Ports window displays.

4. Use the >> button to move ports 25 and 26 from the Ports columnto Ports in the port channel column.


554


5. Confirm that FCoE Port Channels have been created.

6. Click Finish.

Repeat Step 1 through Step 6 to create Port Channels for Fabric B.



Bind Port Channels to VSANsTo bind VSANs to the Port Channels, complete the following steps.

1. In the Cisco Unified Computing System Manager window, SANtab, select SAN > SAN Cloud > Fabric A > FCoE Port Channel >FCoE Port-Channel 10 (Fabric A).

2. In the General tab, fill out the following field:

In the VSAN drop-down menu, select VSAN FCoE-VSAN500(500) and click Save Changes.


556


The Port Channel is now bound to VSAN 500, as shown in thefollowing figure.

Repeat Step 1 and Step 2 to bind Port Channel for Fabric B, as shownin the following three figures.



Preparing and setting up for VF_Port connections to UCSThere are numerous configuration steps required on the Nexus 5Kswitches in order to make these switches able to communicate withUCS FI modules that are running in fibre Channel end-host mode.

There are also many features on the Nexus 5K switches that must beenabled in order for the N5Ks to communicate properly with the UCSsystems. Examples of such features include:

◆ Fibre Channel over Ethernet (FCoE)◆ Link Aggregation Control Protocol (LACP)◆ N_Port ID Virtualization (NVIP)◆ Link Level Discovery Protocol (LLDP)


558


It is recommended that the user of N5K switches to installFC_FEATURE_PKG licenses.

The following summarizes the steps explained in this section toprepare and set up for VF_Port connections to UCS.

1. Enabling Nexus 5K features.

2. Setting up Nexus 5K switches to communicate with UCS.

3. Configuring storage ports on the Nexus 5K switches.

Enabling Nexus 5K featuresTo enable the Nexus 5K features, complete the following steps.

1. Enable the FCoE feature on the Nexus 5K switch.

IOP054050# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# feature fcoeIOP-UCS-Switch(config)#IOP-UCS-Switch(config)# show feature

Feature Name Instance State-------------------- -------- --------...fcoe 1 enabled

2. Make sure the NPIV feature on the Nexus 5K switch is enabled ifthe UCS is set to FC End-Host Mode.

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# feature npivIOP-UCS-Switch(config)# exitIOP-UCS-Switch#IOP-UCS-Switch# show npiv statusNPIV is enabledIOP-UCS-Switch#

3. Ensure that LACP is enabled on the Nexus 5k switch.

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)#IOP-UCS-Switch(config)# feature lacpIOP-UCS-Switch(config)#IOP-UCS-Switch(config)# show feature | include lacplacp 1 enabled



4. Configure Cisco NX-OS quality of service (Qos):

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# system qosIOP-UCS-Switch(config-sys-qos)# service-policy type qos input

fcoe-default-in-policyIOP-UCS-Switch(config-sys-qos)# service-policy type queuing input

fcoe-default-in-policyIOP-UCS-Switch(config-sys-qos)# service-policy type queuing output

fcoe-default-out-policyIOP-UCS-Switch(config-sys-qos)# service-policy type network-qos

fcoe-default-nq-policyIOP-UCS-Switch(config-sys-qos)# exitIOP-UCS-Switch(config)# exitIOP-UCS-Switch#IOP-UCS-Switch# show startup-config ipqos

!Command: show startup-config ipqos...system qos

service-policy type queuing input fcoe-default-in-policyservice-policy type queuing output fcoe-default-out-policyservice-policy type qos input fcoe-default-in-policyservice-policy type network-qos fcoe-default-nq-policy


Setting up Nexus 5K switches to communicate with UCSTo set up the Nexus 5K switches to communicate with UCS, completethe following steps.

1. Create a VLAN to be used for FCoE.

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# vlan 500IOP-UCS-Switch(config-vlan)# name FCoE-VLAN500IOP-UCS-Switch(config-vlan)# exitIOP-UCS-Switch(config)# exitIOP-UCS-Switch# show vlan brief

VLAN Name Status Ports---- ------------------------------ -------------------------------1 default active Po10, Po200, Eth1/1, Eth1/2,

Eth1/3, Eth1/4,Eth1/5, Eth1/6,Eth1/7, Eth1/8

.

.


560


.500 FCoE-VLAN500 active

2. Create a Nexus 5K VSAN to be used for FCoE.

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# vsan databaseIOP-UCS-Switch(config-vsan-db)# vsan 500IOP-UCS-Switch(config-vsan-db)# vsan 500 name FCoE-VSAN500IOP-UCS-Switch(config-vsan-db)# exitIOP-UCS-Switch(config)# exitIOP-UCS-Switch#IOP-UCS-Switch# show vsan 500vsan 500 information

name:FCoE-VSAN500 state:activeinteroperability mode:defaultloadbalancing:src-id/dst-id/oxidoperational state:down

3. Map the Nexus 5K VLAN to the corresponding VSAN.

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# vlan 500IOP-UCS-Switch(config-vlan)# fcoe vsan 500IOP-UCS-Switch(config-vlan)# exitIOP-UCS-Switch(config)# exitIOP-UCS-Switch#IOP-UCS-Switch# show vlan fcoe

Original VLAN ID Translated VSAN ID Association State---------------- ------------------ -----------------

500 500 Operational

4. Create Nexus 5K FCoE Uplink Port and Port Channel.

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# interface ethernet 1/21-22IOP-UCS-Switch(config-if-range)# switchport mode trunkIOP-UCS-Switch(config-if-range)# switchport trunk allowed vlan 500IOP-UCS-Switch(config-if-range)# channel-group 10 mode activeIOP-UCS-Switch(config-if-range)# no shutdownIOP-UCS-Switch(config-if-range)# exitIOP-UCS-Switch(config)# exit

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# interface port-channel 10IOP-UCS-Switch(config-if)# no shutdown



IOP-UCS-Switch(config-if)# exitIOP-UCS-Switch(config)# exit

IOP-UCS-Switch# show port-channel databaseport-channel10

Last membership update is successful2 ports in total, 2 ports upFirst operational port is Ethernet1/21Age of the port-channel is 0d:00h:07m:10sTime since last bundle is 0d:00h:07m:10sLast bundled member is Ethernet1/22Ports: Ethernet1/21 [active ] [up] *

Ethernet1/22 [active ] [up]

5. Create Nexus 5K Virtual Fibre Channel (vfc).

Cisco Nexus 5K requires Virtual Fibre Channel (vfc) creation andthe vfc must be bound to the Port Channel interface for FCoEtraffic. The Port Channel 10 previously created is bound to vfc 10.

It is recommended to match the VSAN ID with VLAN ID for theease of use and troubleshooting.

In the following example, the UCS used was in Fibre ChannelEnd-Host Mode, therefore it had NPIV enabled so the Nexusconfiguration switch was in F mode.

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# interface vfc 10IOP-UCS-Switch(config-if)# bind interface port-channel 10IOP-UCS-Switch(config-if)# switchport trunk allowed vsan 500IOP-UCS-Switch(config-if)# switchport mode FIOP-UCS-Switch(config-if)# no shutdownIOP-UCS-Switch(config-if)# exitIOP-UCS-Switch(config)# exit

IOP-UCS-Switch# show interface vfc 10vfc10 is trunking

Bound interface is port-channel10Hardware is EthernetPort WWN is 20:13:54:7f:ee:a6:68:bfAdmin port mode is F, trunk mode is onsnmp link state traps are enabledPort mode is TEPort vsan is 1Trunk vsans (admin allowed and active) (500)Trunk vsans (up) (500)Trunk vsans (isolated) ()Trunk vsans (initializing) ()1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec


562


0 frames input, 0 bytes0 discards, 0 errors0 frames output, 0 bytes0 discards, 0 errors

last clearing of "show interface" counters neverInterface last changed at Thurs Mar 28 18:37:19 2013

6. Repeat Step 1 through Step 5 on the other Nexus 5K switch.

Configuring storage ports on the Nexus 5K switchesTo configure storage ports on the Nexus 5K switches, complete thefollowing steps.

1. Configure Nexus 5K storage port Ethernet interface.

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# interface ethernet 1/25IOP-UCS-Switch(config-if)# switchport mode trunkIOP-UCS-Switch(config-if)# switchport trunk allowed vlan 500IOP-UCS-Switch(config-if)# spanning-tree port type edge trunkWarning: Edge port type (portfast) should only be enabled on ports

connected to a single host. Connecting hubs, concentrators,switches, bridges, etc... to this interface when edge port type(portfast) is enabled, can cause temporary bridging loops. Use withCAUTION

IOP-UCS-Switch(config-if)# no shutdownIOP-UCS-Switch(config-if)# exitIOP-UCS-Switch(config)# exit

2. Create vfc and bind the storage port to it.

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# interface vfc 125IOP-UCS-Switch(config-if)# switchport trunk allowed vsan 500IOP-UCS-Switch(config-if)# bind interface ethernet 1/25IOP-UCS-Switch(config-if)# no shutdownIOP-UCS-Switch(config-if)# exitIOP-UCS-Switch(config)# exit

3. Map the storage port vfc to the FCoE VSAN.

IOP-UCS-Switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.IOP-UCS-Switch(config)# vsan databaseIOP-UCS-Switch(config-vsan-db)# vsan 500 interface vfc 125IOP-UCS-Switch(config-vsan-db)# exitIOP-UCS-Switch(config)# exit

4. Confirm vfc is created and bound to correct Ethernet Interfaceand VSAN.



IOP-UCS-Switch# show interface vfc 125vfc125 is trunking

Bound interface is Ethernet1/25Hardware is EthernetPort WWN is 20:7c:00:05:73:c6:38:ffAdmin port mode is F, trunk mode is onsnmp link state traps are enabledPort mode is TFPort vsan is 500Trunk vsans (admin allowed and active) (500)Trunk vsans (up) (500)Trunk vsans (isolated) ()Trunk vsans (initializing) ()1 minute input rate 256 bits/sec, 32 bytes/sec, 0 frames/sec1 minute output rate 304 bits/sec, 38 bytes/sec, 0 frames/sec



last clearing of "show interface" counters never

Interface last changed at Tue Mar 26 17:05:55 2013


Configuring the storage devices to a Virtual Machine in an ESXi environmentThe following summarizes the steps explained in this section toconfigure the storage devices to a a Virtual Machine in an ESXienvironment.

1. Creating a SAN Pin Group.

2. Assigning VSANs to vHBAs.

3. Verifying Virtual Interface (VIF) Path on UCS.

4. Creating zoning for vHBAs and storage system.

5. Verifying EMC storage devices and vHBA WWPN from Nexusswitches.

6. Verifying EMC storage devices on the ESXi server.

7. Verifying EMC storage devices on the Virtual Machine.

Creating a SAN Pin GroupTo assign the VSAN created on the UCS to a server, you must create aPIN group. To create a SAN Pin Group, complete the following steps.


564


1. In the Cisco Unified Computing System Manager window, SANtab, select SAN > SAN Cloud > SAN Pin Groups > Create SANPin Group.



The Create SAN Pin Group window displays.

2. In the Create SAN Pin Group window, fill out the followingfields:

a. Name: FCoE_vePort.

b. Target: Enable both Fabric A and Fabric B.

c. Interface: From the drop-down menus select the Port Channelcreated previously.

d. Click OK.



566



Assigning VSANs to vHBAsTo assign the previously created VSANs to the blade server vHBAs,complete the following steps.

1. In the Cisco Unified Computing System Manager window,Server tab, select Servers > Service Profiles > root > IOP_Blade5> vHBAs > vHBA vHBA0.



2. In the General tab, fill out the following fields:

a. Fabric ID: Select A.

b. VSAN: Select FCoE-VSAN500 from drop down menu.

c. Pin Group: Select SAN Pin Group FCoE_vePort from thedrop-down menu.



568


3. In the Cisco Unified Computing System Manager window,Server tab, select Servers > Service Profiles > root > IOP_Blade5> vHBAs > vHBA vHBA1.

4. In this General tab, fill out the following fields:

a. Fabric ID: Select B.

b. VSAN: Select FCoE-VSAN501 from the drop-down menu.

c. Pin Group: Select SAN Pin Group FCoE_vePort fromdrop-down menu.


570


Verifying Virtual Interface (VIF) Path on UCSIn the Cisco Unified Computing System Manager window, Servertab, select Servers > Service Profiles > root > IOP_Blade5 and clickon the VIF Path tab to verify the VIF paths on UCS created byassigning of VSANs to the vHBAs. The following figure shows theVIF for vHBA0 and vHBA1.

Creating zoning for vHBAs and storage systemUse either Cisco DCNM or CLI to create zoning for the vHBAs andFCoE storage ports.

Verifying EMC storage devices and vHBA WWPN from NexusswitchesTo verify zoned ports from storage ports and host ports are visiblefrom both Nexus switches, complete the following steps.

1. Check the vfc from Nexus switches.

IOP-UCS-Switch# show flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------...vfc10 500 0xde0001 22:e0:54:7f:ee:5b:8b:ff 21:f4:54:7f:ee:5b:8b:c1vfc10 500 0xde0003 20:00:00:25:b5:cf:05:3f 20:00:00:25:b5:00:cf:3fvfc125 500 0xde0000 50:00:09:73:00:0e:5d:e8 50:00:09:73:00:0e:5c:00




IOP-UCS-Switch#

IOP-UCS5548-2# show flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------...vfc11 501 0x910001 22:e1:54:7f:ee:5b:8b:7f 21:f5:54:7f:ee:5b:8b:41vfc11 501 0x910002 20:00:00:25:b5:cf:05:2f 20:00:00:25:b5:00:cf:3fvfc125 501 0x910000 50:00:09:73:00:0e:5d:a8 50:00:09:73:00:0e:5c:00


2. Check the WWPNs of ports from Nexus switches.

IOP-UCS5548-1# show zoneset active vsan 501zoneset name Zoneset1 vsan 501

zone name ZoneFCoE_1 vsan 501* fcid 0xde0000 [pwwn 50:00:09:73:00:0e:5d:e8]* fcid 0xde0003 [pwwn 20:00:00:25:b5:cf:05:3f]

IOP-UCS5548-1#

IOP-UCS5548-2# show zoneset active vsan 501zoneset name Zoneset1 vsan 501

zone name Zone_FCoE_2 vsan 501* fcid 0x910000 [pwwn 50:00:09:73:00:0e:5d:a8]* fcid 0x910002 [pwwn 20:00:00:25:b5:cf:05:2f]

IOP-UCS5548-2#


572


Verifying EMC storage devices on the ESXi serverThis step displays the vHBA and storage ports that are visible fromESXi server. Refer to the following two figures for vHBAs and EMCstorage devices.


574


Verifying EMC storage devices on the Virtual MachineFrom the Virtual machine, two of the ESXi raw devices were mappedto the Virtual Machine and EMC storage devices, as shown in thefollowing figure.

Configuring UCS 10G iSCSI using VNX-2 arraysOther targets may be used to configure UCS 10G iSCSI. This exampleuses VNX-2 arrays. These instructions assume that there are no iSCSI10G SLICs installed on VNX-2 arrays. This section contains thefollowing information:

◆ “Installing a new iSCSI 10G SLIC on a VNX-2 array” on page 575◆ “Setting up 10G iSCSI IP address for a VNX-2 array” on page 576◆ “Creating a UCS iSCSI connection” on page 580◆ “Identifying MAC address for the iSCSI NICs” on page 593◆ “Testing the IP link connections on Windows Host” on page 594◆ “Completing the UCS 10G iSCSI on VNX-2 arrays” on page 595◆ “Verifying iSCSI LUNs on the Windows Host” on page 598



Figure 104 shows an example of the configuration used in thissection.

Figure 104 Configuring UCS 10G iSCSI using VNX-2 arrays example

Installing a new iSCSI 10G SLIC on a VNX-2 arrayThis section describes how to install a new iSCSI 10G SLIC on aVNX-2 array. To install the iSCSI 10G SLICs, complete the followingsteps:

1. Update the VNX-2 to the latest code. (The VNX-2 code update isnot shown. These steps assume that users have knowledge of thisprocess). The VNX-2 code used in this example is shown infollowing figure.


576


2. Insert the 10G SLIC into the next available slot on SP B.

3. Issue and run the following cli commands (assuming10.246.52.141 is the IP address of SP B):

naviseccli -h 10.246.52.141 -username -password -AddUserSecurity -scope 0naviseccli -h 10.246.52.141 -username -password -Scope 0 ioportconfig –persist

4. Repeat Step 2 and Step 3 on SP A using the IP address of SP A.

The 10G SLIC appears in the SP Properties dialog box for SP A andSP B. The following figure shows SP Properties dialog box for SP A.

Setting up 10G iSCSI IP address for a VNX-2 arrayThis section describes the initial steps to set up the 10G iSCSI IPaddress for a VNX-2 array with 10G SLICs. The IP addresses used inthis section are just examples.

To create iSCSI target IP addresses, complete the following steps:

1. In EMC Unisphere, Systems tab, select VNX-2 > System >Hardware > Storage Hardware.



The Hardware details display.

2. Select SPs > SP A > IO Modules > Slot A3 > Port 0.

3. Click Properties.


578


The Port Identification dialog box displays.

4. Click Add.

The following screen displays showing the IPv4 configuration.



5. Complete the following fields:

a. IP address: 10.1.1.100

b. Subnet: 255.255.255.0

c. Gateway: Not set

d. Click OK.

6. Add Port 1 by repeating Step 2 and Step 3 and using thefollowing information to complete the fields:

a. IP address: 10.1.2.102

b. Subnet: 255.255.255.0

c. Gateway: Not set

d. Click OK.

7. To view the iSCSI IP addresses, select VNX-2 > Settings >Network > Settings for Block.


580


The following shows the IP addresses and iqn names for thenewly created iSCSI ports.

8. Repeat the above steps to add the following IP addresses to SPB:

a. IP address for port 0: 10.1.1.103

b. IP address for port 1:10.1.2.104

Creating a UCS iSCSI connectionThis section describes the steps required for creating a UCS 10G iSCSIconnection.

Creating a physical connectionTo create a physical connection, physically connect all four 10G NICports to Fabric Interconnect A and B as follows:

VNX SPA 0 (SLIC 10G port 0) to FBRIC interconnect A port 13VNX SPA 1 (SLIC 10G port 1) to FBRIC interconnect B port 13VNX SPB 0 (SLIC 10G port 0) to FBRIC interconnect A port 14VNX SPB 1 (SLIC 10G port 1) to FBRIC interconnect B port 14



PrerequisitesBefore completing the steps to create a UCS 10G iSCSI connection,complete the following steps to verify that the UCS is in EthernetSwitching Mode.

1. In the UCSM Equipment tab, select Equipment > FabricInterconnects > Fabric Interconnect A. In the Status area, setEthernet Mode to Switch.

2. Verify the server adapters that are going to be used are atminimum Cisco UCS M81KRs.

3. In the UCSM Equipment tab, select Equipment > Chassis >Chassis 2 > Servers > Server 2 > Adapters.


582


4. Select Adapter 1.

The Product Name, displayed in the Properties section, is theadapters type.

Creating VLANs for iSCSITo create VLANs for iSCSI, complete the following steps.

1. In the UCSM LAN tab, select LAN > Appliances > VLANs,right-click VLANs, and select Create from the drop-down menu.



The Create VLANs dialog box displays.

2. Complete the following fields to add iSCSI_1:

a. VLAN Name: iSCSI_1

b. Select: Common/Global

c. VLAN IDs: 13

d. Click OK.

When iSCSI_1 is added, the following message displays:

3. Add iSCSI_2. In the UCSM LAN tab, select LAN > Appliances >VLANs, right-click VLANS, and select Create from thedrop-down menu.


584


The Create VLANs dialog box displays.

4. Complete the following fields to add iSCSI_2:

a. VLAN Name: iSCSI_2

b. Select: Common/Global

c. VLAN IDs: 14

d. Click OK.

When iSCSI_2 is added, a Successfully created message displays.

The following figure shows the VLANs were added:



Creating vNICs for iSCSITo create two new vNICs for iSCSI, complete the following steps.

1. In UCSM Servers tab, select Servers > Server Profiles > root >FCoE_Zoning (FCoE_Zoning is the server name) > vNICs.

2. Click Add to create vNICs.


586


The Create vNIC dialog box displays.

3. Complete the following fields to add vNIC_2:

a. Name: vNIC_2

b. MAC Address assignment: Ben_MAC_pools

c. Fabric ID: Fabric A

d. VLANs: iSCSI_1

e. Adapter Policy: Windows

f. Network Control Policy: default

g. Click OK.

4. Repeat Step 1 and Step 2 to add vNIC_3.



The Create vNIC dialog box displays.

5. Complete the following fields to add vNIC_3.

a. Name: vNIC_3

b. MAC Address assignment: Ben_MAC_pools

c. Fibre ID: Fabric B

d. VLAN: iSCSI_2

e. Adapter Policy: Windows

f. Network Control Policy: default

g. Click OK.


588


The following screen shows that vNIC_2 and vNIC_3 wereadded:

6. After the vNIC has been created, you need to enable failover.Click vNIC_2.

The Modify vNIC dialog box displays.



7. In the Fabric ID field, select Enable Failover and click OK.

8. Enable failover in vNIC_3. Click on vNIC_3.

The Modify vNIC dialog box displays.

9. In the Fabric ID field, select Enable Failover and click OK.


590


vNIC_2 and vNIC_3 are now created, as shown in the followingfigure.

Assigning VLANs to physical portsTo assign VLANS to physical ports, you must first convert Ethernetports to Appliance ports by completing the following steps.

After these steps are completed, you will then bind the VLANs to thephysical port, as explained beginning on page 591.

Converting Ethernet ports to Applicant ports

1. In the UCSM Equipment tab, select Equipment > FabricInterconnects > Fabric Interconnect A > Fixed Module >Ethernet Ports.



2. Right-click Port 13 and select Configure as Appliance port.

3. A message box asks you to confirm. Click OK.

4. In the UCSM Equipment tab, select Equipment > FabricInterconnects > Fabric Interconnect A > Fixed Module >Ethernet Ports.

5. Click Port 14 > Configure as Appliance port.

6. Repeat Step 1 through Step 5 for Fabric Interconnect B.

Binding VLANs to the physical ports

Now that you have converted Ethernet ports into Appliance ports,you can bind the VLANs to the physical ports.

1. In the UCSM Equipment tab, select Equipment > FabricInterconnects > Fabric Interconnect A > Fixed Module >Ethernet Ports >Port 13.


592


2. In the General tab, Actions field, select Configure VLAN.

The Configure VLAN for Appliance Interface 1/13 displays.


a. Port Mode: Access

b. Select VLAN: iSCSI_1 (13)

c. Click OK.

4. In the UCSM Equipment tab, select Equipment > FabricInterconnects > Fabric Interconnect A > Fixed Module >Ethernet Ports >Port 14.

5. In the General tab, Actions field, select Configure VLAN.



The Configure VLAN for Appliance Interface 1/13 displays.




c. Click OK.

7. Repeat Step 4 and Step 5 on Fabric Interconnect B and configurethe following Appliance ports and VLANs on ports 13 and 14:



c. Click OK.

Identifying MAC address for the iSCSI NICsTo confirm that the MAC addresses were identified for the two newlycreated iSCSI NICs, complete the following steps.

1. In the UCSM Equipment tab, select Equipment > Chassis >Chassis 2 > Servers > Server 2 > Adapters > Adapter 1.

2. Click on NICs and capture the MAC information:

• MAC for vNIC_2 (NIC 3): 00:25:B5:00:00:2E

• MAC for vNIC_3 (NIC 4): 00:25:B5:00:00:1F


594


The vNICs are displayed, as shown in the following figure:

Testing the IP link connections on Windows Host

Note: This example uses Microsoft initiator to log in to all four VNX-2 targets.These steps assume that users knows how to use Microsoft Initiator.

The Windows host in this example is server 2 on blade server 2. TheWindows host was directly installed on the blade server.

To test the IP link connections on this Windows host, complete thefollowing steps.

1. Log in to the host.

2. From the command prompt, issue the ipconfig /all command.

3. Identify the interface ID #s for the above MAC addresses.Number 12 and 13 on the following example was the interfaceIDs:

• MAC for NIC 3: 00:25:B5:00:00:2E = 12

• MAC for NIC 4: 00:25:B5:00:00:1F = 13

4. Change the IP addresses on the above two NICs on the Windows;for example:

• NIC 12 set to 10.1.1.200

• NIC 13 set to 10.1.2.210

5. Ping the following four VNX-2 iSCSI Ports from the Windows2008 Host to make sure they ping.

• 10.1.1.100

• 10.1.1.103

• 10.1.2.102

• 10.1.2.104

6. If all four IP addresses ping, then the physical links areestablished correctly.



Completing the UCS 10G iSCSI on VNX-2 arraysYou must register the four initiator names on the VNX-2 arrays. Thesesteps assume the users know how to register initiator names on theVNX-2.

Complete the iSCSI initiator registration steps on VNX-2 by creatinghosts using the host iSCSI "iqn" name obtained from the steps in“Testing the IP link connections on Windows Host” on page 594.

Note: Each initiator must be registered individually.

1. Select one of the four new initiator names and click Properties.


596


The Initiator Information dialog box displays.

2. Complete the information and click OK.

The host name is displayed in the Hosts tab.

3. Create two LUNs and add the host name and LUNs to theStorage Group.



The LUNs are displayed in the LUNs tab, as shown in thefollowing figure.


598


Verifying iSCSI LUNs on the Windows HostThe following figures show two different ways that the two iSCSILUNs, created successfully, are visible from Windows host. Thefollowing figure shows the LUNs from the Computer Managementdisplay.

The following figure shows he LUNs from the PowerPath display.



IBM BladeCenterThis section contains the following information:

◆ “Brocade Converged 10 GbE Switch Module supported featuresand topologies” on page 599

◆ “Brocade Converged 10 GbE configuration example” on page 603

◆ “Cisco Nexus 4000 supported features and topologies” onpage 620

◆ “Cisco Nexus 4001I configuration example” on page 622

Brocade Converged 10 GbE Switch Module supported features and topologiesThe supported feature and topologies have been divided into threeareas, as shown in Figure 105 on page 600: FCoE interface, Ethernetuplink interface, and FC interface. This section provides thefollowing information on the supported features and topologies forthe Brocade Converged 10 GbE Switch Module:







Figure 105 is for clarification purposes only. The preferred method ofincorporating the Brocade Converged module into an existingtopology can be found in Figure 107 on page 605, the target topologydiagram for the Brocade Converged 10 GbE Switch Module.

IBM BladeCenter 599

600



Prerequisites FCoE requires the use of specialized host adapters, referred to as aConverged Network Adapter (CNA), and specialized switchhardware known as a Fibre Channel Forwarder (FCF). CNAs areinstalled in the host and then attached to an Ethernet port thatprovides FCF functionality. In an IBM chassis, the connectionbetween CNA ports and embedded Ethernet switch port ishard-wired. Brocade and QLogic CNAs are supported with thisrelease.

CNA prerequisitesRefer to the IBM BladeCenter Interoperability Guide, located onibm.com, for a list of servers compatible with these CNA daughter

MDS Brocade

Storage

FCInterface



FCoEInterface

Nexus 5000

Ethernet Bridge(Switch)

Celerra

10 GbENIC

10 GbENIC

ServerServer

CNA CNA

Server

GEN-001009



cards (CFFh). Refer to the EMC Support Matrix for information on thesupported host platforms.

FCF prerequisites

The Brocade 8470 switch requires:

◆ An IP address for management purposes and a network port towhich to connect the Brocade 8470 management port.

◆ A customer-supplied password for the admin user.

◆ An available Domain ID in the FC fabric.

Management The Brocade Converged 10 GbE Switch Module can be managed withthe following:

◆ Telnet

◆ Web Tools

◆ Connectrix Manager Data Center Edition (v10.4.3 or later).

FCoE interface The FCoE interface is any connection that is on the server-facing(internal) side of the switch module. The following interfaces aresupported:

Note: Always refer to the EMC Support Matrix for the most up-to-datesupport information.

◆ CNAs

• QLogic CNAs– Data Center Bridging (DCB); required

• Brocade CNAs– Data Center Bridging (DCB); required

◆ OS support


Note: The Ethernet uplink interface is any connection that is on thenetwork-facing (external) side of the switch module. Do not connect CNAs orFCoE software initiators from other hosts to any of the external 10 GbEinterfaces on the Brocade Converged 10 GbE Switch Module.


◆ Layer 2 features

• Layer 2 Virtual Local Area Networks (VLANs): 4096• VLAN Encapsulation 802.1Q

IBM BladeCenter 601



602


• Rapid Spanning Tree Protocol (RSTP)• Multiple Spanning Tree MSTP (802.1s): 16 instances• Link Aggregation Control Protocol (LACP) IEEE- 802.3ad• Brocade enhanced frame-based trunking• Advanced PortChannel hashing based on Layer 2, 3, and 4

information• Pause Frames (802.3x)• Storm Control (unicast, multicast, and broadcast)• Address Resolution Protocol (ARP) RFC 826

◆ Layer 2 security

• Ingress Access Control Lists (ACLs)• Standard and extended Layer 2 ACLs• VLAN-based ACLs (VACLs)• Port-based ACLs (PACLs)• ACL statistics• Port-based Network Access Control: IEEE 802.1X

◆ Layer 2 Quality of Service (QoS)

• IEEE 802.1p Class of Service (CoS)• Eight hardware queues per port• Per-port QoS configuration• CoS trust: IEEE 802.1p• Per-port Virtual Output Queuing• CoS-based egress queuing• Egress strict priority queuing• Egress port-based scheduling: Weighted Round-Robin (WRR)

◆ Layer 3 features

• Static IP routes

FC interface External Fibre Channel ports that can operate as F_Ports (fabricports), FL_Ports (fabric loop ports), E_Ports (expansion ports), andN_Ports (when the switch is in Access Gateway mode).


◆ FC fabric services◆ Simple Name Server (SNS)◆ Registered State Change Notification (RSCN)◆ Dynamic Path Selection (DPS)◆ Enhanced Group Management (EGM)◆ ISL Trunking (optional)◆ Fabric Watch (optional)◆ Advanced Performance Monitor (optional)




The Ethernet end-device interface can be any connection that is on thenetwork-facing (external) side of the switch module.

IMPORTANT

Do not connect CNAs or FCoE software initiators from other hoststo any of the external 10 GbE interfaces on the Brocade 8470.

Brocade Converged 10 GbE configuration exampleThis section describes how to configure the Brocade Converged 10GbE Switch Module and contains the following information:




◆ “Configuring the switch modules” on page 607

Note: : In this example, the Brocade Converged module is attached to theenvironment previously configured. For complete configuration informationfor the Brocade switches, refer to the Fibre Channel SAN Topologies TechBook,available through the E-Lab Interoperability Navigator, Topology ResourceCenter tab, at http://elabnavigator.EMC.com.

Summary of configuration stepsThis section provides a summary of the steps to properly configurethe Brocade Converged 10 GbE Switch Module.

1. Disable the switch.

2. Configure FCoE settings on the server-facing interfaces.

3. Enable the external Ethernet interfaces.

4. Set a unique domain ID for the switch.

5. Enable the switch.

Existing topologyThe existing topology is shown in Figure 106 on page 604. Since thiscase study adds FCoE to an existing FC environment, only thosechanges required to add the Brocade Converged module will beshown.

IBM BladeCenter 603


604


For the complete configuration process for FC fabric, refer to the FibreChannel SAN Topologies TechBook, available through the E-LabInteroperability Navigator, Topology Resource Center tab, athttp://elabnavigator.EMC.com. For Brocade MLX configurationinformation, refer to “ED-DCX-B” on page 104.


Target topologyThe target topology for this case study is shown in Figure 107 onpage 605. Another possible topology, although not recommended, isshown in Figure 108 on page 607.

ICO-IMG-000943

StorageSAN BSAN A

Brocade MLX Brocade MLX

Brocade module 7IP = 10.246.60.24


HBA port

NIC port

Module 1(Ethernet)

Module 4(FC)

Module 3(FC)

Module 2(Ethernet)

Int 3 Int 4 Int 3 Int 4Int 3 Int 4 Int 3 Int 4

H H H HN NN N

Server3 Server4

IBM Blade Server chassis

1 3 4 2 3 41 21 22 1

1/1 2/1 2/11/1

1/21/3

1/21/3

1/1

1/1

Brocade DCX-2Domain ID 2



1/21/3

1/1

1 2 3 4


1/21/3

1/1

10 GbE8 Gb/s FC

H

N




Figure 107 Target topology, example 1

This target topology shows a Blade Server chassis containing twoserver blades (Server 3 and Server 4) as well as two BrocadeConverged 10 GbE Switch Modules (Module 7 and Module 9) beingadded to the environment. Note the following important pointsabout the configuration:

◆ Each server has a dual port CNA installed in it.

◆ On each server, one port from the CNA is hard wired to Module 7and the other port is hard wired to Module 9.

Switch module IPs:



Storage ports:Storage port1 (SPA0):WWPN=50:06:01:60:3B:60:03:CB

Storage port1 (SPA1):WWPN=50:06:01:61:3B:60:03:CB

Storage port1 (SPB0):WWPN=50:06:01:68:3B:60:03:Cb

Storage port1 (SPB1):WWPN=50:06:01:69:3B:60:03:Cb

Server 4:CAN_1:WWPN=21:00:00:c0:dd:10:12:1dStorage=Storage port 2 (SPA1)

CAN_2:WWPN=21:00:00:c0:dd:10:12:1fStorage=Storage port 4 (SPB1)

Server 3:CAN_1:WWPN=21:00:00:c0:dd:10:10:c9Storage=Storage port 1 (SPA0)

CAN_2:WWPN=21:00:00:c0:dd:10:10:cbStorage=Storage port 3 (SPB0)


SAN BSAN A


1/1 2/1 2/11/1

1/21/3

1/21/3

1/1

1/1



Storage

1 2 3 4



1/21/3

1/21/3

1/1

1/1

10 GbE8 Gb/s FC

C CNA port

1 2 2 1 2 22 22 21 15 5 6 6 5 63 43 46 5

Module 7Domain ID 5

Interface 0/1-0/14

Module 9Domain ID 6

/Interface 0/1-0 14

Int 4 Int 3 Int 4

C1 C2 C1 C2

Ser

ver

3

4

Ser

erv

Int 3

ICO-IMG-000944

IBM BladeCenter 605

606


• The port that the CNA attaches to corresponds to the slot thatthe server is installed in. For example, the CNA ports fromServer 3 attach to port 3 in Module 7 and port 3 in Module 9.

◆ Interfaces 0/1 - 0/14 will be referred to as server-facing ports.

Figure 108 on page 607 shows another possible target topology. Thistopology is not explained in detail since it a less attractive solution forthe following reasons:

◆ Limited number of FC ports on Brocade 8000 do not allow forscalability.

◆ The Brocade Converged Module as discussed is this section willnever support VE ports (allowing FCoE traffic between switches).

◆ With limited number of FC ports on Brocade 8000, it may notprovide a cost benefit to use one Brocade 8000 per chassis or pertwo chassis.



Figure 108 Target topology, example 2

Configuring the switch modulesThis section describes how to configure the switch modules andcontains the following information:

◆ “Assigning IP address” on page 608

◆ “Configuring switch module 7” on page 611

◆ “Configuring switch module 9” on page 613

◆ “Configuring fabric zoning” on page 615

◆ “Running configurations” on page 616

Module 7Domain ID 5

Module 9Domain ID 6

r e

ICO-IMG-000945



E Ethernet port

Fibre channel portFC


FCStorage SAN BSAN A


FCoEStorage

1 2 3 4

Ser

ve

Ser

vr

3 4

1 2 1 2

Int 3 Int 4 Int 3 Int 4

1 2 2 1 2 21 15 3 4 6 3 46 5

E

E

E

E

E

E

E

E

FC FC FC FC

FC FC FC FC

FC FC FC FC

FC FC FC FC

Brocade 8000Brocade 8000

8 Gb/s FC10 GbE



1 2 3 4

1/11/21/3

1/11/21/3

1/21/3

1/1

1/11/21/3 Brocade DCX-2

Domain ID 2


IBM BladeCenter 607

608


Assigning IP address When you first set up the switch module, use the blade enclosureAdvanced Management Module (AMM) to enter the initial IPinformation. You can then access the switch module through the IPaddress for further configuration.

IMPORTANT

Do not connect devices to the external ports on the switch moduleuntil you have completed the setup procedures and yourconfiguration matches that of the upstream network.

To assign an IP address to each switch module and enable externalports, complete the following steps:

1. Obtain the static IP address and the default gateway address fromyour system administrator.

2. Establish a connection to the AMM, as described in the IBMBladeCenter Advanced Management Module User's Guide. In thelogin window, enter your User ID and password and click Log In.

Note: AMM refers to the switch module as the I/O module.

3. In the Welcome window, click Continue. The System Statussummary pane displays.

4. Expand I/O Module Tasks and click Admin/Power/Restart.

The I/O Module Power/Restart window displays, as shown inFigure 109 on page 609.



Figure 109 AMM I/O Module Power/Restart window

5. In the right pane, from the External ports drop-down list, selectEnabled to enable external ports on the switch module for eachmodule.

6. Click Save, located at the lower, right-hand portion of thewindow, to save your settings.

7. In the left pane, expand I/O Module Tasks and clickConfiguration. The I/O Module Configuration Windowdisplays, as shown in Figure 110 on page 610.

IBM BladeCenter 609

610


Figure 110 I/O Module Configuration window

8. In the right pane, select the panel corresponding to the location ofthe I/O module (switch module) that you are configuring.

Note: The switch module can only be installed in Bays 7 through 10. Theapplicable bay number appears in the pane, followed by other relatedI/O-module information, including the IP address. I/O-moduleinformation is divided into two areas: Current IP Configuration andNew Static IP Configuration (see Figure 110).

9. In the New Static IP Configuration area, enter the new IPaddress, the Subnet mask, and the Gateway address and thenclick Save, located at the lower, right-hand portion of thewindow.



10. Click the Advanced Options link.

This opens up a new window, Advanced configurations for I/Omodule 7. In this window, change the following parametersunder Advance Setup:

a. Enable external management for all ports.b. Preserve the new IP configuration on all resets.

Note: These features are disabled by default.

c. Click Save to save your settings.

11. Repeat Step 9 and Step 10 to configure an I/O module in slot 9-10with an IP address of 10.246.60.25.

12. Exit the AMM web interface.

Configuring switchmodule 7

Complete the following steps to Telnet into the switch module 7 withIP address 10.246.60.24.

Note: The default username is USERID and the password is PASSW0RD.(Both are case-sensitive and the "0" in PASSWORD is the number zero, notthe letter.)

1. Telnet into switch module 7.

C:\>telnet 10.246.60.24Fabric OS (B8470)Fabos Version 6.3.1_cee_rc2_bld06

B8470 login: USERIDPassword: PASSW0RD

2. Change passwords for default accounts. For this example, we willskip this section by pressing Ctrl+C. You can change at thisprompt or at a later time.

Change passwords for switch default accounts now. Press Ctrl+Cto exit or Enter to proceed.

The password was not changed. You will be prompted again atnext login until the password is changed.


IBM_Frisco_6024:USERID> switchdisable

4. Enter CEE configuration menu by entering the cmsh (CEEmanagement Shell) command.

IBM_Frisco_6024:USERID> cmsh

IBM BladeCenter 611

612


5. Enter configure terminal command.

B8470# configure terminalEnter configuration commands, one per line. End with CNTL/Z.

6. Enable FCoE on all the internal (Server facing) ports. The fcoeportcommand will apply the default FCoE profile map for these ports.

B8470(config)# interface intengigabitethernet 0/1B8470(conf-if-int-0/1)#fcoeportB8470(conf-if-int-0/1)#exit

7. Repeat Step 6 for interfaces intengigabitethernet 0/2 tointengigabitethernet 0/14.

8. Enable Ethernet interfaces.

B8470(config)# interface extengigabitethernet 0/18B8470(conf-if-ext-0/18)#no shut

Enable the remaining Ethernet interfaces 15 to 22 by repeatingStep 8.

9. Exit the CEE configuration menu.

B8470(config)# exit

10. Copy configuration data to startup file.

B8470# copy running-config startup-configOverwrite the startup config file (y/n): yBuilding configuration...

B8470# exit11. Configure the domain ID of switch. The domain ID has to be

unique in the SAN.

IBM_Frisco_6024:USERID> configureConfigure...

Fabric parameters (yes, y, no, n): [no] yDomain: (1..239) [1] 5R_A_TOV: (4000..120000) [10000]E_D_TOV: (1000..5000) [2000]WAN_TOV: (0..30000) [0]MAX_HOPS: (7..19) [7]Data field size: (256..2112) [2112]Sequence Level Switching: (0..1) [0]Disable Device Probing: (0..1) [0]Suppress Class F Traffic: (0..1) [0]Per-frame Route Priority: (0..1) [0]Long Distance Fabric: (0..1) [0]BB credit: (1..27) [16]Disable FID Check (yes, y, no, n): [no]



Insistent Domain ID Mode (yes, y, no, n): [no]Configure edge hold time (yes, y, no, n): [no]Virtual Channel parameters (yes, y, no, n): [no]F-Port login parameters (yes, y, no, n): [no]Zoning Operation parameters (yes, y, no, n): [no]RSCN Transmission Mode (yes, y, no, n): [no]Arbitrated Loop parameters (yes, y, no, n): [no]System services (yes, y, no, n): [no]Portlog events enable (yes, y, no, n): [no]ssl attributes (yes, y, no, n): [no]rpcd attributes (yes, y, no, n): [no]cfgload attributes (yes, y, no, n): [no]webtools attributes (yes, y, no, n): [no]Custom attributes (yes, y, no, n): [no]system attributes (yes, y, no, n): [no]System (yes, y, no, n): [no]

WARNING: The domain ID will be changed. The port level zoning may be affected


IBM_Frisco_6024:USERID> switchenable

13. Log out of the session to switch module 7.


Configuring switchmodule 9

Complete the following steps to Telnet into the switch module 9 withIP address 10.246.60.24.

Note: The default username is USERID and the password is PASSW0RD.(Both are case-sensitive and the "0" in PASSWORD is the number zero, notthe letter.)

1. Telnet into switch module 9.

C:\>telnet 10.246.60.24Fabric OS (B8470)Fabos Version 6.3.1_cee_rc2_bld06

B8470 login: USERIDPassword: PASSW0RD

2. Change passwords for default accounts. For this example, we willskip this section by pressing Control C. You can change at thisprompt or at a later time.

Change passwords for switch default accounts now. Press Ctrl+Cto exit or Enter to proceed.

IBM BladeCenter 613

614


The password was not changed. You will be prompted again atnext login until the password is changed.


IBM_Frisco_6024:USERID> switchdisable

4. Enter CEE configuration menu by entering the cmsh command.

IBM_Frisco_6024:USERID> cmsh

5. Enter configure terminal command.

B8470# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

6. Enable FCoE on all the internal (Server facing) ports. The fcoeportcommand will apply the default FCoE profile map for these ports.

B8470(config)# interface intengigabitethernet 0/1B8470(conf-if-int-0/1)#fcoeportB8470(conf-if-int-0/1)#exit

7. Repeat Step 6 for interfaces intengigabitethernet 0/2 tointengigabitethernet 0/14.

8. Enable Ethernet interfaces.

B8470(config)# interface extengigabitethernet 0/18B8470(conf-if-ext-0/18)#no shut

Enable the remaining Ethernet interfaces 15 to 22 by repeatingStep 8.

9. Exit the CEE configuration menu.

B8470(config)# exit

10. Copy configuration data to startup file.

B8470# copy running-config startup-configOverwrite the startup config file (y/n): yBuilding configuration...

B8470# exit11. Configure the domain ID of switch. The domain ID has to be

unique in the SAN.

IBM_Frisco_6024:USERID> configureConfigure...

Fabric parameters (yes, y, no, n): [no] yDomain: (1..239) [1] 6



R_A_TOV: (4000..120000) [10000]E_D_TOV: (1000..5000) [2000]WAN_TOV: (0..30000) [0]MAX_HOPS: (7..19) [7]Data field size: (256..2112) [2112]Sequence Level Switching: (0..1) [0]Disable Device Probing: (0..1) [0]Suppress Class F Traffic: (0..1) [0]Per-frame Route Priority: (0..1) [0]Long Distance Fabric: (0..1) [0]BB credit: (1..27) [16]Disable FID Check (yes, y, no, n): [no]Insistent Domain ID Mode (yes, y, no, n): [no]

Configure edge hold time (yes, y, no, n): [no]Virtual Channel parameters (yes, y, no, n): [no]F-Port login parameters (yes, y, no, n): [no]Zoning Operation parameters (yes, y, no, n): [no]RSCN Transmission Mode (yes, y, no, n): [no]Arbitrated Loop parameters (yes, y, no, n): [no]System services (yes, y, no, n): [no]Portlog events enable (yes, y, no, n): [no]ssl attributes (yes, y, no, n): [no]rpcd attributes (yes, y, no, n): [no]cfgload attributes (yes, y, no, n): [no]webtools attributes (yes, y, no, n): [no]Custom attributes (yes, y, no, n): [no]system attributes (yes, y, no, n): [no]System (yes, y, no, n): [no]

WARNING: The domain ID will be changed. The port level zoning may be affected



13. Log out of the session to switch module 7.


Configuring fabriczoning

Once the connections are completed, zone WWPNs of the CNA onthe server blades to the WWPNs of the storage ports. Zoning willneed to be performed on the fabric using CLI or managementapplication.

The following is a CLI method to zone server and storage ports.

1. Zone each server and storage port.

IBM_Frisco_6025:admin> zonecreate "Srv3_HBA1_SPA0", "21:00:00:c0:dd:10:10:c9;50:06:01:60:3B:60:03:CB"IBM_Frisco_6025:admin> zonecreate "Srv3_HBA2_SPB0", "21:00:00:c0:dd:10:10:cb;50:06:01:68:3B:60:03:Cb"

IBM BladeCenter 615

616


IBM_Frisco_6025:admin> zonecreate "Srv4_HBA1_SPA1", "21:00:00:c0:dd:10:12:1d;50:06:01:61:3B:60:03:CB"IBM_Frisco_6025:admin> zonecreate "Srv4_HBA2_SPB1", "21:00:00:c0:dd:10:12:1f;50:06:01:69:3B:60:03:Cb"

2. Create the zoneset.

IBM_Frisco_6025:admin> cfgcreate "IBM_Brocade_FCoE", "Srv3_HBA1_SPA0;Srv3_HBA2_SPB0; Srv4_HBA1_SPA1; Srv4_HBA2_SPB1"

3. Enable this newly created zoneset onto the fabric,

IBM_Frisco_6025:admin> cfgenable "IBM_Brocade_FCoE"

You are about to enable a new zoning configuration. This action will replace theold zoning configuration with the current configuration selected. If the updateincludes changes to one or more traffic isolation zones, the update may resultin localized disruption to traffic on ports associated with the trafficisolation zone changes.

Do you want to enable 'IBM_Brocade_FCoE' configuration (yes, y, no, n): [no] y

zone config "IBM_Brocade_FCoE" is in effectUpdating flash ...

Runningconfigurations

This section provides output for the following:

◆ “Switch module 7 and 9 CEE configuration” on page 616

◆ “Zoning configuration” on page 619

Switch module 7 and 9 CEE configurationThe following is an example of the output from the show runningconfiguration command for the switch module in slot 7 and 9.

Note: Both modules have the same exact CEE configuration.

B8470# show running-config!no protocol spanning-tree!cee-map defaultpriority-group-table 1 weight 40 pfcpriority-group-table 2 weight 60priority-table 2 2 2 1 2 2 2 2

!fcoe-map defaultfcoe-vlan 1002

!interface Vlan 1shutdown

!



interface InTengigabitEthernet 0/1fcoeportswitchportswitchport mode convergedno shutdown

!interface InTengigabitEthernet 0/2fcoeportswitchportswitchport mode convergedno shutdown







!interface InTengigabitEthernet 0/9fcoeportswitchportswitchport mode converged

IBM BladeCenter 617

618


no shutdown!interface InTengigabitEthernet 0/10fcoeportswitchportswitchport mode convergedno shutdown





!interface ExTengigabitEthernet 0/15shutdown



!interface ExTengigabitEthernet 0/18no shutdown









line vty 0 31login

!end

B8470#

Zoning configurationThe following is an example of zoning configuration output from oneof the switches.

IBM_Frisco_6025:admin> cfgshowDefined configuration:cfg: IBM_Brocade_FCoE

Srv3_HBA1_SPA0; Srv3_HBA2_SPB0; Srv4_HBA1_SPA1; Srv4_HBA2_SPB1zone: Srv3_HBA1_SPA0

21:00:00:c0:dd:10:10:c9; 50:06:01:60:3B:60:03:CBzone: Srv3_HBA2_SPB0

21:00:00:c0:dd:10:10:cb; 50:06:01:68:3B:60:03:Cbzone: Srv4_HBA1_SPA1

21:00:00:c0:dd:10:12:1d; 50:06:01:61:3B:60:03:CBzone: Srv4_HBA2_SPB1

21:00:00:c0:dd:10:12:1f; 50:06:01:69:3B:60:03:CbEffective configuration:cfg: IBM_Brocade_FCoEzone: Srv3_HBA1_SPA0

21:00:00:c0:dd:10:10:c950:06:01:60:3B:60:03:CB

zone: Srv3_HBA2_SPB021:00:00:c0:dd:10:10:cb50:06:01:68:3B:60:03:Cb

zone: Srv4_HBA1_SPA121:00:00:c0:dd:10:12:1d50:06:01:61:3B:60:03:CB

zone: Srv4_HBA2_SPB121:00:00:c0:dd:10:12:1f50:06:01:69:3B:60:03:Cb

IBM_Frisco_6025:admin>

IBM BladeCenter 619

620


Cisco Nexus 4000 supported features and topologiesThe Nexus 4000 supported features and topologies have been brokendown into four areas, as shown in Figure 111 on page 620, and each isdiscussed further in this section.



◆ “FC interface ” on page 622



Figure 111 Nexus 4000 topology example

MDS Brocade

Storage

FCInterface



FCoEInterface

Nexus 4000

Nexus 5000

Celerra

10 GbENIC

10 GbENIC

ServerServer

CNA CNA

Server

SYM-002233



FCoE interface Note: The FCoE interface is any connection that is on the server-facing(internal) side of the switch module. Do not connect CNAs or FCoE softwareinitiators from other hosts to any of the external 10 GbE interfaces on theNexus 4000.


Note: Always refer to the EMC Support Matrix for the most up-to-datesupport information.

◆ QLogic CNAs only– Data Center Bridging (DCB); required

◆ OS support

No FCoE software initiators are supported.


Note: The Ethernet uplink interface is any connection that is on thenetwork-facing (external) side of the switch module. Do not connect CNAs orFCoE software initiators from other hosts to any of the external 10 GbEinterfaces on the Nexus 4000.


◆ Nexus 5548, Nexus 5020, or Nexus 5010 for FCoE connectivity

◆ DCB (Required)• Nexus 7000, Catalyst 6k, Brocade MLX, Brocade 8000 for

non-FCoE network connectivity

• Neither Pause 802.3x nor DCB will be required when on linksthat do not carry FCoE traffic

◆ FIP-Snooping (Required)

◆ Etherchannels for link aggregation• Support up to 256 VLANs

◆ 802.1w, per VLAN Rapid Spanning Tree (PVRST+) (Preferred)

◆ 802.1s Multiple Spanning Tree Protocol (MSTP)

◆ Spanning Tree PortFast and PortFast Guard

◆ Spanning Tree UplinkFast and BackboneFast

◆ Bridge Protocol Data Unit (BPDU) Guard


◆ Internet Group Management Protocol (IGMP) snooping

◆ IGMP snooping querier

IBM BladeCenter 621


622


◆ Protocol Independent Multicast (PIM)

◆ Cisco Etherchannel• 8 members per channel• 12 channels per Nexus 5000 chassis


◆ Advanced port channel hashing


◆ Storm control

◆ Private VLANs / Private VLANs over trunks

FC interface This is not applicable on the Nexus 4000. Connectivity to FibreChannel devices is through the Nexus 5000.


Note: The Ethernet end-device interface can be any connection that is on thenetwork-facing (external) side of the switch module. Do not connect CNAs orFCoE software initiators from other hosts to any of the external 10 GbEinterfaces on the Nexus 4000.





◆ NIC Teaming

Cisco Nexus 4001I configuration example


Note: When the information presented is applicable to only specific switches,the model number is used. Nexus 5000 is used when information is relevantto all Nexus 5000 series switches.

This section details the setup steps required to create theconfiguration shown in Figure 113 on page 625. It includes thefollowing information:







Note: In this example, the Nexus 4000 is attached to the environmentpreviously configured in the Nexus 5020 setup example. For completeconfiguration information for the Nexus 5020, refer to “Nexus 5000direct-connect topology” on page 173.

Summary of configuration stepsThis section provides a summary of the steps to properly configurethe Nexus 4000 and Nexus 5000 to allow for the use of FCoE.

On the Nexus 4001I 1. Enable the fip snooping feature.

2. Create a policy map.

3. Add queuing class types and bandwidth parameters.

4. Add a class map for FCoE and non-FCoE.

5. Add a policy map for FCoE.

6. Enable service policy using system QOS.

7. Define the FCoE VLAN, enable FIP Snooping and define theFC-MAP value.

8. Configure the server facing interfaces (1/1—1/14).

9. Create a port channel for the network facing interfaces.

10. Add the network facing interfaces (1/15—1/20) to the portchannel.

On the Nexus 5000 Note: This example builds on the previous Nexus case study discussed in“Nexus 5000 direct-connect topology” on page 173. Therefore, only thosechanges required to connect the Nexus 4001I will be performed.

1. Create the VFCs.

Note: The VFCs must be bound to the ENode MAC of the CNA and notthe physical interface on the Nexus 5000.

2. Add the VFCs to the appropriate VLAN.

3. Create a port channel for connectivity to the Nexus 4001I.

4. Add interfaces to the port channel.

IBM BladeCenter 623

624


Existing topologyThe existing topology is shown in Figure 112 on page 624. Since thiscase study adds onto an existing FCoE environment, only thosechanges required to add the Nexus 4001I will be shown. For thecomplete configuration process, refer to “Nexus 5000 direct-connecttopology” on page 173 first. In addition, the final configuration filesfor the Nexus 4001I and the Nexus 5020 are included in the Non-EMCSAN Products Data Reference Manual available through the E-LabInteroperability Navigator, Topology Resource Center tab, athttp://elabnavigator.EMC.com.


2/11/1 1/1 2/1


Port-channel 1

Port-channel 2

1/21/31/41/5

1/21/31/41/5


1/1

1/2

1/1

1/2

SAN A SAN BStorage

1 2 3 4

1/1

1/1

DCX_1Domain 1

172.23.185.24

DCX_2Domain 2

172.23.185.25

SAN PortChannels

1/11/21/31/41/5

1/11/21/31/41/5


172.23.185.22


172.23.185.23


Host_1(Windows)

Host_2(Linux)

VLAN / VSAN 100

VLAN / VSAN 200

VLAN / VSAN 300

VLAN / VSAN 400

10 GbE

4 Gb/s FC




CNA_2:

Nexus-5020-1

IP = 172.23.185.113 Domain = 3



Storage port 1: 50:06:04:82:D5:2E:69:79

Storage port 2: 50:06:04:82:D5:2E:69:49


Storage port 4: 50:06:04:82:D5:2E:69:59VSAN = 400 SYM-002234




Target topologyThis section provides the target topology for this case study, alongwith the following information:

◆ “FIP Snooping ” on page 626

◆ “QOS and queuing” on page 627

The target topology for this case study is shown in Figure 113.


2/11/1 1/1 2/1


Port-channel 1

Port-channel 2

1/21/31/41/5

1/21/31/41/5


1/1

1/2

SAN A SAN BStorage

1 2 3 4

1/1

1/1

DCX_1Domain 1

172.23.185.24

DCX_2Domain 2

172.23.185.25

SAN PortChannels

1/11/21/31/41/5

1/11/21/31/41/5


172.23.185.22


172.23.185.23

VLAN / VSAN 100

VLAN / VSAN 200

VLAN / VSAN 300

VLAN / VSAN 400

10 GbE

4 Gb/s FC

SERVER 11:CNA_1:VSAN = 100WWPN= 21:00:00:c0:dd:10:12:1dENode MAC = 00:c0:dd:10:12:1dStorage = Storage port 1

CNA_2:VSAN = 200WWPN = 21:00:00:c0:dd:10:12:1fENode MAC = 00:c0:dd:10:12:1fStorage = Storage port 3

SERVER 12:CNA_1:VSAN = 300WWPN = 21:00:00:c0:dd:10:10:c9ENode MAC = 00:c0:dd:10:10:c9 Storage = Storage port 2

CNA_2:VSAN = 400WWPN = 21:00:00:c0:dd:10:10:cbENode MAC = 00:c0:dd:10:10:cbStorage = Storage port 4

Nexus-5020-1IP = 172.23.185.112Domain = N/A (NPV mode)

Nexus-5020-2IP = 172.23.185.113Domain = 3

Module 7In Slot 7 of Blade server chassisIP = 172.23.199.77

Module 9In Slot 9 of Blade server chassisIP = 172.23.199.78

Storage port 1:50:06:04:82:D5:2E:69:79

Storage port 2:50:06:04:82:D5:2E:69:49


Storage port 4:50:06:04:82:D5:2E:69:59VSAN = 400 SYM-002235

1/11/9 1/13T

T

T

T

T

T

1/21/10 1/14

1/11

1/12

1/9 1/13T

T

T

T

T

T

T T T T T T T T T T T T

1/10 1/14

1/11

1/12

Module 7

Blade Server chassis

Server 11

Interfaces 1/15 – 1/20

Interfaces 1/1 – 1/14Module 9



11 12

1 2

Server 12

1 2

11 12

IBM BladeCenter 625

626


Note: For the sake of clarity, the existing hosts are not shown in Figure 113.

The target topology shows a Blade Server chassis containing twoserver blades (Server 11 and Server 12) as well as two Nexus 4001Is(Module 7 and Module 9) being added to the environment.

Note the following important points about the configuration:

◆ Each server has a dual port CNA installed in it.◆ On each server, one port from the CNA attaches to Module 7 and

the other port attaches to Module 9.• The port that the CNA attaches to corresponds to the slot that

the server is installed in. For example, the CNA ports fromServer 11 attach to port 11 in Module 7 and port 11 in Module9.

◆ Interfaces 1/1 – 1/14 will be referred to as server-facing ports.◆ Interfaces 1/15 – 1/20 are the external ports on the back of the

blade server chassis and are used to connect to the network.Because of this, they will be referred to as network-facing ports.

◆ All of the network-facing ports from each module have beenadded to a single port channel and attached to the same Nexus5020.

FIP Snooping As previously mentioned, there are two basic topologies: directconnect and DCB Cloud. For more information on direct connect andDCB Cloud, refer to the “FCoE Initialization Protocol (FIP)” section inthe Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB)Concepts and Protocols TechBook at http://elabnavigator.EMC.com,under the Topology Resource Center tab.

In the following example, since the CNAs are connected to the Nexus4000 and the Nexus 4000 does not contain an FCF, the resultingtopology is actually an example of a DCB Cloud. Since the Nexus4000 does not contain an FCF, the configuration is susceptible tocertain types of attacks (such as learning and FCID spoofing) that canonly be prevented by the use of FIP Snooping. Because the Nexus4000 is a FIP Snooping bridge, it can prevent these types of attacksfrom being successful.

FIP Snooping is only enabled on the network-facing ports and shouldonly be enabled on those ports that will be directly connected to aNexus 5000. The reason for this is that when you enable FIP Snoopingon a particular interface, you are allowing frames from an FCF to bereceived and forwarded from those interfaces. In other words, you




are saying whatever is attached to the other end of the cable (a Nexus5000) is a trusted port.

Another important security feature of the Nexus 4000 is that it willpin an FCoE device's VN_Port MAC to the interface where itperformed the FIP FLOGI. This prevents a rogue FCoE host fromsending an FCoE frame that contains a victim FCoE host’s SA, whichprevents learning and FCID spoofing attacks.

QOS and queuing When properly configured, the configuration of the Nexus 4000 (asobserved from a show running-config command) will containnumerous QOS-related settings. It is possible to automatically setthese QOS parameters by using the setup script from the Nexus 4000CLI. To do so, answer Y when prompted about FCoE and FIPSnooping. Because the QOS-related setting can be automaticallyconfigured, the following information is provided only to help youunderstand what these parameters are intended to do.

The following configuration parameters assign 50% of the bandwidthto COS 3 (used for FCoE) and divides the remainder of thebandwidth equally among COS values 0-2 and 4-7:

policy-map type queuing policy-fcoe-bandwidthclass type queuing 1p7q4t-out-q-defaultbandwidth percent 50class type queuing 1p7q4t-out-pq1bandwidth percent 7class type queuing 1p7q4t-out-q2bandwidth percent 7class type queuing 1p7q4t-out-q3bandwidth percent 7class type queuing 1p7q4t-out-q4bandwidth percent 7class type queuing 1p7q4t-out-q5bandwidth percent 7class type queuing 1p7q4t-out-q6bandwidth percent 7class type queuing 1p7q4t-out-q7bandwidth percent 7

The following configuration parameters map frames with a COS of 3to the class-fcoe and other COS types to class-non-fcoe:

class-map type network-qos class-fcoematch cos 3class-map type network-qos class-non-fcoematch cos 0-2,4-7

IBM BladeCenter 627

628


The following configuration parameters specify that FCoE will have amaximum MTU of 2500 and a no-drop behavior (lossless):

policy-map type network-qos policy-fcoeclass type network-qos class-fcoepause no-dropmtu 2500class type network-qos class-non-fcoe

The following configuration parameters enable what was configured:

system qosservice-policy type network-qos policy-fcoeservice-policy type queuing output policy-fcoe-bandwidth

Configuring the Nexus 4000This section describes how to configure the switch and includes thefollowing topics:

◆ “Assigning the IP address” on page 628

◆ “Initial setup” on page 631

◆ “Configuring Module 7 (Switch_77)” on page 633

◆ “Configuring Module 9 (Switch_78)” on page 636



◆ “Configure fabric zoning” on page 640

◆ “show running-config output examples” on page 640

Assigning the IPaddress

When you first set up the switch module, use the blade enclosureAdvanced Management Module (AMM) to enter the initial IPinformation. You can then access the switch module through the IPaddress for further configuration.

IMPORTANT

Do not connect devices to the external ports on the switch moduleuntil you have completed the setup procedures and yourconfiguration matches that of the upstream network.

To assign an IP address to the switch module, perform the followingsteps:

1. Obtain the static IP address and the default gateway address fromyour system administrator.



2. Establish a connection to the AMM, as described in the IBM BladeCenter Advanced Management Module User’s Guide. In the loginwindow, enter your User ID and password, and click Log In.

Note: AMM refers to the switch module as the I/O module.

3. In the Welcome window, click Continue.

The System Status summary pane appears.

4. In the I/O Module Advanced Setup area, expand I/O ModuleTasks and click Admin/Power/Restart (see Figure 114).

Figure 114 AMM I/O Module Admin/Power/Restart window

5. From the External ports drop-down list, select Enabled to enableexternal ports on the switch module.

6. Click Save to save your settings.

IBM BladeCenter 629

630


7. In the left pane, expand I/O Module Tasks, and clickConfiguration.

8. In the I/O Module Power/Restart area, select the checkbox for thebay that corresponds to the location of the I/O module (switchmodule) that you are configuring.

Note: The switch module can only be installed in Bays 7 through 10. Theapplicable bay number appears in the pane, followed by other relatedI/O-module information, including the IP address. The I/O-moduleinformation is divided into two areas: Current IP Configuration and NewStatic IP Configuration (see Figure 115).

Figure 115 AMM I/O Module Configuration window

9. In the New Static IP Configuration area, enter the new IPaddress, the Subnet mask, and the Gateway address. Click Save.

Note: The Save button is located in the lower, right portion of thewindow.

10. Click the Advanced Configuration link. Take the followingactions for switch module features:

a. Enable external management for all ports.

b. Preserve the new IP configuration on all resets.



Note: These features are disabled, by default.

11. (Optional) To begin a Telnet session to the switch module, clickthe Start Telnet Session link. Click Start Telnet.

12. Click Save to save your settings. Exit the AMM web interface.

Initial setup To enter the basic configuration parameters, perform the followingsteps:

1. Open the AMM and configure the IP address, subnet mask, andgateway address for the management interface (mgmt1), asdescribed in “Assigning the IP address” on page 628.

2. Telnet to the mgmt1 IP address and log in using the following(default) user ID and password:

• Login ID is USERID.

• Password is PASSW0RD (the 0 in PASSW0RD is a zero).

You can now configure the switch module.

Note: Register the switch module device immediately with yoursupplier. Failure to register may affect response times for the initialservice call. The device must be registered to receive entitled supportservices.

3. Enter the basic configuration information. The following exampleshows how to start the basic configuration setup:

n4k-8# setup---- Basic System Configuration Dialog ----This setup utility will guide you through the basic configuration ofthe system. Setup configures only enough connectivity for managementof the system.

*Note: setup is mainly used for configuring the system initially,when no configuration is present. So setup always assumes systemdefaults and not the current system configuration values.Press Enter at anytime to skip a dialog. Use ctrl-c at anytimeto skip the remaining dialogs.

4. Enter the setup mode by entering yes (or y), as in the followingexample:

Would you like to enter the basic configuration dialog (yes/no): yes

IBM BladeCenter 631

632


5. (Optional) Create additional accounts by entering yes (or y), as inthe following example (no is the default):

Create another login account (yes/no) [n]: yesEnter the User login Id: <ID>Enter the password for "qatest": <password>Confirm the password for "qatest": <password>Enter the user role [network-operator]: <role>

6. (Optional) Configure an SNMP community string by entering yes(or y), as in the following example (no is the default):

Configure read-only SNMP community string (yes/no) [n]: ySNMP community string: <string>

7. Enter a name for the switch, as in the following example:

Enter the switch name: Switch-77

8. Configure out-of-band management by entering yes (or y), as inthe following example:

Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: yMgmt0 IPv4 address: 10.10.10.1Mgmt0 IPv4 netmask: 255.255.255.0

9. Configure the IPv4 default gateway (recommended) by enteringyes (or y). Enter the gateway IP address.

Configure the default gateway? (yes/no) [y]:yIPv4 address of the default gateway: <IP address>

10. Enable the Telnet service by entering yes (or y), as in thefollowing example:

Enable the telnet service? (yes/no) [y]: y

11. Enable the SSH service by entering yes (or y), as in the followingexample (the default is no):

Enable the ssh service? (yes/no) [n]:n

12. Configure the NTP server by entering yes (or y), as in thefollowing example (the default is no):

Configure the ntp server? (yes/no) [n]:n

13. Configure the FCOE service by entering yes (or y), as in thefollowing example (the default is no):

Enable FCOE service? (yes/no) [n]:n



After the prompt for the FCOE service, the configuration appears,as in the following example:

The following configuration will be applied:username qatest password <user-password> role network-operatorsnmp-server community topspin roswitchname ibm-switch-1interface mgmt0ip address 10.10.10.1 255.255.255.0no shutdownip route 0.0.0.0/0 10.10.10.100telnet server enableno ssh server enableWould you like to edit the configuration? (yes/no) [n]:Use this configuration and save it? (yes/no) [y]: y

14. If you want to make changes to the displayed configuration, enteryes (or y); otherwise accept the default (no) by pressing Enter.

If you enter yes, the setup utility returns to the beginning of thesetup, and repeats each step.

15. Save this configuration by entering yes (or y), as in the followingexample (the default is no):

Use this configuration and save it? (yes/no) [y]: ySwitch_77 #

If you do not save the configuration at this point, none of yourchanges are part of the configuration the next time the devicereboots. Saving the configuration also automatically configuresthe boot variables for the kickstart and system images.

Note: The switch module has two out-of-band management interfaces.The AMM configuration is mgmt1. The mgmt0 interface must be placedon a different subnet than mgmt1.

Note: These steps will need to be repeated for each Nexus 4000.

Configuring Module 7(Switch_77)

To configure Module 7 (Switch_77):

1. Telnet into Module 7 (Switch_77).

Note: The default username is USERID and the password is PASSW0RD.(Both are case-sensitive and the "0" in PASSWORD is a zero, and not theletter.)

IBM BladeCenter 633

634


telnet 172.23.199.77Nexus 4000 Switchswitch login: USERIDPassword:Switch_77 #

2. Enter configuration mode.

switch# configure terminalEnter configuration commands, one per line. End with CNTL/Z.Switch_77(config)#

3. Enable the fip snooping feature.

Switch_77(config)# feature fip-snoopingSwitch_77(config)#

4. Create a policy map, add queuing class types and bandwidthparameters.

Switch_77(config)# policy-map type queuing policy-fcoe-bandwidthSwitch_77(config-pmap-que)# class type queuing 1p7q4t-out-q-defaultSwitch_77(config-pmap-c-que)# bandwidth percent 50Switch_77(config-pmap-c-que)# class type queuing 1p7q4t-out-pq1Switch_77(config-pmap-c-que)# bandwidth percent 7Switch_77(config-pmap-c-que)# class type queuing 1p7q4t-out-q2Switch_77(config-pmap-c-que)# bandwidth percent 7Switch_77(config-pmap-c-que)# class type queuing 1p7q4t-out-q3Switch_77(config-pmap-c-que)# bandwidth percent 7Switch_77(config-pmap-c-que)# class type queuing 1p7q4t-out-q4Switch_77(config-pmap-c-que)# bandwidth percent 7Switch_77(config-pmap-c-que)# class type queuing 1p7q4t-out-q5Switch_77(config-pmap-c-que)# bandwidth percent 7Switch_77(config-pmap-c-que)# class type queuing 1p7q4t-out-q6Switch_77(config-pmap-c-que)# bandwidth percent 7Switch_77(config-pmap-c-que)# class type queuing 1p7q4t-out-q7Switch_77(config-pmap-c-que)# bandwidth percent 7

5. Add a class map for FCoE and non-FCoE.

Switch_77(config-pmap-c-que)# class-map type network-qos class-fcoeSwitch_77(config-cmap-nq)# match cos 3Switch_77(config-cmap-nq)# class-map type network-qos class-non-fcoeSwitch_77(config-cmap-nq)# match cos 0-2,4-7

6. Add a policy map for FCoE.

Switch_77(config-cmap-nq)# policy-map type network-qos policy-fcoeSwitch_77(config-pmap-nq)# class type network-qos class-fcoeSwitch_77(config-pmap-nq-c)# pause no-dropSwitch_77(config-pmap-nq-c)# mtu 2500Switch_77(config-pmap-nq-c)# class type network-qos class-non-fcoe



7. Enable service policy using system QOS.

Switch_77(config-pmap-nq-c)# system qosSwitch_77(config-sys-qos)# service-policy type network-qos policy-fcoeSwitch_77(config-sys-qos)# service-policy type queuing output

policy-fcoe-bandwidth8. Define the FCoE VLANs, enable FIP Snooping, and define the

FC-MAP value.

Switch_77(config-sys-qos)# vlan 100Switch_77(config-vlan)# fip-snooping enableSwitch_77(config-vlan)# fip-snooping fc-map 0x0efc00Switch_77(config-vlan)# vlan 300Switch_77(config-vlan)# fip-snooping enableSwitch_77(config-vlan)# fip-snooping fc-map 0x0efc00

9. Configure the server-facing interfaces (1/1 – 1/14).

Note: As indicated in the diagram, interface 1/11 will be added to VLAN100 and interface 1/12 will be added to vlan 300.

Note: After configuring the interface as a Edge port type, the followingwarning message will be displayed. This is normal and can be ignored.

Warning: Edge port type (portfast) should only be enabled on portsconnected to a single host. Connecting hubs, concentrators, switches,bridges, and so on, to this interface when edge port type (portfast) isenabled, can cause temporary bridging loops. Use with CAUTION.

Switch_77(config-vlan)# interface Ethernet1/11Switch_77(config-if)# switchport mode trunkSwitch_77(config-if)# switchport trunk allowed vlan 1,100Switch_77(config-if)# spanning-tree port type edge trunkSwitch_77(config-if)# speed autoSwitch_77(config-if)# interface Ethernet1/12Switch_77(config-if)# switchport mode trunkSwitch_77(config-if)# switchport trunk allowed vlan 1,300Switch_77(config-if)# spanning-tree port type edge trunkSwitch_77(config-if)# speed auto

10. Create a port channel for the network-facing interfaces.

Switch_77(config-if)# interface port-channel 1

11. Add the network-facing interfaces (1/15-1/20) to the portchannel.

Switch_77(config-if)# int e1/15-20Switch_77(config-if-range)# channel-group 1 mode active

IBM BladeCenter 635

636


12. Configure the port channel’s parameters.

Switch_77(config-if-range)# int port-channel 1Switch_77(config-if)# switchport mode trunkSwitch_77(config-if)# switchport trunk allowed vlan 1,100,300Switch_77(config-if)# fip-snooping port-mode fcfSwitch_77(config-if)# no shut

Configuring Module 9(Switch_78)

Follow the same procedures performed in “Configuring Module 7(Switch_77)” on page 633 with the following exceptions:

◆ The IP address to use for Module 9 will be 172.23.199.78.

◆ The switch name should be set to Switch_78.

◆ The VLANs to use should be changed to 200 and 400.


Note: The steps provided below assume that the switches have already beenconfigured as explained in “Nexus 5000 direct-connect topology” onpage 173.


1. Telnet into Nexus-5020-1.

telnet 172.23.185.112Nexus 5000 Switchlogin: adminPassword: xxxxxxNexus-5020-1#

2. Enter the configuration mode.

Nexus-5020-1# config tEnter configuration commands, one per line. End with CNTL/Z.Nexus-5020-1(config)#

3. Create the VFCs.

The VFCs must be bound to the ENode MAC of the CNA and notthe physical interface on the Nexus 5000. The ENode MAC can befound from within SANsurfer.

To determine the ENode MAC of the CNA, use the showfip-snooping vlan-discovery command on each Nexus 4000 asfollows:



• Switch_77:

Switch_77# show fip-snooping vlan-discovery

Legend:-------------------------------------------------------------------------------Interface VLAN FIP MAC-------------------------------------------------------------------------------Eth1/11 1 00:c0:dd:10:12:1dEth1/12 1 00:c0:dd:10:10:c9

• Switch_78:


Legend:-------------------------------------------------------------------------------Interface VLAN FIP MAC-------------------------------------------------------------------------------Eth1/11 1 00:c0:dd:10:12:1fEth1/12 1 00:c0:dd:10:10:cb

Note: For the sake of this example, the vfc interface numbers are beingderived from the blade they will be connected to and the last octet of theswitches IP address. For example, slot 11 switch IP 172.23.199.77 wouldbe vfc1177. You can use any number that makes sense to you providedthat it is unique and falls within the range of 1-8192.

Nexus-5020-1(config)# interface vfc 1177Nexus-5020-1(config-if)# bind mac-address 00:c0:dd:10:12:1dNexus-5020-1(config)# interface vfc 1277Nexus-5020-1(config-if)# bind mac-address 00:c0:dd:10:10:c9

4. Add the VFCs to the appropriate VSAN.

Nexus-5020-1(config-if)# vsan databaseNexus-5020-1(config-vsan-db)# vsan 100 interface vfc 1177Nexus-5020-1(config-vsan-db)# vsan 300 interface vfc 1277


Nexus-5020-1(config-vsan-db)# interface port-channel10Nexus-5020-1(config-if)#

6. Add interfaces to port channel 10.

Nexus-5020-1(config-if)# interface e1/9-14Nexus-5020-1(config-if-range)#channel-group 10 mode active

7. Set the attributes for port channel 10.

Nexus-5020-1(config-if-range)# interface port-channel10Nexus-5020-1(config-if)# switchport mode trunkNexus-5020-1(config-if)# switchport trunk allowed vlan 1,100,300

IBM BladeCenter 637

638


Nexus-5020-1(config-if)# speed 10000Nexus-5020-1(config-if)# duplex fullNexus-5020-1(config-if)# no shut

Example output:

Nexus-5020-1# show port-channel summaryFlags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)s - Suspended r - Module-removedS - Switched R - RoutedU - Up (port-channel)

--------------------------------------------------------------------------------Group Port- Type Protocol Member Ports

Channel--------------------------------------------------------------------------------1 Po1(SU) Eth LACP Eth1/9(P) Eth1/10(P) Eth1/11(P)

Eth1/12(P) Eth1/13(P) Eth1/14(P)Nexus-5020-1#


Note: The steps provided in this section assume that the switches havealready been configured as explained in “Nexus 5000 direct-connecttopology” on page 173.


1. Telnet into Nexus-5020-2.

telnet 172.23.185.113Nexus 5000 Switchlogin: adminPassword: xxxxxxNexus-5020-2#

2. Enter configuration mode.

Nexus-5020-2# config tEnter configuration commands, one per line. End with CNTL/Z.Nexus-5020-2(config)#

3. Create the VFCs.

The VFCs must be bound to the ENode MAC of the CNA and notthe physical interface on the Nexus 5000. The ENode MAC can befound from within SANsurfer.

To determine the ENode MAC of the CNA, use the showfip-snooping vlan-discovery command on each Nexus 4000 asfollows:



• Switch_77:


Legend:-------------------------------------------------------------------------------Interface VLAN FIP MAC-------------------------------------------------------------------------------Eth1/11 1 00:c0:dd:10:12:1dEth1/12 1 00:c0:dd:10:10:c9

• Switch_78:


Legend:-------------------------------------------------------------------------------Interface VLAN FIP MAC-------------------------------------------------------------------------------Eth1/11 1 00:c0:dd:10:12:1fEth1/12 1 00:c0:dd:10:10:cb

Note: For the sake of this example, the vfc interface numbers are beingderived from the blade they will be connected to and the last octet of theswitches IP address. For example, slot 11 switch IP 172.23.199.77 wouldbe vfc1177. You can use any number that makes sense to you providedthat it is unique and falls within the range of 1-8192.

Nexus-5020-2(config)# interface vfc 1178Nexus-5020-2(config-if)# bind mac-address 00:c0:dd:10:12:1fNexus-5020-2(config)# interface vfc 1278Nexus-5020-2(config-if)# bind mac-address 00:c0:dd:10:10:cb

4. Add the VFCs to the appropriate VSAN.

Nexus-5020-2(config-if)# vsan databaseNexus-5020-2(config-vsan-db)# vsan 200 interface vfc 1178Nexus-5020-2(config-vsan-db)# vsan 400 interface vfc 1278


Nexus-5020-2(config-vsan-db)# interface port-channel11Nexus-5020-2(config-if)#

6. Add interfaces to port channel 11.

Nexus-5020-2(config-if)# interface e1/9-14Nexus-5020-2(config-if-range)#channel-group 11 mode active

IBM BladeCenter 639

640


7. Set the attributes for port channel 11.

Nexus-5020-2(config-if-range)# interface port-channel10Nexus-5020-2(config-if)# switchport mode trunkNexus-5020-2(config-if)# switchport trunk allowed vlan 1,200,400Nexus-5020-2(config-if)# speed 10000Nexus-5020-2(config-if)# duplex fullNexus-5020-2(config-if)# no shut

Example output:

Nexus-5020-2# show port-channel summaryFlags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)s - Suspended r - Module-removedS - Switched R - RoutedU - Up (port-channel)

--------------------------------------------------------------------------------Group Port- Type Protocol Member Ports

Channel--------------------------------------------------------------------------------2 Po1(SU) Eth LACP Eth1/9(P) Eth1/10(P) Eth1/11(P)

Eth1/12(P) Eth1/13(P) Eth1/14(P)Nexus-5020-2#

Configure fabriczoning

Zoning of the WWPNs of the CNAs on the server blades to theWWPNs of the storage ports will need to be performed on the fabric.

show running-configoutput examples

To help make the examples clearer, this section shows an example ofthe output from a show running-config command for each switch inthe environment.

◆ “Switch_77 example output” on page 640

◆ “Switch_78 example output” on page 644

◆ “Nexus-5020-1 example output” on page 646

◆ “Nexus-5020-2 example output” on page 650

Switch_77 example outputThe following is an output example from a show running-configcommand for Switch_77.

Switch_77# show running-config summaryversion 4.1(3)N2(1)feature fcoe

feature telnetfeature lacpfeature fex<TRUNCATED>



ip host Nexus-5020-1 172.23.185.113switchname Nexus-5020-2

<TRUNCATED>


vlan 1vlan 200,400fcoe

vsan databasevsan 200vsan 400

interface port-channel1switchport mode trunkswitchport trunk allowed vlan 1,200,400

interface vfc1bind interface Ethernet1/1no shutdown


interface vfc1178bind mac-address 00:c0:dd:10:12:1fno shutdown

interface vfc1278bind mac-address 00:c0:dd:10:10:cbno shutdown

vsan databasevsan 200 interface vfc1vsan 200 interface vfc1177vsan 400 interface vfc2vsan 400 interface vfc1277

interface fc2/1no shutdown




IBM BladeCenter 641

642






interface Ethernet1/1switchport mode trunkswitchport trunk allowed vlan 1,200spanning-tree port type edge trunkspanning-tree bpduguard enable


interface Ethernet1/3






interface Ethernet1/9switchport mode trunkswitchport trunk allowed vlan 1,200,400duplex fullchannel-group 1 mode active













<OUTPUT TRUNCATED>



line consoleboot kickstart bootflash:/n5000-uk9-kickstart.4.1.3.N1.1.binboot system bootflash:/n5000-uk9.4.1.3.N1.1.binip route 0.0.0.0/0 172.23.185.2interface fc2/1interface fc2/2interface fc2/3interface fc2/4interface fc2/5interface fc2/6interface fc2/7interface fc2/8

IBM BladeCenter 643

644


Switch_78 example outputThe following is an output example from a show running-configcommand for Switch_78.

Switch_78# show running-config summaryversion 4.1(2)E1(1)feature telnetno feature sshfeature interface-vlanfeature fip-snooping

<OUTPUT TRUNCATED>

ip domain-lookupip host Switch_77 172.23.199.78switchname Switch_78policy-map type queuing policy-fcoe-bandwidthclass type queuing 1p7q4t-out-q-default

bandwidth percent 50class type queuing 1p7q4t-out-pq1

bandwidth percent 7class type queuing 1p7q4t-out-q2






bandwidth percent 7class-map type network-qos class-fcoe

match cos 3class-map type network-qos class-non-fcoematch cos 0-2,4-7

policy-map type network-qos policy-fcoeclass type network-qos class-fcoe

pause no-dropmtu 2500

class type network-qos class-non-fcoesystem qosservice-policy type network-qos policy-fcoeservice-policy type queuing output policy-fcoe-bandwidth

<OUTPUT TRUNCATED>


vlan 1vlan 200



fip-snooping enablefip-snooping fc-map 0x0efc00

vlan 400fip-snooping enablefip-snooping fc-map 0x0efc00

interface port-channel-1switchport mode trunkswitchport trunk allowed vlan 1,200,400fip-snooping port-mode fcfspeed 10000











interface Ethernet1/11switchport mode trunkswitchport trunk allowed vlan 1,200spanning-tree port type edge trunkspeed auto

interface Ethernet1/12switchport mode trunkswitchport trunk allowed vlan 1,400spanning-tree port type edge trunkspeed auto



interface Ethernet1/15switchport mode trunkswitchport trunk allowed vlan 1,200,400fip-snooping port-mode fcfspeed auto

IBM BladeCenter 645

646








interface mgmt1boot kickstart bootflash:/n4000_kickstart.4.1.2.E1.0.200.binboot system bootflash:/n4000_system.4.1.2.E1.0.200.binsystem health loopback frequency 60ip route 0.0.0.0/0 172.23.199.2

Nexus-5020-1 example outputThe following is an output example from a show running-configcommand for the Nexus-5020-1 switch.

Nexus-5020-1# show running-config summaryversion 4.1(3)N2(1)feature fcoe

feature telnetfeature lacpfeature fex

<TRUNCATED>




<TRUNCATED>







interface vfc1177bind mac-address 00:c0:dd:10:12:1dno shutdown

interface vfc1277bind mac-address 00:c0:dd:10:10:c9no shutdown

vsan databasevsan 100 interface vfc1vsan 100 interface vfc1177vsan 100 interface fc2/1vsan 100 interface fc2/3vsan 100 interface fc3/1vsan 100 interface fc3/3vsan 300 interface vfc2vsan 300 interface vfc1277vsan 300 interface fc2/2vsan 300 interface fc2/4vsan 300 interface fc3/2vsan 300 interface fc3/4

feature npvnpv enablenpv auto-load-balance disruptive

IBM BladeCenter 647

648































<OUTPUT TRUNCATED>



line consoleboot kickstart bootflash:/n5000-uk9-kickstart.4.1.3.N1.1.binboot system bootflash:/n5000-uk9.4.1.3.N1.1.binip route 0.0.0.0/0 172.23.185.2interface fc2/1interface fc2/2

IBM BladeCenter 649

650


interface fc2/3interface fc2/4interface fc2/5interface fc2/6interface fc2/7interface fc2/8

Nexus-5020-2 example outputThe following is an output example from a show running-configcommand for the Nexus-5020-2 switch.

Nexus-5020-2# show running-config summaryversion 4.1(3)N2(1)feature fcoefeature telnetfeature lacpfeature fex

<TRUNCATED>


<TRUNCATED>







interface vfc1178bind mac-address 00:c0:dd:10:12:1fno shutdown



interface vfc1278bind mac-address 00:c0:dd:10:10:cbno shutdown

vsan databasevsan 200 interface vfc1vsan 200 interface vfc1177vsan 400 interface vfc2vsan 400 interface vfc1277















IBM BladeCenter 651

652















<OUTPUT TRUNCATED>



interface Ethernet1/40interface mgmt0ip address 172.23.185.113/24

line consoleboot kickstart bootflash:/n5000-uk9-kickstart.4.1.3.N1.1.binboot system bootflash:/n5000-uk9.4.1.3.N1.1.binip route 0.0.0.0/0 172.23.185.2interface fc2/1interface fc2/2interface fc2/3interface fc2/4interface fc2/5interface fc2/6interface fc2/7interface fc2/8

IBM BladeCenter 653

654


HP Virtual Connect FlexFabric and Flex-10This section provides the following informationon HP's VirtualConnect Flex-10 technology and the HP BladeSystem-integratedVirtual Connect FlexFabric network connectivity device:

◆ “Virtual Connect FlexFabric solution” on page 654

◆ “Flex-10 technology” on page 658

◆ “FlexFabric components and data center communication” onpage 659

◆ “Using FlexHBAs for FC SAN connectivity” on page 666

◆ “Creating Server Profiles” on page 668

◆ “Case study” on page 671

◆ “References” on page 687

Virtual Connect FlexFabric solutionHP Virtual Connect (VC) extends the benefits of virtualizationbeyond the server to the rest of your infrastructure, virtualizingserver-to-network connections.

The HP Virtual Connect FlexFabric solution has been designed tomeet the demands of converged infrastructure and provide serveradministrators with a flexible way to connect HP c-class server bladesto a LAN or SAN. Since it eliminates the need to deploy separateEthernet-based and Fibre Channel-based interconnect modulesoffered by HP, this solution reduces management requirements, thenumber of modules needed, and power and operational costs.

The LAN convergence functionality of the FlexFabric design is basedon the HP Virtual Connect Flex-10 technology, discussed further in“Flex-10 technology” on page 658. The Flex-10 technology is ahardware-based solution that enables a 10 Gb server networkconnection to be split into four variable partitions, allowing users toreplace multiple lower bandwidth physical NIC ports with a singleFlex-10 port.

The FlexFabric suite is comprised of products that extend this Flex-10technology into LAN and SAN convergence. FlexFabric adapters andFlexFabric modules can converge Ethernet, Fibre Channel overEthernet (FCoE), or accelerated iSCSI traffic into a single 10 Gb datastream and yet partition that 10 Gb transmit bandwidth into multiple



adjustable bandwidths while preserving the routing information forall the individual data classes.

This section will briefly describe the following Virtual ConnectFlexFabric hardware:

◆ “FlexFabric interconnect module” on page 655

◆ “FlexFabric Adapter” on page 656

FlexFabric interconnect moduleThe FlexFabric adapters consolidate the FCoE and IP traffic into one10 Gb data stream. The FlexFabric interconnect module separates thisconverged traffic, allowing Fibre Channel and IP traffic to continuebeyond the server-network edge using the existing native Ethernetand Fibre Channel infrastructure.

The FlexFabric 10 Gb/24-port module (model 571956-B21) has 24ports; 16 internal and 8 external. Figure 116 shows the FlexFabric 10Gb 24 port module.

Figure 116 FlexFabric interconnect module

Note the following:

External ports include eight SFP+ connections for uplinks to external10 Gb Ethernet and native 2/4/8 Gb Fibre Channel switches overfiber optic or copper cables.

◆ Four of the external uplink SFP+ ports are Ethernet-only and canonly accommodate Ethernet SFP or SFP+ transceivers.

◆ Four other external uplink SFP+ ports can be configured as eitherEthernet or Fibre Channel by installing the appropriate Ehernetor Fibre channel SFP or SFP+ transceivers into the SFP+transceiver socket on the module.

HP Virtual Connect FlexFabric and Flex-10 655

656


If you swap the SFP/SFP+ transceiver to a different type (Ethernet orFibre Channel) after it has been configured and provisioned with theVC management software, the system will disable the port and flagan error event to the management system until the configuration hasbeen corrected or the correct transceiver type is installed.

Virtual Connect FlexFabric modules provide up to four physicalconnections for each internal 10 Gb port, with the unique ability tofine-tune bandwidth to adapt to virtual server workload demands onthe fly. System administrator can define the hardware personalities ofthese connections as coming from either:

◆ FlexHBAs, which support only Fibre Channel or acceleratediSCSI protocol, or

◆ FlexNICs, which support only Ethernet traffic

“FlexFabric Adapter” on page 656 discusses the concept of FlexHBAsand FlexNICS in more detail.

Contact your HP Customer Representative for specific support-basedinformation about stacking or interconnecting these modules acrossdifferent enclosures, or for plugging these modules in a chassis withother Virtual Connect product offerings.

FlexFabric AdapterThe HP FlexFabric Adapter addresses the growing need of expensiveEthernet connections caused by increased network capacityrequirements, especially for virtual machines.

The FlexFabric Adapter has more functionalities than a standardconverged network adapter (CNA) because in addition to thestandard NIC functions, it provides both Flex-10 NIC devicecapabilities and FCoE or iSCSI FlexHBA capabilities.

FlexFabric Adapters converge FCoE and IP data streams. They useData Center Bridging (DCB) to ensure losslessness and standardpackets to encapsulate Fibre Channel as FCoE. The adaptersconsolidate the FCoE and IP traffic into one 10 Gb data stream.

Each FlexFabricAdapter contains two 10 Gb Ethernet ports. Eachphysical 10 Gb Ethernet port has four Flex-10 physical functions (PFs)that can be either FlexNICs or FlexHBAs.

◆ FlexHBAs support only Fibre Channel or iSCSI protocol

◆ FlexNICs support onlyEthernet traffic



The four PFs can either be in the form of three FlexNICs and oneFlexHBA or four FlexNICs.

IMPORTANT

A single adapter cannot converge both FCoE and iSCSI accelerationtraffic at the same time since there can only be one FlexHBAfunction in any FlexFabric Adapter. However, a software iSCSIinitiator can always be used along with FCoE due to the presence ofthe FlexNIC functionality.

The FlexHBA is an actual PCIe physical function on the FlexFabricAdapter that can be configured to handle storage (SAN) traffic. Theserver ROM, OS, and hypervisor recognize the PCIe physicalfunction as an HBA device.

The FlexFabric Adapter uses Flex-10 technology, explained onpage 658, to configure FlexNICs. Figure 117 shows the HP and EMCsupported FlexFabric Adapters that can be programmed to act asFlexHBAs and FlexNICs.FlexHBAs can also be configured withsimilar transmit bandwidth adjustment capabilities. FlexNIC andFlexHBAs are capable of adjustable bandwidths from:

◆ 100 Mb to 10 Gb in 100 Mb increments when connected to the HPNC553m/NC553i FlexFabric CNAs or any of the Flex-10 NICs

◆ 1 Gb to 10 Gb in 100 Mb increments when connected to theNC551m/NC551i FlexFabric CNAs.

Figure 117 FlexFabric adapters

The FlexFabric Adapters shown in Figure 117 are:

◆ Emulex-based 10 Gb dual port HP NC551m and integratedversion HP NC551i


658


◆ Emulex-based 10 Gb dual port NC553m and integrated versionNC553i.

Flex-10 technologyFlex-10 technology allows a user to configure a single 10 Gb Ethernetport to represent four FlexNICs. A FlexNIC is an actual PCIe functionthat appears to the system ROM, OS, or hypervisor as a discretephysical NIC with its own driver instance. It is not a virtual NICcontained in a software layer.

Since the Flex-10 technology is not software-based; there is noprocessor overhead required to operate virtualized NICs in virtualmachines and with traditional operating systems. The individualFlexNIC bandwidth allocations can be set by using the VirtualConnect Manager CLI or GUI. This is non-disruptive.

The bandwidth can be changed dynamically without having toreboot the server. The transmit bandwidth of each FlexNIC can beadjusted in 100 Mb increments, according to the server's workload.The user has the flexibility to add multiple NICs without addingmore server NIC mezzanine cards and associated interconnectmodules. Figure 118 shows how the FlexNIC compares to a standardNIC iwith regards to dynamic transmit bandwidth allocation.

Figure 118 FlexNIC



The currently available Flex-10 NIC devices are dual-portLAN-on-motherboard NICs (LOMs) or mezzanine cards that supportup to four FlexNICs per port.

The FlexFabric Adapter uses this Flex-10 technology to configureFlexNICs. In addition, users can also configure a single FlexHBA perFlexFabric Adapter with similar transmit bandwidth adjustmentcapabilities.

Flex-10 interconnect modules support interconnectivity with up to 64FlexNICs (16 server blades with 4 FlexNICs per server blade). Themodules also support traditional (non-Flex-10) 10 Gb and 1 Gb NICdevices.

FlexFabric components and data center communicationThis section discusses the communication between the Flex-10 basedNIC (FlexNIC), the interconnect module, and the external data centerswitch.

This section further explains this process by explaining the followingconcepts:

◆ “Overview” on page 659

◆ “FlexNIC VLAN tags” on page 661

◆ “Connection modes” on page 661

OverviewThe FlexNIC functionality applies to the LAN-based (Ethernet)communication of the FlexFabric Adapter and modules.

In addition to Ethernet, the FlexFabric Adapters and modules canconverge Fibre Channel or accelerated iSCSI traffic into a single 10 Gbdata stream.

◆ The FlexFabric Adapter uses Data Center Bridging (DCB) toensure losslessness and standard packets to encapsulate FibreChannel as Fibre Channel over Ethernet (FCoE) and consolidatesthe Fibre Channel and IP traffic into one 10 Gb data stream.

◆ The FlexFabric interconnect module separates the convergedtraffic. Fibre Channel and IP traffic continue beyond theserver-network edge using the existing native Ethernet and FibreChannel infrastructure deployed within the HP blade systemenclosures.


660


This converged networks technology is only deployed within the HPblade system enclosures. The VC technology splits the convergedstream into native Ethernet LANs and Fibre Channel SANs at theFlexFabric modules before going to external connections.

Figure 119 shows how the LAN-based and SAN-based traffic isseparated at the external interface of the FlexFabric interconnectmodule.

Figure 119 Communication between FlexFabric technology in HP BladeServerenclosure and the data center



FlexNIC VLAN tagsThere is just one 10 Gb physical pathway internal to the chassisbetween a physical port on the Flex-10 NIC device (LOM ormezzanine card) and the corresponding port on the Flex-10interconnect module. The traffic flows along this pathway. Thepathway uses the 10GBase-KR (IEEE specification 802.3ap) singlelane, serial Ethernet backplane connection standard.

Although a set of FlexNICs mapped to the same physical port sharethis one 10 Gb pathway, the traffic flow for each is isolated with itsown MAC address and FlexNIC VLAN tags. Users must map eachFlexNIC to one or more Virtual Connect networks (vNets). The datatraffic automatically gets isolated due to OS or FlexNIC drivergenerated VLAN tags.

There are two VLAN tags associated with a packet coming out of aFlexNIC device.

◆ Service/Provider tagsThe FlexNIC device adds its own outer VLAN tag before thepacket travels to the associated VC Flex-10 module. This VLANtag is an industry-standard "service" or "provider" VLAN tag,which is transparent to users and administrators. In addition tothis FlexNIC specific tag, there are the OS-generated VLAN tags.

◆ OS-generated tagsThe outer VLAN tag is placed in the packet by the host OS orFlexNIC driver. The Flex-10 interconnect module that receives thepacket from the Flex-10 NIC preserves or translates the VLAN tagwhen it travels through the module. The interconnect module canact as a VLAN bridge and translate between the VLANnumbering scheme used on the servers and the scheme used onthe upstream data center switches.

Connection modesWhether or not the interconnect module preserves or translates thetag depends on the connection mode. There are two connectionmodes, each discussed briefly, along with a table comparing the two.

◆ “Mapping mode” on page 662

◆ “Tunneling mode” on page 664

◆ “Mapping and tunneling mode comparison” on page 666


662


Mapping mode All VLAN tags are examined in the mapping mode. For specific tagsadded or stripped at different interfaces, such as the server OS, theFlexNIC, or the Flex-10 module, refer to Figure 120.

Figure 120 Mapping mode

The mapping mode is mainly designed to carry traffic for "SharedUplink Sets," which is the Virtual Connect terminology for a VLANtrunk. A VLAN trunk carries traffic to and from multiple VLANsacross different Virtual Connect networks (vNets) in the data center.An example can be seen in Figure 121 on page 663.

The mapping mode makes the server-side VLANs independent of thenetwork-side VLANs. However, on a mapped connection, the VCfirmware limits the connections to 128 VLANs from an uplink set.The tunneling mode has to be deployed if more than 128 VLANs areneeded on a single link.



Figure 121 shows an example of the HP Blade System enclosureconfigured with both mapping and trunking mode, using SharedUplink.

Figure 121 Mapping Mode used for Shared Uplinks example


664


Tunneling mode In tunneling mode, the VLAN tags are not examined, which is whythere are no limits on the number of VLANs supported up to thestandard limit of 4096 VLANs.

For specific tags added or stripped at different interfaces, such as atthe server OS, the FlexNIC, and the VC Flex-10 module in thetunneling mode, refer to Figure 122.

Figure 122 Tunneling mode

The tunneling mode is mainly designed to carry traffic for "DedicatedUplinks." An example can be seen in Figure 123 on page 665. Thededicated uplink can carry multiple VLANs over the same link, butthese VLANs are transparent to the Virtual Connect based adapterand module and are associated with a single vNet.



Figure 123 shows an example of the HP Blade System enclosureconfigured with both mapping and trunking mode, using bothShared and Dedicated Uplink.

Figure 123 Tunneling Mode used for Dedicated Uplinks example


666


Mapping andtunneling mode

comparison

Table 15 provides a high-level comparison of the mapping andtunneling mode. Mapping is preferred over tunneling because it ismore efficient by design.

Using FlexHBAs for FC SAN connectivityAs discussed previously, each physical 10 Gb Ethernet port has fourFlex-10 physical functions (PFs) that can be either FlexNICs orFlexHBAs. This section describes how FlexHBAs are used for FCSAN conenctivity and includes the following information:

◆ “FlexHBAs” on page 666

◆ “Features and functionalities” on page 667

◆ “Support restrictions” on page 667

FlexHBAsEach 10 Gb Ethernet port on a FlexFabric adapter has four Flex-10Physical Functions (PFs), either FlexHBAs or FlexNICs.

FlexHBAs carry the storage traffic (Fibre Channel or SCSI). Only thesecond PF of each FlexFabric adapter port can be configured as aFlexHBA. HP uses the second PF of each port as the storage functionbecause in a traditional CNA this is the PF used for storage access.

If storage access is not required, the user has the flexibility to disablethe FlexFabric adapter storage function and configure the second PFas another FlexNIC function. More information is provided in“FlexFabric components and data center communication” onpage 659.

The first, third, and fourth PFs work only as FlexNIC devices andalways have a NIC personality.

The like-numbered PFs (F1, F2, F3, F4) of each port on the sameFlexFabric adapter must have the same personality, even in adisabled state. This is why Fibre Channel FlexHBA and iSCSI

Table 15 Mapping and Tunneling mode comparison

Mapping mode Tunneling mode

• VLAN-based• Recommended for Shared Uplinks• VLAN trunking supported• Only 128 VLANs supported

• MAC address-based• Recommended for Dedicated Uplinks• VLAN trunking not supported• Up to 4096 VLANs supported



FlexHBA cannot exist on the same FlexFabric adapter at the sametime.

Features and functionalitiesOther FlexHBA features and functionalities include:

◆ Maximum of two Fibre Channel or iSCSI FlexHBA functions aresupported per FlexFabric adapter, one per port

◆ Failover support

◆ Support for Multipath I/O (MPIO) logic in OS for FlexHBAs

◆ CHAP3 and Mutual CHAP support for iSCSI FlexHBApoint-to-point protocol authentication

◆ Up to 128 targets per iSCSI or Fibre Channel FlexHBA

◆ Target discovery for FlexHBAs

◆ Boot from SAN parameters and target information specified inVirtual Connect, iSCSI BIOS Utility, and OS for each FlexHBA

Support restrictionsNote the following support restrictions:

Connectivity support ◆ The FlexFabric modules are not supported for direct connect toany type of storage (FC/FCoE/iSCSI) due to their NPIV Gatewayfunctionality

◆ No zoning can be configured on NPIV Gateways, thus allowingall attached devices to access each other; hence the supportrestriction for direct connect storage

FC driver support The same drivers used for the FlexHBA function on a FlexFabricadapter are same as those for a traditional Fibre Channel HBAadapter given the drivers are from the same vendor. FlexHBAssupport boot from SAN capabilities just like legacy Fibre Channel oriSCSI HBAs.

iSCSI support FlexFabric adapters handle iSCSI data in the same way as they doFC-the converged traffic leaves the FlexFabric adapter as a single datastream. However, iSCSI can share the same networks and uplinksfrom the FlexFabric module where the LAN/IP data streams flowbecause it does not need the DCB-enhanced networks.


668


Creating Server ProfilesThis section discusses using Server Profiles to configure a FlexFabricmodule-based environment.

Virtual Connect management tools, such as the VC Manager (VCM)or VC Enterprise Manager (VCEM), are used to create an I/Oconnection profile for each server blade after physically making theLAN and SAN connections to the FlexFabric modules.

VCM management capabilities run on a processor in anEthernet-based module. This means each blade system enclosuremust have at least one VC module that can act as an Ethernet switchor Ethernet pass-through. Therefore, a FlexFabric module can be usedfor VC Management.

The I/O connection profile, referred to as the Server Profile on theVCM interface, defines the linkage between the converged networkadapter ports on the server and the data center componentsconnected to the ports of a VC module. Server Profiles containinformation about server addresses, connections, and bootparameters for a specific HP c-Class blade server enclosure. They alsopreserve the identity of the network connections in a data center.

Using the VCM or VCEM, Server Profiles can be created and assignedto any blade server in the chassis. Unique HP-generated MACaddresses and World Wide Names (WWNs) can be allocated to theseprofiles. These override the physical MAC and WWN addresses onthe NIC and embedded CNA or HBA mezzanine cards. This enablesnetwork and storage administrators to establish all LAN and SANconnections once during deployment and they need not makeconnection changes later if servers are swapped or new servers areadded. VC keeps the I/O profile for a LAN and SAN connectionconstant. A Server Profile can be created or deleted as required.

IMPORTANT

It is essential to create a Server Profile for every server bladeaccessing the data center.



Server Profiles contain the following key information about a server:

◆ MAC and WWN addresses

The VC assigned WWNs and MAC addresses are real, not virtual.They are the only WWNs and MAC addresses seen by the system,the OS, and the networks. The VCM can be used to reset thedefault physical MAC addresses and WWNs prior to boot, soPXE/SAN boot and all operating systems will see only the VCassigned addresses.

It is during the VC environment setup that a user can select one ofthe following set of addresses:

• Factory default MACs/WWNs

• A specific, user-defined range of MACs/WWNs

• One of several HP pre-defined ranges of MACs/WWNs

Using the HP pre-defined ranges of MACs/WWN addresses isrecommended since the factory default MACs/WWNs cannot bemoved from one blade to another. It is also difficult to keep trackof user-defined ranges and to verify that each reserved range isonly being used once within a data center environment.

IMPORTANT

Local MAC addresses and WWNs automatically return to theoriginal factory defaults if a server is moved from aVC-managed enclosure to an unmanaged enclosure. When aserver is moved from one bay to another, the server gets a newset of addresses that were configured for that server baylocation.

◆ Connections to LANs and SANs for each NIC and FC HBA,respectively

The connections to LANs and SANs are defined using Ethernetnetworks and SAN Fabrics, respectively.

• An Ethernet network is used to map an Ethernet interface onthe FlexFabric module to a specific network or VLAN.

• A SAN Fabric is used to map an FC interface on the module toa specific SAN. Each of the four FC ports on a FlexFabricmodule can be connected to a different SAN, which cancomprise of a single switch or a fabric.


670


◆ VLAN, PXE, and SAN Boot parameters

The boot parameters can be configured for each of the followingdifferent kind of connections in a Service Profile:

• FC HBA

• iSCSI HBA

• FCoE HBA

◆ Profile failover resources

Using VCM, one can upgrade or replace a server by simplyreassigning its VC Server Profile to a server in another enclosurebay or another enclosure in the same VC Domain group. A VCdomain group includes up to four blade system enclosures thatconnect to common LAN and SAN environments independent oftheir physical location. The VCM or VCEM can be used to createVC domains.

When a VC server connection profile is moved, the associatedMAC and WWN boot from SAN parameters and the relatedworkload moves with the profile. A profile move can be initiatedmanually from VCM and VCEM. However, VCEM can automatethis process using its profile failover and spare server allocationfeatures.

Profile movement and failover can be deployed to providecost-effective server blade recovery. This functionality can also beused to perform proactive hardware maintenance with reduceddowntimes and reallocate servers promptly to meet changingworkload and application priorities.

VC Server Profiles can be moved rapidly if the correspondingsource and target servers can boot-from-SAN. Therefore, theautomated profile failover functionality delivered in VCEMrequires a boot-from-SAN environment to be configured.

A sample Server Profile is shown in Figure 124 on page 671 anddiscussed further in “Case study” on page 671, which offers acase study for configuring a FlexFabric for SAN connectivity.

Case studyThis case study covers a SAN connectivity-based configuration of theHP Virtual Connect FlexFabric modules. The high-level target



configuration for this case study is shown in Figure 124.

Figure 124 High-level target topology example

This section will not focus on the LAN connectivity-basedconfiguration in the above topology, but only on the FC portconnectivity between the FlexFabric module and the Brocade andCisco SANs. In this case study, to reduce complexity, only aone-switch fabric for the Brocade and Cisco SAN is used, as shown inFigure 125 on page 672.


672


Figure 125 Specific target topology example

1. Install and set up the HP VC Flex adapters, FlexFabric module,and the VC Manager (VCM).

Refer to the HP documentation/guide for the initial setup andinstallation of the VC hardware and management application,available at:

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c01732252/c01732252.pdf



a. For this case study, we have selected the recommended "VCassigned WWNs" setting for the ports on the FlexAdaptersinstalled in this enclosure.

When assigning WWNs to FC ports on the mezzanineadapters, Virtual Connect assigns both a port WWN and anode WWN. Since the port WWN is typically used forconfiguring fabric zoning, it is the WWN displayedthroughout the Virtual Connect user interface. The assignednode WWN is always the same as the port WWN,incremented by one.

b. The FlexFabric modules have been embedded in Bays 5 and 6of the enclosure and the NC553m Flex Adapter has beeninstalled in the mezzanine card slot labeled 'C' on the HPProliant G7 server blades in slots 2 and 15.

To execute this step, physically connect the FlexFabricmodules to the external switches. In this example, the Brocadeswitch is connected to interfaces X1,X2 of the FlexFabricmodules in bay 5, 6, while the Cisco switch is connected tointerfaces X3,X4 of the FlexFabric modules in bay 5, 6.

2. Create SAN fabrics for the Brocade and Cisco fabric.

a. On the left-hand side pane on the VCM interface, clickConnections > SAN Fabrics.

b. To define a SAN Fabric, click Add in the External Connectiontab on the main SAN Fabrics page.

• We have defined the Brocade Fabrics as follows:

Fabric Name: B300_b5X1X2

a. Add the Uplink ports under the Enclosure Uplink portsby selecting ports X1, X2 from bay 5.

b. Allow the Configured Speed to stay at the default, Auto.

Fabric Name: B300_b6X1X2

c. Add the Uplink ports under the Enclosure Uplink portsby selecting ports X1, X2 from bay 6.

d. Allow the Configured Speed to stay at the default, Auto.

e. After the fabrics have been configured, click Apply.


674


The WWN of the Brocade switch should now displayunder the Connected To column, as shown in Figure 126and in Figure 127 on page 675.

Figure 126 Edit SAN Fabric screen for defining B300_b5X1X2 fabric



Figure 127 Edit SAN Fabric screen for defining B300_b6X1X2 fabric

• We have defined the Cisco Fabrics as follows:

Fabric Name: C9134_b5X3X4

a. Add the Uplink ports under the Enclosure Uplink portsby selecting ports X3, X4 from bay 5.

b. Allow the Configured Speed to stay at the default, Auto.

Fabric Name: C9134_b6X3X4

c. Add the Uplink ports under the Enclosure Uplink portsby selecting ports X3, X4 from bay 6.

d. Allow the Configured Speed to stay at the default, Auto.

e. After the fabrics have been configured, click Apply.


676


The WWN of the Cisco switch should now display underthe Connected To column, as shown in Figure 128 and inFigure 129 on page 677.

Figure 128 Edit SAN Fabric screen for defining C9148_b5X3X4 fabric



Figure 129 Edit SAN Fabric screen for defining C9148_b6X3X4 fabric

Now that the Brocade and Cisco Fabrics have been defined,click SAN Fabrics in the left pane to verify that the followingfour defined SAN fabrics are displaying with the negotiatedlink speed and show a green check mark next to it, whichrepresents an operational link status, as shown in Figure 130on page 678:

– B300_b5X1X2– B300_b6X1X2– C9134_b5X3X4– C9134_b6X3X4


678


Figure 130 All SAN fabrics and link status

3. Create Server Profiles.

This example configures two Server Profiles. One of the ServerProfiles will be deployed for servers that will access the Brocadefabric, while the other Server Profile will be for servers that willaccess the Cisco fabric.

a. On the left-hand side pane on the VCM interface, clickConnections > Server Profiles.

b. On the main Server Profiles page, click Add to define a ServerProfile.

c. On the Define Server Profile page, assign a Profile name.

As stated earlier, this example will create two Server Profiles:one for blade 2 and one for blade 15 in the enclosure.

– Blade 2 will be configured to access the Brocade SAN.– Blade 15 will be configured to access the Cisco SAN.



Only one profile will be defined at a time so all changes madeto the profile are saved and will take effect.

The first profile will be named Profile_Brocade_Server. ForSAN connectivity, we are only interested in modifying theFCoE HBA Connection section.

The Ethernet Adapter Connection section will display twodefault "Undefined" network connections from the LOM onany given server blade to the Ethernet modules (Ethernetswitch or Ethernet pass-thru modules) embedded in bays 1and 2 respectively, on the backplane of the blade serverenclosure. These two ports are used as management ports forthat server blade. When this Server Profile is mapped to aserver, VC-defined MAC addresses will be allocated to theLOM ports of that server, as shown in Figure 131.

Figure 131 Defining the Server Profile for Blade 2

d. Make the folloiwng changes in the FCoE HBA Connectionssection for Profile_Brocade_Server:

– For port 1 connected to Bay 5, assign the defined SANFabric B300_b5X1X2.


680


– For port 2 connected to Bay 6, assign the defined SANFabric B300_b6X1X2.

The following port speed settings are available:

1 - 1 Gb2 - 2 Gb4 - 4 Gb8 - 8 Gb

Custom — Allows user to specify the speed or allocatedbandwidth in increments of 0.1 Gb between 1 to 10 Gb forFCoE/iSCSI connections.

Disabled —The port is not connected.

This example sets the port to 8 Gb for connectivity to theBrocade 8 Gb switch. The VC-assigned WWNs will beallocated to the FlexHBA ports on the server blade's Flexadapter once the Server Profile is applied to the server.

e. In Assign Profile to Server Bay, select Bay 2 from thedrop-down menu to apply the Server Profile configured to theserver in Bay 2, and click Apply, as shown in Figure 132.

Figure 132 Configuring the Server Profile for Blade 2 screen

f. Repeat Step b to Step e for the server in Bay 15 with thefollowing specs: Refer to Figure 133 and Figure 134 onpage 681.

Profile name: Profile_Cisco_Server

Under the FCoE HBA Connections section:

– For Port 1 connected to Bay 5: FC SAN: C9134_b5X3X4;



Port speed: 4 Gb

– For Port 2 connected to Bay 6: FC SAN: C9134_b6X3X4Port speed: 4 Gb

g. In the Assign Profile to Server Bay select Bay 15 from thedrop-down menu and click Apply.

Figure 133 Defining the Server Profile for Blade 15 screen

Figure 134 Configuring the Server Profile for Blade 15 screen


682


4. Verify the connectivity status.

a. View the Interconnect Bays screen (under Hardware >OA-xxxon left-pane), shown in Figure 135 and in Figure 136 onpage 683, for Bays 5 and 6 on the VC Manager to verify thestatus of the connectivity between the Flex Adapter and thedata center switches via the FlexFabric module.

Figure 135 Status of internal/external ports on FlexFabric module, Bay 5



Figure 136 Status of internal/external ports on FlexFabric module, Bay 6

b. Click Device Bays in the left-hand pane and then click on thebay numbers the Server Profiles are applied to, such as Bay 2and Bay 15, as shown in Figure 137 on page 684 and inFigure 139 on page 686.


684


Figure 137 Status between FlexFabric adapter ports in Server 2 and the externalBrocade SAN switch



Figure 138 Status between FlexFabric adapter ports in Server 15 and the externalCisco SAN switch

5. Zone the NPIV logins (HP-generated WWNs or native WWNs) toEMC storage (FC/FCoE).


686


a. Check the Brocade name server for FlexHBA port logins fromserver blade 2. Figure 139 shows the Brocade Name Serverdisplaying four physical logins from the ports on the VCFlexFabric modules (two from each module) and two NPIVlogins from the FlexFabric adapter in Server 2 coming viathese physical ports.

Figure 139 Brocade Name Server FlexHBA port logins, Server 2

b. Check the Cisco name server for FlexHBA port logins fromserver blade 15. Figure 140 shows the Cisco Name Serverdisplaying four physical logins from the ports on the VCFlexFabric modules (two from each module) and two NPIVlogins from the FlexFabric adapter in Server 15 coming viathese physical ports.

Figure 140 Cisco Name Server FlexHBA port logins, Server 15

c. Zone the FlexHBA WWNs to the FCoE storage attached to theBrocade switch and to the FC storage attached to the Ciscoswitch.



ReferencesRefer to the HP website, http://hp.com, for further information anddocumentation on the subjects covered in this chapter.

Information on this website includes:

◆ HP Virtual Connect for c-Class BladeSystem Setup and InstallationGuide

◆ HP Virtual Connect Technology

◆ Converged networks with Fibre Channel over Ethernet and Data CenterBridging

◆ HP Virtual Connect Flex-10 technology: Convergence with FlexFabricComponents


688



5

This chapter provides basic information, supported features,topologies, and detailed setup steps for Juniper QFX3500 switches.

Note: Although the case study in this chapter focuses on the QFX3500, thesame steps are applicable to Juniper QFX3600 switches as well.

The following information is included.

◆ Juniper QFX3500 overview............................................................. 690◆ Juniper QFX3500 supported features and topologies................. 692◆ QFX3500 Gateway and Transit mode topology........................... 695◆ Configuration steps ......................................................................... 699◆ Useful JUNOS commands .............................................................. 711

Juniper QFX3500Switches Setup

Examples

Juniper QFX3500 Switches Setup Examples 689

690

Juniper QFX3500 Switches Setup Examples

Juniper QFX3500 overviewThis chapter provides an introduction to the Juniper QFX3500 switchthat delivers a high-performance, low-latency, feature rich L2 and L3solution for supporting a wide range of deployment scenarios.Examples of situations include traditional and virtualized datacenters, high-performance computing, network-attached storage,converged server I/O, and cloud computing. A versatile, compact,high-density 10GbE platform that runs the same Junos OS as otherJuniper switches, routers and security platforms, the QFX3500 alsodelivers a fabric-ready edge solution for the Juniper NetworksQFabric System.

The QFX3500 offers 63 dual-mode small form-factor pluggabletransceiver (SFP+/SFP) ports and four quad small form-factorpluggable plus (QSFP+) ports in a 1 U form factor, deliveringfeature-rich L2 and L3 connectivity to networked devices such as rackservers, blade servers, storage systems and other switches in highlydemanding, high-performance data center environments. Forconverged server edge access environments, the QFX3500 is also astandards-based Fibre Channel over Ethernet (FCoE) transit switchand FCoE to Fibre Channel (FCoE-FC) gateway, protecting customerinvestments in existing data center aggregation and Fibre Channelstorage area network (SAN) infrastructures.

When deployed with other components of the Juniper NetworksQFabric System, the QFX3500, which is manageable by Junos Space,delivers a fabric-ready QFabric Node edge solution that contributesto a high-performance, low latency fabric, unleashing the power ofthe exponential data center for users migrating from traditionalmultitier networks.

The Juniper QFX3500 can operate in two modes; (Gateway andTransit switch).

◆ Gateway Mode

The Gateway mode of operation is similar to Cisco's N_PortVirtualizer (NPV) mode and Brocade's Access Gateway (AG)mode. While in this mode all incoming FCoE Frames arede-capsulated and forwarded to either the Brocade or Cisco"Core" switch that it is connected to via the FC Interface. Inaddition, all requests that are destined to the FC well knownaddresses (e.g., FLOGI, Name Server registrations and queries)



are forwarded to and serviced by the Core switch. Zoningoperations are also performed using the Core switches nativemanagement tools.

◆ Transit Switch Mode

The Transit switch mode of operation allows the QFX3500 tooperate as a FIP Snooping Bridge. While in this mode theQFX3500 must be connected to another QFX3500 that is runningin Gateway mode or to an FCF that supports connectivity fromFIP Snooping Bridges.

Note: QFabric Support: In terms of FCoE connectivity, a QFabric shouldbe configured as a switch running in Transit Mode. See the Configuretransit mode switch for more information.

Juniper QFX3500 overview 691

692


Juniper QFX3500 supported features and topologiesThe following supported features and topologies as shown inFigure 141 are discussed in this section.

◆ “FCoE Interface” on page 693◆ “Ethernet uplink Interface” on page 693◆ “FC Interface” on page 694◆ “Ethernet end device Interface” on page 694

Figure 141 is for clarification purposes only. Actual referencetopologies are provided in the appropriate sections.

Figure 141 Juniper QXF3500 overview



FCoE InterfaceThe following interfaces are supported:

◆ CNAs directly connected via physical media to the QFX3500

• Hardware-based FCoE CNAs from Brocade, Emulex andQLogic

• Software-based CNAs from Intel and Broadcom


• Fabric A / Fabric B connectivity is supported (e.g., a pair ofQFX3500s connected to different fabrics)

• SFP+ optical (short reach only)

• Twinax cable is supported:

– Active 1, 3, 5, 7 and 10m– Passive 1, 3 and 5m

For more information, refer to:

http://www.juniper.net/techpubs/us/en/release-independent/junos/topics/reference/specifications/interface-qfx3500-support.htm

◆ FCoE Storage connections to the QFX3500 may be supportable.Contact your EMC sales representative to determine support viathe RPQ process.

◆ A QFX3500 operating in Transit Switch mode may be connectedto a QFX3500 running in Gateway mode

Ethernet uplink InterfaceThe following are supported:

◆ Traffic Monitoring

• Port-based

• LAG port

• VLAN-based

• ACL-based

• Mirror to local and remote destinations (L2 over VLAN)

◆ QoS Queues / Port 12 (8 unicast; 4 multicast)

◆ MAC Addresses 120,000

Juniper QFX3500 supported features and topologies 693

694


◆ Jumbo Frames 9,216 Bytes

◆ IPv4 Unicast Routes 12,000 prefixes and 8,000 host routes

◆ Number of VLANs 4,095

◆ ARP Entries 8,000

Refer to the Juniper website, www.juniper.net, for more information.

FC InterfaceThe following are supported:

◆ The FC ports on the QFX3500 may only be used to connect to anFC Switch. Currently, only Brocade and Cisco switches aresupported.

See the EMC Simple Support Matrix, located athttp://elabnavigator.EMC.com, for more information.

The following are NOT supported:

◆ N_Port or E_Port connectivity (e.g., no hosts, storage ports orISLs)

Ethernet end device InterfaceThe following are supported:







QFX3500 Gateway and Transit mode topologyThis section details the setup steps required to create theconfiguration shown in Figure 143 on page 698. It includes thefollowing information:




Summary of configuration stepsThis section provides a summary and examples of the steps to createthe configuration shown in the target topology.

Configure Gateway mode switch (54.80)The following steps are required to configure a Juniper QFX3500 toact as a gateway mode switch.


2. Set switch name.

3. Set IP Address.


5. Configure the appropriate interfaces to be of type "fibre-channel" .

6. Configure VLANs.

7. Create a layer 3 interface and associate it with the appropriateVLAN.

8. Create a default VLAN and assign a name to it.

9. Enable FCoE on the FCoE VLAN and configure interfaces.

a. Enable the FCoE vlan.

b. Configure MTU.

c. Allow the appropriate vlans.

d. Set the native vlan ID.

10. Configure the FC interfaces.

11. Configure FC-Fabrics.

QFX3500 Gateway and Transit mode topology 695

696


12. Enable LLDP, DCBX and disable IGMP snooping on the FCoEVLAN.

13. Configure Class of Server parameters.

Configure transit mode switch (54.81)Perform the following required steps to configure a Juniper QFX3500to act as a transit mode switch


2. Set switch name.

3. Set IP Address.


5. Configure VLANs.

6. Create a default VLAN and assign a name to it.

7. Enable FCoE on the FCoE VLAN and configure interfaces.

a. Enable the FCoE vlan.

b. Configure MTU.

c. Allow the appropriate vlans.

d. Set the native vlan ID.

8. Enable LLDP, DCBX and disable IGMP snooping on the FCoEVLAN.

9. Configure Class of Server parameters.



Existing topologyFigure 142 on page 697 provides a view of the existing topology.


The topology consists of a Cisco 9148, two hosts each containing adual port CNA and two FC interfaces from an EMC SymmetrixVMAX storage array.

Target topologyFigure 143 on page 698 provides a view of Target Topology.

The Target Topology will be created by adding two Juniper QFX 3500switches. The QFX3500 in the middle of the topology (54.80) is actingas a gateway (N_Port virtualizer).

The QFX3500 at the bottom of the topology (54.81) is acting as atransit switch (FIP Snooping Bridge). VLANs 1 and 101 will beconfigured and used on the switch for testing. VLAN 1 will be usedfor FIP VLAN Discovery as well as non-FCoE traffic and VLAN 101will be used for FCoE traffic.

QFX3500 Gateway and Transit mode topology 697

698



Zone 1:Host1 - I1 (A9EB)Storage Port 1 - 8E0 (CD1C)

Zone 2:Host1 - I2 (A9EC)Storage Port 2 - 8E1 (CD1D)

Zone 3:Host2 - I1 (F3AF)

Storage Port 1 - 8E0 (CD1C)

Zone 4:Host2 - I2 (F3AD)Storage Port 2 - 8E1 (CD1D)

Note: The above zoning configuration adheres to the EMC best practice ofSingle Initiator Zoning. However, the actual topology shown is only forillustration purposes and is not intended to indicate a recommendedtopology.



Configuration stepsThe following steps are required to configure a Juniper QFX3500switch:

◆ “Configure Gateway mode switch (54.80)” on page 699

◆ “Configure transit mode switch (54.81)” on page 706

Configure Gateway mode switch (54.80)The following steps are required to configure a Juniper QFX3500 toact as a Gateway.

Basic switchconfiguration

1. Power on the QFX3500.

2. Using a serial cable, configure the IP Address (10.222.54.80),subnet mask (/24) and static route to the gateway (0.0.0.0/010.222.54.1).

set interfaces me4 unit 0 family inet address 10.222.54.80/24set routing-options static route 0.0.0.0/0 next-hop 10.222.54.1

3. Enable ssh, disconnect serial cable and start an ssh session to theswitch from the system that will be used to configure the switch(e.g., a laptop).

4. Set the system name using

set system host-name qfx-top-gw.

Configure thephysical interfaces

Enter the switch configuration mode using the configure command.

user@qfx-top-gw> configureEntering configuration mode[edit]user@qfx-top-gw#

5. Edit chassis configuration using the edit chassis command

user@qfx-top-gw# edit chassis

[edit chassis]user@qfx-top-gw#

6. Configure the appropriate interfaces to be of type "fibre-channel"as follows.

Configuration steps 699

700


Note: Although physical ports 0-5 and 42-47 will be configured forfibre-channel, only interfaces 0 and 1 will be used in this case study.

user@qfx-top-gw# set fpc 0 pic 0 fibre-channel port-range 0 5

[edit chassis]user@qfx-top-gw# set fpc 0 pic 0 fibre-channel port-range 42 47

[edit chassis]user@qfx-top-gw# exit

[edit]user@qfx-top-gw# commit

Configure VLANs 1. Enter the VLAN configuration mode by using the command

edit vlans.

user@qfx-top-gw# edit vlans[edit vlans]user@qfx-top-gw#

2. Create a VLAN and assign a name to it using the set<vlan-name> vlan-id <VLAN#> command.

user@qfx-top-gw# set FCoE_VLAN_100 vlan-id 100[edit vlans]user@qfx-top-gw#

3. Create a layer 3 interface and associate it with the VLAN createdin Step 2. Use the set <vlan-name> l3-interface vlan.<l3 interfacedesignation> command.

user@qfx-top-gw# set FCoE_VLAN_100 l3-interface vlan.100

edit vlans]user@qfx-top-gw#

4. Create a default VLAN and assign a name to it using the set<vlan-name> vlan-id <VLAN#> command.

user@qfx-top-gw# set vlan_default vlan-id 1

edit vlans]user@qfx-top-gw# exit

[edit]user@qfx-top-gw#



Enable FCoE on theFCoE VLAN and

configure interfaces

1. Enable FCoE on the FCoE VLAN using the set vlan unit<interface-unit-number> enable family fibre-channel port-modef-port command.

[edit]user@qfx-top-gw# edit interfaces

[edit interfaces]user@qfx-top-gw# set vlan unit 100 enable family

fibre-channel port-mode f-port

2. Enable the FCoE vlan.

[edit interfaces]user@qfx-top-gw# set vlan unit 100 enable

3. Configure MTU on interface xe-0/0/6.

[edit interfaces]user@qfx-top-gw# set xe-0/0/6 mtu 2500

4. Configure interface xe-0/0/6 as a trunk port.

[edit interfaces]user@qfx-top-gw# set xe-0/0/6 unit 0 family ethernet-switchingport-mode trunk

5. Allow the appropriate vlans on interface xe-0/0/6.

[edit interfaces]user@qfx-top-gw# set xe-0/0/6 unit 0 family ethernet-switching vlanmembers FCoE_VLAN_100

6. Set the native vlan ID for interface xe-0/0/6.

[edit interfaces]user@qfx-top-gw# set xe-0/0/6 unit 0 family ethernet-switchingnative-vlan-id 1

[edit interfaces]user@qfx-top-gw#

7. Copy the interface configuration from xe-0/0/6 and apply it tointerface xe-0/0/7.

[edit interfaces]user@qfx-top-gw# copy xe-0/0/6 to xe-0/0/7

8. Configure the FC interfaces.

[edit interfaces]user@qfx-top-gw#set fc-0/0/0 unit 0 family

fibre-channel port-mode np-port


702


[edit nterfaces]user@qfx-top-gw#set fc-0/0/1 unit 0 family

fibre-channel port-mode np-port

[edit interfaces]user@qfx-top-gw# exit

[edit]

Configure FC-Fabrics 1. Create the proxy fabric using the set <fabric-name> fabric-id<fabric-id> fabric-type proxy command.

[edit]user@qfx-top-gw# edit fc-fabrics

[edit fc-fabrics]user@qfx-top-gw# set MDS-Proxy fabric-id 100fabric-type proxy

2. Associate the proxy fabric with the FCoE VLAN.

[edit fc-fabrics]user@qfx-top-gw# set MDS-Proxy interface vlan.100

3. Associate the FC interfaces with the appropriate FC proxy fabric.

[edit fc-fabrics]user@qfx-top-gw# set MDS-Proxy interface fc-0/0/0.0

[edit fc-fabrics]user@qfx-top-gw# set MDS-Proxy interface fc-0/0/1.0

[edit fc-fabrics]user@qfx-top-gw# exit


Enable LLDP, DCBXand disable IGMP

snooping on the FCoEVLAN

Enable LLDP and DCBX.

[edit]user@qfx-top-gw# edit protocols

[edit protocols]user@qfx-top-gw# set lldp interface all

[edit protocols]user@qfx-top-gw# set dcbx interface all



[edit protocols]user@qfx-top-gw# set igmp-snooping vlan FCOE_VLAN_100disable

[edit protocols]user@qfx-top-gw# exit

[edit]user@qfx-top-gw#

Configure aggregateethernet interface

Note: A total of five aggregated ethernet "ae" interfaces will be created in thissection but only ae0 will be used in this example. Interface ae0 will be used toconnect to the transit switch and as a result, this step is not necessary in orderto get FCoE up and running. It is being included at this point to avoid havingto configure the QoS settings twice.

1. Create a range of aggregated ethernet interfaces.

[edit]user@qfx-top-gw# edit chassis

[edit chassis]user@qfx-top-gw# set aggregated-devices ethernetdevice-count 5

[edit chassis]user@qfx-top-gw# exit

2. Add the appropriate interfaces to ae0.

Note: In this case interfaced xe-0/0/40 and xe-0/0/41 will be added toae0.

Note: You may need to remove interfaces xe-0/0/40.0 and xe-0/0/41.0before this step is performed.

[edit]user@qfx-top-gw# edit interfaces

[edit interfaces]user@qfx-top-gw# set interface-range LAG member-range xe-0/0/40 to

xe-0/0/41

3. Configure ae0 interface parameters.

[edit interfaces]user@qfx-top-gw# set interface-range LAG ether-options 802.3ad ae0


704


[edit interfaces]user@qfx-top-gw# set ae0 mtu 2500

[edit interfaces]user@qfx-top-gw# set ae0 unit 0 family ethernet-switching port-mode

trunk

[edit interfaces]user@qfx-top-gw# set ae0 unit 0 family ethernet-switching vlan members

FCoE_VLAN_100

[edit interfaces]user@qfx-top-gw# set ae0 unit 0 family ethernet-switching

native-vlan-id 1

[edit interfaces]user@qfx-top-gw# exit


Configure Class ofServer parameters

1. Configure the Class of service settings.

[edit]user@qfx-top-gw# edit class-of-service

Note: To avoid errors, cut and paste the following commands directly intothe CLI interface you are using. If this is not possible, be very careful anddouble-check that you have entered everything exactly as it is shown below.Refer to the Juniper documentation for more information about the followingcommands.

set classifiers ieee-802.1 fcoe forwarding-class fcoe loss-priority lowcode-points 011

set classifiers ieee-802.1 fcoe forwarding-class best-effort loss-priority highcode-points 000









set traffic-control-profiles fcoe_tcp scheduler-map fcoe_mapset traffic-control-profiles fcoe_tcp guaranteed-rate percent 60set traffic-control-profiles lan_tcp scheduler-map lan_mapset traffic-control-profiles lan_tcp guaranteed-rate percent 40set forwarding-class-sets fcoe_fc_set class fcoeset forwarding-class-sets lan_fc_set class best-effortset congestion-notification-profile cnp input ieee-802.1 code-point 011 pfcset scheduler-maps fcoe_map forwarding-class fcoe scheduler fcoe_schset scheduler-maps lan_map forwarding-class best-effort scheduler lan_schset schedulers fcoe_sch transmit-rate 6gset schedulers lan_sch transmit-rate 4g

2. Configure CoS settings on interface xe-0/0/6.

set interfaces xe-0/0/6 forwarding-class-set fcoe_fc_setoutput-traffic-control-profile fcoe_tcpset interfaces xe-0/0/6 forwarding-class-set lan_fc_setoutput-traffic-control-profile lan_tcpset interfaces xe-0/0/6 congestion-notification-profile cnpset interfaces xe-0/0/6 unit 0 classifiers ieee-802.1 fcoe

3. Copy CoS parameters to interfaces xe-0/0/7 and ae0.

[edit class-of-service]user@qfx-top-gw# copy interfaces xe-0/0/6 to xe-0/0/7

[edit class-of-service]user@qfx-top-gw# copy interfaces xe-0/0/6 to ae0

[edit class-of-service]user@qfx-top-gw# exit

[edit]user@qfx-bottom-ts# commit

4. Configure ae0 as a trusted FCoE port.

user@qfx-bottom-ts# edit ethernet-switching-options

[edit ethernet-switching-options]user@qfx-bottom-ts# set secure-access-port interface ae0.0

fcoe-trusted

[edit ethernet-switching-options]user@qfx-top-gw# exit

[edit ethernet-switching-options]user@qfx-top-gw# commit


706


Configure transit mode switch (54.81)The following steps are required to configure a Juniper QFX3500 toact as a transit mode switch.

Basic switchconfiguration

1. Power on the QFX3500.

2. Using a serial cable, configure the IP Address (10.222.54.80),subnet mask (/24) and static route to the gateway (0.0.0.0/010.222.54.1).

set interfaces me4 unit 0 family inet address 10.222.54.81/24set routing-options static route 0.0.0.0/0 next-hop 10.222.54.1

3. Enable ssh, disconnect serial cable and start an ssh session to theswitch from the system that will be used to configure the switch(e.g., a laptop).

4. Set the system name using

set system host-name qfx-bottom-ts.

Configure VLANs 1. Enter the VLAN configuration mode by using the command editvlans.

user@qfx-bottom-ts# edit vlans

[edit vlans]user@qfx-bottom-ts#

2. Create a VLAN and assign a name to it using the set<vlan-name> vlan-id <VLAN#> command.

user@qfx-bottom-ts# set FCoE_VLAN_100 vlan-id 100

[edit vlans]user@qfx-bottom-ts#

3. Create a default VLAN and assign a name to it using the set<vlan-name> vlan-id <VLAN#> command.

user@qfx-bottom-ts# set vlan_default vlan-id 1

[edit vlans]user@qfx-bottom-ts# exit

[edit]user@qfx-bottom-ts#



Configure FCoEinterfaces and assign

to the FCoE VLAN

1. Enable the FCoE vlan.

[edit interfaces]user@qfx-bottom-ts# set vlan unit 100 enable

2. Configure MTU on interface xe-0/0/9.

[edit interfaces]user@qfx-bottom-ts# set xe-0/0/9 mtu 2500

3. Configure interface xe-0/0/9 as a trunk port.

[edit interfaces]user@qfx-bottom-ts# set xe-0/0/9 unit 0 family ethernet-switching

port-mode trunk

4. Allow the appropriate vlans on interface xe-0/0/9.

[edit interfaces]user@qfx-bottom-ts# set xe-0/0/9 unit 0 family ethernet-switchingvlan members FCoE_VLAN_100

5. Set the native vlan ID for interface xe-0/0/9.

[edit interfaces]user@qfx-bottom-ts# set xe-0/0/9 unit 0 family ethernet-switching

native-vlan-id 1[edit interfaces]

user@qfx-bottom-ts#

6. Copy the interface configuration from xe-0/0/9 and apply it tointerface xe-0/0/12.

[edit interfaces]user@qfx-bottom-ts# copy xe-0/0/6 to xe-0/0/12

[edit interfaces]user@qfx-bottom-ts# exit

[edit]

Enable LLDP, DCBXand disable IGMP

snooping on the FCoEVLAN

1. Enable LLDP and DCBX.

[edit]user@qfx-bottom-ts# edit protocols

[edit protocols]user@qfx-bottom-ts# set lldp interface all


708


[edit protocols]user@qfx-bottom-ts# set dcbx interface all

[edit protocols]user@qfx-bottom-ts# set igmp-snooping vlan FCoE_VLAN_100 disable

[edit protocols]user@qfx-bottom-ts# exit

[edit]user@qfx-bottom-ts#

Configure aggregateethernet interface

Note: A total of five aggregated ethernet "ae" interfaces will be created in thissection but only ae0 will be used in this example. Interface ae0 will be usedto connect to the transit switch and as a result, this step is not necessary inorder to get FCoE up and running. It is being included at this point to avoidhaving to configure the QoS settings twice.

1. Create a range of aggregated ethernet interfaces.

[edit]user@qfx-bottom-ts# edit chassis

[edit chassis]user@qfx-bottom-ts# set aggregated-devices ethernet device-count 5

[edit chassis]user@qfx-bottom-ts# exit

2. Add the appropriate interfaces to ae0.

Note: : In this case interfaced xe-0/0/40 and xe-0/0/41 will be added toae0.

Note: You may need to delete interfaces xe-0/0/40.0 and xe-0/0/41.0before this step is performed.

[edit]user@qfx-bottom-ts# edit interfaces

[edit interfaces]user@qfx-bottom-ts# set interface-range LAG member-range xe-0/0/40 to

xe-0/0/41

3. Configure ae0 interface parameters.

[edit interfaces]



user@qfx-bottom-ts# set interface-range LAG ether-options 802.3ad ae0

[edit interfaces]user@qfx-bottom-ts# set ae0 mtu 2500

[edit interfaces]user@qfx-bottom-ts# set ae0 unit 0 family ethernet-switching port-mode

trunk

[edit interfaces]user@qfx-bottom-ts# set ae0 unit 0 family ethernet-switching vlan

members FCoE_VLAN_100

[edit interfaces]user@qfx-bottom-ts# set ae0 unit 0 family ethernet-switching

native-vlan-id 1

[edit interfaces]user@qfx-bottom-ts# exit

[edit]

Configure Class ofServer parameters

1. Configure the Class of service settings.

[edit]user@qfx-bottom-ts# edit class-of-service

Note: To avoid errors, cut and paste the following commands directlyinto the CLI interface you are using. If this is not possible, be very carefuland double-check that you have entered everything exactly as it is shownbelow. Refer to the Juniper documentation for more information aboutthe following commands.

set classifiers ieee-802.1 fcoe forwarding-class fcoe loss-priority lowcode-points 011








710



set traffic-control-profiles fcoe_tcp scheduler-map fcoe_mapset traffic-control-profiles fcoe_tcp guaranteed-rate percent 60set traffic-control-profiles lan_tcp scheduler-map lan_mapset traffic-control-profiles lan_tcp guaranteed-rate percent 40set forwarding-class-sets fcoe_fc_set class fcoeset forwarding-class-sets lan_fc_set class best-effortset congestion-notification-profile cnp input ieee-802.1 code-point 011 pfcset scheduler-maps fcoe_map forwarding-class fcoe scheduler fcoe_schset scheduler-maps lan_map forwarding-class best-effort scheduler lan_schset schedulers fcoe_sch transmit-rate 6gset schedulers lan_sch transmit-rate 4g

2. Configure CoS settings on interface xe-0/0/9.

set interfaces xe-0/0/9 forwarding-class-set fcoe_fc_setoutput-traffic-control-profile fcoe_tcpset interfaces xe-0/0/9 forwarding-class-set lan_fc_set

output-traffic-control-profile lan_tcpset interfaces xe-0/0/9 congestion-notification-profile cnpset interfaces xe-0/0/9 unit 0 classifiers ieee-802.1 fcoe

3. Copy CoS parameters to interfaces xe-0/0/12 and ae0.

[edit class-of-service]user@qfx-bottom-ts# copy interfaces xe-0/0/6 to xe-0/0/12

[edit class-of-service]user@qfx-bottom-ts# copy interfaces xe-0/0/6 to ae0

[edit class-of-service]user@qfx-bottom-ts# exit

[edit]user@qfx-bottom-ts# commit

Configure ae0 as atrusted FCoE port andenable FIP Snooping

1. Configure ae0 as a trusted FCoE port.

user@qfx-bottom-ts# edit ethernet-switching-options

[edit ethernet-switching-options]user@qfx-bottom-ts# set secure-access-port interface ae0.0

fcoe-trusted

2. Enable FIP Snooping on interface ae0.

[edit ethernet-switching-options]user@qfx-bottom-ts# set secure-access-port vlan FCoE_VLAN_100

examine-fip fc-map 0x0efc00



Useful JUNOS commandsThe following commands assume the switch is in configurationmode:

On Gateway switch show fibre-channel fabric

show fibre-channel ftp detail

show fibre-channel ftp statistics

show fibre-channel interfaces detail

show fibre-channel flogi statistics

show fibre-channel proxy login-table

show fibre-channel proxy statistics

show fibre-channel fc2 sessions

show interfvaces fc-0/0/1 detail

On transit switch show ethernet-switching-options

show ftp snooping detail

show ftp snooping statistics

show dcbx neighbors interface xe-0/0/8

Note: For enode interfaces, it does not matter if transit orgateway.

General commands show version

show chassis hardware

show interfaces terse

show vlans

run show ethernet-switching table

show

show | compare

rollback 0

rollback 1

Useful JUNOS commands 711

712



6

This chapter provides basic information, supported features,topologies, and detailed setup steps for Brocade VDX switches. Thefollowing information is included.

◆ Brocade VDX data center switches................................................ 714◆ Brocade VCS fabric technology...................................................... 718◆ Brocade VDX switches supported and unsupported features .. 722◆ Brocade VDX switches in a VCS fabric case study...................... 726◆ References.......................................................................................... 741

Brocade VDX SwitchesSetup Examples

Brocade VDX Switches Setup Examples 713

714

Brocade VDX Switches Setup Examples

Brocade VDX data center switchesThe Brocade VDX switches are designed to utilize the Brocade VCSFabric technology (discussed in more detail in “Brocade VCS fabrictechnology” on page 718) to build Ethernet Fabrics. Before gettinginto any specifics around the Brocade VDX product line and the VCStechnology, this section will briefly review the concept of Ethernetfabrics. Then, a brief description of the three switches in the BrocadeVDX family is provided. Finally, since Layer 2 features make theBrocade VDX switches ideal platforms for traditional Top-of-Rack(ToR) switch deployments, modes of operation will be discussed.

◆ “Ethernet fabrics” on page 714

◆ “VDX 6710” on page 715

◆ “VDX 6720” on page 715

◆ “VDX 6730” on page 715

◆ “Modes of operation” on page 716

Ethernet fabricsCompared to classic hierarchical Ethernet architectures, Ethernetfabrics have been designed to provide higher levels of performance,utilization, availability, and simplicity. They have the followingminimum characteristics:

◆ Flatter: Ethernet fabrics eliminate the need for Spanning TreeProtocol, while still being completely interoperable with existingEthernet networks.

◆ Flexible: Can be architected in any topology to best meet theneeds of any variety of workloads.

◆ Resilient: Just like in Fibre Channel fabrics, multiple "least cost"paths are used for high performance and high reliability.

◆ Elastic: The fabric can easily be scaled up and down at need.

More advanced Ethernet fabrics borrow further from Fibre Channelfabric constructs. The ones that especially apply to the Brocade VCSfabrics are:

◆ Self-forming and function as a single logical entity, in which allswitches forming the fabric automatically know about each otherand all connected physical and logical devices.



◆ Management can then be domain-based rather than device-based,and defined by policy rather than repetitive procedures.

◆ Protocol convergence, such as Fibre Channel over Ethernet(FCoE), may also be a feature, intended as a means of betterbridging LAN and SAN traffic.

All of these features present the VCS Fabric as a solution that can:

◆ Scale in real time

◆ Minimize downtime

◆ Integrate with existing Ethernet networks

VDX 6710The Brocade VDX 6710 Data Center Switch is a high-performance, 1Gigabit Ethernet (GbE) fixed port switch that provides a reliable,scalable, and flexible foundation for supporting demanding businessapplications. It offers a classic ToR server connectivity and the optionto connect servers to an Ethernet fabric using Brocade VCS Fabrictechnology.

The Brocade VDX 6710 Data Center Switch is available in a 1Urack-mounted model with 48 GbE (Gigabit Ethernet) and six 10-GbEports.

VDX 6720The Brocade VDX 6720 Data Center Switch is a high-performance,ultra-low latency wire-speed 10 Gigabit Ethernet (GbE) fixed portswitch. It supports several connectivity options, including classic ToRserver deployments, Ethernet fabrics, and Ethernet storageconnectivity.

The Brocade VDX 6720 Data Center Switch is available in either a 1Urack-mounted model with 24 10-GbE ports or a 2U rack-mountedmodel with 60 10-GbE ports.

VDX 6730The Brocade VDX 6730 Data Center Switch is a 10 Gigabit Ethernet(GbE) fixed port switch with LAN and native Fibre Channel ports. Itsupports multiple connectivity options, including classic ToR serverdeployments, Ethernet fabrics, and Ethernet storage connectivity for

Brocade VDX data center switches 715

716


Fibre Channel over Ethernet (FCoE), iSCSI, NAS, and bridging FibreChannel Storage Area Networks (SANs) and Ethernet fabrics.

The Brocade VDX 6730 Data Center Switch is available in either a 1Urack-mounted model with 24 10-GbE ports and eight 8-Gbps FibreChannel ports, or a 2U rack-mounted model with 60 10-GbE and 168-Gbps Fibre Channel ports.

Modes of operationBrocade VDX switches can operate in one of two modes:

◆ Classic (Standalone)

This mode represents the classic 10 GbE server access and 1 GbEToR aggregation deployments with support for a broad set ofEthernet Layer 2 features.

Integrating Brocade VDX into the classic 10 GbEaccess/aggregation architecture requires no change to theexisting enterprise data center network architecture.

◆ VCS

VCS mode is enabled by default on all VDX switches. However, aVCS Fabric license is required when there are more than twoVCS-enabled switches in one VCS fabric. If there are not morethan two switches in the fabric, then this license is not required toimplement a VCS cluster.

In VCS Fabric or cluster mode, the switch is part of an Ethernetfabric involving two or more VCS-enabled switches. Distributedintelligence and logical chassis are concepts used in VCStechnology.

• With distributed intelligence, all configuration and destinationinformation is automatically distributed to each memberswitch to provide unified fabric functionality.

Distributed intelligence has three major characteristics:

– The fabric is self-forming. When two or more VCS-enabledswitches with unique Rbridge IDs are connected to form anVCS fabric, the fabric is automatically created and theswitches discover the common fabric configuration.



– The fabric is masterless. No single switch storesconfiguration information or controls fabric operations.Any switch can fail or be removed without causingdisruptive fabric downtime or delayed traffic.

– The fabric is aware of all members, devices, and VirtualMachines (VMs). Automatic Migration of Port Profiles(AMPP) supports VM migration to another physical server.If the VM moves, it is automatically reconnected to all of itsoriginal resources.

For example, when a host connects to the fabric for the firsttime, all switches in the fabric learn about that server. In thisway, fabric switches can be added or removed and physical orvirtual servers can be relocated-without the fabric requiringmanual reconfiguration.

• With logical chassis, regardless of the number of VCS fabricmode-enabled switches in the VCS fabric, they are going to bemanaged as if they were a single logical chassis. From thevisibility of the network, the fabric looks no different than anyother Ethernet switch.

More information on the VCS Ethernet fabric, including distributedintelligence, logical chassis, and deployment examples, can be foundin the Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB)Concepts and Protocols TechBook, available on the E-Lab Navigator,Topology Resource Center tab.

Brocade VDX data center switches 717

718


Brocade VCS fabric technologyThis section includes the following information:

◆ “Introduction” on page 718

◆ “Terminology” on page 720

◆ “VCS Ethernet fabric” on page 720

IntroductionThe Brocade VCS Fabric is a new Layer 2 Ethernet technology. Itleverages the emerging TRILL standard as well as other standardsfrom IEEE and T11, such as Data Center Bridging (DCB) and FibreChannel over Ethernet (FCoE). VCS eliminates many limitations ofclassic Ethernet networks in the data center.

TRILL (Transparent Interconnect of Lots of Links) is an IETF standardfor improved bridging loop prevention and having Layer 2multipathing function in an Ethernet fabric. Unlike the Spanning TreeProtocol, with TRILL, all the paths through the network are active,and traffic is automatically distributed across the equal-cost paths.

More information on TRILL can be found in the Fibre Channel overEthernet (FCoE) Data Center Bridging (DCB) Concepts and ProtocolsTechBook, available on the E-Lab Navigator, Topology ResourceCenter tab.

Figure 144 on page 719 shows a classic Ethernet architecture and thecorresponding Brocade VCS Fabric architecture. The Brocade VCSFabric combines the Access layer and Aggregation layers. It is morescalable especially as you add and expand the network.



Figure 144 Classic Ethernet and corresponding VCS Fabric architecture

Brocade VCS Fabric technology comprises the following concepts:

◆ Ethernet fabric

◆ Distributed intelligence

◆ Logical chassis

When two or more Brocade VCS Fabric mode-enabled switches (suchas VDX 6720 or 6730) are connected together, they form an Ethernetfabric and exchange information among each other to implementdistributed intelligence. To the rest of the network, the Ethernet fabricappears as a single logical chassis.

More information on Ethernet fabric, distributed intelligence, andlogical chassis can be found in the Fibre Channel over Ethernet (FCoE)Data Center Bridging (DCB) Concepts and Protocols TechBook, availableon the E-Lab Navigator, Topology Resource Center tab.

Brocade VCS fabric technology 719

720


TerminologyTable 16 lists terms used throughout this chapter associated with VCSarchitecture.

VCS Ethernet fabricCompared to the traditional tier-designed Ethernet or the classicEthernet, Ethernet fabrics provide higher levels of networkperformance and utilization, availability, and simplicity.

Ethernet fabrics have the following characteristics that are improvedover classical Ethernet:

◆ Flatter — Ethernet fabrics eliminate the need for Spanning TreeProtocol (STP), Ethernet fabrics can attach to traditional Ethernetnetworks, but it is huge overstatement to say they are completelyinteroperable.

◆ Greater flexibility — Ethernet fabrics can be designed in anytopology to best meet different network needs and requirements.

Table 16 VCS terminology

Terms Description

Edge ports In an Ethernet (VCS) fabric, all switch ports used to connect external equipment,including end stations, servers and storage.

Ethernet fabric A group of connected Ethernet switches exchanging information amongeach other to implement distributed intelligence.

Fabric ports The ports on either end of an interswitch link (ISL) in an Ethernet fabric.

Interswitch link (ISL) An interface connected between switches in an Ethernet fabric. The portson either end of the interface are called ISL ports or Fabric ports. The ISLcan be a single link or a Brocade-proprietary hardware-based trunk.

RBridge A physical (VDX) switch in an Ethernet fabric.

RBridge ID A unique identifier for an RBridge. In commands, the RBridge ID is used inreferencing all interfaces in the Ethernet fabric

VCS ID A unique identifier for an Ethernet (VCS) fabric. The factory default VCS ID is 1.All switches in an Ethernet fabric must have the same VCS ID.

WWN World Wide Name. A globally unique ID that is burned into the switch at the factory.



◆ Better resiliency — Multiple "least cost" paths are used for highperformance and high reliability.

◆ Improved performance — With TRILL the shortest paths throughthe network are all active, and traffic is automatically distributedacross the equal-cost paths, unlike classic Ethernet runningSpanning Tree Protocol, only 50% of the links are forwardingtraffic while the rests are blocked and waiting for the primary linkto fail.

◆ Easier to scale — Ethernet fabrics easily scale up and down ondemand.

More advanced Ethernet fabrics borrow further from Fibre Channelfabric concepts. They are self-forming and function as a single logicalentity, in which all switches automatically know about each other andall connected physical and logical devices. Therefore, managementcan be domain-based rather than device-based and defined by policyrather than by repetitive procedures.

Examples of Brocade VCS deploynments in a data center can befound in the Fibre Channel over Ethernet (FCoE) Data Center Bridging(DCB) Concepts and Protocols TechBook, available on the E-LabNavigator, Topology Resource Center tab.

Brocade VCS fabric technology 721

722


Brocade VDX switches supported and unsupported featuresThis section contains the following information:

◆ “Supported features and configurations” on page 722

◆ “Unsupported features and configurations” on page 724

Supported features and configurationsThe Brocade VDX Data Center switches, along with the VCS fabrictechnology, is scalable and flexible network building blocks thatnetwork and SAN architects can apply in three important use cases:

◆ Classic 10-GbE Ethernet Access and Aggregation

• An existing hierarchical network design can be preserved, butnow with the benefit of no STP, resulting in an active-activenetwork and reduced management overhead.

• A two-switch VCS configuration can be deployed at the top ofeach server rack.

◆ Scale-out Fabrics for Virtual Data Centers

• Scale-out VCS fabrics can be deployed instead of a hierarchicalnetwork to flatten the network design. These fabrics provideseamless application mobility, and the user has the ability tomanage the entire fabric as a single Logical Chassis.

◆ LAN/SAN Convergence

• The VCS fabric provides end-to-end DCB capabilities, whichallows traditional IP and storage traffic (Fibre Channel overEthernet and/or iSCSI traffic), to exist on the same network.

• The Brocade VDX 6730 allows FCoE traffic in a VCS Ethernetfabric to connect with Fibre Channel ports in a Fibre Channelfabric via Fibre Channel routing. This allows traffic flowbetween devices using FCoE and Fibre Channel, whileavoiding the need to merge a VCS Ethernet fabric with a FibreChannel fabric.

Refer to the “Brocade VDX switches in a VCS fabric case study”on page 726.



Currently, the following features and topologies are supported byBrocade:

◆ VDX/VCS and Brocade FC SAN connectivity is enabled via FCR

• An E_Port on a In VCS mode, the switch is part of an Ethernetfabric involving two or more VCS-enabled switches.30connects to an EX-port on an FCR

There are no restrictions to the platform used as the router, aslong as that switch is operating with FOS v7.0.1 or later.However, one must note that some platforms have greatercapabilities than others, and thus the supported FCR limitsmay be different depending on the platform used. TheBrocade FOS Scalability Matrix calls out these limits. (And theenhanced BB limits are platform dependent)

• Integrated Routing license is not required for device sharingbetween VDX/VCS and FC SAN

• Switches that support IR: 5100, 5300, DCX, DCX-4S, 6510,DCX-8510-8, DCX-8510-4

◆ Integrated Routing (IR) license is not required for FCRconnectivity between VCS/VDX 6730 and Brocade FC (FOSbased) fabrics

• FOS v7.0.1 allows EX_Port configuration in absence of IRlicense

• A new FCR EX_Port mode 5 is used for VCS connectivity at anEX-port

• IR license requirement is not imposed in any routing scenariowhether it is edge to backbone or edge to edge routing for VCSto FC SAN device sharing

• IR license is still required to share devices between FC fabricsvia FCR

• When IR license exists, device sharing is allowed bothbetween VCS and FC fabrics and also between FC fabrics

◆ Increase in FCR Backbone fabric scalability limits for devices toenable Backbone to edge routing configurations

• Support for up to 2000 devices in FCR backbone fabric (anincrease from 512 devices) with reduced edge fabric count(max of eight)

• Allows existing flat L2 FC fabric environments to easilyinterop with VCS without disruption

Brocade VDX switches supported and unsupported features 723

724


• Increased FCR scalability is applicable to 5300, DCX, DCX-4Sand DCX8510 platforms

• Increased scalability is not applicable to 5100, 7800 and 6510platforms

◆ More number of VCS /FC edge fabrics can be supported whenusing a dedicated FCR backbone (Edge to Edge routing scenario)

• Up to 48 edge fabrics supported when DCX/DCX8510 is usedas an FCR backbone

• Refer to the scalability matrix for additional details

◆ FCR must be running FOS v7.0.1 or later and VDX 6730 must berunning NOS v2.1.1 or later

◆ Edge to Edge Connectivity

• VDX/VCS and FC fabrics connect as edge fabrics to FCR (nohosts/targets in FCR backbone)

◆ Edge to Backbone Connectivity

• VDX/VCS can connect to an FCR backbone that hashosts/targets Devices are shared between VDX/VCS and FCfabrics using LSAN zones

◆ Servers connected to VCS can communicate with FC targets in anFC SAN

◆ Servers in FC SAN cannot communicate with targets attached toVCS

Unsupported features and configurations◆ FCoE devices connected to Brocade 8000 and FCOE10-24 cannot

communicate with FCoE devices in VCS

◆ Communication with M-EOS edge fabrics, or fabrics operating inInteropmode 2 or Interopmode 3, or Admin Domains enabledFOS fabrics will not be supported

◆ VDX 6730 does not support Access Gateway mode with currentrelease

◆ E_Port on VDX 6730 cannot connect to E_Port on an FC switchwith current release

◆ E_Port on VDX 6730 cannot connect to an E_Port on another VDX6730 with current release



◆ FC host/storage cannot be directly attached to FC-ports on VDX6730 with current release

◆ Sharing devices between two VCS fabrics using FCR is notsupported

◆ Sharing devices between VCS/VDX and an FC fabric across FCIPconnection (within the FC fabric) is not supported with currentrelease

• A VCS connected to an EX-port cannot share devices with anFC SAN connected to a VEX port

• SAN/VCS interop is not intended as a distance solution forVCS for the current release

◆ Long distance links between the VDX 6730 and FOS FC router(FCR) are not supported (IFL cannot be configured for longdistance mode)

Brocade VDX switches supported and unsupported features 725

726


Brocade VDX switches in a VCS fabric case studyThis section provides a case study for VCS LAN/SAN convergence.The following information is included:


◆ “VCS LAN/SAN Convergence case study” on page 726

Summary of configuration stepsThis section provides a summary of the steps needed to configureindividual VDX switches and prepare them to participate in a VCSfabric. Each step is explained further in “Configuration steps” onpage 730.

1. Assign static IP addresses.

2. Verify VCS fabric license.

3. Configure VDX switch attributes within a VCS.

4. Verify configuration on Brocade FCR capable switches.

5. Verify connectivity of all devices.

6. Zone end devices.

7. Verify devices can access each other across the Backbone fabric.

VCS LAN/SAN Convergence case studyThis section defines the source topology and prerequisites andexplains the configurations steps in detail.

Source topologyThe source topology comprises of a Brocade FC SAN that has aConnectrix DS5100B switch and an ED-DCX-4S-B switch with VirtualFabrics enabled. The logical switch with FID 40 on the ED-DCX-4S-Bis ISLed to the DS5100B as shown in Figure 145 on page 727. ThisSAN is connected to a Brocade FC Routing backbone fabric. The FCRBB fabric comprises of the FID 127 (base switch) on the sameED-DCX-4S-B and a DS5300B switch.

There are multiple end devices other than the ones shown inFigure 145 that are connected to these switches. Only the ones shownwill be used in this case study.



The FC SAN has an active zoneset named "ZonesetA."

Figure 145 Source topology example

Target topologyThe target topology comprises of a VCS cluster configured with twoVDX 6720 and four VDX 6730 switches connected to the FC SAN,(described in the source topology), via a Brocade FC Routingbackbone, as illustrated in Figure 146 on page 728.

Brocade VDX switches in a VCS fabric case study 727

728



PrerequisitesBefore configuring the target topology, shown in Figure 146, note thefollowing prerequisites:

◆ Brocade FC routing capable switches are required to form thebackbone (BB) router fabric between the VCS and the Brocade FCSAN. The VCS fabric and FC SAN can be viewed as edge fabricsfrom this BB fabric.

◆ The BB router switches should be running FOS v7.0.1 and later.

◆ The VCS license should be present on the VDX switches. This willbe verified as part of this case study implementation.



AssumptionsNote the following assumptions:

◆ This case study example begins by configuring the individualVDX switches and getting them ready to participate in a VCSfabric. Once e a VCS is configured, it will be integrated with anexisting Brocade FC SAN environment.

◆ The VCS will comprise of the two VDX 6720 switches and fourVDX 6730 switches shown in the target topology, Figure 145 onpage 727.

◆ A core-edge topology was used to create the VCS fabric. There aretwo VDX 6730s in the core and two VDX 6730s, two VDX 6720s inthe edge.

◆ The BB fabric has a DCX4S switch with Virtual Fabric (VF)enabled. The 'base switch' on that physical switch is configured toact as the BB router.

◆ As discussed in “Source topology” on page 726, the Brocade FCSAN comprises of a Brcd 5100 switch and the logical switch (fid=4) on the DCX-4S (with VFs enabled). Since the primaryobjective of this case study is to discuss the VCS configuration, itis assumed that this Brocade FC SAN is already integrated withthe FCR Backbone fabric as shown in Figure 147 on page 730.

◆ In the detailed topology diagram shown in Figure 147, the zonedserver and storage ports are shaded in the same color.


730


Figure 147 Detailed topology

Configuration stepsThe steps summarized in “Summary of configuration steps” onpage 726, are detailed next.

Step1. Assign static IP addressesTo assign static IP addresses to the VDX 6720 and 6730 switches,complete the following steps.



1. Connect the Brocade serial cable (with the RJ-45 and RS-232 ends)to the RJ-45 serial port on the switch and to an RS-232 serial porton the workstation or terminal device.

2. Open a terminal emulator application (such as HyperTerminal ona PC), and configure the application as follows:

On a Windows environment, enter the following values:

9600 bits per second, 8 databits, no parity, 1 stop bit, and no flowcontrol.

3. Once the switch CLI is accessible, complete the following steps toset a static IP address.

a. Log in to the switch using the default password. The defaultlogin is admin/password.

b. Use the ip address command to set the Ethernet IP address.

If you are going to use an IPv4 IP address, enter the IP addressin dotted decimal notation. You should also disable DHCPand enter a gateway address as well. In the following examplethe IP is set on one of the VDX 6730 switches to 10.246.54.60with 10.246.54.2 set as the gateway IP.

switch # conf tswitch(config)# interface Management 1/0switch(config-Management-1/0)# no ip address dhcpswitch(config-Management-1/0)# ip address 10.246.54.60/24switch(config-Management-1/0)# ip gateway-address 10.246.54.2

The other switches, including the VDX 6720 can be configuredin a similar manner. Note that the IPs of the individualswitches have been specified inFigure 147 on page 730.

Step 2. Verify VCS Fabric licenseTo verify that the VCS Fabric license is installed to enable FC portaccess and management on all the VDX 6730 switches, enter the showlicense command, as shown next.

VDX6730# show licenseNote that in the following output, the root bridge ID is 1. This will bechanged later.

Look for the license in bold.

rbridge-id: 1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

FCoE Base licenseFeature name:FCOE_BASE


732


xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxPorts on Demand license - additional 8 port upgrade licenseFeature name:PORTS_ON_DEMAND_1

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxError:Invalid or unsupported licenseFeature name:VCS_2_NODE

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxVCS Fabric licenseFeature name:VCS_FABRIC

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxError:Invalid or unsupported licenseFeature name:VCS_MULTI_NODE

Step 3. Configure VDX switch attributes within a VCSPerform the following steps to enable VCS mode. It should beenabled by default.

1. Assign a root-bridge ID to the VDX switches that will participatein a VCS cluster.

Unique rbridge-ids have to be assigned for all switches in sameVCS cluster. Below is the command to configure the rbridge-id.Configure the rbridge-id on the same VDX730 on to which the IPwas configured, to 3.

switch# vcs rbridge-id 10This operation will change the configuration to default and reboot the switch. Do

you want to continue? [y/n]:ySuccessfully set rbridge-id.

Similarly, assign unique rbridge-ids on the other VDX switches.

2. Assign a switch name to a VDX by running the followingcommand. In this example, the name of the switch is set toVDX_60. '3' is the rbridge-id allocated in Step 1.

switch# conf tswitch(config)# switch-attributes 3switch(config-switch-attributes-3)# host-name VDX_60

3. Physically connect the VDX switches to construct the topologyillustrated in Figure 147 on page 730. Note that the inter-switchlinks (ISLs) in this case will be run between the 10 GbE ports onthe VDX.

Follow the EMC best practice to have at least two ISLs betweenany two VDX switches.

4. To enable VCS, enter the vcs enable command, including therbridge parameter (configured in step1), as shown next.



Note: VCS is enabled by default.

switch# vcs rbridge-id 10 enable

The switch reboots when you confirm that you want to enableVCS mode. Run this command on all the VDX switches.

5. Once a switch comes back up, if it is connected to otherVCS-enabled switches then the negotiation protocols begin,determining which switch in the fabric is the principal switch andensuring that all the RBridge IDs are unique.

a. Should the insistent domain ID not be unique, you can changeit. The switch with the lowest World Wide Name (WWN)becomes the principal switch, primarily for purposes ofdetermining the uniqueness of the ID of the other switches inthe fabric.

b. If necessary, you can change the VCS ID. This identifies theVCS fabric of which the VDX switch can be a part of. Bydefault, the VCS ID of every Brocade VDX 6730 is 1. Changethe VCS ID if you need to create a new, separate VCS fabric.This can be done by entering the following command:

switch# vcs vcs-id x

where x is the new VCS ID number.

Step 4. Verify configuration on Brocade FCR capable switchesVerify that the Brocade FCR-capable switches, which in this case arethe Brocade 5300 and Brocade DCX-4S (with VFs configured), arerunning FOS 7.0.1 and up. You must configure an inter-fabric linkwith EX ports on the Backbone (BB) router switches and the FC porton the VDX switch acting as the E_Port.

1. Provide the command for enabling EX ports to be run on thenon-VFs enabled Brcd 5300 switch.

For this example, consider that ports 4, 5, 6, and 7 on the Brcd5300 have to be connected to the VDX_70 as shown in Figure 147on page 730. These two ports must be configured as EX ports.

a. A fabric ID (FID) must be assigned to every edge fabricconnected to a router switch. This example assigns a FID of 6to the edge fabric, which in this case is the VCS cluster. This iswhat the '-f 6' implies in the following command.


734


b. The interoperability mode parameter must be supplied for theEX port that is being configured, which is decided based onthe edge switch or fabric it is connected to. The '-m 5' is mode5, which is the Brocade NOS fabric mode introduced with FOS7.0.1.

The series of commands to be run on the Brcd 5300 are:

Brcd5300_BB> portdisable 4-7Brcd5300_BB> portcfgexport 4 -a 1 -f 6 -m 5Brcd5300_BB> portcfgexport 5 -a 1 -f 6 -m 5Brcd5300_BB> portcfgexport 6 -a 1 -f 6 -m 5Brcd5300_BB> portcfgexport 7 -a 1 -f 6 -m 5

2. Provide the command for the VF-enabled DCX4S switch.

The previous commands can be run on the Brcd DCX4S switches thathave VFs enabled, but the ports configured as EX ports have to be apart of a base switch.

Some of the points to note when configuring EX ports on switcheswith VFs enabled are as follows:

◆ In case a base switch is not already present, it must be configured

◆ An existing logical switch can be converted into a base switch

◆ Note that FID 128 is not a base switch; it is the default FID

◆ No F_Ports or end devices can be present on a base switch

Note: Refer to the "Virtual Fabrics" case study in the EMC Fibre Channel SANTopologies TechBook, located on the E-Lab Navigator, Topology ResourceCenter tab, for the actual steps involved in configuring a base switch on aswitch with VFs enabled.

Step 5. Verify connectivity of all devicesSteps in this section check the name-servers on all the switches andcopy and paste the WWPNs of the logged in devices into a text file.This example uses CLI for configuring the zoning; therefore, it will beeasy to paste the WWPNs into the LSAN zones when they are beingdefined on the individual switches.

The following commands have to be run to verify that your CNA andstorage port show up in the respective name-servers.

For simplicity, in all the upcoming zoning steps the sample WWPNs10:10:10:10:10:10:10:10 and 50:50:50:50:50:50:50:50 are used.



Consider that the WWPN of one of the CNA ports on Host 2 is asfollows:

WWPN of CNA_1: 10:10:10:10:10:10:10:10

The WWPN on the storage port SPA4 for VNX_West to which theCNA_1 defined above will be zoned is:

50:50:50:50:50:50:50:50:50

◆ On the VDX_60:

# show name-server detailPID: 0d0200Port Name: 10:10:10:10:10:10:10:10Node Name: 20:20:20:20:20:20:20:20SCR: 3FC4s: FCPPortSymb: [92] "Brocade-1020 | 3.0.3.0 | WIN-9I2KM0KQSSS | Windows Server (R)2008 Standard | Service Pack 1"NodeSymb: NULLFabric Port Name: 50:00:53:3A:F3:AD:50:20Permanent Port Name: 10:10:10:10:10:10:10:10Device type: Physical InitiatorInterface: Fcoe 1/3/1Physical Interface: Te 3/0/5Share Area: YesRedirect: No

PID: 0b0240Port Name: 10:00:00:05:1E:9A:A9:EBNode Name: 20:00:00:05:1E:9A:A9:EBFC4s: FCPPortSymb: [92] "Brocade-1020 | 3.0.3.0 | WIN-9I2KM0KQSSS | Windows Server (R)2008 Standard | Service Pack 1"NodeSymb: NULLFabric Port Name: 50:00:53:3B:B6:00:90:21Permanent Port Name: 10:00:00:05:1E:9A:A9:EBDevice type: Physical InitiatorInterface: Fcoe 1/1/2Physical Interface: Te 1/0/5Share Area: No

◆ On the Brocade FC DS-5100B switch and FC routers:

> nsshow{Type Pid COS PortName NodeName TTL(sec)N 022000; 2,3;50:50:50:50:50:50:50:50; 50:06:01:60:C6:E0:01:49; na

FC4s: IPFC FCP


736


PortSymb: [28] "DGC LUNZ 0531"NodeSymb: NULLFabric Port Name: 50:00:51:E7:90:4C:1A:7APermanent Port Name: 50:50:50:50:50:50:50:50Port Index: 32Share Area: NoDevice Shared in Other AD: NoRedirect: NoPartial: NoLSAN: No

The Local Name Server has 1 entry }

These are the WWPNs that have to be pasted into a text file. For thisexample, the text file will comprise of sixteen WWPNs since there areeight end devices with dual ports.

Another check that may be worthwhile to execute, is to verify that alldevices in either edge fabric, the FC SAN and the VCS cluster, arevisible from the Backbone fabric. This can be done by running thefollowing command on the FCR capable Brcd 5300 or the DCX4S baseswitch:

> fcrphydevshowDevice WWN PhysicalExists PID

in Fabric-----------------------------------------

4 50:50:50:50:50:50:50:50 022000...

6 10:10:10:10:10:10:10:10 0d0200...Total devices displayed: 7

In this example, expect to see a list of fourteen WWPNs displayedsince there are a total of seven dual port end devices distributedacross the two edge fabrics. The eighth device - server 53.181 isconnected to the backbone and therefore will not be seen as an outputfor this command.

What will also be displayed is the edge fabric ID the devices belongin, which will help the user to know whether the device belongs tothe FC SAN or the VCS cluster.

Step 6. Zone end devicesAs shown in Figure 147 on page 730, the servers and storage portsshaded in the same colors are the ones that will be zoned as a part ofthis case study.

Note how these zone pairs cover:



◆ End devices spread across the VCS cluster

◆ Across the VCS cluster and backbone router

◆ Across the VCS cluster and the edge FC SAN

Depending on where the end devices are placed, the zones have to becreated in one or two different places in this configuration. Except forthe case where both end devices are in the VCS cluster, you will haveto create Brocade FCR based LSAN zones on the VCS cluster whereone device is and on the Backbone fabric or on the Edge fabric, basedon where the second end device is logged in.

In this example, the pairs of end devices zoned, their locations, andthe kind of zones are as follows:

Note: The LSAN zones created in both areas of the configuration should haveidentical names and zone members.

In this case, the LSAN zones have to be created on the VDX_60 wherethe server port is connected, and on the Brcd 5100 where the storageport is connected and added into the respective active zoningconfigurations on those switches. Since they are part of the VCScluster and FC SAN edge fabrics respectively, they can have differentactive zoning configurations but must contain the identical LSANzone required to zone these end devices.

For this example, it is assumed that there is no active zoningconfiguration on the VCS cluster edge fabric. As stated in the sourcetopology description on page 726, the FC SAN edge fabric has anactive ZonesetA.

To complete the zoning, complete the following steps:

1. Create the identical LSAN zones on the VDX_60 and Brcd 5100switch. The exact steps to be run on the VDX for zoningconfiguration are provided below.

Servers Server at Storage Storage at Zones configured are

Host 1 FCR Backbone VMAX_East VCS Cluster LSAN zones on FCR BB and VCS

Host 2 FC SAN VMAX_West VCS Cluster LSAN zones on FC SAN and VCS

Host 3 VCS Cluster VNX_East VCS Cluster Regular (non LSAN) zones on VCS

Host 4 VCS Cluster VNX_West FC SAN LSAN zones on VCS and FC SAN


738


"lsan_zone1" will be the zone created on both the VDX_60 andBrcd 5100 switch.

2. Create a zone configurations on the VDX switches.

"ZonesetB" will be created on the VDX_60.

3. Add the LSAN zones into this zone configuration.

4. Activate the zone configuration so that it propagates to the otherswitches in the edge fabric.

Zoning on VDX_60


1. Create the LSAN zones.

switch(config)# zoning defined-configuration zone lsan_zone1switch(config-zone-zone1)# member-entry 10:10:10:10:10:10:10:10switch(config-zone-zone1)# member-entry 50:50:50:50:50:50:50:50switch(config-zone-zone1)# exit

2. Create the zone configuration adding the LSAN zone andactivating the configuration.

switch(config)# zoning defined-configuration cfg ZonesetBswitch(config-cfg-config1)# member-zone lsan_zone1switch(config-cfg-config1)# exit

switch(config)# zoning enabled-configuration cfg-name ZonesetBswitch(config)# zoning enabled-configuration default-zone-access allaccessswitch(config)# zoning enabled-configuration cfg-action cfg-save

3. Verify the active configuration has been saved and is running onat least two or more switches in the VCS

# show running-config zoningzoning defined-configuration cfg ZonesetBmember-zone lsan_zone1...

!zoning defined-configuration zone lsan_zone1member-entry 10:10:10:10:10:10:10:10member-entry 50:50:50:50:50:50:50:50

!zoning enabled-configuration cfg-name ZonesetBzoning enabled-configuration enabled-zone lsan_zone1member-entry 10:10:10:10:10:10:10:10member-entry 50:50:50:50:50:50:50:50

!...



Zoning on Brcd 5100


1. Create the LSAN zones.

B5100> zonecreate "lsan_zone1","10:10:10:10:10:10:10:10;50:50:50:50:50:50:50:50"

2. Create the zone configuration adding the LSAN zone andactivating the configuration.

B5100> cfgcreate "ZonesetA", "lsan_zone1"B5100> cfgenable "ZonesetA"B5100> cfgsave

3. Verify the active configuration has been saved.

DCX4S_178:FID127:admin> cfgactvshowEffective configuration:cfg: ZonesetBzone: lsan_zone1

50:50:50:50:50:50:50:5010:10:10:10:10:10:10:10

...

Step 7. Verify devices can access each other across BackbonefabricEMC recommends running some commands on the FCR Backbone toverify that the lsan zones have been appropriately activated and theindividual edge fabrics have gained access to the zoned devicesacross the Backbone fabric. This can be done by running thefollowing two commands on the FCR Backbone switch.

Note: In this case study, the VCS cluster has a fabric ID of 4 based on how theEX ports connected to the switches in this cluster were configured in “Step 4.Verify configuration on Brocade FCR capable switches” on page 733, andassumes that the FC SAN has a fabric ID of 6.

FCR_BB> lsanzoneshowFabric ID: 4 Zone Name: lsan_zone1

10:10:10:10:10:10:10:1050:50:50:50:50:50:50:50

Fabric ID: 6 Zone Name: lsan_zone150:50:50:50:50:50:50:5010:10:10:10:10:10:10:10

The above output verifies that the identical lsan zones have beencreated in both edge fabrics.

FCR_BB> fcrproxydevshowProxy WWN Proxy Device Physical State


740


Created PID Exists PIDin Fabric in Fabric--------------------------------------------------------------------------------4 10:10:10:10:10:10:10:10 02f301 6 2d1d00 Imported6 50:50:50:50:50:50:50:50 01f201 4 b2f9c0 Imported

The above output validates that the device from FID 4 is importedinto FID 6 and vice-versa.

Step 8. Repeat Steps 1 through 7 for the remaining host and storageport pairs.



ReferencesMuch of the information provided in this section was derived fromthe Brocade website, http://www.brocade.com, which providesdetails on VCS Fabric technology, its technical architecture, Ethernetfabrics, configuration guides, guides, case studies, and deploymentscenarios.

◆ EMC Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB)Concepts and Protocols TechBook, available on the E-Lab Navigator,Topology Resource Center tab.

◆ EMC Fibre Channel SAN Topologies TechBook, available on the E-LabNavigator, Topology Resource Center tab.

◆ EMC host connectivity guides, available on the E-Lab Navigator,PDFs and Guides tab.

◆ Network OS Administrator's Guide, located at www.brocade.com,for information on the following:

• Brocade VCS Fabric formation• Ethernet fabrics• Automatic Migration of Port Profiles• Configuring Classic Ethernet or IEEE 802.x standards like STP,

VLANs, Link Aggregation, ACL, IGMP, etc.• Configuring Fibre Channel (for example, Zoning and FC

ports)• Configuring vLAGs

◆ Network OS Command Reference, located at www.brocade.com, forinformation on the following:

• Configuring Classic Ethernet or IEEE 802.x standards like STP,VLANs, Link Aggregation, ACL, IGMP, etc.

• Configuring Fibre Channel (for example, Zoning and FCports)

• Configuring vLAGs

◆ Products section at www.brocade.com ror more productinformation on Brocade VDX data center switches.

◆ Technical documents located at www.brocade.com and internetdrafts at www.ietf.org for more information on TRILL(Transparent Interconnect of Lots of Links).

References 741

742



A

This appendix contains the following information:

◆ Installing the Nexus 5000................................................................ 744◆ Installing the MP-8000B .................................................................. 749◆ Installing the MLX ........................................................................... 758

Note: More detailed Fibre Channel over Ethernet information can be found inthe Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts andProtocols TechBook at http://elabnavigator.EMC.com, under the TopologyResource Center tab. Information in this TechBook includes FCoE andEthernet basics, EMC storage, RecoverPoint and Celerra MPFS as solutions inan FCoE environement, andtroubleshooting basic FCoE and CEE problems.

Installation Notes

Installation Notes 743


744

Installation Notes

Installing the Nexus 5000This section describes how to use the rackmount kit provided withthe switch to install the Nexus 5000 switch into a cabinet or rack. AllNexus Series switch switches use the same installation procedure.

IMPORTANT

If the rack is on wheels, ensure that the brakes are engaged or thatthe rack is otherwise stabilized.

Table 17 lists the items contained in the rackmount kit provided withthe switch.

To install Nexus 5020, complete the following steps:

1. Unpack the Nexus 5020s, install the mounting ears to the chassis,and install the chassis into the rack.

To install the switch in a cabinet or rack using the rackmount kitprovided with the switch, follow these steps:

a. Install the front rackmount brackets as follows:

– Position a front rackmount bracket against the chassis andalign the screw holes as shown in Figure 148 on page 745.Then attach the front rackmount bracket to the chassis withsix of the M4 screws.

– Repeat with the other front rackmount bracket on the otherside of the switch.

Table 17 Nexus 5020 switch rackmount kit

Quantity Part description

2 rackmount brackets

16 M4x0.7 x 8-mm Phillips countersunk screws

2 rackmount guides

12 12-24 x 3/4-inch Phillips countersunk screws

12 10-32 x 3/4-inch Phillips countersunk screws

1 Slider rails


Installation Notes

Figure 148 Attaching a front rackmount bracket to a Nexus 5020 switch

b. Install the rackmount guides on the switch as follows:

– Position one of the rackmount brackets against the side ofthe switch and align the screw holes as shown inFigure 148. Then attach the bracket to the switch with twoof the flat-head M4 screws.

– Repeat with the other rackmount bracket on the other sideof the switch.

c. Attach the slider rails to the rack as shown in Figure 149 onpage 746. Use two 12-24 screws or two 10-32 screws,depending on the rack rail thread type. For racks with squareholes, insert the 12-24 cage nuts in position behind themounting holes in the slider rails.

– Repeat with the other slider rail on the other side of therack.

– Use the tape measure and level to verify that the rails arehorizontal and at the same height.

Installing the Nexus 5000 745

746

Installation Notes

Figure 149 Installing the slider rails

d. Insert the switch into the rack:

– Using both hands, position the back of the switch betweenthe front posts of the rack.

– Align the two rackmount guides on either side of theswitch with the slider rails installed in the rack. Slide therackmount glides onto the slider rails, and then gentlyslide the switch all the way into the rack, as shown inFigure 150 on page 747. If the switch does not slide easily,try realigning the rackmount glides on the slider rails.


Installation Notes

Figure 150 Sliding the chassis into the rack

e. Stabilize the switch in the rack by attaching the frontrackmount brackets to the front rackmounting rails:

– Insert two screws (12-24 or 10-32, depending on rack type)through the cage nuts and the holes in one of the frontrackmount brackets and into the threaded holes in therackmounting rail, as shown in Figure 151 on page 748.

– Repeat for the front rackmount bracket on the other side ofthe switch.

Installing the Nexus 5000 747

748

Installation Notes

Figure 151 Attaching the switch to the rack

2. Install the expansion modules.

3. Connect the power cords.


Installation Notes

Installing the MP-8000BThis section contains the following information to guide you throughthe installation process of the MP-8000B.


◆ “Installing power to the MP-8000B” on page 751

◆ “Interpreting POST results” on page 751

PrerequisitesConsider the following before installing the MP-8000B, eachdiscussed further in this section:

◆ “Installation and mounting considerations” on page 749

◆ “Electrical considerations” on page 749

◆ “Environmental considerations” on page 750

◆ “Cabinet considerations” on page 750

Installation and mounting considerationsYou can install the MP-8000B switch in the following ways:

◆ As a standalone unit on a flat surface.

◆ In a standard EIA-310 computer equipment cabinet with theoptional rail kit.

Electrical considerationsTo install and operate the switch successfully, ensure the following:

◆ The primary outlet is correctly wired, protected by a circuitbreaker, and grounded in accordance with local electrical codes.

◆ The supply circuit, line fusing, and wire size are adequate, asspecified by the electrical rating on the switch nameplate.

◆ The power supply standards provided in Table 18 on page 750 aremet.

Installing the MP-8000B 749

750

Installation Notes

Environmental considerationsFor successful installation and operation of the switch, ensure that thefollowing environmental requirements are met:

◆ At a minimum, adequate cooling requires that you install theswitch with the non-port side, which contains the air intake vents,facing the cool-air aisle.

◆ All equipment in the rack should force air in the same direction toavoid intake of exhaust air.

◆ A maximum of 102 cubic meters/hour (60 cubic feet/minute) anda minimum of 74.8 cubic meters/hour (44 cubic feet/minute) ofair flow is available to the air intake vents on the non-port side ofthe switch.

◆ The ambient air temperature does not exceed 40° C (104° F) whilethe switch is operating.

Cabinet considerationsFor successful installation and operation of the switch in a cabinet,ensure the following cabinet requirements are met:

◆ The cabinet must be a standard EIA cabinet.◆ A cabinet space that is one rack unit (1U) high; 4.45 cm (1.75

inches) high and 48.3 cm (19 inches) wide.

Table 18 Power supply specifications

Specification Value

Maximum output of one power supply 350 watts

System DC power consumption (excluding power supplyand fan FRUs)

Idle: 182.4 WMaximum: 306 W

Input system power (including power supply and fan FRUs) Idle: 206 WMaximum: 306 W

Input voltage 85-264 VAC

Input line frequency 47-63 Hz

BTU rating 1044.11 BTU/hr

Inrush current Maximum of 60A for period of10-15mS

Input line protection Both AC lines are fused


Installation Notes

◆ The equipment in the cabinet is grounded through a reliablebranch circuit connection and maintain ground at all times. Donot rely on a secondary connection to a branch circuit, such as apower strip.

◆ Airflow and temperature requirements are met on an ongoingbasis, particularly if the switch is installed in a closed ormulti-cabinet assembly.

◆ The additional weight of the switch does not exceed the cabinet’sweight limits or unbalance the cabinet in any way.

◆ The cabinet is secured to ensure stability in case of unexpectedmovement, such as an earthquake.

Installing power to the MP-8000BComplete the following steps to provide power to the MP-8000B.

1. Connect the power cords to both power supplies, and then topower sources on separate circuits to protect against AC failure.Ensure that the cords have a minimum service loop of 6in.available and are routed to avoid stress.

2. Power on the power supplies by flipping both AC switches to the“I” symbol. The power supply LEDs display amber until POST iscomplete, and then change to green. The switch usually requiresseveral minutes to boot and complete POST.

Note: Power is supplied to the switch as soon as the first power supply isconnected and turned on.

3. After POST is complete, verify that the switch power and statusLEDs on the left of the port side of the switch are green. SeeFigure 152 on page 754 for the specific location of these LEDs.

Interpreting POST resultsThis section includes the following information for interpretyingPOST results:

◆ “Post” on page 752◆ “Boot” on page 752◆ “LED activity interpretation” on page 753


752

Installation Notes

PostPOST is a system check that is performed each time the switch ispowered on, rebooted, or reset. During POST, the LEDs flash eitheramber or green. Any errors that occur during POST are listed in theerror log.

Complete the following steps to determine whether POST hascompleted successfully and whether any errors were detected.

1. Verify that the switch LEDs indicate that all components arehealthy.

See Table 19 on page 755 and Table 20 on page 756 fordescriptions and interpretations of LED patterns. If one or moreLEDs do not display a healthy state, verify that the LEDs on theswitch are not set to “beacon” by entering the switchShowcommand to detect if beaconing is active.

2. Verify that the switch prompt displays on the terminal of acomputer workstation connected to the switch.

If there is no switch prompt when POST completes, press Enter.

If the switch prompt still does not display, try opening a Telnetsession or accessing the switch through another managementtool. If this is not successful, the switch did not successfullycomplete POST. Contact your switch supplier for repair.

3. Review the switch system log for errors.

Any errors detected during POST are written to the system log,accessible through the errShow command.

For information about all referenced commands and how to accessthe error log, refer to the EMC Connectrix B Fabric OS Administrator’sGuide. For information about error messages, refer to the EMCConnectrix B Series System Error Message Reference Manual.

BootIn addition to POST, boot includes the following tasks after POST iscomplete:

◆ Performs universal port configuration.

◆ Initializes links.

◆ Analyzes fabric. If any ports are connected to other switches, theswitch participates in a fabric configuration.

◆ Obtains a domain ID and assigns port addresses.


Installation Notes

◆ Constructs unicast routing tables.

◆ Enables normal port operation.

LED activity interpretationSystem activity and status can be determined through the activity ofthe LEDs on the switch.

There are three possible LED states: no light, a steady light, and aflashing light. Flashing lights may be slow, fast, or flickering. Thelights are green or amber.

Sometimes, the LEDs flash either of the colors during boot, POST, orother diagnostic tests. This is normal; it does not indicate a problemunless the LEDs do not indicate a healthy state after all boot processesand diagnostic tests are complete.

MP-8000B LEDs The MP-8000-B has the following LEDs:

◆ One system status LED (above) on the left side.

◆ One power status LED (below) on the left side.

◆ Two Ethernet Port LEDs (one green and one amber).

◆ One bicolor (green/amber) port status LED for each port on theswitch. These LEDs are arrayed above each pair of Ethernet orFibre Channel ports. The port LEDs are located in the array in thesame relative positions as the ports.

◆ One power supply LED above the AC power switch on eachpower supply on the non-port side of the switch.

◆ One fan status LED at the bottom of each fan assembly on thenon-port side of the switch.

LED locations Figure 152 on page 754 shows the port side of the MP-8000-B. Theport status LEDs for both the GbE and FC ports are verticallystaggered to correspond to the upper and lower ports in each pair.


754

Installation Notes

Figure 152 LEDs on the port side of MP-8000-B

Figure 153 shows the LEDs on the non-port side of the switch.

Figure 153 LEDs on the non-port side of MP-8000-B

1 System power LED 4 Ethernet link LED

2 System status LED 5 GbE port status LED (port 0)

3 Ethernet speed LED 6 GbE port status LED (port 4)

2

1

5 6

3 4

3 4 5

21

1 Power supply # 2 status LED 4 Fan assembly # 2 status LED

2 Power supply # 1 status LED 5 Fan assembly # 1 status LED

3 Fan assembly # 3 status LED


Installation Notes

LED patterns Table 19 describes the port side LEDs and their behavior.

Table 19 LED patterns (page 1 of 2)

LED name LED color Status of hardware Recommended action

Power Status No light System is off or there is an internalpower supply failure.

Verify the system is powered on (power supplyswitches to “1”), the power cables attached,and your power source is live.If the system power LED is not green, the unitmay be faulty.Contact your switch service provider.

Steady green System is on and power supplies arefunctioning properly.

No action required.

System Status No light System is off or there is no power. Verify the system is on and has completedbooting.

Steady green System is on and functioning properly. No action required.

Steady amber (formore than fiveseconds)

Unknown state, boot failed, or thesystem is faulty.This LED displays steady amberduring POST, this is normal and doesnot indicate a fault.

Perform the following steps:Connect a serial cable to the system.Reboot the system.Check the failure indicated on the systemconsole.Contact your switch service provider.

Flashing amber Attention is required. A number ofvariables can cause this statusincluding a single power supply failure,a fan failure, or one or moreenvironmental ranges has beenexceeded.

Check the management interface and theerror log for details on the cause of status.Contact your switch service provider.

Ethernet Speed(Managementport)

No light Port speed is 10 Mb/s. No action required.

Steady green Port speed is 100/1000 Mb/s. No action required.

EthernetActivity/Link(Managementport)

No light There is no link. Verify that the Ethernet cable is connectedcorrectly.

Steady green There is a link. No action required.

Blinking green There is link activity (traffic). No action required.


756

Installation Notes

Table 20 describes the LEDs on the non-port side of the switch.

Optical mediaport status(bi-color LED foreach FC andCEE port)

Off No light or signal carrier on mediainterface.

Verify that the transceiver is installed correctlyand that the cable is connected correctly.

Steady amber Receiving light or carrier, but notonline.

No action required.

Slow blinking amber(2 sec)

Disabled (by diagnostics or byportDisable command).

Verify that the diagnostic tests are not beingrun. Reenable the port using the portEnablecommand.

Fast blinking amber(1/2 sec)

Port failure. Check the management interface and theerror log for details on the cause of the failure.Contact Technical Support if necessary.

Steady green Online. No action required.

Slow blinking green (2sec)

Online but segmented (loopback cableor incompatible switch).

No action required.

Fast blinking green(1/2 sec)

Internal loop-back (diagnostic). No action required.

Flickering green Online, frames flowing through port. No action required.

Table 19 LED patterns (page 2 of 2)


Table 20 LEDs on non-port side of switch (page 1 of 2)


Power SupplyStatus

No light Power supply is not receiving power oris off.

Verify the power supply is on and seated andthe power cord is connected to a functioningpower source.

Steady green Power supply is operating normally. No action required.

Steady amber (formore than 5 seconds)

Power supply is faulty.Note: When the switch is firstpowered on the power supply statusLED will show amber until POST hascompleted.

Try the following:Check the power cable connection.Verify that the power supply is powered on.Replace the power supply FRU.


Installation Notes

Fan Status No light Fan assembly is not receiving power. Try the following:Verify the fan FRU is seated correctly.Verify the switch is powered on.

Steady green Fan assembly is operating normally. No action required.

Steady amber (formore than 5 seconds)

Fan fault for one of the followingreasons:One or more of the fan(s) in the fanassembly has failedThe fan FRU was disabled by the userNote: When the switch is firstpowered on the fan status LED willshow amber until POST hascompleted.

Try the following:Verify the fan FRU is enabled (use thefanEnable command).Replace the fan FRU.

Table 20 LEDs on non-port side of switch (page 2 of 2)



758

Installation Notes

Installing the MLXThis section provides the steps needed to successfully complete thelink aggregation configurations on the Brocade NetIron MLX-16switching router. A Link Aggregation Group (LAG), also known asan Etherchannel, is a way in which to combine the interfaces into asingle virtual link, therefore combining its bandwidth.

In the following example we are configuring the LAG and associatedconfigurations on the Core-MLX-1. The same configuration steps willalso need to be completed on CORE-MLX-2.

To configure the LAG and associated configurations on theCore-MLX-1, complete the following steps:

1. Enter the configuration mode and configure the LAG.

telnet@CORE-MLX-1# Config t

2. Create the LAG, apply the name, and configure it as dynamic.

telnet@CORE-MLX-1(config)# Lag MP8000B_1 dynamic

3. Assign the correct ports to the LAG.

telnet@CORE-MLX-1(config-lag-MP8000B_1)# Ports eth 1/1 eth 2/1

4. Assign the primary port.

Note: A LAG must have an interface configured as a primary port beforeit can be deployed.

telnet@CORE-MLX-1(config-lag-MP8000B_1)# primary-port 1/1

5. Enable LACP for this LAG.

telnet@CORE-MLX-1(config-lag-MP8000B_1)# lacp-timeout short

6. Enable the LAG.

telnet@CORE-MLX-1(config-lag-MP8000B_1)#)# Deploy

7. Add VLAN 100 to the LAG. This is done by adding the physicalinterfaces of the LAG to the correct VLAN as “tagged” to allowfor trunking of the VLAN over the LAG.


Installation Notes

Note: This configuration step must be completed for any VLANtraversing a 802.1Q trunk, which also includes any 802.1Q configuredLAG interfaces.

telnet@CORE-MLX-1(config-lag-MP8000B_1)# vlan 100telnet@CORE-MLX-1(config-vlan-100)# tagged eth 1/1 eth 2/1

8. Enable the physical interfaces of the LAG.

Note: Only the primary port of the LAG needs to be configured asenabled, and only the primary port will show up in the runningconfiguration as enabled.

telnet@CORE-MLX-1(config-vlan-100)# interface ethernet 1/1telnet@CORE-MLX-1(config-if-e10000-1/1)# enable

9. Verify that the configuration steps Step 1through Step 8 havebeen completed successfully.

Note: The following output is truncated to show only the appropriateconfigurations.

telnet@CORE-MLX-1(config-if-e10000-1/1)# sho run

lag "MP8000B_1" dynamicports ethernet 1/1 ethernet 2/1primary-port 1/1lacp-timeout shortdeploy..vlan 100tagged ethe 1/1 ethe 2/1

Installing the MLX 759

760

Installation Notes


Glossary

This glossary contains terms related to EMC products and EMCnetworked storage concepts.

access control A service that allows or prohibits access to a resource. Storagemanagement products implement access control to allow or prohibitspecific users. Storage platform products implement access control,often called LUN Masking, to allow or prohibit access to volumes byInitiators (HBAs). See also “persistent binding” and “zoning.”

active domain ID The domain ID actively being used by a switch. It is assigned to aswitch by the principal switch.

active zone set The active zone set is the zone set definition currently in effect andenforced by the fabric or other entity (for example, the name server).Only one zone set at a time can be active.

agent An autonomous agent is a system situated within (and is part of) anenvironment that senses that environment, and acts on it over time inpursuit of its own agenda. Storage management software centralizesthe control and monitoring of highly distributed storageinfrastructure. The centralizing part of the software managementsystem can depend on agents that are installed on the distributedparts of the infrastructure. For example, an agent (softwarecomponent) can be installed on each of the hosts (servers) in anenvironment to allow the centralizing software to control andmonitor the hosts.

alarm An SNMP message notifying an operator of a network problem.


762

Glossary

any-to-any portconnectivity

A characteristic of a Fibre Channel switch that allows any port on theswitch to communicate with any other port on the same switch.

application Application software is a defined subclass of computer software thatemploys the capabilities of a computer directly to a task that userswant to perform. This is in contrast to system software thatparticipates with integration of various capabilities of a computer,and typically does not directly apply these capabilities to performingtasks that benefit users. The term application refers to both theapplication software and its implementation which often refers to theuse of an information processing system. (For example, a payrollapplication, an airline reservation application, or a networkapplication.) Typically an application is installed “on top of” anoperating system like Windows or LINUX, and contains a userinterface.

application-specificintegrated circuit

(ASIC)

A circuit designed for a specific purpose, such as implementinglower-layer Fibre Channel protocols (FC-1 and FC-0). ASICs contrastwith general-purpose devices such as memory chips ormicroprocessors, which can be used in many different applications.

arbitration The process of selecting one respondent from a collection of severalcandidates that request service concurrently.

ASIC family Different switch hardware platforms that utilize the same port ASICcan be grouped into collections known as an ASIC family. Forexample, the Fuji ASIC family which consists of the ED-64M andED-140M run different microprocessors, but both utilize the sameport ASIC to provide Fibre Channel connectivity, and are therefore inthe same ASIC family. For inter operability concerns, it is useful tounderstand to which ASIC family a switch belongs.

ASCII ASCII (American Standard Code for Information Interchange),generally pronounced [aeski], is a character encoding based onthe English alphabet. ASCII codes represent text in computers,communications equipment, and other devices that work withtext. Most modern character encodings, which support manymore characters, have a historical basis in ASCII.

audit log A log containing summaries of actions taken by a ConnectrixManagement software user that creates an audit trail of changes.Adding, modifying, or deleting user or product administrationvalues, creates a record in the audit log that includes the date andtime.


Glossary

authentication Verification of the identity of a process or person.

Bbackpressure The effect on the environment leading up to the point of restriction.

See “congestion.”

BB_Credit See “buffer-to-buffer credit.”

beaconing Repeated transmission of a beacon light and message until an error iscorrected or bypassed. Typically used by a piece of equipment whenan individual Field Replaceable Unit (FRU) needs replacement.Beaconing helps the field engineer locate the specific defectivecomponent. Some equipment management software systems such asConnectrix Manager offer beaconing capability.

BER See “bit error rate.”

bidirectional In Fibre Channel, the capability to simultaneously communicateat maximum speeds in both directions over a link.

bit error rate Ratio of received bits that contain errors to total of all bitstransmitted.

blade server A consolidation of independent servers and switch technology in thesame chassis.

blocked port Devices communicating with a blocked port are prevented fromlogging in to the Fibre Channel switch containing the port orcommunicating with other devices attached to the switch. A blockedport continuously transmits the off-line sequence (OLS).

bridge A device that provides a translation service between two networksegments utilizing different communication protocols. EMC supportsand sells bridges that convert iSCSI storage commands from a NIC-attached server to Fibre Channel commands for a storage platform.

broadcast Sends a transmission to all ports in a network. Typically used inIP networks. Not typically used in Fibre Channel networks.

broadcast frames Data packet, also known as a broadcast packet, whose destinationaddress specifies all computers on a network. See also “multicast.”


764

Glossary

buffer Storage area for data in transit. Buffers compensate for differences inlink speeds and link congestion between devices.

buffer-to-buffer credit The number of receive buffers allocated by a receiving FC_Port to atransmitting FC_Port. The value is negotiated between Fibre Channelports during link initialization. Each time a port transmits a frame itdecrements this credit value. Each time a port receives an R_Rdyframe it increments this credit value. If the credit value isdecremented to zero, the transmitter stops sending any new framesuntil the receiver has transmitted an R_Rdy frame. Buffer-to-buffercredit is particularly important in SRDF and Mirror View distanceextension solutions.

CCall Home A product feature that allows the Connectrix service processor to

automatically dial out to a support center and report systemproblems. The support center server accepts calls from the Connectrixservice processor, logs reported events, and can notify one or moresupport center representatives. Telephone numbers and otherinformation are configured through the Windows NT dial-upnetworking application. The Call Home function can be enabled anddisabled through the Connectrix Product Manager.

channel With Open Systems, a channel is a point-to-point link thattransports data from one point to another on the communicationpath, typically with high throughput and low latency that isgenerally required by storage systems. With Mainframeenvironments, a channel refers to the server-side of theserver-storage communication path, analogous to the HBA inOpen Systems.

Class 2 Fibre Channelclass of service

In Class 2 service, the fabric and destination N_Ports provideconnectionless service with notification of delivery or nondeliverybetween the two N_Ports. Historically Class 2 service is not widelyused in Fibre Channel system.

Class 3 Fibre Channelclass of service

Class 3 service provides a connectionless service without notificationof delivery between N_Ports. (This is also known as datagramservice.) The transmission and routing of Class 3 frames is the sameas for Class 2 frames. Class 3 is the dominant class of communicationused in Fibre Channel for moving data between servers and storageand may be referred to as “Ship and pray.”


Glossary

Class F Fibre Channelclass of service

Class F service is used for all switch-to-switch communication in amultiswitch fabric environment. It is nearly identical to class 2 from aflow control point of view.

community A relationship between an SNMP agent and a set of SNMP managersthat defines authentication, access control, and proxy characteristics.

community name A name that represents an SNMP community that the agent softwarerecognizes as a valid source for SNMP requests. An SNMPmanagement program that sends an SNMP request to an agentprogram must identify the request with a community name that theagent recognizes or the agent discards the message as anauthentication failure. The agent counts these failures and reports thecount to the manager program upon request, or sends anauthentication failure trap message to the manager program.

community profile Information that specifies which management objects areavailable to what management domain or SNMP communityname.

congestion Occurs at the point of restriction. See “backpressure.”

connectionless Non dedicated link. Typically used to describe a link betweennodes that allows the switch to forward Class 2 or Class 3 framesas resources (ports) allow. Contrast with the dedicated bandwidththat is required in a Class 1 Fibre Channel Service point-to-pointlink.

Connectivity Unit A hardware component that contains hardware (and possiblysoftware) that provides Fibre Channel connectivity across a fabric.Connectrix switches are example of Connectivity Units. This is a termpopularized by the Fibre Alliance MIB, sometimes abbreviated toconnunit.

Connectrixmanagement

software

The software application that implements the management userinterface for all managed Fibre Channel products, typically theConnectrix -M product line. Connectrix Management software is aclient/server application with the server running on the Connectrixservice processor, and clients running remotely or on the serviceprocessor.

Connectrix serviceprocessor

An optional 1U server shipped with the Connectrix -M product lineto run the Connectrix Management server software and EMC remotesupport application software.


766

Glossary

Control Unit In mainframe environments, a Control Unit controls access to storage.It is analogous to a Target in Open Systems environments.

core switch Occupies central locations within the interconnections of a fabric.Generally provides the primary data paths across the fabric and thedirect connections to storage devices. Connectrix directors aretypically installed as core switches, but may be located anywhere inthe fabric.

credit A numeric value that relates to the number of available BB_Creditson a Fibre Channel port. See“buffer-to-buffer credit”.

DDASD Direct Access Storage Device.

default Pertaining to an attribute, value, or option that is assumed whennone is explicitly specified.

default zone A zone containing all attached devices that are not members of anyactive zone. Typically the default zone is disabled in a Connectrix Menvironment which prevents newly installed servers and storagefrom communicating until they have been provisioned.

Dense WavelengthDivision Multiplexing

(DWDM)

A process that carries different data channels at different wavelengthsover one pair of fiber optic links. A conventional fiber-optic systemcarries only one channel over a single wavelength traveling through asingle fiber.

destination ID A field in a Fibre Channel header that specifies the destinationaddress for a frame. The Fibre Channel header also contains a SourceID (SID). The FCID for a port contains both the SID and the DID.

device A piece of equipment, such as a server, switch or storage system.

dialog box A user interface element of a software product typically implementedas a pop-up window containing informational messages and fieldsfor modification. Facilitates a dialog between the user and theapplication. Dialog box is often used interchangeably with window.

DID An acronym used to refer to either Domain ID or Destination ID. Thisambiguity can create confusion. As a result E-Lab recommends thisacronym be used to apply to Domain ID. Destination ID can beabbreviated to FCID.


Glossary

director An enterprise-class Fibre Channel switch, such as the ConnectrixED-140M, MDS 9509, or ED-48000B. Directors deliver highavailability, failure ride-through, and repair under power to insuremaximum uptime for business critical applications. Major assemblies,such as power supplies, fan modules, switch controller cards,switching elements, and port modules, are all hot-swappable.

The term director may also refer to a board-level module in theSymmetrix that provides the interface between host channels(through an associated adapter module in the Symmetrix) andSymmetrix disk devices. (This description is presented here only toclarify a term used in other EMC documents.)

DNS See “domain name service name.”

domain ID A byte-wide field in the three byte Fibre Channel address thatuniquely identifies a switch in a fabric. The three fields in a FCID aredomain, area, and port. A distinct Domain ID is requested from theprincipal switch. The principal switch allocates one Domain ID toeach switch in the fabric. A user may be able to set a Preferred IDwhich can be requested of the Principal switch, or set an InsistentDomain ID. If two switches insist on the same DID one or bothswitches will segment from the fabric.

domain name servicename

Host or node name for a system that is translated to an IP addressthrough a name server. All DNS names have a host name componentand, if fully qualified, a domain component, such as host1.abcd.com. Inthis example, host1 is the host name.

dual-attached host A host that has two (or more) connections to a set of devices.

EE_D_TOV A time-out period within which each data frame in a Fibre Channel

sequence transmits. This avoids time-out errors at the destinationNx_Port. This function facilitates high speed recovery from droppedframes. Typically this value is 2 seconds.

E_Port Expansion Port, a port type in a Fibre Channel switch that attaches toanother E_Port on a second Fibre Channel switch forming anInterswitch Link (ISL). This link typically conforms to the FC-SWstandards developed by the T11 committee, but might not supportheterogeneous inter operability.


768

Glossary

edge switch Occupies the periphery of the fabric, generally providing the directconnections to host servers and management workstations. No twoedge switches can be connected by interswitch links (ISLs).Connectrix departmental switches are typically installed as edgeswitches in a multiswitch fabric, but may be located anywhere in thefabric

Embedded WebServer

A management interface embedded on the switch’s code that offersfeatures similar to (but not as robust as) the Connectrix Manager andProduct Manager.

error detect time outvalue

Defines the time the switch waits for an expected response beforedeclaring an error condition. The error detect time out value(E_D_TOV) can be set within a range of two-tenths of a second to onesecond using the Connectrix switch Product Manager.

error message An indication that an error has been detected. See also “informationmessage” and “warning message.”

Ethernet A baseband LAN that allows multiple station access to thetransmission medium at will without prior coordination and whichavoids or resolves contention.

event log A record of significant events that have occurred on a Connectrixswitch, such as FRU failures, degraded operation, and port problems.

expansionport See “E_Port.”

explicit fabric login In order to join a fabric, an Nport must login to the fabric (anoperation referred to as an FLOGI). Typically this is an explicitoperation performed by the Nport communicating with the F_port ofthe switch, and is called an explicit fabric login. Some legacy FibreChannel ports do not perform explicit login, and switch vendorsperform login for ports creating an implicit login. Typically logins areexplicit.

FFA Fibre Adapter, another name for a Symmetrix Fibre Channel director.

F_Port Fabric Port, a port type on a Fibre Channel switch. An F_Port attachesto an N_Port through a point-to-point full-duplex link connection. AG_Port automatically becomes an F_port or an E-Port depending onthe port initialization process.


Glossary

fabric One or more switching devices that interconnect Fibre ChannelN_Ports, and route Fibre Channel frames based on destination IDs inthe frame headers. A fabric provides discovery, path provisioning,and state change management services for a Fibre Channelenvironment.

fabric element Any active switch or director in the fabric.

fabric login Process used by N_Ports to establish their operating parametersincluding class of service, speed, and buffer-to-buffer credit value.

fabric port A port type (F_Port) on a Fibre Channel switch that attaches to anN_Port through a point-to-point full-duplex link connection. AnN_Port is typically a host (HBA) or a storage device like Symmetrixor VNX series or CLARiiON.

fabric shortest pathfirst (FSPF)

A routing algorithm implemented by Fibre Channel switches in afabric. The algorithm seeks to minimize the number of hops traversedas a Fibre Channel frame travels from its source to its destination.

fabric tree A hierarchical list in Connectrix Manager of all fabrics currentlyknown to the Connectrix service processor. The tree includes allmembers of the fabrics, listed by WWN or nickname.

failover The process of detecting a failure on an active Connectrix switch FRUand the automatic transition of functions to a backup FRU.

fan-in/fan-out Term used to describe the server:storage ratio, where a graphicrepresentation of a 1:n (fan-in) or n:1 (fan-out) logical topology lookslike a hand-held fan, with the wide end toward n. By conventionfan-out refers to the number of server ports that share a single storageport. Fan-out consolidates a large number of server ports on a fewernumber of storage ports. Fan-in refers to the number of storage portsthat a single server port uses. Fan-in enlarges the storage capacityused by a server. A fan-in or fan-out rate is often referred to as just then part of the ratio; For example, a 16:1 fan-out is also called a fan-outrate of 16, in this case 16 server ports are sharing a single storage port.

FCP See “Fibre Channel Protocol.”

FC-SW The Fibre Channel fabric standard. The standard is developed by theT11 organization whose documentation can be found at T11.org. EMCactively participates in T11. T11 is a committee within theInterNational Committee for Information Technology (INCITS).


770

Glossary

fiber optics The branch of optical technology concerned with the transmission ofradiant power through fibers made of transparent materials such asglass, fused silica, and plastic.

Either a single discrete fiber or a non spatially aligned fiber bundlecan be used for each information channel. Such fibers are often calledoptical fibers to differentiate them from fibers used innon-communication applications.

fibre A general term used to cover all physical media types supported bythe Fibre Channel specification, such as optical fiber, twisted pair, andcoaxial cable.

Fibre Channel The general name of an integrated set of ANSI standards that definenew protocols for flexible information transfer. Logically, FibreChannel is a high-performance serial data channel.

Fibre ChannelProtocol

A standard Fibre Channel FC-4 level protocol used to run SCSI overFibre Channel.

Fibre Channel switchmodules

The embedded switch modules in the back plane of the blade server.See “blade server” on page 763.

firmware The program code (embedded software) that resides and executes ona connectivity device, such as a Connectrix switch, a Symmetrix FibreChannel director, or a host bus adapter (HBA).

F_Port Fabric Port, a physical interface within the fabric. An F_Port attachesto an N_Port through a point-to-point full-duplex link connection.

frame A set of fields making up a unit of transmission. Each field is made ofbytes. The typical Fibre Channel frame consists of fields:Start-of-frame, header, data-field, CRC, end-of-frame. The maximumframe size is 2148 bytes.

frame header Control information placed before the data-field when encapsulatingdata for network transmission. The header provides the source anddestination IDs of the frame.

FRU Field-replaceable unit, a hardware component that can be replaced asan entire unit. The Connectrix switch Product Manager can displaystatus for the FRUs installed in the unit.

FSPF Fabric Shortest Path First, an algorithm used for routing traffic. Thismeans that, between the source and destination, only the paths that


Glossary

have the least amount of physical hops will be used for framedelivery.

Ggateway address In TCP/IP, a device that connects two systems that use the same

or different protocols.

gigabyte (GB) A unit of measure for storage size, loosely one billion (109) bytes. Onegigabyte actually equals 1,073,741,824 bytes.

G_Port A port type on a Fibre Channel switch capable of acting either as anF_Port or an E_Port, depending on the port type at the other end ofthe link.

GUI Graphical user interface.

HHBA See “host bus adapter.”

hexadecimal Pertaining to a numbering system with base of 16; valid numbers usethe digits 0 through 9 and characters A through F (which representthe numbers 10 through 15).

high availability A performance feature characterized by hardware componentredundancy and hot-swappability (enabling non-disruptivemaintenance). High-availability systems maximize systemuptime while providing superior reliability, availability, andserviceability.

hop A hop refers to the number of InterSwitch Links (ISLs) a FibreChannel frame must traverse to go from its source to its destination.Good design practice encourages three hops or less to minimizecongestion and performance management complexities.

host bus adapter A bus card in a host system that allows the host system to connect tothe storage system. Typically the HBA communicates with the hostover a PCI or PCI Express bus and has a single Fibre Channel link tothe fabric. The HBA contains an embedded microprocessor with onboard firmware, one or more ASICs, and a Small Form FactorPluggable module (SFP) to connect to the Fibre Channel link.


772

Glossary

II/O See “input/output.”

in-band management Transmission of monitoring and control functions over the FibreChannel interface. You can also perform these functions out-of-bandtypically by use of the ethernet to manage Fibre Channel devices.

information message A message telling a user that a function is performing normally orhas completed normally. User acknowledgement might or might notbe required, depending on the message. See also “error message” and“warning message.”

input/output (1) Pertaining to a device whose parts can perform an input processand an output process at the same time. (2) Pertaining to a functionalunit or channel involved in an input process, output process, or both(concurrently or not), and to the data involved in such a process.(3) Pertaining to input, output, or both.

interface (1) A shared boundary between two functional units, defined byfunctional characteristics, signal characteristics, or othercharacteristics as appropriate. The concept includes the specificationof the connection of two devices having different functions. (2)Hardware, software, or both, that links systems, programs, ordevices.

Internet Protocol See “IP.”

interoperability The ability to communicate, execute programs, or transfer databetween various functional units over a network. Also refers to aFibre Channel fabric that contains switches from more than onevendor.

interswitch link (ISL) Interswitch link, a physical E_Port connection between any twoswitches in a Fibre Channel fabric. An ISL forms a hop in a fabric.

IP Internet Protocol, the TCP/IP standard protocol that defines thedatagram as the unit of information passed across an internet andprovides the basis for connectionless, best-effort packet deliveryservice. IP includes the ICMP control and error message protocol asan integral part.

IP address A unique string of numbers that identifies a device on a network. Theaddress consists of four groups (quadrants) of numbers delimited by


Glossary

periods. (This is called dotted-decimal notation.) All resources on thenetwork must have an IP address. A valid IP address is in the formnnn.nnn.nnn.nnn, where each nnn is a decimal in the range 0 to 255.

ISL Interswitch link, a physical E_Port connection between any twoswitches in a Fibre Channel fabric.

Kkilobyte (K) A unit of measure for storage size, loosely one thousand bytes. One

kilobyte actually equals 1,024 bytes.

Llaser A device that produces optical radiation using a population inversion

to provide light amplification by stimulated emission of radiationand (generally) an optical resonant cavity to provide positivefeedback. Laser radiation can be highly coherent temporally, spatially,or both.

LED Light-emitting diode.

link The physical connection between two devices on a switched fabric.

link incident A problem detected on a fiber-optic link; for example, loss of light, orinvalid sequences.

load balancing The ability to distribute traffic over all network ports that are thesame distance from the destination address by assigning differentpaths to different messages. Increases effective network bandwidth.EMC PowerPath software provides load-balancing services for serverIO.

logical volume A named unit of storage consisting of a logically contiguous set ofdisk sectors.

Logical Unit Number(LUN)

A number, assigned to a storage volume, that (in combination withthe storage device node's World Wide Port Name (WWPN))represents a unique identifier for a logical volume on a storage areanetwork.


774

Glossary

MMAC address Media Access Control address, the hardware address of a device

connected to a shared network.

managed product A hardware product that can be managed using the ConnectrixProduct Manager. For example, a Connectrix switch is a managedproduct.

management session Exists when a user logs in to the Connectrix Management softwareand successfully connects to the product server. The user mustspecify the network address of the product server at login time.

media The disk surface on which data is stored.

media access control See “MAC address.”

megabyte (MB) A unit of measure for storage size, loosely one million (106) bytes.One megabyte actually equals 1,048,576 bytes.

MIB Management Information Base, a related set of objects (variables)containing information about a managed device and accessedthrough SNMP from a network management station.

multicast Multicast is used when multiple copies of data are to be sent todesignated, multiple, destinations.

multiswitch fabric Fibre Channel fabric created by linking more than one switch ordirector together to allow communication. See also “ISL.”

multiswitch linking Port-to-port connections between two switches.

Nname server (dNS) A service known as the distributed Name Server provided by a Fibre

Channel fabric that provides device discovery, path provisioning, andstate change notification services to the N_Ports in the fabric. Theservice is implemented in a distributed fashion, for example, eachswitch in a fabric participates in providing the service. The service isaddressed by the N_Ports through a Well Known Address.

network address A name or address that identifies a managed product, such as aConnectrix switch, or a Connectrix service processor on a TCP/IPnetwork. The network address can be either an IP address in dotted


Glossary

decimal notation, or a Domain Name Service (DNS) name asadministered on a customer network. All DNS names have a hostname component and (if fully qualified) a domain component, suchas host1.emc.com. In this example, host1 is the host name and EMC.comis the domain component.

nickname A user-defined name representing a specific WWxN, typically used ina Connectrix -M management environment. The analog in theConnectrix -B and MDS environments is alias.

node The point at which one or more functional units connect to thenetwork.

N_Port Node Port, a Fibre Channel port implemented by an end device(node) that can attach to an F_Port or directly to another N_Portthrough a point-to-point link connection. HBAs and storage systemsimplement N_Ports that connect to the fabric.

NVRAM Nonvolatile random access memory.

Ooffline sequence

(OLS)The OLS Primitive Sequence is transmitted to indicate that theFC_Port transmitting the Sequence is:

a. initiating the Link Initialization Protocol

b. receiving and recognizing NOS

c. or entering the offline state

OLS See “offline sequence (OLS)”.

operating mode Regulates what other types of switches can share a multiswitch fabricwith the switch under consideration.

operating system Software that controls the execution of programs and that mayprovide such services as resource allocation, scheduling,input/output control, and data management. Although operatingsystems are predominantly software, partial hardwareimplementations are possible.

optical cable A fiber, multiple fibers, or a fiber bundle in a structure built to meetoptical, mechanical, and environmental specifications.


776

Glossary

OS See “operating system.”

out-of-bandmanagement

Transmission of monitoring/control functions outside of the FibreChannel interface, typically over ethernet.

oversubscription The ratio of bandwidth required to bandwidth available. When allports, associated pair-wise, in any random fashion, cannot sustainfull duplex at full line-rate, the switch is oversubscribed.

Pparameter A characteristic element with a variable value that is given a constant

value for a specified application. Also, a user-specified value for anitem in a menu; a value that the system provides when a menu isinterpreted; data passed between programs or procedures.

password (1) A value used in authentication or a value used to establishmembership in a group having specific privileges. (2) A unique stringof characters known to the computer system and to a user who mustspecify it to gain full or limited access to a system and to theinformation stored within it.

path In a network, any route between any two nodes.

persistent binding Use of server-level access control configuration information topersistently bind a server device name to a specific Fibre Channelstorage volume or logical unit number, through a specific HBA andstorage port WWN. The address of a persistently bound device doesnot shift if a storage target fails to recover during a power cycle. Thisfunction is the responsibility of the HBA device driver.

port (1) An access point for data entry or exit. (2) A receptacle on a deviceto which a cable for another device is attached.

port card Field replaceable hardware component that provides the connectionfor fiber cables and performs specific device-dependent logicfunctions.

port name A symbolic name that the user defines for a particular port throughthe Product Manager.

preferred domain ID An ID configured by the fabric administrator. During the fabricbuild process a switch requests permission from the principalswitch to use its preferred domain ID. The principal switch can


Glossary

deny this request by providing an alternate domain ID only ifthere is a conflict for the requested Domain ID. Typically aprincipal switch grants the non-principal switch its requestedPreferred Domain ID.

principal downstreamISL

The ISL to which each switch will forward frames originating fromthe principal switch.

principal ISL The principal ISL is the ISL that frames destined to, or coming from,the principal switch in the fabric will use. An example is an RDIframe.

principal switch In a multiswitch fabric, the switch that allocates domain IDs toitself and to all other switches in the fabric. There is always oneprincipal switch in a fabric. If a switch is not connected to anyother switches, it acts as its own principal switch.

principal upstream ISL The ISL to which each switch will forward frames destined for theprincipal switch. The principal switch does not have any upstreamISLs.

product (1) Connectivity Product, a generic name for a switch, director, or anyother Fibre Channel product. (2) Managed Product, a generichardware product that can be managed by the Product Manager (aConnectrix switch is a managed product). Note distinction from thedefinition for “device.”

Product Manager A software component of Connectrix Manager software such as aConnectrix switch product manager, that implements themanagement user interface for a specific product. When a productinstance is opened from the Connectrix Manager software productsview, the corresponding product manager is invoked. The productmanager is also known as an Element Manager.

product name A user configurable identifier assigned to a Managed Product.Typically, this name is stored on the product itself. For a Connectrixswitch, the Product Name can also be accessed by an SNMP Manageras the System Name. The Product Name should align with the hostname component of a Network Address.

products view The top-level display in the Connectrix Management software userinterface that displays icons of Managed Products.


778

Glossary

protocol (1) A set of semantic and syntactic rules that determines the behaviorof functional units in achieving communication. (2) A specificationfor the format and relative timing of information exchanged betweencommunicating parties.

RR_A_TOV See “resource allocation time out value.”

remote access link The ability to communicate with a data processing facility through aremote data link.

remote notification The system can be programmed to notify remote sites of certainclasses of events.

remote userworkstation

A workstation, such as a PC, using Connectrix Management softwareand Product Manager software that can access the Connectrix serviceprocessor over a LAN connection. A user at a remote workstation canperform all of the management and monitoring tasks available to alocal user on the Connectrix service processor.

resource allocationtime out value

A value used to time-out operations that depend on a maximum timethat an exchange can be delayed in a fabric and still be delivered. Theresource allocation time-out value of (R_A_TOV) can be set within arange of two-tenths of a second to 120 seconds using the Connectrixswitch product manager. The typical value is 10 seconds.

SSAN See “storage area network (SAN).”

segmentation A non-connection between two switches. Numerous reasons exist foran operational ISL to segment, including interop modeincompatibility, zoning conflicts, and domain overlaps.

segmented E_Port E_Port that has ceased to function as an E_Port within amultiswitch fabric due to an incompatibility between the fabricsthat it joins.

service processor See “Connectrix service processor.”

session See “management session.”

single attached host A host that only has a single connection to a set of devices.


Glossary

small form factorpluggable (SFP)

An optical module implementing a shortwave or long wave opticaltransceiver.

SMTP Simple Mail Transfer Protocol, a TCP/IP protocol that allows users tocreate, send, and receive text messages. SMTP protocols specify howmessages are passed across a link from one system to another. Theydo not specify how the mail application accepts, presents or stores themail.

SNMP Simple Network Management Protocol, a TCP/IP protocol thatgenerally uses the User Datagram Protocol (UDP) to exchangemessages between a management information base (MIB) and amanagement client residing on a network.

storage area network(SAN)

A network linking servers or workstations to disk arrays, tapebackup systems, and other devices, typically over Fibre Channel andconsisting of multiple fabrics.

subnet mask Used by a computer to determine whether another computerwith which it needs to communicate is located on a local orremote network. The network mask depends upon the class ofnetworks to which the computer is connecting. The maskindicates which digits to look at in a longer network address andallows the router to avoid handling the entire address. Subnetmasking allows routers to move the packets more quickly.Typically, a subnet may represent all the machines at onegeographic location, in one building, or on the same local areanetwork.

switch priority Value configured into each switch in a fabric that determines itsrelative likelihood of becoming the fabric’s principal switch.

TTCP/IP Transmission Control Protocol/Internet Protocol. TCP/IP refers to

the protocols that are used on the Internet and most computernetworks. TCP refers to the Transport layer that provides flow controland connection services. IP refers to the Internet Protocol level whereaddressing and routing are implemented.

toggle To change the state of a feature/function that has only two states. Forexample, if a feature/function is enabled, toggling changes the state todisabled.


780

Glossary

topology Logical and/or physical arrangement of switches on a network.

trap An asynchronous (unsolicited) notification of an event originating onan SNMP-managed device and directed to a centralized SNMPNetwork Management Station.

Uunblocked port Devices communicating with an unblocked port can log in to a

Connectrix switch or a similar product and communicate withdevices attached to any other unblocked port if the devices are in thesame zone.

Unicast Unicast routing provides one or more optimal path(s) between any oftwo switches that make up the fabric. (This is used to send a singlecopy of the data to designated destinations.)

upper layer protocol(ULP)

The protocol user of FC-4 including IPI, SCSI, IP, and SBCCS. In adevice driver ULP typically refers to the operations that are managedby the class level of the driver, not the port level.

URL Uniform Resource Locater, the addressing system used by the WorldWide Web. It describes the location of a file or server anywhere on theInternet.

Vvirtual switch A Fibre Channel switch function that allows users to subdivide a

physical switch into multiple virtual switches. Each virtual switchconsists of a subset of ports on the physical switch, and has all theproperties of a Fibre Channel switch. Multiple virtual switches can beconnected through ISL to form a virtual fabric or VSAN.

virtual storage areanetwork (VSAN)

An allocation of switch ports that can span multiple physicalswitches, and forms a virtual fabric. A single physical switch cansometimes host more than one VSAN.

volume A general term referring to an addressable logically contiguousstorage space providing block IO services.

VSAN Virtual Storage Area Network.


Glossary

Wwarning message An indication that a possible error has been detected. See also “error

message” and “information message.”

World Wide Name(WWN)

A unique identifier, even on global networks. The WWN is a 64-bitnumber (XX:XX:XX:XX:XX:XX:XX:XX). The WWN contains an OUIwhich uniquely determines the equipment manufacturer. OUIs areadministered by the Institute of Electronic and Electrical Engineers(IEEE). The Fibre Channel environment uses two types of WWNs; aWorld Wide Node Name (WWNN) and a World Wide Port Name(WWPN). Typically the WWPN is used for zoning (path provisioningfunction).

Zzone An information object implemented by the distributed

Nameserver(dNS) of a Fibre Channel switch. A zone contains a set ofmembers which are permitted to discover and communicate with oneanother. The members can be identified by a WWPN or port ID. EMCrecommends the use of WWPNs in zone management.

zone set An information object implemented by the distributedNameserver(dNS) of a Fibre Channel switch. A Zone Set contains aset of Zones. A Zone Set is activated against a fabric, and only oneZone Set can be active in a fabric.

zonie A storage administrator who spends a large percentage of hisworkday zoning a Fibre Channel network and provisioning storage.

zoning Zoning allows an administrator to group several devices by functionor by location. All devices connected to a connectivity product, suchas a Connectrix switch, may be configured into one or more zones.


782

Glossary


Index

BBlade Server 420boot 752Brocade Converged 10 GbE Switch Module 423Brocade VCS Fabric technology 718

CCisco unified computing system (UCS) 421command

diagDisablePost 48errShow 752fastboot 48switchShow 752

Control plane protocol 33

DData plane protocol 33diagDisablePost command 48

EerrShow 752Etherchannel configuration, Nexus 7000 203

Ffastboot command 48FCoE initiator, configuring 405FCoE target, configuring 405Fibre Channel over Ethernet

benefits 28goal 26

Flex-10 Technology 658

FlexFabric 654FlexHBAs 666FlexNICs 666interconnect module 655Server Profiles 668

FlexNIC 661VLAN tags 661

front panel LEDs 753

HHP

Blade System enclosure 663Flex-10 technology 658FlexFabric 654Virtual Connect 654

II/O consolidation 26

technologies and protocols 33IGMP (Internet Group Management Protocol)

207interface

Ethernet end device 140, 142ethernet uplink 138, 142FC 139FCoE 137, 141

Internet Group Management Protocol (IGMP)207

interpretingLED activity 753POST results 751

IPSAN concepts 173


784

iSCSI initiator, configuring 388iSCSI target, configuring 388

JJuniper QFX3500 690

LLEDs

activity 753interpretation 755location 753patterns 755port side of the switch 753

Linux host, configuring 203, 271logical chassis 717

Mmonitoring through LED activity 753MP-8000B, installation 749MP-8000B, installation prerequisites 749

Nnetwork parameters, configuring 203, 271Nexus 2232PP 242Nexus 4000 424Nexus 5010 118Nexus 5020 117

configuring virtual interfaces 178Etherchannel configuration 203firmware upgrade procedure 280management 121spanning-tree protocol 206VLAN configuration 203

Nexus 5020-1configuring 180, 246

Nexus 5020-2configuring 191, 259

Nexus 7000, Etherchannel configuration 203

PPOST 48POST and boot specifications 48

SServer Profiles 668switchShow 752

TTarget topology

prerequisites 177, 214, 245target topology

example 176TRILL 718

UUCS, Cisco Unified Computer System 421

VVCS Ethernet fabric 720VCS Fabric 718Virtual Connect 654virtual PortChannel

case studies 324cross-connect topology 307host vPC topology 309overview 299peer-keepalive 306straight through topology 306terminology 301underlying protocols 294

VLAN tagsmapping mode 662tunneling mode 664

VMware host, configuring 203, 271

WWindows host, configuring 202, 270


fibre channel over ethernet (fcoe) data center bridging (dcb) case studies techbook

Technology

emc logo

trademarks of emc corporation

emc software

fcoe interface

emc online supporthttps

ethernet fcoe data center

contentsfibre channel

b overview