brocade fabric technology with the pure storage … · network os layer 2 switching configuration...

53-1004168-01February 2016

Brocade Fabric Technologywith the Pure StorageFlashArray//m20Validation Test Report

Supporting Fabric OS 7.4.1

© 2016, Brocade Communications Systems, Inc. All Rights Reserved.

Brocade, Brocade Assurance, the B-wing symbol, ClearLink, DCX, Fabric OS, HyperEdge, ICX, MLX, MyBrocade, OpenScript, VCS, VDX,Vplane, and Vyatta are registered trademarks, and Fabric Vision is a trademark of Brocade Communications Systems, Inc., in the UnitedStates and/or in other countries. Other brands, products, or service names mentioned may be trademarks of others.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning anyequipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this documentat any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not becurrently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained inthis document may require an export license from the United States government.

The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to theaccuracy of this document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs thataccompany it.

The product described by this document may contain open source software covered by the GNU General Public License or other opensource license agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable tothe open source software, and obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.

http://www.brocade.com/support/oscd

Contents

Preface..................................................................................................................................... 5Document History..............................................................................................5Overview........................................................................................................... 5Purpose of This Document................................................................................5Audience........................................................................................................... 5Objectives......................................................................................................... 6Related Documents ..........................................................................................6About Brocade.................................................................................................. 6About Pure Storage...........................................................................................6

Configure DUT and Test Equipment ........................................................................................... 7Task 1: Brocade FC Fabric Configuration.........................................................7Task 2: Pure Storage Array Configuration........................................................ 9Task 3: Host Configuration .............................................................................11

Test Plan.................................................................................................................................16Scope.............................................................................................................. 16What's New in This Report .............................................................................16Test Configuration...........................................................................................17DUT Descriptions............................................................................................ 17DUT Specifications..........................................................................................18

Test Cases.............................................................................................................................. 201.0 Fabric Initialization—Base Functionality................................................... 21

1.0.1 Storage Device—Physical and Logical Login with SpeedNegotiation.................................................................................... 21

1.0.2 Zoning and LUN Mapping.......................................................... 221.0.3 Storage Device Fabric I/O Integrity............................................ 221.0.4 Storage Device Multipath Configuration—Path Integrity............23

1.1 Fabric—Advanced Functionality............................................................... 241.1.1 Storage Device Bottleneck Detection Using MAPS FPI—

With Congested Host.....................................................................241.1.2 Storage Device Bottleneck Detection Using MAPS FPI—

With Congested Fabric.................................................................. 251.1.3 Storage Device—QoS Integrity with QoS Zone-Based

Traffic Prioritization........................................................................261.1.4 Storage Device—QoS Integrity with CS_CTL-Based Traffic

Prioritization...................................................................................261.1.5 Storage Device—FC Protocol Jammer Test Suite.....................291.1.6 Clear Link Diagnostics (D_Port) Test.........................................30

1.2 Stress and Error Recovery with Device Multipath.....................................321.2.1 Storage Device Fabric I/O Integrity—Congested Fabric............ 321.2.2 Storage Device Nameserver Integrity—Device Recovery

with Port Toggle.............................................................................321.2.3 Storage Device Nameserver Integrity—Device Recovery

with Device Relocation.................................................................. 33

Brocade Fabric Technology with the Pure Storage FlashArray//m20 Validation Test Report 353-1004168-01

1.2.4 Storage Device—Device Recovery with Device PortToggle—Extended Run............................................................... 34

1.2.5 Storage Device Recovery—ISL Port Toggle............................351.2.6 Storage Device Recovery—ISL Port Toggle (Entire Switch)... 361.2.7 Storage Device Recovery—Switch Offline...............................371.2.8 Storage Device Recovery—Switch Firmware Download......... 38

1.3 Storage Device—Fibre Channel Routing (FCR) InternetworkingTests........................................................................................................39

1.3.1 Storage Device Internetworking Validation with the FCHost............................................................................................. 39

1.3.2 Storage Device Internetworking Validation with FCoE TestUsing VDX FlexPort.....................................................................40

1.3.3 Storage Device Edge Recovery After FCR Disruptions...........411.3.4 Storage Device Backbone Recovery After FCR Disruptions... 42

1.4 Optional/Additional Tests........................................................................ 431.4.1 Storage Device Firmware Update............................................431.4.2 Performance I/O Testing with Varying Workload

Dimensions..................................................................................431.4.3 VMware I/O Analyzer Workload Test.......................................44

Test Conclusions...................................................................................................................45

4 Brocade Fabric Technology with the Pure Storage FlashArray//m20 Validation Test Report53-1004168-01

Preface

● Document History..............................................................................................................5● Overview........................................................................................................................... 5● Purpose of This Document................................................................................................5● Audience........................................................................................................................... 5● Objectives......................................................................................................................... 6● Related Documents ..........................................................................................................6● About Brocade.................................................................................................................. 6● About Pure Storage...........................................................................................................6

Document History

Date Part Number Description

February 2016 53-1004168-01 Initial version.

OverviewThe Solid State Ready (SSR) program is a comprehensive testing and configuration initiative to provideFibre Channel SAN and IP interoperability with flash storage. This program provides testing of multiplefabrics, heterogeneous servers, NICs, and HBAs in large port-count Brocade environments. Testingcovers future Brocade OS versions and vendor software releases. The SSR qualification program helpsverify seamless interoperability and optimum performance with solid-state storage in Brocade SANfabrics.

Purpose of This DocumentThis document provides the validation of Brocade Fabric technology with the Pure StorageFlashArray//m20 all-flash storage array, using multiple switch platforms, HBAs, and server operatingsystems. This validation shows that the //m20 array interoperates properly within a Brocade FibreChannel fabric, while supporting the performance and low latency associated with solid-state storage.

AudienceThis document is written for a technical audience, including solution architects, system engineers, andtechnical development representatives.


Objectives1. Test the Pure Storage FlashArray//m20 with Brocade FC fabrics, in single and routed configurations

for different stress and error recovery scenarios, to validate their interoperability and integration.2. Validate the performance of FC fabric in a solid-state storage environment for high-throughput and

low-latency applications.

Related Documents

• Brocade Fabric OS Administrator's Guide, 7.4.1• Brocade SAN Design and Best Practices• Brocade Monitoring and Alerting Policy Suite Administrator's Guide, 7.4.1• Brocade Network OS Layer 2 Switching Configuration Guide, 6.0.1• Emulex ExpressLane Configuration• Emulex OneCommand Manager User Manual• QLogic QLE2672 Adapter User Guide• QLogic BR-1860 Adapter Administrator's Guide

About BrocadeBrocade networking solutions help the world's leading organizations transition smoothly to a worldwhere applications and information reside anywhere. This vision is realized through the BrocadeOne™ strategy, which is designed to deliver key business benefits such as unmatched simplicity, non-stop networking, application optimization, and investment protection.

Innovative Ethernet and storage networking solutions for data center, campus, and service providernetworks help reduce complexity and cost while enabling virtualization and cloud computing toincrease business agility.

To help ensure a complete solution, Brocade partners with world-class IT companies and providescomprehensive education, support, and professional services offerings.

To learn more, visit www.brocade.com.

About Pure StoragePure Storage has a simple mission: to enable the widespread adoption of flash in the enterprise datacenter. We're a team of some of the brightest minds in storage, flash memory, and related technicalindustries. Founded on principles of teamwork and knowledge sharing, we focus on producing anexceptional result while we transform the landscape of the enterprise storage market (and have somefun along the way).

Objectives


http://www.brocade.com/content/html/en/administration-guide/fos-741-adminguide/index.html

http://www.brocade.com/content/brocade/en/backend-content/pdf-page.html?/content/dam/common/documents/content-types/product-design-guide/san-design-best-practices.pdf

http://www.brocade.com/content/html/en/administration-guide/fos-741-maps/index.html

http://www.brocade.com/content/html/en/configuration-guide/nos-601a-l2guide/index.html

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&uact=8&ved=0CCMQFjAB&url=http%3A%2F%2Fwww.emulex.com%2Fartifacts%2FBBE3C568-802C-4CD0-8471-C7B1680232CC%2Felx_wp_all_expresslane.pdf&ei=K-BLVcDZIpfpoAT41oC4DQ&usg=AFQjCNEpdBY_8qiiBUcmP9Puh2o4EC0elQ&sig2=W_7n3oUL3TtegJYFUYbEPA&bvm=bv.92765956,d.aWw

http://www-dl.emulex.com/support/elx/rt10.4.0/ga/Docs/final/apps/ocm_gui_manual_elx.pdf

http://filedownloads.qlogic.com/Files/TempDownlods/90481/UsersGuide_2600_FCAdapter_FC0054609-00G.pdf

http://filedownloads.qlogic.com/Files/TempDownlods/90546/AdminGuide_BRSeriesAdapters_BR0054501-00C.pdf

http://www.brocade.com

Configure DUT and Test Equipment

● Task 1: Brocade FC Fabric Configuration.........................................................................7● Task 2: Pure Storage Array Configuration........................................................................ 9● Task 3: Host Configuration .............................................................................................11

Task 1: Brocade FC Fabric Configuration1. Zoning is configured using the Peer Zoning feature in Brocade Fabric OS 7.4.1. Peer zoning allows a

"principal" device to communicate with the rest of the devices in the zone. The principal devicemanages a peer zone. Other "non-principal" devices in the zone can communicate with the principaldevice only; they cannot communicate with each other.

root> zonecreate –peerzone pure_peer –principal “52:4a:93:7d:f3:5f:61:00; 52:4a:93:7d:f3:5f:61:01;52:4a:93:7d:f3:5f:61:02; 52:4a:93:7d:f3:5f:61:03;52:4a:93:7d:f3:5f:61:10; 52:4a:93:7d:f3:5f:61:11” –members “10:00:8c:7c:ff:24:4c:00; 10:00:8c:7c:ff:24:4c:01 …”root> zoneshow --peerzone allzone: pure_peer Property Member: 00:02:00:00:00:03:00:08 Created by: User Principal Member(s): 52:4a:93:7d:f3:5f:61:00; 52:4a:93:7d:f3:5f:61:01; 52:4a:93:7d:f3:5f:61:02; 52:4a:93:7d:f3:5f:61:03; 52:4a:93:7d:f3:5f:61:10; 52:4a:93:7d:f3:5f:61:11; 52:4a:93:7d:f3:5f:61:12; 52:4a:93:7d:f3:5f:61:13 Peer Member(s): 10:00:8c:7c:ff:24:4c:00; 10:00:8c:7c:ff:24:4c:01 10:00:00:90:fa:61:92:3b; 10:00:00:90:fa:61:92:3c; 10:00:8c:7c:ff:05:60:01; 10:00:8c:7c:ff:05:60:00; 10:00:8c:7c:ff:14:e0:01; 10:00:8c:7c:ff:14:e0:00; 10:00:8c:7c:ff:03:bc:01; 10:00:8c:7c:ff:03:bc:00; 10:00:8c:7c:ff:03:9b:00; 10:00:8c:7c:ff:03:9b:01; 10:00:8c:7c:ff:05:72:02; 10:00:8c:7c:ff:05:72:03

2. Configure MAPS on switches (requires a Fabric Vision license). MAPS enables reporting of latencyand congestion alerts on each switch, as well as a number of switch and fabric health metrics. See Monitoring and Alerting Policy Suite Administrator's Guide, 7.4.1, for more information on configuringand using MAPS.Enable the desired MAPS policy using any of the available default policies or create a custom policy.For this test, the default Aggressive Policy is used. This policy provides the most sensitive thresholdlevels for detection of latency and congestion.

> mapsconfig --enablemaps -policy dflt_aggressive_policy

Define the reporting actions that MAPS will take. In this test, errors discovered by MAPS willgenerate a RASLog entry and send e-mail to the configured recipients.

> mapsconfig --actions raslog,email

The Fabric Performance Impact (FPI) category within MAPS monitors the current condition of thelatency seen on F_Ports over different time windows and uses that information to determine theperformance impact to the fabric and network.


http://www.brocade.com/content/html/en/administration-guide/fos-741-maps/index.html

In order to use FPI monitoring, legacy "bottelneckmon" must be disabled:

> bottleneckmon –disable

Enable FPI monitoring:

> mapsConfig –enableFPImon

To examine the MAPS configuration, use the following command:

> mapsconfig –showConfigured Notifications: RASLOG,EMAILMail Recipient: [email protected],[email protected] Monitoring: EnabledPaused members :===============PORT :CIRCUIT :SFP :

Use this command to view a summary of events or rules triggered and the objects on which therules were triggered over a specified period of time.

> mapsdb –show1 Dashboard Information:=======================DB start time: Wed Dec 17 20:59:09 2014Active policy: dflt_aggressive_policyConfigured Notifications: RASLOG,EMAILFenced Ports : NoneDecommissioned Ports : None

3. Configure FlexPorts on VDX switches for Fibre Channel support (an FCoE Base license isrequired).These ports are used as uplinks from the VCS fabric to the Fibre Channel fabric in this test. See the Network OS Layer 2 Switching Configuration Guide, 6.0.1, for detailed information on configuringVDX 6740 switches for Fibre Channel support.Example of configuring a Fibre Channel port on a Brocade VDX 6740 switch:

# conf t(config)# hardware(config-hardware)# flexport 112/0/38(config-flexport-112/0/38)# type fibre-channel

FlexPort speeds are configured in connector groups. In this test bed example, the speed HighMixedis chosen. This speed allows the Fibre Channel FlexPort to operate at 16-Gb speed, which isoptimal because the uplink is attaching to a 16-Gb Fibre Chanel SAN. Also, not all of the ports inthis connector group are configured as Fibre Channel ports; some are still configured as Ethernetports, and the HighMixed setting supports the Ethernet speeds on these ports as well. See the Network OS Layer 2 Switching Configuration Guide, 6.0.1, for detailed information on configuringVDX 6740 connector-group speeds.

• LowMixed—2/4/8G Fibre Channel and Ethernet speeds (default)• HighMixed—16G Fibre Channel and Ethernet speeds• FibreChannel—2/4/8/16G Fibre Channel

Use the following command to set the connector-group speed to HighMixed:

(config-hardware)# connector-group 111/0/3(config-connector-group-111/0/3)# speed HighMixed

4. Configure zones for FCoE initiators on VDX switches.





Example zone on a Brocade VDX 6740 switch:

# show zoning enabled-configurationzoning enabled-configuration cfg-name NOS_SSRzoning enabled-configuration enabled-zone lsan_hb067166_pure member-entry 10:00:8c:7c:ff:1f:7b:00 member-entry 10:00:8c:7c:ff:1f:7b:01 member-entry 52:4a:93:7d:f3:5f:61:00 member-entry 52:4a:93:7d:f3:5f:61:01 member-entry 52:4a:93:7d:f3:5f:61:10 member-entry 52:4a:93:7d:f3:5f:61:11

5. Configure Fibre Channel Routing (an Integrated Routing license required). Detailed information onFCR setup can be found in the Fabric OS Administrator's Guide, 7.4.1.Example FCR configuration is shown below:

> fcrconfigure –bbfid 100> fosconfig --enable fcr> portcfgexport [port#] -a1 –m[0/5] -f 10 {0=Brocade FC fabric; 5=Brocade NOS fabric}

The prefix Isan is used when configuring zones for use in Fibre Channel Routing. Example of a zoneprefixed with lsan:

> zoneshow lsan_hb067166_pure zone: lsan_hb067166_pure 10:00:8c:7c:ff:23:b7:00; 10:00:8c:7c:ff:23:b7:01; 52:4a:93:7d:f3:5f:61:00; 52:4a:93:7d:f3:5f:61:01 52:4a:93:7d:f3:5f:61:10; 52:4a:93:7d:f3:5f:61:11

Example output of exported devices:

> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric---------------------------------------------------------------------------- 10 21:00:00:24:ff:48:b9:6a 02f001 20 551a00 Imported 10 21:00:00:24:ff:48:b9:6b 02f101 20 541e00 Imported 10 52:4a:93:7d:f3:5f:61:00 02f201 20 550e00 Imported 10 52:4a:93:7d:f3:5f:61:01 02f401 20 540400 Imported

Task 2: Pure Storage Array Configuration1. Create a host initiator group for each host:

Task 2: Pure Storage Array Configuration


http://www.brocade.com/content/html/en/administration-guide/fos-741-adminguide/index.html

FIGURE 1 New Host Creation on the Pure Storage Array

2. Add WWNs to the host:

FIGURE 2 Adding Host Ports to the Host Configuration on the Pure Storage Array

3. Create new volumes. For this test configuration, 8 x 10GB volumes are created:



FIGURE 3 Creating New Volumes on the Pure Storage Array

FIGURE 4 Finished LUN Configuration and Presentation on the Pure Storage Array

Task 3: Host Configuration

1. Provision a minimum of two uplinks from the host to the FC fabric for redundancy and use nativemultipath tools to manage the available paths and load-balance across them.

2. Multipath configuration on Linux hosts:

This configuration allows all paths to be used in a round-robin fashion. This provides superiorperformance to the default Linux settings which would only use a single active path per LUN.

Task 3: Host Configuration


Recommended /etc/multipath.conf entry on Linux systems:

device { vendor "PURE" path_selector "round-robin 0" path_grouping_policy multibus rr_min_io 1 path_checker tur fast_io_fail_tmo 10 dev_loss_tmo 60 }

Example multipath configuration on Linux:

# multipath -llmpathp (3624a9370a15a66e949f7d1440001009a) dm-19 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:13 sdm 8:192 active ready running |- 1:0:2:13 sdu 65:64 active ready running |- 1:0:3:13 sdac 65:192 active ready running |- 1:0:4:13 sdak 66:64 active ready running |- 10:0:2:13 sddm 71:64 active ready running |- 10:0:4:13 sdec 128:64 active ready running |- 10:0:3:13 sddu 71:192 active ready running `- 10:0:5:13 sdek 128:192 active ready runningmpatho (3624a9370a15a66e949f7d1440001009c) dm-20 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:15 sdo 8:224 active ready running |- 1:0:2:15 sdw 65:96 active ready running |- 1:0:3:15 sdae 65:224 active ready running |- 1:0:4:15 sdam 66:96 active ready running |- 10:0:2:15 sddo 71:96 active ready running |- 10:0:3:15 sddw 71:224 active ready running |- 10:0:4:15 sdee 128:96 active ready running `- 10:0:5:15 sdem 128:224 active ready runningmpathn (3624a9370a15a66e949f7d14400010099) dm-16 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:12 sdl 8:176 active ready running |- 1:0:2:12 sdt 65:48 active ready running |- 1:0:3:12 sdab 65:176 active ready running |- 1:0:4:12 sdaj 66:48 active ready running |- 10:0:2:12 sddl 71:48 active ready running |- 10:0:3:12 sddt 71:176 active ready running |- 10:0:4:12 sdeb 128:48 active ready running `- 10:0:5:12 sdej 128:176 active ready runningmpathm (3624a9370a15a66e949f7d1440001009d) dm-6 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:16 sdp 8:240 active ready running |- 1:0:2:16 sdx 65:112 active ready running |- 1:0:3:16 sdaf 65:240 active ready running |- 1:0:4:16 sdan 66:112 active ready running |- 10:0:2:16 sddp 71:112 active ready running |- 10:0:3:16 sddx 71:240 active ready running |- 10:0:4:16 sdef 128:112 active ready running `- 10:0:5:16 sden 128:240 active ready runningmpathl (3624a9370a15a66e949f7d1440001009b) dm-18 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:14 sdn 8:208 active ready running |- 1:0:2:14 sdv 65:80 active ready running |- 1:0:3:14 sdad 65:208 active ready running |- 1:0:4:14 sdal 66:80 active ready running |- 10:0:3:14 sddv 71:208 active ready running |- 10:0:2:14 sddn 71:80 active ready running |- 10:0:4:14 sded 128:80 active ready running `- 10:0:5:14 sdel 128:208 active ready runningmpathk (3624a9370a15a66e949f7d1440001009e) dm-7 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:17 sdq 65:0 active ready running |- 1:0:2:17 sdy 65:128 active ready running



|- 1:0:4:17 sdao 66:128 active ready running |- 1:0:3:17 sdag 66:0 active ready running |- 10:0:3:17 sddy 128:0 active ready running |- 10:0:2:17 sddq 71:128 active ready running |- 10:0:4:17 sdeg 128:128 active ready running `- 10:0:5:17 sdeo 129:0 active ready runningmpathj (3624a9370a15a66e949f7d14400010097) dm-3 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:10 sdj 8:144 active ready running |- 1:0:2:10 sdr 65:16 active ready running |- 1:0:3:10 sdz 65:144 active ready running |- 1:0:4:10 sdah 66:16 active ready running |- 10:0:2:10 sddj 71:16 active ready running |- 10:0:3:10 sddr 71:144 active ready running |- 10:0:4:10 sddz 128:16 active ready running `- 10:0:5:10 sdeh 128:144 active ready runningmpathi (3624a9370a15a66e949f7d14400010098) dm-8 PURE ,FlashArray size=10.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 1:0:1:11 sdk 8:160 active ready running |- 1:0:2:11 sds 65:32 active ready running |- 1:0:3:11 sdaa 65:160 active ready running |- 1:0:4:11 sdai 66:32 active ready running |- 10:0:2:11 sddk 71:32 active ready running |- 10:0:3:11 sdds 71:160 active ready running |- 10:0:5:11 sdei 128:160 active ready running `- 10:0:4:11 sdea 128:32 active ready running

3. Multipath configuration on Windows hosts:

Use the Windows MPIO dialog to discover and manage multipath entries for Pure Storage.



FIGURE 5 Windows MPIO Multipathing with the Pure Storage Array

4. Multipath configuration on VMware hosts:

Change the path selection to Round Robin. Round Robin provides superior performance to theMost Recently Used setting, which uses only a single active path per LUN.



FIGURE 6 Round Robin Multipath Configuration on VMware

5. Apply any additional host tuning (for Linux systems). The following settings help to support high IOPSoperation of the Pure Storage flash array with Linux hosts.

Create a file named /etc/udev/rules.d/99-pure-storage.rules, and add the following lines:

# Recommended settings for Pure Storage flash array.# Use noop scheduler for high-performance solid-state storageACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/scheduler}="noop"# Reduce CPU overhead due to entropy collectionACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/add_random}="0"# Spread the CPU load by redirecting completions to the originating CPUACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/rq_affinity}="2"# Set the HBA timeout to 60 secondsACTION=="add", SUBSYSTEMS=="scsi", ATTRS{model}=="FlashArray", RUN+="/bin/sh -c 'echo 60 > /sys$DEVPATH/device/timeout'"

6. Set up workload generators.

On Windows and Linux systems, Medusa Labs Test Tools is installed. On VMware systems, theVMware I/O Analyzer is installed.



Test Plan

● Scope............................................................................................................................16● What's New in This Report ...........................................................................................16● Test Configuration.........................................................................................................17● DUT Descriptions..........................................................................................................17● DUT Specifications........................................................................................................18

ScopeTesting focuses on interoperability of the Pure Storage array and determining an optimal configurationfor performance and availability. Performance is observed within the context of best-practice fabricconfiguration; however, absolute maximum benchmark reporting of storage performance is beyond thescope of this test. Details of the test steps are covered under the "Test Cases" section. Standard test-bed setup includes IBM/HP/Dell chassis server hosts with Brocade/QLogic/Emulex HBAs with twouplinks from every host to a Brocade FC fabric. I/O generators included Medusa Labs Test Tools andVMware I/O Analyzer.

The storage array is connected to a Brocade 16-Gb FC fabric and multiple server hosts to drive I/O ina multipath configuration. Error injection is introduced, and failover and recovery behaviors areobserved. I/O performance is observed across different workload configurations.

What's New in This Report1. The array hardware under test is the Pure Storage FlashArray//m20.2. The array firmware version under test is Purity 4.5.5.3. The Brocade Fabric OS (FOS) version under test is 7.4.1.4. Testing includes new operating system versions RHEL 6.6, RHEL 7.1, SLES 12, and VMware ESXi

6.0.5. Emulex, QLogic, and Broadcom adapters have updated firmware and drivers.

For detailed information, see the "DUT Specifications" section.

The test report covering Brocade Fabric OS 7.4.0 and the Pure Storage FA-420 array running Purityversion 3.4.11 is available at http://www.brocade.com/content/html/en/validation-test-report/brocade-pure-storage-fa-420-fos-740-vt/index.html.

Test Plan


http://www.brocade.com/content/html/en/validation-test-report/brocade-pure-storage-fa-420-fos-740-vt/index.html

http://www.brocade.com/content/html/en/validation-test-report/brocade-pure-storage-fa-420-fos-740-vt/index.html

Test Configuration

DUT Descriptions

The following tables provide details about the devices under test (DUTs).

Storage Array TABLE 1

DUT ID Model Vendor Description

Pure StorageFlashArray//m20

FlashArray//m20 Pure Storage The Pure Storage FlashArray//m20 is an all-flash storage array that supports up to 120TB of useable capacity. The system undertest is configured with 2.79 TB of useablecapacity and 4x8Gb Fibre Channel targetconnections.

Switches TABLE 2


6510-1...9 BR-6510 Brocade 48-port 16-Gb FC switch

DCX-1 DCX 8510-8 Brocade 8-slot 16-Gb FC chassis

DCX-2 DCX 8510-4 Brocade 4-slot 16-Gb FC chassis

Test Configuration


Switches (Continued)TABLE 2


VDX-1,2 VDX 6740 Brocade 48-port 10-Gb switch (48x10Gb/4x40Gb)

DUT Specifications

Storage Version

Pure Storage FlashArray//m20 solid-state flash array Purity version 4.5.5

Brocade Switches Version

DCX 8510-8 Fabric OS 7.4.1

DCX 8510-4 Fabric OS 7.4.1

6510 + Integrated Routing, Fabric Vision licenses Fabric OS 7.4.1

VDX 6740 Network OS 6.0.1

Adapters Version

Brocade 1860 2-port 16-Gb FC HBA Driver and firmware version 3.2.6.0

QLogic QLE2672 2-port 16-GB FC HBA Driver 8.07.00.18.07.0-k, firmware 8.00.00

QLogic QLE2562 2-port 8-GB FC HBA Driver 9.1.13.20, firmware 7.04.00

Emulex LPe 16202-X 2-port 16-Gb FC HBA Driver 10.4.255.16, firmware 10.4.255.23

Brocade 1020 2-port CNA adapter Driver and firmware version 3.2.6.0

DUT ID Servers RAM Processor OS

SRV-1 HP ProLiant DL380P G8 160 GB E5-2640 VMware 6.0[cluster]

SRV-2 HP ProLiant DL380P G8 160 GB E5-2640 VMware 6.0[cluster]

SRV-3 HP ProLiant DL380P G8 32 GB E5-2690v2 Windows Server2012 R2

SRV-4 Dell PowerEdge R720 16 GB E5-2620 RHEL 7.1 x86_64

SRV-5 Dell PowerEdge R720 16 GB E5-2620 SLES 11.3 x86_64

SRV-6 HP ProLiant DL380P G8 16 GB E5-2620 SLES 12.0 x86_64

SRV-7 HP ProLiant DL380P G8 16 GB E5-2620 RHEL 6.5 x86_64

SRV-8 IBM System x3750 M4 16 GB E5-4603 CentOS 6.6 x86_64

DUT Specifications


Device/Software Tools Version

Finisar 16-Gb Analyzer/Jammer XGIG5K2001153

Medusa Labs Test Tools 7.1.0.165067

VMware I/O Analyzer 1.6.2

Test Plan


Test Cases

● 1.0 Fabric Initialization—Base Functionality................................................................. 21● 1.1 Fabric—Advanced Functionality............................................................................. 24● 1.2 Stress and Error Recovery with Device Multipath...................................................32● 1.3 Storage Device—Fibre Channel Routing (FCR) Internetworking Tests..................39● 1.4 Optional/Additional Tests........................................................................................ 43

1.0 Fabric Initialization—Base Functionality

Confirm basic Fibre Channel functionality of the storage array.

1.0.1 Storage Device—Physical and Logical Login with Speed Negotiation

1.0.2 Zoning and LUN Mapping

1.0.3 Storage Device Fabric I/O Integrity

1.0.4 Storage Device Multipath Configuration—Path Integrity

1.1 Fabric—Advanced Functionality

Examine the storage behavior related to more advanced fabric features suchas QoS, bottleneck detection, and advanced frame recovery.

1.1.1 Storage Device Bottleneck Detection Using MAPS FPI—With Congested Host

1.1.2 Storage Device Bottleneck Detection Using MAPS FPI—With CongestedFabric

1.1.3 Storage Device—QoS Integrity with QoS Zone-Based Traffic Prioritization

1.1.4 Storage Device—QoS Integrity with CS_CTL-Based Traffic Prioritization

1.1.5 Storage Device—FC Protocol Jammer Test Suite

1.1.6 Clear Link Diagnostics (D_Port) Test

1.2 Stress and Error Recovery with Device Multipath

Confirm proper HA/failover behavior of storage in a multipath environment.

1.2.1 Storage Device Fabric I/O Integrity—Congested Fabric

1.2.2 Storage Device Nameserver Integrity—Device Recovery with Port Toggle

1.2.3 Storage Device Nameserver Integrity—Device Recovery with DeviceRelocation

1.2.4 Storage Device—Device Recovery with Device Port Toggle—Extended Run

1.2.5 Storage Device Recovery—ISL Port Toggle

1.2.6 Storage Device Recovery—ISL Port Toggle (Entire Switch)

1.2.7 Storage Device Recovery—Switch Offline

1.2.8 Storage Device Recovery—Switch Firmware Download

Test Cases


1.3 Storage Device—Fibre Channel Routing (FCR) Internetworking Tests

Confirm proper storage functioning within routed fabrics.

1.3.1 Storage Device Internetworking Validation with the FC host

1.3.2 Storage Device Internetworking Validation with FCoE Using VDX FlexPort

1.3.3 Storage Device Edge Recovery After FCR Disruptions

1.3.4 Storage Device Backbone Recovery After FCR Disruptions

1.4 Optional/Additional Tests

1.4.1 Storage Device Firmware Update

1.4.2 Performance I/O Testing with Varying Workload Dimensions

1.4.3 VMware I/O Analyzer Workload Test


1.0.1 Storage Device—Physical and Logical Login with SpeedNegotiation

Test ObjectiveVerify device login to the switch and nameserver with all supported speed settings.

Test ConfigurationTest Execution

Set switch ports to 2/4/8/Auto_Negotiate speed settings. In order to test the 2-Gb speed setting, an 8-Gb SFP is required. The 16-Gb SFP supports speeds of 4/8/16.

portcfgspeed <port> [2/4/8/0]

Result Validation

1. Validate link states on the array and verify speed negotiation and device login at different speeds.Use the portshow command on the switch to check the link state and speed, and use thenscamshow command to verify device login to the fabric.

2. Check switch port status and verify the "actual" and "configured" link speed. Check the name serverfor device login.

# nscamshow# portshow Xroot> portshow 14portIndex: 14portName: port14portHealth: HEALTHYportState: 1 OnlineProtocol: FC



portWwn of device(s) connected: 52:4a:93:75:8c:54:50:10Distance: normalportSpeed: N8Gbps

Test ResultsPASS. Storage logs in to the fabric and is link up at 2Gb/4Gb/8Gb.


Test ObjectiveVerify that host-to-LUN access exists with valid zoning.


1. Create an FC zone on the fabric with the initiator and target WWNs.2. Create host groups and LUNs on the array with access to the initiator WWN.

Result Validation

Verify that LUNs are discovered on the hosts using host-specific tools.

- Linux: Check the output of lsscsi - Windows: Check the output of Computer Management > Storage > Disk Management- VMware: Check the output of Configuration > Storage > Devices

Test ResultsPASS. Host has read/write access to the presented LUNs.

1.0.3 Storage Device Fabric I/O Integrity

Test ObjectiveValidate single path host-to-LUN I/O with read/write/verify testing.


1. Set up read/write I/O to the LUN using Medusa.2. Perform link disruptions with port toggles/cable pulls.

Result Validation

Check Medusa I/O logs and verify that I/O resumes after a short downtime. Medusa I/O may pause,but should recover without errors.

Test ResultsPASS. I/O resumes without errors.




Test ObjectiveVerify that multipath configures successfully, and verify the integrity of each path.


1. Set up the host with at least two initiator ports zoned with two target ports on the array.2. Set up multipath on the host.3. Start I/O.4. Isolate paths individually, and run I/O on the separate paths.

Result Validation

1. Check the host multipath properties to verify that the toggled path recovers.

- Windows: mpclaim –s –d - Linux: multipath –ll- VMware: Check the paths at Configuration > Storage > Devices > Manage Paths

2. Check the host and storage logs for failures.3. Check the switch error logs and switch port status after toggling.

# errdumpall# portstatsshow X# portshow Xroot> portshow 14portIndex: 14portName: port14portHealth: HEALTHYportState: 1 OnlineProtocol: FCportWwn of device(s) connected: 52:4a:93:75:8c:54:50:10Distance: normalportSpeed: N8Gbps

4. Check I/O logs, and verify that I/O continues without errors.

Test ResultsPASS. Each individual path is valid, and the paths combine successfully in a multipath configuration.




1.1.1 Storage Device Bottleneck Detection Using MAPS FPI—WithCongested Host

Test ObjectiveVerify that congestion on host ports is detected. Verify storage device and fabric behavior duringcongestion.


1. Configure MAPS and FPI on all switches. For more information, see the "Configure DUT and TestEquipment" section.

2. Start I/O from a single host initiator to multiple targets.3. Monitor switch logs for I/O_PERF_IMPACT/ I/O_FRAME_LOSS warnings.

Result Validation

Check switch error logs and the MAPS dashboard for bottleneck warnings.

2015/10/20-11:56:00:672622, [MAPS-1003], 117148/115018, FID 128, WARNING, B6510_066_088, Port 16, Condition=ALL_F_PORTS(DEV_LATENCY_IMPACT==IO_PERF_IMPACT), Current Value:[DEV_LATENCY_IMPACT,IO_PERF_IMPACT, 30.0% in 10 secs], RuleName=defALL_F_PORTS_IO_PERF_IMPACT, Dashboard Category=Fabric Performance Impact., actionHndlr.c, line: 755, comp:md, ltime:2014/12/17-11:56:00:671909# mapsdb --show allroot> mapsdb --show1 Dashboard Information:========================Active policy: dflt_aggressive_policy. . .2 Switch Health Report:=======================Current Switch Policy Status: HEALTHY3.1 Summary Report:===================Category |Today |Last 7 days |--------------------------------------------------------------------------------Port Health |No Errors |Out of operating range |Fru Health |In operating range |In operating range |Security Violations |No Errors |In operating range |Fabric State Changes |No Errors |In operating range |Switch Resource |In operating range |In operating range |Traffic Performance |In operating range |In operating range |FCIP Health |Not applicable |Not applicable |Fabric Performance Impact|Out of operating range |Out of operating range |

Test ResultsPASS. With MAPS reporting configured, performance warnings are reported as expected. Thebottlenecked ports are displayed on the MAPS dashboard, and a RASLog warning is created.



1.1.2 Storage Device Bottleneck Detection Using MAPS FPI—WithCongested Fabric

Test ObjectiveValidate bottleneck detection with congested fabric.


1. Configure MAPS on all switches. For more information, refer to the "Configure DUT and TestEquipment" section.

2. Isolate a single ISL in the fabric.3. Start I/O from multiple host initiators to multiple targets.4. Monitor switch logs for traffic performance warnings.

Result Validation

Check switch error logs, e-mail, and the MAPS dashboard for bottleneck warnings.

2015/10/20-14:56:02, [MAPS-1003], 11869, FID 128, WARNING, B6510_066_083, T-Port 1, Condition=ALL_E_PORTS(TX/hour>60.00), Current Value:[TX,61.07 %], RuleName=defALL_E_PORTSTX_60, Dashboard Category=Fabric Performance Impact.

Affected Entity: E-Port 46Rule Name: defALL_PORTS_IO_PERF_IMPACTCondition: ALL_PORTS(DEV_LATENCY_IMPACT==IO_PERF_IMPACT)Current Value: [DEV_LATENCY_IMPACT,IO_PERF_IMPACT, 13 ms Frame Delay]Dashboard Category: Fabric Performance Impact Switch Name: B6510_066_082Switch WWN: 10:00:00:05:33:13:96:5aSwitch IP: 10.38.66.82Fabric Name: SSRVFID: 128# mapsdb --show allroot> mapsdb --show1 Dashboard Information:========================Active policy: dflt_aggressive_policy. . .2 Switch Health Report:=======================Current Switch Policy Status: HEALTHY3.1 Summary Report:===================Category |Today |Last 7 days |--------------------------------------------------------------------------------Port Health |No Errors |Out of operating range |Fru Health |In operating range |In operating range |Security Violations |No Errors |In operating range |Fabric State Changes |No Errors |In operating range |Switch Resource |In operating range |In operating range |Traffic Performance |In operating range |In operating range |FCIP Health |Not applicable |Not applicable |Fabric Performance Impact|Out of operating range |Out of operating range |

1.1.2 Storage Device Bottleneck Detection Using MAPS FPI—With Congested Fabric


Test ResultsPASS. With MAPS reporting configured, performance warnings are reported as expected.

1.1.3 Storage Device—QoS Integrity with QoS Zone-Based TrafficPrioritization

Test ObjectiveValidate QoS functionality.


1. Set up initiator-target pairs with Low/Medium/High QoS zones in the fabric.2. Start I/O across all pairs, and validate traffic priority.

Result Validation

1. Check I/O logs and verify that I/O continues without errors.2. Check the switch logs and switch port status for errors.

# porterrshowroot> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

3. Verify traffic prioritization by examining performance and VC utilization.

Test ResultsPASS. I/O completes without errors in low, medium, and high priority zones.

1.1.4 Storage Device—QoS Integrity with CS_CTL-Based TrafficPrioritization

Test ObjectiveValidate CS_CTL QoS functionality.


1.1.3 Storage Device—QoS Integrity with QoS Zone-Based Traffic Prioritization


1. Enable ExpressLane on the Emulex adapter driver by setting the value to 1 and the priority to 3(high).

FIGURE 7 Configuring Driver Parameters in the Emulex OneCommand Interface

2. Enable CS_CTL QoS Auto-Mode at the chassis level on all switches in the Brocade fabric.

> configurechassisConfigure...cfgload attributes (yes, y, no, n): [no]Custom attributes (yes, y, no, n): [no] system attributes (yes, y, no, n): [no] fos attributes (yes, y, no, n): [no] y Reboot needed to effect new CSCTL Mode CSCTL QoS Mode (0 = default; 1 = auto mode): (0..1) [0] 1

To verify:root> configshow -all | grep csctlfos.csctlMode:1

3. Enable CS_CTL QoS on initiator and target ports.

root> portcfgqos --enable 7 csctl_modeEnabling CSCTL mode flows causes QoS zone flows to lose priority on such ports.

Test Cases


Do you want to proceed?(y/n):y

To verify:root> portcfgshow 7 | grep -i csctlCSCTL mode: ON

4. Enable ExpressLane QoS on selected LUNs. For multipath LUNs, enable ExpressLane on eachdevice path.

FIGURE 8 Enabling LUNs for ExpressLane in the Emulex OneCommand Interface

5. Start high throughput I/O to all LUNs.

Result Validation

1. Check I/O logs and verify that I/O continues without errors for all LUNs.2. Verify that I/O performance is improved on ExpressLane-enabled LUNs.

Test Cases


3. Verify CS_CTL prioritization in the fabric by monitoring the high VC buffer credits on the ISLs.4. Check the switch logs and switch port status for errors.

# porterrshowroot> porterrshow 14 frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Test ResultsPASS. Prioritized LUNs display higher performance, the traffic receives prioritization through the fabric,and I/O completes without errors.


Test ObjectivePerform FC Jammer tests including areas such as CRC corruption, packet corruption, missing frame,host error recovery, and target error recovery.


1. Insert the Jammer device in the I/O path on the storage link.2. Execute the following Jammer scenarios:

• Delete one frame.• Delete R_RDY.• Replace CRC of data frame.• Replace EOF of data frame.• Replace "good status" with "check condition".• Replace IDLE with LR.• Truncate frame.• Create S_ID/D_ID error of data frame.

3. Verify Jammer operations and recovery using Analyzer.

Result Validation

Check the Medusa log and verify that I/O recovers and completes without errors.

Test ResultsPASS. Packet anomalies are introduced, and proper recovery is verified.




Test ObjectiveValidate the Clear Link Diagnostics feature on supported HBAs. The feature is tested on the EmulexLPe16202, QLogic 2672, and QLogic/Brocade 1860.


1. The Brocade switch can be configured in static, dynamic, and on-demand D_Port modes. Theswitch supports dynamic D_Port by default at the chassis level.

root> configureConfigure... Fabric parameters (yes, y, no, n): [no] D-Port Parameters (yes, y, no, n): [no] y Dynamic D-Port (on, off): [on]On Demand D-Port (on, off): [off]

2. With the Emulex LPe16202 adapter, the switch port is in dynamic mode, and the test is manuallyinitiated on the HBA port from the Emulex OneCommand Manager.

root> switchshow0 0 030000 id N16 In_Sync FC D-Port 10:00:00:90:fa:61:92:3b Dynamic

3. With the QLogic 2672 adapter, the HBA port is in dynamic D_Port mode, and the test is initiated byconfiguring the switch port as a static D_Port. The test starts automatically after enabling the D_Portat the switch.

> portdisable <port#>



> portcfgdport ––enable <port#>Caution: D_Port functionality is only available on 16Gb-capable platforms with 16Gb FC SFPs, 10Gb FC SFPs, 8Gb LWL/ELWL FC SFPs, QSFPs or QSFP+.> portenable <port#>(Reverse the above steps to disable D_Port mode.)

To verify:root> portcfgshow <port#> | grep -i d-portD-Port mode: ONroot> switchshow7 7 581300 id N16 Online FC D-Port Loopback->Port 7

4. With the QLogic/Brocade 1860 adapter, the HBA port can be in static or dynamic D_Port mode.

Enabling static D_Port on the HBA:# bcu port --disable 2/0port disabled# bcu diag --dportenable 2/0D-port mode for port 2/0 enabled.

5. Execute the D_Port test, and verify that all tests pass and that no port and link problems arereported.

Result Validation

Check the D_Port test results on the switch ports and host HBA diagnostic utilities.

root> portdporttest --show 14D-Port Information:===================Port: 14Remote WWPN: 10:00:8c:7c:ff:23:b7:00Mode: AutomaticNo. of test frames: 1 MillionTest frame size: 1024 BytesFEC (enabled/option/active): Yes/No/NoCR (enabled/option/active): No/No/NoStart time: Wed Apr 15 12:02:53 2015End time: Wed Apr 15 12:03:52 2015Status: PASSED================================================================================Test Start time Result EST(HH:MM:SS) Comments================================================================================Electrical loopback 12:03:01 PASSED -------- --------Optical loopback 12:03:31 PASSED -------- --------Link traffic test 12:03:39 PASSED -------- --------================================================================================Roundtrip link latency: 157 nano-secondsEstimated cable distance: 3 metersBuffers required: 1 (for 2112 byte frames at 16Gbps speed)Egress power: Tx: -3.4 dBm, Rx: Not Avail.Ingress power: Rx: -3.3 dBm, Tx: Not Avail.

Test ResultsPASS. The D_Port feature is validated on all tested HBAs.

Test Cases



1.2.1 Storage Device Fabric I/O Integrity—Congested Fabric

Test ObjectiveValidate I/O integrity in a congested fabric environment.


From all initiators, start a mixture of Read, Read/Write, and Write traffic continuously to all their targetsfor a 24-hour period.

Result Validation

1. Check the host and storage logs for errors.2. Verify the link congestion, and check the switch logs for errors.

# errdumpall# portperfshow# porterrshowroot> porterrshow 15 frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

3. Check I/O generator tool logs to verify that I/O runs without errors.

Test ResultsPASS. In a congested fabric, I/O runs successfully without errors or failover.

1.2.2 Storage Device Nameserver Integrity—Device Recovery with PortToggle

Test ObjectiveValidate path recovery behavior on storage and host ports.




1. Set up multipath on the host, and start I/O.2. Perform multiple iterations of sequential port toggles across initiator and target switch ports.

Result Validation

1. Check the switch port status after toggling, and check for any errors in the switch logs.

# errdumpall# portstatsshow X# portshow Xroot> portshow 14portIndex: 14portName: port14portHealth: HEALTHYportState: 1 OnlineProtocol: FCportWwn of device(s) connected: 52:4a:93:75:8c:54:50:10Distance: normalportSpeed: N8Gbps

2. Check the multipath status on hosts to verify whether the toggled path recovers.


3. Check host and storage error logs, and verify that I/O continues without errors.

Test ResultsPASS. Failover between 8 logical paths (2 host x 4 storage) tests successfully. Paths recover, and I/Ocompletes without errors.

1.2.3 Storage Device Nameserver Integrity—Device Recovery withDevice Relocation

Test ObjectiveValidate storage device path recovery after physical port relocation.


1. Perform the test sequentially for each storage device port.2. Disconnect a port and reconnect it to different switch port in the same fabric.

Result Validation

1. Check for errors in the switch logs and the switch port status at the new switch port.

# errdumpall# portstatsshow X# portshow Xroot> portshow 8portHealth: HEALTHY <-

1.2.3 Storage Device Nameserver Integrity—Device Recovery with Device Relocation


. . .portState: 1 Online <-. . .portWwn of device(s) connected: 52:4a:93:7d:f3:5f:61:11 <-Distance: normalportSpeed: N8Gbps

2. Check the multipath status on hosts to verify that the toggled path recovers.


3. Confirm that there are no errors in the Medusa log.

Test ResultsPASS. A physical move of the storage port shows successful recovery. Paths recover, and I/Ocompletes without errors.


Test ObjectiveValidate path recovery behavior on storage and host ports for an extended duration run.


1. Set up multipath on the host, and start I/O.2. Sequentially toggle each initiator and target port in the fabric (multiple iterations).3. Run the test for an extended period.

Result Validation

1. Check the switch port status after toggling, and check for errors in the switch logs.

# errdumpall# portstatsshow X# portshow Xroot> portshow 19portHealth: HEALTHY <-. . .portState: 1 Online <-. . .portWwn of device(s) connected: 10:00:8c:7c:ff:22:f7:81 <-Distance: normalportSpeed: N16Gbps

2. Check host multipath properties for iSCSI hosts to verify that the toggled path recovers.

On Windows: mpclaim –s –d On Linux: multipath –llOn VMware: Check the paths at Configuration > Storage > Devices > Manage Paths




Test ResultsPASS. 48-hour run; paths recover and I/O completes without errors.


Test ObjectiveValidate path recovery and I/O integrity when ISL links are disabled.


1. Set up multipath on the host, and start I/O.2. Sequentially toggle each ISL path (one at a time) on all switches.

Result Validation

1. Check the FC fabric status after ISL toggling. Verify that all nodes are online.

# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073" 19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"The Fabric has 5 switches <-Fabric Name: SSR

2. Check the switch logs for errors, and verify that I/O fails over to an alternate ISL path in the fabric.

# errdumpall# portperfshow# porterrshowroot> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

3. Check host and storage logs, and verify that I/O continues without errors.

Test ResultsPASS. Paths recover, and I/O completes without errors.




Test ObjectiveValidate path recovery and I/O integrity when all ISL links on a switch are disabled.


1. Ensure ISL redundancy by provisioning multiple ISLs connected to different switches to providemultiple paths through the fabric.

2. Set up multipath on the host, and start I/O.3. On each switch, disable all ISL links at once.

Result Validation

1. Check the FC fabric status after ISL toggling. Verify that all nodes are online.

# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name-------------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073" 19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"The Fabric has 5 switches <-Fabric Name: SSR

2. Check the switch logs for errors, and verify that /O fails over to an alternate ISL path in the fabric.

# errdumpall# portperfshow# porterrshowroot> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0


Test ResultsPASS. Paths recover, and I/O completes without errors.




Test ObjectiveValidate path recovery and I/O integrity during switch offline events.


1. Toggle each switch in sequential order.2. Include switch enable/disable, power on/off, and reboot testing.

Result Validation

1. Check the FC fabric status after the switch toggles. Verify that all nodes are present in the fabric.

# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name-------------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073" 19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"The Fabric has 5 switches <-Fabric Name: SSR

2. Check the switch logs for errors, and verify that the toggled switch has recovered.

# errdumpall# switchshowroot> switchshowswitchName: B6510_066_088switchType: 109.1switchState: Online <-switchMode: NativeswitchRole: SubordinateswitchDomain: 88switchId: fffc58switchWwn: 10:00:00:27:f8:06:23:28zoning: ON (SSR)switchBeacon: OFFFC Router: ONFC Router BB Fabric ID: 100Address Mode: 0Fabric Name: SSR_2


Test ResultsPASS. Paths fail over and recover, and I/O completes without errors.




Test ObjectiveVerify that I/O continues with minimal disruption throughout the switch firmware upgrade process.


1. Set up host multipath with links on different switches in the FC fabric, and start I/O.2. Sequentially perform firmware upgrades on all switches in the fabric.

Result Validation

1. Verify that the firmware upgrade completes successfully on each switch node and that the switchnodes merge back in the FC fabric.

# versionroot> versionKernel: 2.6.14.2Fabric OS: v7.4.1Made on: Wed Sep 23 14:56:29 2015Flash: Thu Oct 15 14:54:24 2015BootProm: 1.0.11# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name-------------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073" 19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"The Fabric has 5 switches <-Fabric Name: SSR

2. Check I/O generator tool logs to verify that I/O runs without errors throughout the firmware upgrade.3. Check the switch logs for errors, and verify that I/O resumes on the node after the firmware upgrade

is complete.

# errdumpall# portperfshow

Test ResultsPASS. I/O failover and recovery are successful during the firmware download. Paths recover, and I/Ocompletes without errors.



1.3 Storage Device—Fibre Channel Routing (FCR) InternetworkingTests

1.3.1 Storage Device Internetworking Validation with the FC Host

Test ObjectiveValidate that storage targets are imported successfully in a routed Fibre Channel environment.


1. Set up FCR in an edge-backbone-edge configuration.2. Set up LSAN zoning, verify host access to target LUNs, and start I/O.

Result Validation

1. Verify the name server and FCR fabric state.

# fcrfabricshowroot> fcrfabricshowFC Router WWN: 10:00:00:05:33:13:96:5a, Dom ID: 82,Info: 10.38.66.82, "B6510_066_082" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------------- 40 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 41 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92"FC Router WWN: 10:00:00:05:33:5b:1d:1d, Dom ID: 83,Info: 10.38.66.83, "B6510_066_083" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------------- 40 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 41 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088"# fcrproxydevshowroot> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric------------------------------------------------------------------------ 40 10:00:00:05:1e:60:b4:6b 02ff02 100 132400 Imported 40 10:00:00:05:1e:60:b4:6c 02fe02 100 032400 Imported. . . 100 50:05:07:60:5e:80:76:52 01fe02 40 612800 Imported 100 50:05:07:60:5e:80:76:71 01ff02 40 642300 ImportedTotal devices displayed: 20

2. Verify that I/O runs successfully without errors.

Test ResultsPASS. All devices are available via the routed fabric. Running I/O confirms successful routing.

1.3 Storage Device—Fibre Channel Routing (FCR) Internetworking Tests


1.3.2 Storage Device Internetworking Validation with FCoE Test UsingVDX FlexPort

Test ObjectiveVerify storage device availability from FCoE initiators routed through the VCS fabric.


1. Set up FCoE host initiator ports on the VCS fabric using VDX 6740 switches.2. Enable Fibre Channel support on VDX switch ports (for more detail, see the "Configure DUT and

Test Equipment" section).3. Connect the VCS fabric to the FC fabric via FCR ports.4. Configure LSAN zones in the VCS and FC fabrics.

Result Validation

1. Verify the name server and FCR fabric state.

# fcrfabricshowroot> fcrfabricshowFC Router WWN: 10:00:00:05:33:5b:7d:06, Dom ID: 97,Info: 10.38.66.97, "B6510_066_097" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ---------------------------------------------------------------------------- 18 110 10.38.66.120 10:00:50:eb:1a:62:8c:33 "VDX6740_066_120"FC Router WWN: 10:00:00:05:33:5b:10:e8, Dom ID: 100,Info: 10.38.66.100, "B6510_066_100" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ---------------------------------------------------------------------------- 8 110 10.38.66.119 10:00:50:eb:1a:62:83:7b "VDX6740_066_119"# fcrproxydevshowroot> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric------------------------------------------------------------------------ 110 50:05:07:60:5e:80:76:52 02f001 120 612800 Imported 110 50:05:07:60:5e:80:76:71 02f101 120 642300 Imported 120 10:00:8c:7c:ff:00:48:00 03f101 110 781000 Imported 120 10:00:8c:7c:ff:00:48:01 03f001 110 771000 ImportedTotal devices displayed: 4

2. Verify that I/O runs successfully without errors.

Test ResultsPASS. Storage targets are available through the FCoE/FC routed configuration; I/O completes withouterrors.

1.3.2 Storage Device Internetworking Validation with FCoE Test Using VDX FlexPort



Test ObjectiveValidate that storage target paths recover successfully from disruptions in a routed Fibre Channelenvironment. Edge-backbone-edge configuration.


1. Set up FCR in an edge-backbone-edge configuration.2. Set up LSAN zoning.3. With I/O running, perform sequential reboots, switch disables, and ISL port toggles on the switches in

the backbone fabric.

Result Validation

1. Verify the FCR fabric state throughout the disruptions.


2. Check the switch logs for errors.


3. Check the host and storage logs, and verify that I/O runs without errors.

Test ResultsPASS. Paths fail over and recover with FCR disruptions, and I/O completes without errors.




Test ObjectiveValidate that storage target paths recover successfully from disruptions in a routed Fibre Channelenvironment. Edge-backbone configuration.


1. Set up FCR in an edge-backbone configuration.2. Set up LSAN zoning.3. With I/O running, perform sequential reboots, switch disables, and ISL port toggles on the switches

in the backbone fabric.

Result Validation

1. Verify the FCR fabric state throughout the disruptions.


2. Check the switch logs for errors.


3. Check the host and storage logs, and verify that I/O runs without errors.

Test ResultsPASS. Paths fail over successfully, and I/O completes without errors.




1.4.1 Storage Device Firmware Update

Test ObjectiveExecute a non-disruptive firmware update on the array while running I/O, and confirm that there are noI/O errors.


1. Run continuous I/O to the array.2. Execute the code update procedure as described in vendor documentation.

Result Validation

1. Confirm the updated version on all array components.2. Check the I/O generator tool logs to verify that I/O completes without errors.3. Check the host and storage logs for errors throughout the I/O operations.4. Check the switch error logs and port stats for errors or I/O drops.

# errdumpall

Test ResultsPASS. The update is successful (from 3.4.10 to 3.4.11) with proper failover and no I/O errors.

1.4.2 Performance I/O Testing with Varying Workload Dimensions

Test ObjectiveRun I/O workload generation with varying dimensions (block size, read/write/mix), and verify thatperformance characteristics are as expected.


1. Run an I/O loop at block transfer sizes of 512, 1k, 2k, 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, and1m.

2. Include a nested loop of 100-percent read, 100-percent write, and 50-percent read/write.

Repeat the test for the following configurations:

• One host port to one target port (single path)• Two host ports to multiple target ports (multipath)• Multiple hosts to multiple target ports (multihost multipath)

Result Validation



1. Check the I/O generator tool logs to verify that I/O completes without errors.2. Check the host and storage logs for errors throughout the I/O operations.3. Check the switch logs and port stats for errors or I/O drops.

Test ResultsPASS. All workload runs are monitored at the host, storage, and fabric, and they complete without I/Oerrors or faults.


Test ObjectiveRun the workload test suite including varying application workloads generated from multiple VMs.


1. Configure a two-host VMware cluster with multipath on two initiator ports per host, and four targetports.

2. Configure workload generation from eight worker VMs using VMware I/O Analyzer.3. Run a variety of application I/O workload patterns from the I/O Analyzer suite. For this round of

testing, the following workloads are used:

• Workstation• Webserver• Video on Demand• Max Write Throughput• Max Write IOPS• Max IOPS• Max Throughput• SQL Server 16k• 4k Read• Exchange 2007• OLTP 4k

Result Validation

1. Check the I/O generator tool logs to verify that I/O completes without errors.2. Check the host and storage logs for errors throughout the I/O operations.3. Check the switch logs and port stats for errors or I/O drops.

Test ResultsPASS. All workload runs are monitored at the host, storage, and fabric, and they complete without I/Oerrors.



Test Conclusions

1. Achieved a 100-percent pass rate on all test cases in the SSR qualification test plan. The networkand the storage were able handle the various stress and error recovery scenarios without issues.

2. Different I/O workload scenarios were simulated using Medusa and VMware I/O Analyzer tools, andsustained performance levels were demonstrated across all workload types. One caveat should benoted; when the array is hit very hard with a write-only workload using the smallest block sizes (512B or 1 K) and a sustained high thread count, the I/O may occasionally pause for a few secondsbefore recovering. This situation represents a corner case with synthetic workload generation, and itis not expected to occur in real-world workload cases.

3. The results confirm that the Pure Storage FlashArray//m20 array interoperates seamlessly withBrocade Fibre Channel fabrics, and together they demonstrate high availability, performance, andlow latency.

4. The Brocade Gen 5 16-Gb FC switches were able to handle the sustained throughput and latencyperformance requirements efficiently with fewer ISL trunks. Multiple ISLs to different switches in thefabric should be set up to provide path redundancy through the fabric.

5. The Monitoring and Alerting Policy Suite (MAPS) heath monitor should be enabled on all switches inthe FC fabric to report fabric-wide events and traffic performance metrics. The additional MAPSfeature of Fabric Performance Impact monitoring should also be enabled to detect bottlenecks in theform of timeouts and latency. Using MAPS is recommended to maximize the benefit of high-performance low-latency storage.

6. Peer zoning helps reduce the zone database size and the zoning complexity, while providing theRSCN and hardware resource efficiencies of single-initiator zoning.

7. QoS zoning should be used to classify host-target traffic into high, medium, or low priority zones toprovide traffic prioritization through the FC fabric for the desired host-target pair by allocating moreresources to the traffic in the higher priority zone.

8. Enabling Emulex ExpressLane on a LUN provides prioritized queuing on the HBA for traffic to thatLUN and also sets the CS_CTL tag on the frame, which allows the traffic to be prioritized through theFC fabric based on the value of the CS_CTL tag and the corresponding priority level.

9. For optimal availability and performance, consideration should be given to multipath configuration onthe host side. While Windows 2008 and 2012 provide Round-Robin behavior by default, Linuxsystems benefit from adding a custom entry to /etc/multipath.conf, and VMware host systems shouldbe changed from the default Most Recently Used (VMware) setting to Round-Robin (VMware).Actively using all available paths provides a significant improvement in performance throughput.

10.The ClearLink D_Port diagnostic test is a valuable tool for validating link quality before adding newdevices or ISLs to the fabric and for troubleshooting existing links.
