installation tuning guide - supermicro

NetXen, Inc.

Intelligent NIC™ Installation and Tuning Guide

Version 3.4

October 30, 2007

This document is applicable to releases 3.4.216 and up

NetXen, Inc.3965 Freedom CircleFourth FloorSanta Clara, CA 95054Telephone: (408) 330-0010Web Site: http://www.netxen.com

Copyright© Copyright 2007. NetXen, Inc. — All Rights Reserved.

DisclaimerThis is a preliminary document and is subject to change without prior notification and does not represent a commitment on the part of NetXen, Inc. The information contained herein is the proprietary and confidential information of NetXen or its licensors, and is supplied subject to, and may be used only by NetXen’s customer in accordance with, a written agreement between NetXen and its customer. Except as may be explicitly set forth in such agreement, NetXen does not make, and expressly disclaims, any representations or warranties as to the completeness, accuracy or usefulness of the information contained in this document.

Please contact NetXen, Inc. to obtain the latest revision of this document.

Trademarks NetXen, the NetXen logo, and Intelligent NIC are trademarks of NetXen, Inc. Third Party brands and names are the property of their respective owners.

NetXen, Inc

Revision History

Version Date Editor Notes

3.0 4/13/074/20/076/21/077/12/07

SR First release of document.Update screen shots and Advanced Properties table.Miscellaneous small edits from internal review.Updated selected screen shots to reflect new software version.

3.1 9/10/07 SR Updated Table 3, System and Software Requirements.Updated Table 4, New NIC Driver Installation.Updated Table 5, Updating from an Existing Driver and Firmware.Updated selected screen shots in Section 4.2.1, Windows NIC Driver Installation.Updated Table 6, Description of Advanced Properties Parameters.Updated Table 7, NetXen Flash Utility Commands.Updated Section 6, Linux Socket Accelerator (LSA).Added Section 7, Teaming Driver.Added Section 8.5, Additional Support.Updated Section A.1.1, Unidirectional: One Server to One Client.Added Appendix B, PXE Boot with the NetXen Card.Added Appendix C, Using Accelerated iSCSI Via the Linux Open iSCSI Initiator.Updated Appendix D, Glossary.Added Section 4.3.2, NIC Driver Options.Reworked Section 9, User Diagnostics.Added Table 12, NetXen Windows Diagnostic Utility Commands.Added Table 13, NetXen Linux Diagnostic Utility Commands.

3.2 9/18/07 SR Added Table 6, Enabling Windows Chimney and Linux LSA.Added Section 4.4.3, Uninstalling the LSA Driver in Linux.Updated Table 9, NetXen Flash Utility Commands.

3.3 10/03/07 SR Integrated miscellaneous edits from internal review cycle.Added TCPv4 Stateful Offload entry to Table 8, Description of Advanced Properties Parameters.Added Appendix Section B.2.2 Windows Remote Installation Services (RIS) Installation.Updated Table 3, NXB-10GXSR XFP Module Specifications.Reorganized material in Section B.2, ‘PXE Configuration — Setting Up Windows Server 2003’ into three subsections.

3.4 10/25/07 SR Updated Section 4.3, Linux Installation.Removed Section on Teaming Driver for Linux and Windows.Added footnote #1 to Table 5, Quick Start Instructions — New NIC Driver Installation.Remove appendix on PXE boot.

THIS PAGE INTENTIONALLY LEFT BLANK

NetXen, Inc

Table of Contents

1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Related Documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 NIC Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Currently Supported Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 System Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.1 Software Release Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2 Hardware and Software Installation Requirements . . . . . . . . . . . . . . . . . . . 2

2.2.1 Cable Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2.2 XFP Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2.3 System and Software Requirements . . . . . . . . . . . . . . . . . . . . . . 3

3 Quick-Start Installation Instructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Detailed Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4.1 Installing and Connecting the Intelligent NIC Adapter. . . . . . . . . . . . . . . . . 74.2 Windows Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.2.1 NIC Driver Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.2.2 Configuring the Network Interface Card . . . . . . . . . . . . . . . . . . . 16

4.3 Linux Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.3.1 Linux NIC Driver Installation (TAR format) . . . . . . . . . . . . . . . . . 184.3.2 Linux NIC Driver Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.3.3 Linux NIC Driver Installation (RPM format). . . . . . . . . . . . . . . . . 204.3.4 Configuring the NIC on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.4 Uninstalling the Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.4.1 Uninstalling Windows Drivers for Both NIC and Chimney . . . . . 224.4.2 Uninstalling the NetXen NIC Driver in Linux. . . . . . . . . . . . . . . . 224.4.3 Uninstalling the LSA Driver in Linux . . . . . . . . . . . . . . . . . . . . . . 22

5 Firmware Updates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.1 Using the NetXen Flash Utility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.2 Windows Firmware Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255.3 Linux Firmware Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.4 Updating the Flash on Multiple Standalone Cards in Windows. . . . . . . . . 26

6 Linux Socket Accelerator (LSA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.1 LSA Driver Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

6.1.1 LSA Driver Installation Prerequisites . . . . . . . . . . . . . . . . . . . . . 276.1.2 TAR Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

6.2 LSA Offload Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.2.1 LSA Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.2.2 LSA Configuration Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.2.3 nxoffload Caveats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

6.3 Running LSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.4 LSA Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

6.4.1 Example Read Out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Troubleshooting Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

7.1 Unable to Boot the Host System with the NIC Card Installed . . . . . . . . . . 337.2 Unable to ping the NIC Installed on the Host . . . . . . . . . . . . . . . . . . . . . . 337.3 Gather System Information and Check for Errors on Linux Host . . . . . . . 347.4 Caveats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

7.4.1 MSI Interrupts for Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

NetXen, Inc. i

Table of Contents

7.4.2 PCI-Express Slot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357.5 Additional Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

8 User Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378.1 Windows Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378.2 Linux Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Appendix A Tuning Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40A.1 Windows Performance Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40A.1.1 NTttcp (Windows) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40A.1.2 nttcps and ntttcpr Command Line Options . . . . . . . . . . . . . . . . . . . 41A.2 Windows Optimization — Tcp1323Opts . . . . . . . . . . . . . . . . . . . . . . 42A.3 Linux Performance Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42A.3.1 IPERF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43A.4 Linux Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Appendix B Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

ii NetXen, Inc.

1. Overview

1 OverviewThis guide describes the system requirements, installation, and setup required for the NetXen Intelligent NIC™ network interface card product release.

1.1 Related DocumentsRelease notes included in the software release.

Ethernet Driver Developer Reference Guide: For Open Source Software (on the web).

Check NetXen web site for updates.

1.2 NIC VersionsThe NIC products support Windows, Linux, and other operating systems. See the release notes for an updated list.

1.3 Currently Supported Features

Note: Check with NetXen periodically for software updates and additional features.

Product Number Product Definition

NXB-10GSXR Single 10-GE port (optical fiber)

NXB-10CX4 Single 10-GE port (copper)

Feature Windows Linux

TCP, UDP, and IP checksum offload Yes Yes

TCP segmentation offload (TSO / LSO) Yes Yes

Interrupt coalescing Yes Yes

Interrupt distribution No Yes

VLAN Yes Yes

SNMP Ethernet MIB Yes Yes

Jumbo frames Yes Yes

Teaming and Failover Yes Yes

Linux Socket Accelerator (LSA) N/A Yes

NetXen, Inc. Intelligent NIC Installation and Tuning Guide 1

2. System Requirements

2 System RequirementsBefore installing the product, verify the minimum system configuration requirements outlined below.

2.1 Software Release Package Contents

2.2 Hardware and Software Installation Requirements

2.2.1 Cable Specifications

Contents Format Location

Documentation Release Notes NetXen web site

NIC Linux Driver [version].tar.gz or [version].rpm [version]/driver/nx_nic.ko or modprobe for nx_nic

NIC Windows Driver [version].zip [version]\win2k3\x64 or [version]\win2k3\x86

NX Intercept Driver [version].-lsa.tar.gz or [version]-lsa.rpm

[version]-lsa/driver/nx_intercept.ko or modprobe for nx_intercept

LSA Driver [version].-lsa.tar.gz or [version]-lsa.rpm

[version]-lsa/driver/nx_lsa.ko or modprobe for nx_lsa

Firmware cx4_romimage or xfp_romimage [version]\win2k3\utils or [version]/bin

Diagnostics nxudiag [version]\win2k3\utils or [version]/bin

Tools nxflash [version]\win2k3\utils or [version]/bin

Table 1 NXB-10GXSR Cable Specifications — Optical1

1. Check with NetXen for information on the NXB-10GXLR long range optical cable specifications.

ParameterOptical Core Thickness

50 μm 62.5 μm

Wavelength 850 nm 850 nm

Bandwidth (min) 500 MHz 500 MHz

Operating Range 300 m 300 m

Table 2 NXB-10CX4 Cable Specifications — Copper

802.3 Physical Layout Network Type Cable Length

10GBASE-CX4 LAN 15m (max)

2 Intelligent NIC Installation and Tuning Guide NetXen, Inc.

2. System Requirements

2.2.2 XFP Module

2.2.3 System and Software Requirements

Table 3 NXB-10GXSR XFP Module Specifications

Manufacturer Name Part Number

JDS Uniphase Multi-rate 10 Gb/s Short Wave Optical XFP Transceiver

64P0194

PicoLight Multi-rate 10 Gb/s 850 nm Optical XFP Transceiver Module

PL-XXL-SC-S45-22

Table 4 System and Software Requirements

Hardware Systems Validated Platforms

Validated OS (32bit and x64) Firmware1

1. NIC and LSA use the same firmware.

Driver

PCIe: (x8)

Memory: Minimum

2GB

AMD, Intel Windows Server: 2003 SP1,SP2,R2

Linux Distros: RHEL4 U3,U4 SLES9 SP3,

SLES10, RHEL5

NICLSA

Linux:2 NIC, LSA

Windows3

NIC

2. For Linux NIC and LSA, install the kernel source and header files

3. For Windows NIC, SP1 must be installed. SP2 is recommended.


3. Quick-Start Installation Instructions

3 Quick-Start Installation InstructionsThis section is for users already familiar with the Windows or Linux operating systems. For those user’s requiring more detailed information, refer to Section 4.

Table 5 Quick Start Instructions — New NIC Driver Installation

NetXen NIC Hardware Installation Windows NIC Driver Installation Linux NIC Driver Installation1

1.Certain Linux distributions may contain a NetXen GPL driver. This driver will be names netxen_nic.ko. This driver must be removed before using the NetXen commercial driver. Please refer to Section 4.4 of this document, as well as the NetXen release notes.

Pre-Installation1. Verify the system meets the requirements

described in Section 2.2. Power down the system.

Installing the NetXen NIC Adapter1. Insert the NetXen NIC adapter card into an

available PCI Express slot.2. Connect the appropriate cables to the

board.3. Power on the system.

Note that the NIC card comes with the firmware already installed.

Driver Installation1. Copy all of the provided NIC driver files

to a directory.2. From the Device Manager, scan for new

hardware.3. When prompted to connect to Windows

Update, Click No, not at this time, then click Next.

4. In the next window, click Install from a list of specific location and then click Next.

5. Select Search for the test driver in these locations. Select Include this location in the search.

6. Provide the path to the directory where the driver NXP2NIC.sys is located. For 32-bit operating systems, choose the x86 directory. For 64-bit operating systems, choose the x64 directory.

NIC Card Configuration1. Open the Network Connections window

from the Control Panel.2. Verify that NetXen NIC adapter icon says

Connected. If not, right-click on the NIC Adapter icon and select Properties, then assign the IP and subnet addresses and click OK.

Driver Installation (tar.gz)1. Untar the file, build, install, and load the

driver: tar xzf [version].tar.gz cd [version] make make install modprobe nx_nic

Driver Installation (source rpm)1. Install the RPM as follows2

rpm -iv nx_nic-[version].src.rpm2. Create the binary RPM as follows:

rpmbuild -bb /usr/src/redhat/SPECS/ nx_nic-[version].spec, (Redhat) or

rpmbuild -bb /usr/src/packages/SPECS nx_nic-[version].spec (SuSE)

3. Load the nx_nic driver. modprobe nx_nic

NIC Card Configuration1. Identify the NetXen Ethernet interface:

ethtool -i eth[n]2. Assign an IP address:

ifconfig eth[n] [IP address]

2. x.yy.zz is an example of [version]



If a NetXen Intelligent NIC card is already installed on the system and a driver and/or firmware update is required, refer to the procedure in Table 6 below.

Table 6 Quick Start Instructions — Updating from an Existing Driver and Firmware

Windows Driver/Firmware Update Linux Driver/Firmware Update

Verify Link Status1. With the existing driver and firmware installed,

verify that the interface is up before continuing with the firmware update.

Update to New Firmware1. Open a Command window.2. Go to the utils directory in the driver package

provided.3. The following binaries are required to be in the

same directory from where the Flash command will be executed: nxflash.exe xfp_romimage cx4_romimage

4. Run the following commands: nxflash -i NX_NIC --all1

5. Verify the firmware has been updated using the following command. nxflash -i NX_NIC --info

Update to the New Driver1. Install the new driver version.2. Verify the new driver version.

1. Writing of the Flash is a critical operation. Please do not interrupt or reboot during this time.

Verify Link Status1. With the existing driver and firmware installed, verify

that the interface is up using ifconfig before continuing with the firmware update.

Update to New Firmware1. Run the following command from the bin directory of

the driver: ./nxflash -i eth[n] --all1

Unload and Remove the Existing Driver1. Unload the existing driver.

ifconfig eth[n] down rmmod nx_nic2. Remove the existing driver module. rm -f /lib/modules/[kernel version]/kernel/

drivers/net/nx_nic.ko2

Reboot the System1. Once the existing driver has been removed, reboot the

system.

Update to the New Driver2. Update to the new driver as follows:

tar xzf [version].tar.gz cd [version] make make install modprobe nx_nic

3. Ensure that the Ethernet interface is up. ifconfig eth[n] <ip addr> up

2. For Linux kernel 2.4, the driver will appear as a nx_nic.o file.



Once the WIndows or Linux NIC installation is complete as described in the previous tables, the following procedures in Table 7 can be used to enable Windows Chimney or Linux LSA. Table 7 Quick Start Instructions — Enabling WIndows Chimney and Linux LSA

Enabling Windows Chimney Enabling Linux LSA

The following steps can be used to enable Windows Chimney on the driver. Note that the Windows NIC driver must be installed before executing the procedure below.1

1. Open the Network Connections window.

2. Right click on the icon for the NetXen adapter and select Properties.

3. Click on the Configure button.4. Click on the Advanced tab.5. Change the “TCPv4 Stateful offload” from disabled

to enabled.6. To verify the offload has been enabled and is

operating: C:\>netsh int ip show offload

1.A key is required to be loaded onto the card before Chimney can be enabled. Please refer to the Release Notes for more information on how to obtain and verify this key.

The following steps can be used to enable Linux LSA on the driver. Note that the Linux NIC driver must be installed before executing the procedure below.

1. Untar the file, build, and load the drivers.tar xzf [version]-lsa.tar.gzcd [version]-lsamakecd driverinsmod ./nx_intercept.koinsmod ./nx_lsa.ko

2. To verify the LSA driver is operating, execute the following command:cat /proc/net/nx_nic/lsa_1/stats

3. Example for offloading iperf.cd ..\nxoffload./nxoffload –a –n iperf –z

4. To display the current offload rules execute the following:./nxoffload -s


4. Detailed Installation and Configuration

4 Detailed Installation and Configuration

4.1 Installing and Connecting the Intelligent NIC Adapter1 Verify that the system is powered off.

2 Remove the system cover.3 Locate the x8 PCIe slot if one is available.

4 Once the bracket is removed, insert the adapter into the PCI Express slot1. Apply uniform pressure on top of the board until it is firmly seated in the bus connector.

1. Due to space constraints in some systems, when installing an XFP board, it may be necessary to first remove theXPF module from the board, insert the board into the PCI slot, then reconnect the XFP module to the board.

Identify x8 slots for optimal performance



5 For optical installations, once the board is inserted into the slot, connect the XFP module. Make sure the optical module clicks when inserted into the cage.

6 Secure the bracket.7 Connect the appropriate cables to the NIC and replace the cover.

8 Power on the system.

XFP Cable connection

CX4 Cable connection

XFP Module

CX4Connector



4.2 Windows Driver InstallationThis section describes the Windows installation procedure. For the Linux driver installation procedure, refer to Section 4.3.

4.2.1 NIC Driver InstallationNote that the following procedure is for a new driver installation.

1 Copy the NIC release package to a directory on the host.

2 After the server system is powered on, Windows detects the new card and displays the Found New Hardware Wizard window.

When this window appears, select No, not at this time, and click Next.



3 Select Install from a list or specific location (Advanced) and click Next, as follows:

4 Select Search for the best driver in these locations.Select Include this location in the search.

Type the appropriate path to the NIC driver directory in the accompanying field and click Next. An example is shown below:

If the exact directory path is not known, click Browse and select the NIC driver directory.The NIC card supports both 32-bit and 64-bit operating systems. The x86 directory contains 32-bit driver and the x64 directory contains the 64-bit driver.

D:\3.4.216\SNP\x64



Select the appropriate directory (x86 or x64) and click OK. Then click Next in the Hardware Update Wizard window to install the driver.

5 The following screen is displayed while the driver is installed.



6 Once the driver installation is complete, the following window will appear. Click Finish to exit.

7 To verify that the driver was properly installed, Right-click on My Computer, then select Device Manager. A listing of devices appears in right side of the window. Under Network Adapters, the NetXen 10Gb Ethernet Adapter should be listed as shown below:



8 In the above window, right-click on the NetXen 10Gb Ethernet adapter and select Properties. The General tab will be displayed as shown.

9 There are four tabs — General, Advanced, Driver, and Resources. Click on the Advanced tab to display the driver properties as shown below.



The Advanced tab selections displayed above are defined as follows:Table 8 Description of Advanced Properties Parameters

Parameter Description

Flow Control Sets flow control parameters.Registry Key: FlowControlDefault: Enabled.Values: Enabled/Disabled

Large Send Offload TCP Large send offload – Enable/Disable options.Registry Key: LargeSendOffloadDefault: Enabled.Values: Enabled/DisabledSize of LSO: 64K

Locally Administered Address This is the Locally Administered Address (LAA) that users/administrators can set on this interface and this address overrides the permanent address of the adapter (that may have been flashed in to the hardware).Registry Key: NetworkAddressFormat: HexOptions: Value (a value is required, enter the 12 hex bytes of the MAC address to be used), or not present.

Max Ethernet Frame Size Ethernet frame size for packet transmission and receive (does not include MAC Header).Reg Key: MaxFrameSize)Default: 1500 (corresponds to 1514 bytes on the wire + 4 bytes of CRC)Max: 8000 Min: 1500

Max Jumbo Buffers Number of buffers per Jumbo frame.Reg Key: RxJumboRingSizeDefault: 4096 Values: 1024, 2048, 4096, 8192.

Maximum Receive buffers (in K) This is packet buffers used by the driver for receive (DMA memory and indicated to stack). Size of each buffer is 1536 bytes.Registry Key: MaxRxBuffersMin: 1KMax: 16KDefault: 8192

802.1P/Q Packet Tagging For supporting 802.1p/q priority tagging. Should be enabled to set VLAN ID.Registry Key: PQTaggingDefault: EnabledValues: Enable/Disabled



10 Click on the Driver tab in the Advanced Properties window. The driver can be updated, rolled back, or uninstalled from this window.

Receive IP Checksum Offload This parameter is for enabling/disabling the receive IP checksum offload.Registry Key: RxIPChecksumOffloadDefault: EnabledValues: Enabled/Disabled

Receive TCP/UDP Checksum Offload This parameter is for enabling/disabling the receive TCP and receive UDP checksum offload.Registry Key: RxTCPChecksumOffloadDefault: EnabledValues: Enabled/Disabled

Status Ring Size Receive status ring size from hardware.Registry Key: StsRingSizeDefault: 8192Values: 1024, 2048, 4096, 8192, 16384, 32768

TCPv4 Stateful Offload TCPv4 stateful offload.Registry Key: EnableTcpOffloadDefault: DisabledValues: Enabled/Disabled

Transmit IP Checksum Offload This parameter is for enabling/disabling the transmit IP checksum offload.Registry Key: TxIPChecksumOffloadDefault: EnabledValues: Enabled/Disabled

Transmit Ring Size Transmit command ring size provided to hardware.Registry Key: TxRingSizeDefault: 1024Values: 1024, 2048.

Transmit TCP/UDP Checksum Offload

This parameter is for enabling/disabling the transmit TCP and transmit UDP checksum offload.Registry Key: TxTCPChecksumOffloadDefault: EnabledValues: Enabled/Disabled

VLAN ID For setting the VLAN ID for this interface (also exposed through the standard OID).Registry Key: VlanIdDefault: 0 (no VLAN)Range: 0 - 4094

Table 8 Description of Advanced Properties Parameters (Continued)

Parameter Description



This completes the installation of the NetXen NIC software. Refer to the next section for information on configuring the NIC.

4.2.2 Configuring the Network Interface Card

1 Configure the network connections by opening the Network Connections window in the control panel, or directly from the desktop if the icon exists. The following window is displayed.



Note that if an IP address has not been assigned, the NetXen 10Gb Ethernet adapter entry above will say “Limited or no connectivity”.If a red ‘X’ appears in the NetXen 10Gb Ethernet10 Adapter icon, this indicates that the link is down. If this occurs, check the following:

Verify that the XFP module is installed correctly.

Verify that the card is seated in the PCI slot properly.

Make sure the network cable is connected properly at both ends.

Is a known good cable being used?

Check the XFP module and replace if necessary.

Has an IP address been assigned to the interface?

Refer to Section 7, "Troubleshooting Guidelines" for more information.To check the status of the connection, right-click on the NetXen 10 Gb NIC adapter icon above and select properties. This causes the Local Area Connection Properties window to be displayed.2 To assign an IP address manually, select the Internet Protocol (TCP/IP) entry below

and click the Properties tab.



3 In the Internet Protocol (TCP/IP) Properties window, assign the appropriate IP, subnet mask, and default gateway values.

4 Using the ping utility, verify connectivity with another card or switch.

4.3 Linux InstallationThe Linux sources can be installed in either TAR or RPM format depending on the release package you received as described below.IMPORTANT NOTE: In the case of SLES 10.1 SP1 and RHEL5U1, the NetXen GPL driver netxen-nic.ko exists in the kernel and may be installed automatically when the O/S comes up. Other Linux distributions above kernel version 2.6.18 may also contain the GPL driver. This driver must be removed before installing the commercial driver from this release.

To check if the netxen-nic has been installed on the system:

1 Check the /lib/modules/<kernel-version>/kernel/net/netxen directory for the netxen-nic.ko driver and either remove or rename this file then reboot.

2 Or to uninstall the driver, perform a “rmmod netxen-nic” before continuing with the following steps in this section.

The following sections describe the procedure for a new driver installation.

4.3.1 Linux NIC Driver Installation (TAR format)

Note: Root level permissions are required for the driver installation.



1 Copy the file [version]1 tar.gz to the host system and follow the installation command sequence:

To build, install, and load the driver:

tar xvzf [version].tar.gz

cd [version]

make

make install

(driver module is installed in: /lib/modules/<kernel version>/kernel/drivers/net)

cd driver

modprobe nx_nicVerify that nx_nic driver is loaded:

lsmod | grep -i nx_nic

2 Identify the NetXen Ethernet interface:

ethtool –i eth[n]

3 Assign an IP address and bring up the Ethernet interface

ifconfig eth[n] [IPaddress] up

4.3.2 Linux NIC Driver OptionsStep 1 above uses the insmod command. The following optional parameters can be passed to the driver with the modprobe or insmod commands:

insmod nx_nic <parameter> or

modprobe nx_nic <parameter>

The optional parameters are listed below.

1. Where [version] is the numbered release version of the software.

Table 9 insmod and modprobe Command Optional Parameters

Parameter Description Default Range

tx_desc1

1.Example usage: modprobe nx_nic tx_desc=1024 or insmod nx_nic tx_desc=1024

Sets the size of the command descriptor rings (Tx). 1024 256 - 4096 (must be power or two)

jumbo_desc2 Sets the size of the receive ring for jumbo frames. 1024 64 - 4096(must be power or two)

rdesc_1g3 Sets the size of the receive ring for normal packets (1 GbE ports).

8192 64 - 32,768

rdesc_10g4 Sets the size of the receive ring for normal packets (10 GbE port).

32,768 64 - 32,768



4.3.3 Linux NIC Driver Installation (RPM format)The following subsections describe how to install the NetXen Linux driver source in RPM format, create the binary, and install the binary.

Installing the Linux Driver Source in RPM Format

To install the Linux driver source in RPM format, execute the following command.

rpm -iv nx_nic-[version].src.rpm

Create the Binary

To create the binary RPM for Redhat systems, execute the following command:

rpmbuild -bb usr/src/redhat/SPECS/nx_nic-[version].spec

To create the binary RPM for SuSE systems, execute the following command:

rpmbuild -bb usr/src/packages/SPECS/nx_nic-[version].spec

Install the Binary

To install the binary RPM for Redhat systems, execute the following command:

rpm -i usr/src/redhat/RPMS/x86_64/nx_nic-[version]-1.x86_64.rpm

To create the binary RPM for SuSE systems, execute the following command:

rpm -i usr/src/packages/RPMS/i586/nx_nic-[version]-1.i586.rpm

Note that the i586 designation in the path above could also be i686 or i386 or x86_64 depending on the system.

Install the Tools

To install the binary tools, execute the following command:

rpm -ivh nx_tools-[version].rpm

The tools can be found in the following directory:

cd /opt/netxen

4.3.4 Configuring the NIC on LinuxAssign an IP address to the interface from the command line. For example:

1 Identify the Ethernet interface (eth1, eth2, etc.) and the nx_nic driver:

ethtool -i eth[n]

ifconfig -eth[n] -a

2 Assign an IP Address and activate the interface:

2.Example usage: modprobe nx_nic jumbo_desc=1024 or insmod nx_nic jumbo_desc=1024

3.Example usage: modprobe nx_nic rdesc_1g=8192 or insmod nx_nic rdesc_1g=8192

4.Example usage: modprobe nx_nic rdesc_10g=32768 or insmod nx_nic rdesc_10g=32768



ifconfig eth[n] down

ifconfig eth[n] <IP Address> <netmask> up

3 Verify network services are running, and ping:

service network restart

ping <IP address>



4.4 Uninstalling the SoftwareThis section describes how to uninstall the NetXen NIC software from Windows and Linux systems.

4.4.1 Uninstalling Windows Drivers for Both NIC and ChimneyTo uninstall the NetXen NIC adapter driver software in Windows, perform the following steps:

1 Right-click on My Computer and click on Manage. The Computer Management window is displayed.

2 Double click on Device Manager. The list of devices in the computer is displayed on the right side of the window.

3 Locate and right-click on NetXen 10Gb Ethernet Server Adapter.4 Click Uninstall to perform the operation.

4.4.2 Uninstalling the NetXen NIC Driver in LinuxNote that if the LSA driver is installed, it must be uninstalled before the NetXen Linux NIC driver is uninstalled. Refer to Section 4.4.3 below for information on how to uninstall the LSA driver.

TAR FormatIn the Linux system, the TAR archive may be removed as follows:

rm -rf [version]

rm -f [version].tar.gz

ifconfig eth[n] downrmmod nx_nic.ko

The driver may be removed from /lib/modules as follows:

rm -f /lib/modules/$(uname -r)/kernel/drivers/net/nx_nic.ko

RPM FormatThe RPM package may be removed as follows:

rpm -e nx_nic-[version]

4.4.3 Uninstalling the LSA Driver in Linux

TAR FormatNote that the LSA driver must be removed before the Linux NIC driver can be uninstalled as described in Section 4.4.1 above. The removal procedure for the LSA driver is shown below.

rm -rf [version]-lsa

rm -f [version]-lsa.tar.gz



rmmod nx_lsa.koThe driver may be removed from /lib/modules as follows:

rm -f /lib/modules/$(uname -r)/kernel/drivers/net/nx_lsa.ko

RPM FormatThe RPM package may be removed as follows:

rpm -e nx_lsa-[version]


5. Firmware Updates

5 Firmware UpdatesThe hardware shipped with the product contains the latest firmware. The firmware must be properly flashed in order for the adapter to function properly. Check with NetXen periodically for firmware updates.

5.1 Using the NetXen Flash UtilityThe NetXen Flash utility can be used to modify the contents of the Flash. When using the nxflash.exe utility, the following syntax should be used.nxflash.exe -i NX_NIC [command] (for Windows OS)1

./nxflash -i eth[n] [command] (for Linux OS)In this command line, the -i indicates the interface command and the NX_NIC specifies the actual interface for Windows. The [n] indicates NetXen’s Ethernet interface.Along with the interface, the available commands are listed in Table 10.

The following are some examples of how the nxflash tool may be used.

Update firmware image

Get information about the Flash contents

Each of these functions is described in detail below.

1 Update firmware image

The following files must be present in the directory from where nxflash is executed:

1. For Windows, note that the interface type MUST BE ALL CAPITALS to be recognized as a validinterface.

Table 10 NetXen Flash Utility Commands

Command Definition Description

-i--interface

Interface Specifies the interface type, either [NX_NIC, NX_NIC1, etc.] or [eth1, eth2, etc.].

--info Information Prints board information such as board type, chip revision, serial ID, firmware version, BIOS version, MAC address, etc.

-a--all

Firmware update Complete update of BIOS and firmware.

-r--restore

Restore Restores the existing firmware version on the card.

--opt-rom-on Enable ROM Enable expansion ROM.--opt-rom-off Disable ROM Disable expansion ROM.

-h--help

Help Displays the help menu.

-v--version

Version Displays the version information.


5. Firmware Updates

xfp_romimage - This is the firmware image for an XFP card.cx4_romimage - This is the firmware image for a CX4 card.These files are supplied as part of the software release. To update the firmware, change directory where the above files are located, and type the following: nxflash.exe -i NX_NIC --all (for Windows) ./nxflash -i eth[n] --all (for Linux)This above update command displays the current firmware version on the card and the new firmware version to which the card will be updated upon completion. The Flash update takes a few minutes to complete and MUST NOT BE INTERRUPTED.2 Get Flash related informationThe --info option may be used to get information about the Flash. e.g. nxflash.exe -i NX_NIC --info (for Windows) ./nxflash -i eth[n] --info (for Linux)The following is an example of a Windows nxflash output:

D:\3.4.216\utils>nxflash.exe -i NX_NIC --infoVerifying flash...Done.Board Type : XGb XFPBoard Chip Rev : C1Serial Id : MX6AMK0006Firmware version (on Flash) : 3.4.216Active Firmware version : 3.4.216BIOS version : 1.0.2MAC Addr 0 : 00:0E:1E:01:03:D4Subsystem ID : 0x00014040Expansion ROM : Disabled

5.2 Windows Firmware UpdateThe following steps are used for updating the Windows OS firmware.1 Open Command window.2 Go to the utils directory in the driver package provided.3 The following binaries are required to be in the same directory from where the Flash

command will be executed: nxflash.exe xfp_romimage cx4_romimage4 Run the following command:

nxflash.exe -i NX_NIC --all

Writing of the flash requires a few minutes. Please do not interrupt or reboot during this time.5 Reboot the system.


5. Firmware Updates

Verify the firmware has been updated using the following command.

nxflash.exe -i NX_NIC --info

5.3 Linux Firmware UpdatesThe following steps are used for updating the Linux OS firmware.

1 Using the existing firmware and driver on the host, bring the Ethernet interface up using the following command:

ifconfig eth[n] <IP addr> netmask <netmask> up

2 Update to the new firmware from the bin directory:

cd <new release version directory>/bin

./nxflash -i eth[n] --all

3 Bring down the Ethernet interface.

ifconfig eth[n] down

4 Remove the existing or old driver module.

rmmod nx_nic

5 Build and install the new driver.

cd <new release version directory>

make

make install

6 Reboot the host system.7 Load the new driver.

cd <new release version directory>/driver

insmod ./nx_nic.ko

5.4 Updating the Flash on Multiple Standalone Cards in WindowsThe Flash can be updated on multiple cards on the same host using one of the following procedures.

1 If the updates are being done from the same directory tree, the following commands must be run sequentially after the Flash update has completed.

nxflash.exe -i NX_NIC0 --all

nxflash.exe -i NX_NIC1 --all

2 Updates can be done simultaneously from two different directory trees. Note that the following two commands are being executed from two separate directory trees.

c:\[version]-a\utils\ nxflash.exe -i NX_NIC0 --all

c:\[version]-b\utils\ nxflash.exe -i NX_NIC1 --all


6. Linux Socket Accelerator (LSA)

6 Linux Socket Accelerator (LSA)The purpose of LSA is to allow the user to offload a significant amount of TCP network processing on to the Netxen NIC, significantly reducing host CPU utilization. Note that the LSA does not require the user to rebuild the Linux kernel, as simple loadable modules are used.

As described below, the user has very fine-grained control over what can be offloaded. Offload decision can be based on application name, local TCP port number, or remote TCP port number.

6.1 LSA Driver InstallationThis section describes the LSA installation instructions for the tar and RPM formats.

NOTE: The LSA Intercept driver must be running at all times for the LSA driver to work. Whenever the server is rebooted, check if the NIC and Intercept driver is running lsmod before using the LSA driver. The LSA intercept driver cannot be removed with the command rmmod; Remove the driver file from /lib/modules/<kernel>/drivers/net/, and reboot the host.

6.1.1 LSA Driver Installation PrerequisitesThe following prerequisites are required before installing the LSA driver.

At least 1 GB of memory.

At least 64 MB of free disk space.

Any of the following Linux distributions

- RHEL4, U3, U4- RHEL5- SLES9 SP3- SLES10

In all cases, Linux kernel 2.6.5 or later is required.

6.1.2 TAR FormatThis section describes the LSA driver installation from the tar package.

1 First Load the NIC driver (nx_nic.ko)

2 Build and load the Intercept and LSA driver

% tar xvzf <version>-lsa.tar.gz

% cd <version>-lsa

% make

% make install

% insmod./driver/nx_intercept.ko

% insmod./driver/nx_lsa.ko



3 Use offload rules to offload the applications

4 Use rmmod nx_lsa to remove the LSA driver

6.2 LSA Offload ConfigurationLSA can be configured via the nxoffload utility. This utility works by maintaining a list of rules that dictate which applications to offload. Each rule can take one of the following parameters:

Application name

Local TCP port number

Local and Remote TCP port number

6.2.1 LSA Configuration OptionsA list of parameters to this utility can be obtained by typing ./nxoffload -help

usage: ./nxoffload <options>

The options are listed below.

6.2.2 LSA Configuration Examples

To offload the iperf application, type the following: ./nxoffload -a -n iperf

To offload the application that uses local TCP port 80, type the following: ./nxoffload -a -p80

To remove the application that uses local TCP port 80, type the following: ./nxoffload -r -p80

To list the current offload rules, type the following: ./nxoffload -s

Table 11 LSA Configuration Options

<option> Definition Comments

-p Port number to be offloaded.-n Application name to be offloaded.-t TCP tuple to be offloaded. Format: local IP, local port, remote

OP, remove port. Example: ./nxoffload -a -t'0, 0, 0, 5001'./nxoffload -a -t'0, 2000, 0, 5001'

-a Add an offload rule.-r Remove and offload rule.-z Enable zero copy.-s List all ports and applications offloaded.



To offload an application with zero copy enabled, type the following: ./nxoffload -a -z -n <application-name>.

Zero copy is a way to significantly reduce CPU utilization in the send path for some applications. Not all applications may benefit from using zero copy, and some reduction in throughput may be observed. Applications that do well with zero copy are those that have large write sizes and a large number of individual connections.

6.2.3 nxoffload Caveatsnxoffload has the following caveats:

To add or remove offload rules, nxoffload requires root access

Rules may be added or removed at any time. The addition or removal of a rule, however, will not affect connections that have already been established. For example, if the user starts a non-offloaded iperf application and then types ./nxoffload -a -n iperf, this instantiation of iperf will NOT be subsequently offloaded. Future instantiations, however, will be offloaded.

6.3 Running LSAOnce the appropriate rule has been entered into nxoffload, no further user intervention is required. The offload occurs automatically. The offload statistics can be monitored by reading the file /proc/net/nx_nic/lsa_x/stats, where 'x' is NetXen card number.

6.4 LSA StatisticsTable 12 lists the different types of LSA statistics, along with the current support status for each type.

Table 12 LSA Statistics

Name Description Status

TCP StatisticsTcpMaxConn Total number of connections since initial power up. Not supported.

TcpActiveOpens Total number of connections opened via connect. Supported.TcpPassiveOpens Total number of connections opened via accept. Not supported.TcpAttemptFails Number of unsuccessful connection attempts. Supported.TcpEstabResets Resets received by connections in the established

state.Supported.

TcpCurrEstab Number of connections currently in the established state.

Supported.

TcpInSegs Number of received segments. Supported.TcpOutSegs Number of transmitted segments. Supported.

TcpSlowOutSegs Number of transmit segments processed by the slow firmware path.

Supported.

TcpRetransSegs Number of retransmitted Supported.



TcpInErrs Number of illegal received segments. Supported.TcpOutRsts Number of transmitted resets. Supported.

TcpOutCollapsed Number of collapsed transmitted segments. A collapsed segment is generated when smaller segments are combined to form a single large segment

Supported.

IP StatisticsIpInReceives The total number of input datagrams received from

interfaces, including those received in error.Supported.

IpInHdrErrors The number of input datagrams discarded due to errors in their IP headers, including bad checksums, version number mismatch, other format errors, time-to-live exceeded, errors discovered in processing their IP options, etc.

Supported.

IpInAddrErrors The number of input datagrams discarded because the IP address in their IP header's destination field was not a valid address to be received at this entity. This count. includes invalid addresses (e.g., 0.0.0.0) and addresses of unsupported Classes (e.g., Class E). For entities which are not IP routers and therefore do not forward datagrams, this counter includes datagrams discarded because the destination address was not a local address.

Not supported.

IpInNoRoutes (Not in ip-mib)

Number of input datagrams discarded because there was a problem with either determination of inbound interface or a problem with determination of a route back to the source of the datagram.

Supported.

IpInDiscards The number of input IP datagrams for which no problems were encountered to prevent their continued processing, but which were discarded (e.g., for lack of buffer space). Note that this counter does not include any datagrams discarded while awaiting re-assembly.

Not supported.

IpInDelivers The total number of input datagrams successfully delivered to IP user-protocols (including ICMP).

Supported.

IpOutRequests The total number of IP datagrams which local IP user-protocols (including ICMP) supplied to IP in requests for transmission. Note that this counter does not include any datagrams counted in ipForwDatagrams.

Supported.

IpOutDiscards The number of output IP datagrams for which no problem was encountered to prevent their transmission to their destination, but which were discarded (e.g., for lack of buffer space). Note that this counter would include datagrams counted in ipForwDatagrams if any such packets met this (discretionary) discard criterion.

Supported.

Table 12 LSA Statistics (Continued)




6.4.1 Example Read OutThe following is a sample of the TCP and IP statistics displayed when the stats command is executed.

cat /proc/net/nx_nic/lsa_1/stats

IpOutNoRoutes The number of IP datagrams discarded because no route could be found to transmit them to their destination. Note that this counter includes any packets counted in ipForwDatagrams which meet this `no-route' criterion. Note that this includes any datagrams which a host cannot route because all of its default routers are down.

Supported.

IpReasmTimeout The maximum number of seconds which received fragments are held while they are awaiting reassembly at this entity.

Not Supported.

IpReasmReqds The number of IP fragments received which needed to be reassembled at this entity.

Not Supported.

IpReasmOKs The number of IP datagrams successfully re-assembled.

Not Supported.

IpReasmFails The number of failures detected by the IP re-assembly algorithm (for whatever reason: timed out, errors, etc). Note that this is not necessarily a count of discarded IP fragments since some algorithms (notably the algorithm in RFC 815) can lose track of the number of fragments by combining them as they are received.

Not Supported.

IpFragOKs The number of IP datagrams that have been successfully fragmented at this entity.

Not Supported.

IpFragFails The number of IP datagrams that have been discarded because they needed to be fragmented at this entity but could not be, e.g., because their Don't Fragment flag was set.

Not Supported.

IpFragCreates The number of IP datagram fragments that have been generated as a result of fragmentation at this entity.

Not Supported.

TCP Statistics Port 0 Port 1 Port 2 Port 3

TcpMaxConn 0 0 0 0

TcpActiveOpens 180 0 0 0

TcpPassiveOpens 0 0 0 0

TcpAttemptFails 0 0 0 0

Table 12 LSA Statistics (Continued)




TcpEstabResets 0 0 0 0

TcpCurrEstab 0 0 0 0

TcpInSegs 54593130 0 0 0

TcpOutSegs 173931132 0 0 0

TcpSlowOutSegs 2690 0 0 0

TcpRetransSegs 743 0 0 0

TcpInErrs 0 0 0 0

TcpOutRsts 0 0 0 0

TcpOutCollapsed 0 0 0 0

TcpTimeWaitConns 0 0 0 0

IP Statistic Value

IpInReceives 54593098

IpInHdrErrors 0

IpInAddrErrors 0

IpInNoRoutes 0

IpInDiscards 0

IpInDelivers 54593130

IpOutRequests 173933823

IpOutDiscards 0

IpOutNoRoutes 0

IpReasmTimeout 0

IpReasmReqds 0

IpReasmOKs 0

IpReasmFails 0

IpFragOKs 0

IpFragFails 0

IpFragCreates 0

TCP Statistics Port 0 Port 1 Port 2 Port 3


7. Troubleshooting Guidelines

7 Troubleshooting Guidelines

7.1 Unable to Boot the Host System with the NIC Card InstalledRemove the NIC and reboot. Does the system boot without the NIC?

If so, power down the system, insert the NIC into a different PCIe slot and boot-up the system.

Is the recommended minimum memory installed on the host system?

Note any error message during the OS boot sequence.

Check for system error messages:

Dbgview.exe (Windows event viewer)

dmesg (Linux)

Use the NetXen diagnostics utility for further testing.

7.2 Unable to ping the NIC Installed on the HostType lspci -v and look for vendor ID ‘4040’.

- Is the NIC visible on the PCI bus?

- Is the NIC is placed in the PCIe x8 slot?

Check if the driver is loaded correctly and the card is initialized

- dmesg (display system messages on Linux)

- Windows Event Viewer tool

Run the NetXen user diagnostics

Check the NIC XFP module and optical cables; For CX4, check the IB cable

Check Network Configuration

- Check for valid IP Address and subnet

- Check for duplicate IP address and subnet numbers

- If connected via gateway, check gateway address and network path

- Check the NIC MAC address values

- Network link status as seen by the host OS

- Check for LED activity (using ping)

- Using network monitoring tools, analyze the Tx and Rx packets for the NIC

- Verify the ARP entry and response



7.3 Gather System Information and Check for Errors on Linux HostNetXen Product Information

- Card type

- NIC firmware and driver version

Operating System Information

- Windows OS (32 / 64) and Service Pack #

- Linux Distribution and Kernel Release (uname –a)

Host System Information

- CPU, Cores, Chipset – (Run System Information tool)

- BIOS information

- Host memory

- PCIe x8, x4, x1 …

- System setup description

- How many servers, clients etc

- Model of the host system

- Network setup (back to back, router, switch, backplane etc)

Check for error messages from the kernel:

- Type dmesg

- Type cat /var/log/messages

Check routing table, gateway, and IP Address conflicts for all interfaces:

- Type route

Check network interface TX and RX counts during pings:

- Type ifconfig <network interface>

Use ethereal, tethereal, or tcpdump utilities to capture packets:

- Analyze the ping packets

7.4 CaveatsJumbo frames of maximum size 8000 are supported.

The current version of the Windows Win2k3 OS does not support MSI in the network drivers (NDIS 5.1). However, hardware support is provided for NDIS versions 5.2 and 6.0.



7.4.1 MSI Interrupts for LinuxThe Linux OS version supports MSI, and is used by the Intelligent NIC driver (if available).

On the Linux host, type cat /proc/interrupts to check that PCI-MSI is enabled for the Ethernet interface

Determine if the installed Linux kernel has the MSI configuration option enabled (earlier versions of the kernel may require an MSI patch).Type make menuconfig from the directory /usr/src/kernels/<kernel version> and verify that MSI configuration option is supported/enabled.NetXen device ID = 4040

$ lspci -n | grep 4040

7.4.2 PCI-Express SlotThe NetXen reference card can be located on the PCI-Express bus using the following Linux procedures.

Determine Vendor and Device IDTo check if the PCIe slot has an x8 link for optimal PCIe performance, first determine the PCI vendor ID and device ID codes using the following command:

% lspci -n

A message similar to the following should be displayed:

06:00.0 0200: 4040:0001 (rev 24)

The NetXen vendor ID is 0x4040. A list of NetXen device ID’s is shown below.

Determine Interface AttributesTo determine the attributes of the interface, execute the following command:

% lspci -n -d 4040:0002 -vvv | grep -i width

A message similar to the one below should be displayed.

Link: Supported Speed 2.5 Gb/s, Width x8, ASPM L0s, Port 0

Link: Speed 2.5Gb/s, Width x8

Determine Width ParametersCheck if the lspci output "negotiated width" (the second width in the output) reflects the actual "supported width". In an XFP system for example, execute the following command to determine if these two widths are the same:

Table 13 NetXen Device IDs

Product Device ID Description

NXB-10GXSR 0x0001 10 Gbit Ethernet NIC with XFP optical interface.

NXB-10GCX4 0x0002 10 Gbit Ethernet NIC with CX4 copper interface.



% lspci -xxx -vvv -d 4040:0002

where 4040 indicates the NetXen vendor ID and 0002 indicates a CX4 card device ID. The XFP card device ID is 0001.Examine the following hex output:

From the lspci hex output:

d0: 10 00 01 00 02 80 00 00 40 20 01 00 81 04 00 00 /\

||_________ capability. "81 04" = x8

e0: 01 00 81 00 00 00 00 00 00 00 00 00 00 00 00 00 /\

||_________ negotiated. "81 04" = x8. "41 04" = x4, "11 04" = x1, "01

05" = x16

Note that some PCI-e chip sets with a x16 link may not support a x8 PCI-Express link width. A link is only required to support its maximum width and x1. Check the motherboard specification to determine if its PCI-e x16 slot can autonegotiate to x8. On some motherboards, the PCI-e slots are all the same physical size required by a x16 slot, but electrically they are different widths. Check the motherboard specification for the supported width of each of the PCI-e slots.

7.5 Additional SupportIf problems are encountered with server applications when either Chimney or RSS are enabled under Microsoft Win2K3 with SP2 (in Offload Mode), please follow the link below to see if Microsoft has addressed the specific issue.

http://support.microsoft.com/kb/936594

...... 81 04 00 00 01 00 81

Supported link width

Negotiated link width


8. User Diagnostics

8 User DiagnosticsThe user diagnostics can be invoked in Windows and Linux using the following command:

nxudiag.exe (Windows)

[version]/bin/nxudiag -i eth[n] (Linux)

8.1 Windows Tests The Windows user diagnostics utility performs the following tests using the nxudiag.exe command.

Table 14 NetXen Windows Diagnostic Utility Commands

Command Description

-i--interface

Specifies the interface type [NX_NIC, NX_NIC1, etc.]

-a--all

Perform all test, regardless of default.

-n--nodefault

No defaults, all tests will be set implicitly.

-R--CRegs

Test all control registers.

-F-Flash

Test the Flash.

-I--IRQS

Test interrupt mechanism.

-L--IntLB

Internal loopback test

-S--LinkST

Link status test.

-I--LED

LED test.

-nR--noCRegs

No control registers test.

-nF--noFLASH

No Flash test.

-nl--noIRQS

No interrupt test.

-nL--noIntLB

No internal loopback test.

-nS--noLinkSt

No Link status test.

-nl--noLED

No LED status test.


8. User Diagnostics

8.2 Linux Tests The Linux user diagnostics utility performs the following tests using the ./nxudiag.exe -i eth[n] command.

-E--ExtLB

External full duplex loop-back test.

-h--help

Prints the help message.

Table 15 NetXen Linux Diagnostic Utility Commands

Command Description

-i--interface

Specifies the interface type [eth1, eth2, etc]

-a--all

Perform all test, regardless of default.

-n--nodefault

No defaults, all tests will be set implicitly.

-s--silent

Silent mode. No output to console.

-v--verbose

Verbose mode. Output result to console.

-R--CRegs

Test all control registers.

-F-Flash

Test the Flash.

-I--IRQS

Test interrupt mechanism.

-L--IntLB

Internal loopback test

-H--Hw

Hardware test.

-S--LinkST

Link status test.

-nR--noCRegs

No control registers test.

-nF--noFLASH

No Flash test.

-nl--noIRQS

No interrupt test.

-nL--noIntLB

No internal loopback test.

Table 14 NetXen Windows Diagnostic Utility Commands

Command Description


8. User Diagnostics

-nH--noHw

No hardware test.

-nS--noLinkSt

No Link status test.

-E--ExtLB

External full duplex loop-back test.

-C--Cmdif

Command interface status test.

-h--help

Prints the help message.

Table 15 NetXen Linux Diagnostic Utility Commands

Command Description


Appendix A. Tuning Guidelines

Appendix A Tuning Guidelines

A.1 Windows Performance Tools

A.1.1 NTttcp (Windows)There are two switches on ntttcp that boost performance: -a and -rb. Running ntttcp without an optimal value for the -a switch and without the -rb switch is acceptable for the Gigabit NIC product.In the 10G NIC product, however, and in particular with fewer threads on Windows, performance is limited by the Receive window for the most part and by the outstanding I/O (-a) value, regardless of hardware limitations.For receive, an optimal value is 16 for -a and 512000 for -rb (or something similar). With these values, a single thread should perform closer to a multi-thread configuration.

ntttcpr -m 16,0,<rcv ipaddr> -n <# of tcp segs for snd/rcv> -a 16

For transmit, -a should be a value less than 8. The more threads, the lower the number should be.

Note: The send and receive side NTTTCP commands must use the receive side’s IP address.

Steps for NIC Performance Testing using NTTTCP on Windows

1 Install Win2k3 SP1

2 Enable Windows Scaling Set TCp1323Opts parameter value to 1, if the parameter does not exist, create it: "regedit" to HKLM\system\currentcontrolset\services\Tcpip\Parameters3 The drivers property sheet default is 1500. For Jumbo, set the frame size to 8000.

Table 16 ntttcpr Command Options

Command Optionsntttcpr Description

-l Message size in bytes. Default is 64K.

-m Number of threads.

0 CPU number.

IPAddress

Receive IP address.(Send side also uses receive side's IP Address).

-n Number of TCP segments.

-a Number of requests in parallel.(Send and receive side values must match).

-rb Receive window size on the receiver.



Unidirectional: One Server to One ClientReceiver:

ntttcpr.exe -m 16,1,<Receive Side IPAddr> -a 16 -p 5001 -fr -n 200000

Sender:ntttcps.exe -m 16,1,<Receive Side IPAddr> -a 16 -p 5001 -n 200000

The above commands can be used in both Chimney and non-Chimney modes. With Chimney enabled, performance is higher and CPU utilization is also reduced.

Unidirectional: One Server to One Client — Single Thread, Multiple ProcessorsIf the system has more than one CPU, the load can be distributed across two CPUs and port numbers.Send Sidentttcps.exe -m 1,0,192.168.20.2 -n 300000 -a 6 -p 5001ntttcps.exe -m 1,0,192.168.20.2 -n 300000 -a 6 -p 5002

Receive Sidentttcpr.exe -m 1,1,192.168.20.2 -n 300000 -a 16 -rb 512000 -p 5001ntttcpr.exe -m 1,1,192.168.20.2 -n 300000 -a 16 -rb 512000 -p 5002

A.1.2 nttcps and ntttcpr Command Line Options

C:\NTTTCP_NEW>ntttcps.exe <parameters>orC:\NTTTCP_NEW>ntttcpr.exe <parameters>

Version 2.6ntttcpr.exe: [-l|-n|-p|-a|-x|-rb|-sb|-i|-f|-u|-w|-d|-t|-v|-6|-fr|-mb] -m <mapping> [mapping]

-l <Length of buffer> [default: 64K] -n <Number of buffers> [default: 20K] -p <Port base> [default: 5001] -a [outstanding I/O] [default: 2] -x [PacketArray size] [default: 1] -rb <Receive buffer size> [default: 64K] -sb <Send buffer size> [default: 0] -i Infinite Loop [Only UDP mode] -f <File Name> [default: output.txt] -u UDP send/recv -w WSARecv/WSASend -d Verify Flag -t <Runtime> in seconds -v enable verbose mode -6 enable IPv6 mode



-fr Full buffers posted on reads -mb Multiple buffer post mode -m <mapping> [mapping] where a mapping is a session(s),processor,receiver IP set e.g. -m 4,0,1.2.3.4 sets up: 4 sessions on processor 0 to test a network on 1.2.3.4

A.2 Windows Optimization — Tcp1323OptsThis parameter determines whether TCP uses the time-stamping and window scaling features described in RFC 1323, TCP Extensions for High Performance.Window scaling permits TCP to negotiate a scaling factor for the TCP receive window size, allowing for a very large TCP receive window of up to 1 Gbyte. The TCP receive window is the amount of data the sending host can send at one time on a connection.To set the window scaling and timestamp parameters, click Start -> Run, then type regedit to start the registry editor. Open the following location:HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Window ScalingFor more efficient use of high bandwidth networks, a larger TCP window size may be used. The TCP window size field controls the flow of data and is limited to 2 bytes, or a window size of 65,535 bytes.Since the size field cannot be expanded, a scaling factor is used. TCP window scale is an option used to increase the maximum window size from 65,535 bytes to 1 Gigabyte.When the value for window size is added to the registry and its size is larger than the default value, Windows attempts to use a scale value that accommodates the new window size.

A.3 Linux Performance ToolsOnce connectivity between the two cards is established, common performance tools can be used for performance measurements.

Data Type Range Recommended Value

REG_DWORD 0 | 1 | 2 | 3 1

Hex Value Meaning

0 Timestamps and window scaling are disabled.

1 Window scaling is enabled.

2 Timestamps are enabled.

3 Timestamps and window scaling are enabled.



Note: Various factors including tools, kernel, stacks, and related parameters affect the performance of a system. A discussion on performance criteria is beyond the scope for the getting started section.

Add the following for performance tuning on Linux systems under --/etc/sysctl.conf

A.3.1 IPERFThe -P option determines the number of threads; default is one.Server Side:

iperf -s -w256k -l 56k

Client Side:

iperf -c <ipAddr of Server> -w256k -P12 -l 56k

A.4 Linux OptimizationWhile optimization of the Linux operating system is a topic beyond the scope of this guide, below are a few common guidelines.

PCIe x8 provides optimal performance.

MSI interrupts are preferred over legacy interrupts.

Linux Kernel versions 2.6.15 or later provide better performance optimization for the NIC.

Check CPU utilization on the host for CPU-bound bottlenecks.

The tunable performance parameters are system and application specific. Understanding the relationship between the parameter values, target workload, performance applications, OS and system configuration provides a better understanding of performance optimization and bottlenecks.

net.core.rmem_max = 16777216net.core.wmem_max = 16777216

net.ipv4.tcp_rmem = 4096 87380 16777216net.ipv4.tcp_wmem = 4096 87380 16777216

net.core.netdev_max_backlog = 250000

net.ipv4.tcp_timestamps = 0net.ipv4.tcp_sack = 1net.ipv4.tcp_rmem = 4096 87380 16777216net.ipv4.tcp_wmem = 4096 87380 16777216net.ipv4.tcp_low_latency = 0net.ipv4.tcp_window_scaling = 1net.ipv4.tcp_moderate_rcvbuf = 1net.ipv4.route.flush = 1

net.core.optmem_max = 524287


Appendix B. Glossary

Appendix B Glossary

Name Description

cx4_romimage NetXen firmware image for a CX4 card

dbgview.exe Debug viewer for Windows

dmesg Linux system messages

ethereal Packet analyzer

ethtool Linux Ethernet tool for card settings. See man ethtool in Linux

iperf Linux performance tool

LKM Linux Kernel Module

Load balancing Allows the load to be balanced between 2 or more ports in a system.

LSA Linux Socket Accelerator

NIC Network Interface Card

nxflash NetXen Flash utility

nx_intercept.ko Linux LSA intercept driver

nx_lsa.ko Linux LSA driver

nx_nic.ko Linux NIC driver

nxp2nic.sys Windows NIC driver

nxudiag NetXen user diagnostic utility

xfp_romimage NetXen firmware image for an XFP card


Appendix B. Glossary
