working with hardware resource limitations

WWoorrkkiinngg wwiitthh HHaarrddwwaarree RReessoouurrccee LLiimmiittaattiioonnss

By the Ardence Publications Department

Ardence, a Citrix® company

Document Number: RTX-810-004

ii

Working With Hardware Resource Limitation

Notice

The information in this publication is subject to change without notice.

THIS PUBLICATION IS PROVIDED “AS IS” WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT. CITRIX SYSTEMS, INC. (“CITRIX”), SHALL NOT BE LIABLE FOR TECHNICAL OR EDITORIAL ERRORS OR OMISSIONS CONTAINED HEREIN, NOR FOR DIRECT, INCIDENTAL, CONSEQUENTIAL OR ANY OTHER DAMAGES RESULTING FROM THE FURNISHING, PERFORMANCE, OR USE OF THIS PUBLICATION, EVEN IF CITRIX HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES IN ADVANCE.

This publication contains information protected by copyright. Except for internal distribution, no part of this publication may be photocopied or reproduced in any form without prior written consent from Citrix.

The exclusive warranty for Citrix products, if any, is stated in the product documentation accompanying such products. Citrix does not warrant products other than its own.

Product names mentioned herein may be trademarks and/or registered trademarks of their respective companies.

Copyright © 2008 by Ardence, a Citrix company. All rights reserved.

Version History

November 15, 2007 Ross Philipson Original version

December 19, 2007 Hope Nilsson Added Device Manager info, made general flow and language edits.

iii

Working With Hardware Resource Limitation

Table of Contents INTRODUCTION........................................................................................................................................................................ 1

BACKGROUND .......................................................................................................................................................................... 1

RESOLVING RESOURCE ISSUES.......................................................................................................................................... 2 DEVICE MANAGER OVERVIEW ................................................................................................................................................... 3 PCISCOPE OVERVIEW................................................................................................................................................................. 5 EXAMPLE SCENARIO ................................................................................................................................................................... 8 OTHER TOOLS........................................................................................................................................................................... 15

MESSAGE SIGNAL INTERRUPTS ....................................................................................................................................... 16

RESOURCES ............................................................................................................................................................................. 18 PCISCOPE ................................................................................................................................................................................. 18 PCI UTILITIES ........................................................................................................................................................................... 18

Working With Hardware Resource Limitation 1

Introduction This whitepaper provides information that is helpful in solving problems related to line-based interrupt request (IRQ) resource limitations on customer systems. It is important to note that line-based IRQ scarcity problems are not restricted to RTX. These issues have existed for all PC hardware and software from the beginning.

Customers who are having difficulty configuring their hardware for use with RTX due to line-based IRQ resource limitations should review this information. The primary approach discussed uses Device Manager and a third-party tool, PCIScope, to identify hardware conflicts and possible reconfiguration scenarios that would allow RTX usage. This approach is detailed in "Resolving Resource Issues," starting on page 2.

An alternative to hardware reconfiguration is the use of message-based interrupts. PCI version 2.2 introduced a new interrupt scheme called Message Signal Interrupts (MSI). MSI does not rely on actual interrupt lines between peripheral devices and the interrupt controller hardware. Instead, MSI uses the system's main memory to write interrupt messages directly to the interrupt controller hardware. This solution completely removes the device dependence on line-based IRQs. Customers with newer systems and hardware may be able to move away from the line-based IRQ limitations entirely if their devices support MSI. All readers are encouraged to read "Message Signal Interrupts" on page 16 to see if using MSI will solve their problems.

Note: The approach described in "Resolving Resource Issues" may not work in all situations. The underlying issue with line-based IRQ resources is a hardware limitation and the exact details are specific to the particular system that the customer is using. In some cases, there may be no way to reconfigure the system for proper use with RTX. Also note that this approach is also limited to reconfiguration scenarios for peripheral component interconnect (PCI) devices only.

Background On most PC systems, there are few options for configuring the association between IRQ lines and devices. On older systems with programmable interrupt controller (PIC) hardware, the primary limitation comes from having a maximum of 16 lines (many of which are reserved for a specific device in the system). Newer systems with advanced programmable interrupt controller (APIC) hardware introduced more IRQ lines (for example, 24) and PCI devices are generally wired to IRQ lines beyond the 16 “legacy” lines. However, while there are more lines, there are not usually more configuration possibilities because most PCI devices are hardwired to a particular IRQ line and cannot be moved.

In an effort to mitigate line-based resource limitations, PCI hardware is designed to share IRQ lines. Problems arise, however, when the operating system has to cycle through multiple devices to determine which one generated the interrupt. This cycling can introduce latency that can negatively affect device performance. Because of these potential issues, Microsoft does not recommend sharing IRQ lines and considers it a necessary evil (see the article "The Importance of Implementing APIC-Based Interrupt Subsystems on Uniprocessor PCs" for more information).

As recommended by Microsoft, RTX does not currently support sharing IRQ lines with Windows devices. This means that the IRQ lines that RTX uses must be available for exclusive use. Finding an exclusive IRQ often requires physically moving hardware around in the system and/or disabling other Windows devices. Using Device Manager and other third-party tools, such as PCIScope, to view the PCI configuration is helpful in determining what reconfigurations are possible. PCI expansion slot cards are obviously more configurable than hardware integrated into the motherboard of a system. In the latter case, the only method available is disabling (or possibly moving) the other device(s) that might be sharing the IRQ line.


Resolving Resource Issues When you attempt to configure a PCI device for use with RTX and that device is sharing IRQ resources with other Windows devices, you get the following status message in the RTX PCI Device Properties dialog box, indicating that the IRQ requested is not available for exclusive use:

The existing method for dealing with this situation is to either move the hardware to another PCI slot where, hopefully, the requested IRQ is not in use by other devices and/or to disable the other devices that are using your requested IRQ. However, this trial-and-error method may not solve the problem in the end because, ultimately, the limitation is that of the hardware you are using and its wiring. This document tries to improve on the trial-and-error method by using tools that allow you to view your system’s layout and make more informed decisions about which devices to move or disable. In particular, this document focuses on the use of Device Manager and PCIScope, an inexpensive tool that shows in detail the PCI information and layout for a given system.

Note: See the page 18 for information on acquiring PCIScope.

This status message indicates that the requested IRQ is not available for exclusive use


Device Manager Overview Windows Device Manager provides a host of information about the devices that are attached to your system. This document focuses on the IRQ information you can find in Device Manager.

To view IRQ information in Device Manager:

1. Start Device Manager. 2. From the View menu, choose Resources by Type, then expand the Interrupt Request (IRQ) node. You see a

window similar to the following:

For each device, you can see the type of device (PCI or ISA), the IRQ the device uses, and the name of the device. Devices that have the same IRQ number share an IRQ line and are not exclusive. You should avoid these IRQs when trying to find resources for exclusive use in RTX. For example, IRQ 16 in the illustration below would be a poor choice for RTX:


It is important to note that not all devices appear in Device Manager; specifically, some devices on the motherboard and some legacy ISA devices may not appear. In this case, you can use PCIScope to view all devices.

Multiple devices share IRQ 16


PCIScope Overview PCIScope presents all of the PCI-related information for a system in several well laid out graphical views. This presentation makes it easy to view and understand the connections and interactions between various PCI devices and busses. The following illustration shows the main view presented after you start PCIScope.

The PCI busses, and devices on those busses, are show in a tree view on the left. Clicking a device displays extended information about that device in the right pane. The device selected in the illustration above is a USB device that uses IRQ 10h. The device's location is on the motherboard.

IRQ Location


Click the System tab on the bottom to see general summary information about the system. The following illustration shows this view. Because we are concerned with IRQ limitations, we have selected the IRQ option from the tree in the left pane so that we see the IRQ Resource Summary in the right pane.

As with Device Manager, it becomes obvious in the IRQ Resource Summary that IRQ 10h has many devices attached to it. Again, this IRQ resource would be a poor choice when trying to find resources for exclusive use in RTX.

Click here to see the IRQ Resource Summary

IRQ 10h has many devices attached to it

System tab


Note: PCIScope shows its IRQ numbers in hexadecimal values while Device Manager uses decimal values, therefore, IRQ 10h in PCIScope is equivalent to IRQ 16 in Device Manager.

Another extremely useful view is the IRQ wiring diagram (invoked by clicking the button in the toolbar) which shows the actual IRQ line wiring of the various PCI devices with the interrupt controller. The diagram represents the individual IRQ wires in different colors, showing what devices share the lines. The following is an example of the diagram on an APIC system.

Note: PCI Scope's wiring diagram will only show information for busses that have cards installed in them. To see all of your busses, you can temporarily add cards to any empty slots.

The next section walks through an example scenario using the PCIScope features to reconfigure the hardware in a system to allow RTX usage of a specific device.


Example Scenario Our example scenario starts with the same system seen in the illustrations above and assumes that we want to configure the Intel PRO/100 Ethernet Controller for use in RTX. Unfortunately, this won't work because the Intel PRO/100 device is currently sharing IRQ resources with other devices, as is evidenced by the status message in the RTX PCI Device Properties dialog box:

The Status message and Disposition setting in the RTX PCI Device Properties dialog box tell us that:

• The Intel PRO/100 device is configured for exclusive use but resources cannot be acquired.

• The Intel PRO/100 device keeps getting an IRQ resource from the Windows plug-and-play manager that is shared with other devices.

Our next step is to find out more information about the Intel PRO/100 device and other devices that we may be able to reconfigure in our efforts to find an exclusive IRQ.

This status message indicates that the requested IRQ is not available for exclusive use


PCIScope reports the following information for the Intel PRO/100 device on the PCI Information tab:

PCIScope shows that the Intel PRO/100 uses IRQ line 10h. As was noted on page 6, many of the devices in this system share IRQ 10h. Moving or disabling all of these devices is not possible. Additionally, the location shown above, PCI Slot 5, indicates that the Intel PRO/100 device is in an expansion slot. In fact, inspection of the other devices on bus 04h shows that they are all in expansion slots. Given this situation, a reasonable next step is to look at these other devices, see what IRQ lines they use, and determine if they are good candidates for swapping with the Intel PRO/100. We will start by looking at the information for the VIA Fire II IEEE-1394 firewire controller.

IRQ 10h Intel PRO/100 Location


PCIScope show the following information for the VIA Fire II IEEE-1394 firewire controller:

The VIA Fire II IEEE-1394 firewire controller is using IRQ 13h and is located in an expansion slot, PCI Slot 4. Because it's installed in an expansion slot, the firewire controller is a good candidate for swapping, so we can go to the next step and determine if its IRQ is exclusive or not.

IRQ 13h VIA Fire II IEEE 1394 Location


To determine IRQ exclusivity, we return to the IRQ Resource Summary and look at the information for IRQ 13h. The IRQ Resource Summary shows that the VIA Fire II IEEE-1394 firewire controller is the only device using IRQ 13h; again, making it a good candidate for swapping:

VIA Fire II IEEE 1394 is the only device using IRQ 13h


An alternate method for determining IRQ exclusivity is to look at the IRQ wiring view. Because the wiring view can be difficult to parse, the following illustration has been altered to show only the dark purple and red wires connected to IRQ 13h and IRQ 10h, respectively:

You can see from this illustration that the 1394 firewire controller is the only device using IRQ 13h. At this point, we have determined that the 1394 firewire controller is installed in an expansion slot and uses an exclusive IRQ. These two characteristics make it an ideal candidate for swapping and we go ahead and move the Intel PRO/100 card to the slot where the 1394 firewire controller is and vice versa.


After moving the cards and restarting the system, PCIScope shows that the change is promising. The IRQ Summary Resource view shows that the Intel PRO/100 device is now using IRQ 13h exclusively. The 1394 firewire device has moved (as expected) to the crowded IRQ 10h:

VIA Fire II IEEE 1394 now shares IRQ 10h

Intel PRO/100 now uses IRQ 13h exclusively


The IRQ wiring diagram also shows that the Intel PRO/100 is the exclusive user of the 13h IRQ (again, the illustration below has been altered to show only the dark purple and red wires connected to IRQ 13h and IRQ 10h, respectively):

Finally, the RTX PCI Device Properties dialog box for the Intel PRO/100 device (on the following page) now shows no conflicts and indicates that the hardware is ready for use in RTX.


This example scenario is a relatively simple case where the device we wanted to configure for RTX was easily moved to another slot. There will be cases where there simply is no way to move the devices around to satisfy the exclusive IRQ requirements. In these cases, the PCIScope tool is useful in determining exactly why the problem cannot be resolved. You should also determine if you can leverage RTX support for MSI to solve your IRQ resource issues. See "Message Signal Interrupts" on page 16 for more details.

Other Tools This document has focused on the PCIScope tool, however, there are other potentially useful tools for viewing PCI configuration information. One of these tool packages is a free alternative called “PCI Utilities”. This is an open source GNU package for use mainly with Linux but it is easily compiled for Win32. It comes with instructions on building and using the command line tools the package produces. See "Resources" on page 18 for information on acquiring PCI Utilities.

Status indicates an exclusive IRQ has been obtained for this device


Message Signal Interrupts Message Signal Interrupts (MSI) do not rely on physical interrupt lines between peripheral devices and the interrupt controller hardware. MSI instead uses the system's main memory to write interrupt messages directly to the interrupt controller hardware. This solution completely removes the device dependence on line-based IRQs.

Going forward, more and more devices will start supporting MSI, leaving only legacy devices using the line-based IRQs. Most modern PC systems support use of MSI and Microsoft introduced operating system support for it in Windows Vista. RTX offers MSI support in version 8.1 on Windows 2000 and later. If you have MSI-capable hardware, this will greatly simplify hardware configuration in RTX. If you are planning to buy new hardware for use with RTX, you should consider purchasing hardware with MSI support.

When devices have MSI support, Windows Vista, as well as RTX 8.1, will use it by default. There are no manual steps to configure it. You can use either Device Manager or PCIScope to determine if a device is MSI-capable.

To use Device Manager to verify MSI-capability:

1. Start Device Manager. 2. From the View menu, choose Resources by Type, then expand the Interrupt Request (IRQ) node. You see a

window similar to the following. A negative IRQ number indicates that a device is using MSI.

A negative IRQ number indicates that this is an MSI-capable device.


You can use PCIScope to determine if your hardware supports MSI and to see specific MSI capabilities that the hardware has. The following is the PCI information for an MSI enabled network device.

The PCIScope reports MSI capabilities for this particular device under the New Capabilities section.

Note: If you see an “MSI Enable” value of “Disabled,” it simply means that MSI is not initialized for the device yet (that is, nothing is using it).

Indicates that this device is MSI-capable


When an MSI device is used in RTX, the RTX PCI Device Properties dialog box will indicate that MSI is present and RTX is using it.

Resources

PCIScope The following is a link to the home page for APSoft, the company that makes and sells the PCIScope tool:

http://www.tssc.de/

PCI Utilities You can find the main download site for PCI Utilities here:

ftp://atrey.karlin.mff.cuni.cz/pub/linux/pci/

Status indicates that this device is using MSI

working with hardware resource limitations

Documents