scaling network performance to 1000 mbps
TRANSCRIPT
Gigabit Ethernet Comes of Age
T e c h n i c a l P a p e r
Scaling Network Performance to 1000 Mbps
11
Gigabit Ethernet Comes of AgeScaling Network Performance to 1000 Mbps
Contents
Factors Driving Network Bandwidth 2
What Is Gigabit Ethernet? 2
Why Gigabit Ethernet? 3
Gigabit Ethernet Delivers Cost-Effective 1000 Mbps Ethernet Performance 4
Gigabit Ethernet Leverages Ethernet Equipment Costs 4
The Gigabit Ethernet Standard 4
Leveraging Experience and Expertise: Lowering the Cost of Ownership (box) 5
Gigabit Ethernet Cabling and Distance Specifications 6
Multimode vs. Single-Mode Fiber Cabling (box) 8
Implementing Gigabit Ethernet in the Network 8
Gigabit Ethernet and Link Aggregation 8
Optimizing Servers to Handle Gigabit Ethernet 9
PCI Bus Bandwidth 9
Network Operating System 10
CPU Utilization at Gigabit Speeds 10
Memory Subsystems 11
Partnering Gigabit Ethernet with ATM in Enterprise Networks (box) 11
Guidelines for Using Gigabit Ethernet and ATM (box) 11
Storage Subsystems 12
Gigabit Ethernet and Layer 3 Switching 12
Gigabit Ethernet Migration Scenario 12
Summary 13
For Further Reference 15
Gigabit Ethernet Comes of Age
Scaling Network Performance to 1000 Mbps
Is there a cost-effective way to accommodateexploding application and user bandwidthdemands on network backbones while limitingnetwork complexity? This white paper presents3Com’s strategy for IT managers faced withbandwidth capacity constraints and consideringmigrating to a Gigabit Ethernet (GbE) technol-ogy solution. The paper briefly examines theforces driving the adoption of Gigabit Ethernet,the concepts behind the technology, and the keyadvantages of implementing Gigabit Ethernetover other technologies. The paper also explainsthe recently ratified Institute of Electrical andElectronics Engineers (IEEE) 802.3z GigabitEthernet standard and accompanying cablingspecifications and their significance in networkimplementation. Finally, the paper covers someimportant deployment issues and presents anincremental migration path to a Gigabit Ether-net infrastructure.
Factors Driving Network Bandwidth
Progress is relentless; new technologies spurnew application development, and new appli-cations, in turn, fuel the need for further tech-nology advancements. In a relatively shorttime span, network connections have evolvedfrom shared or switched Ethernet to shared orswitched Fast Ethernet to accommodate risingbandwidth demand. Within the enterprisenetworks, business applications are nowadvancing to embrace high-resolution graph-ics, video, and other rich media types thatexceed the capacity of even Fast Ethernet per-formance. This latest generation of band-width-intensive applications can be organizedinto four categories: • Scientific modeling, publication, and med-
ical imaging applications produce multime-dia and graphics files that are ballooning insize from megabytes to gigabytes to terabytes.
• Internet and intranet applications createunpredictable any-to-any traffic patterns.Increasingly, with servers distributed acrossthe enterprise and users accessing Web sitesinside and outside the corporate network,
there is no way to predict where traffic willgo.
• Data warehousing and backup applicationshandle gigabytes or terabytes of data distrib-uted among hundreds of servers and storagesystems.
• Bandwidth-intensive groupware applica-tions such as desktop video conferencing,interactive whiteboarding, and real-timevideo that support mission-critical businessapplications not only require more rawbandwidth, they also demand low latencyand limited jitter to be effective.
Table 1 summarizes the applications andtheir impact on the network. As you can see,these applications are driving the demand forincreased bandwidth, applying pressure at thedesktop, the server, and the hub, and creatinghigher capacity demands at the network core.
Another critical factor spurring backbonebandwidth demand is the cost effectiveness ofedge and workgroup switching. Switching isfast becoming the predominant form of Ether-net connection; switched Ethernet shipmentsare predicted to outnumber shared Ethernetshipments by the year 2000 (Dell’Oro Group,Portola Valley, California). Switching at thenetwork edge puts a tremendous load on thenetwork backbone where traffic aggregates.
What Is Gigabit Ethernet?
Gigabit Ethernet is Ethernet that providesspeeds of 1000 Mbps—one billion bits persecond. It uses the same Ethernet frame for-mat and media access control technology as allother 802.3 Ethernet technologies. It also usesthe same 802.3 full-duplex Ethernet technol-ogy and 802.3 flow control. In terms of datarate, Gigabit Ethernet simply moves the deci-mal point, increasing the Fast Ethernet trans-mission rate tenfold, from 100 Mbps to 1000Mbps. Table 2 compares Fast Ethernet andGigabit Ethernet technologies.
Like its Ethernet and Fast Ethernet pre-cursors, Gigabit Ethernet is a physical (PHY)and media access control (MAC) layer tech-nology, specifying the Layer 2 data link layerof the OSI protocol model. It complementsupper-layer protocols TCP and IP, which
22
Acronyms andAbbreviations
ASIC
application-specific
integrated circuit
ATM
Asynchronous Transfer
Mode
CoS
Class of Service
CPU
central processing unit
CSMA/CD
carrier sense multiple
access/collision detection
FDDI
Fiber Distributed Data
Interface
GbE
Gigabit Ethernet
GBIC
gigabit interface connector
GMII
Gigabit Media Independent
Interface
IEEE
Institute of Electrical and
Electronics Engineers
IP
Internet Protocol
MAC
media access control
MAN
metropolitan area network
MM
multimode
MTU
maximum transmission unit
specify the Layer 4 transport and Layer 3 net-work portions and enable reliable communica-tion services between applications.
The network topology for Gigabit Ether-net follows the traditional rules of Ethernet.At Layer 2, the Spanning Tree Protocol is usedto ensure that there are no logical loops in thenetwork, creating a hierarchical tree topology.More complex LAN configurations, includingparallel data paths, are created through the useof routing technology. At any speed, Ethernetis designed for transporting data in the localarea environment. Like Ethernet and FastEthernet, Gigabit Ethernet leverages othertechnologies and standards to provide higher-
level services such as Class of Service (CoS)traffic prioritization and Quality of Service(QoS) data delivery that limit jitter andlatency.
Why Gigabit Ethernet?
There are three compelling reasons to deployGigabit Ethernet for greater bandwidth at thebackbone:• Gigabit Ethernet delivers the scalable per-
formance of Ethernet technology.• Gigabit Ethernet leverages the installed base
of Ethernet equipment.• Gigabit Ethernet leverages the historical cost
drivers of Ethernet switching equipment.
33
Table 1. Application Bandwidth Drivers
Scientific Modeling,
Publications, Medical
Internet/Intranet
Data Warehousing,
Backup
DVC, Interactive
Whiteboard
Data Types/Size
• Data Files• MB to GB
• Data files now• Audio now• Video future• High transaction rate
• Large files• Data files• GB to TB
• Constant data stream• Up to 3+ Mbps to the
desktop
Traffic Implication
• Increased bandwidthneeded for large files
• Increased bandwidth needed for large files
• Low latency• CoS/QoS
• Many data streams• Increased bandwidth
needed for large files• Search and access
require low latency• Backup performed
during fixed period
• CoS/QoS• Many data streams
Network Requirement
• Higher bandwidth for desktops, servers, backbone
• Higher bandwidth for desktops, servers, backbone
• Low latency
• Higher bandwidth for servers and backbone
• Low latency
• Higher bandwidth for servers and backbone
• Low latency• Predictable latency
Table 2. Gigabit Ethernet Is Ethernet
Fast Ethernet Gigabit Ethernet
Speed 100 Mbps 1000 Mbps
Frame Format 802.3 Ethernet 802.3 Ethernet
MAC Layer 802.3 Ethernet 802.3 Ethernet
Flow Control 802.3x Ethernet 802.3x Ethernet
Primary Mode Full duplex Full duplex
Signaling FDDI Fibre Channel
Acronyms andAbbreviations
NDIS
network driver interface
specification
NIC
network interface card
NOS
network operating system
OSI
Open Systems
Interconnection
PCI
Peripheral Component
Interconnect
PCS
physical coding sublayer
PHY
physical
PMA
physical medium attachment
QoS
Quality of Service
RMON
Remote Monitoring
RSVP
Resource Reservation
Protocol
SM
single-mode
TCP
Transmission Control
Protocol
UTP
unshielded twisted pair
Source: Gigabit Ethernet Alliance
Gigabit Ethernet Delivers Cost-Effective 1000
Mbps Ethernet Performance
Ethernet is the dominant and most ubiquitousLAN technology today, and it is clearly themost widely understood LAN technology.According to industry analyst InternationalData Corporation (IDC, Framingham, MA),more than 85 percent of all installed networkconnections were Ethernet by the end of1997, representing more than 118 millioninterconnected PCs, workstations, and servers.Ethernet is popular because it offers the bestcombination of price, simplicity, scalability,and management ease of use. Because GigabitEthernet is Ethernet, network managers canleverage their investment in staff training andskills, reducing their total cost of ownership.
Gigabit Ethernet extends Ethernet’s scal-able 10/100 Mbps performance to 1000 Mbps.Figure 1 shows third-party tests results for the3Com SuperStack® II Switch 9300, a twelve-port Gigabit Ethernet switch. The switchachieved an aggregate throughput of morethan 17.8 million frames per second in a meshconfiguration using all 12 full-duplex switchports, or 1.48 million frames per second foreach port. Once the bit overhead of the 12-byte interframe gap and the 8-byte preambleare included in the calculation, the test shows
that the Gigabit Ethernet link performs at fullwire speed of 1.0 billion bits per second with64-byte frames.
Gigabit Ethernet Leverages Ethernet
Equipment Costs
The goal of the IEEE 802.3z Task Force,which developed the Gigabit Ethernet stan-dard, was to specify connections that delivered10 times the performance of Fast Ethernet atvery affordable prices. Because Gigabit Ether-net leverages existing Ethernet technologies, italso leverages Ethernet’s fiercely competitiveindustry cost curve.
Gigabit Ethernet products are a good valuetoday. Currently, switched 1000 Mbps GigabitEthernet provides the best price performanceof all the leading high-speed LAN technologies.Figure 3 illustrates data from the Dell’OroGroup showing that switched Gigabit Ethernetdelivers the lowest cost per data rate—less than$2 U.S. per Mbps—compared to switchedFDDI, switched 155 Mbps ATM, switched622 Mbps ATM, and switched Fast Ethernet.
The Gigabit Ethernet Standard
The 1000BASE-X (IEEE 802.3z) Gigabit Eth-ernet standard was ratified in June 1998, aftermore than two years of intense effort within
44
0
2
4
6
8
10
12
14
16
18
20
4 6 8
Number of ports under test
Aggregate switching performanceFull-duplex Gigabit Ethernet ports (64-byte frames)
Source: The Tolly Group, January 1998
10 12
Theoretical maximum
Switch throughput
Thro
ughp
ut (m
illio
ns o
f fra
mes
per
sec
ond)
Figure 1. Gigabit Ethernet Wire-Speed Performance
55
The total cost of ownership is an importantfactor in evaluating any new networking tech-nology. Total cost of ownership includes notonly the purchase price of equipment, but alsothe cost of network administration, applicationsupport, training, maintenance, and trouble-shooting. Figure 2 shows the results ofresearch by the Gartner Group that quantifies
the total annual cost of ownership for anetwork-attached desktop. Note that 73percent of the total is “people costs”: 33percent in managing the NOS and supportingthe applications that run on the network, 13percent in administering the network, and 27percent in “hidden” costs incurred whennetwork problems cause productivity losses.
With familiar technologies like Ethernet,people costs are lower because network
managers are able to leverage the skillsets and training of their existingnetwork administration staff andleverage existing network analysisand network management tools.And because increasing thenumber of protocols and tech-nologies always increases network
complexity, deploying an Ethernet-based backbone solution is simpler
than deploying FDDI or ATM. A lesscomplex network, in turn, lowers the
cost of ownership.
Leveraging Experience and Expertise:Lowering the Cost of Ownership
33%
27%13% 27%
NOS/Application support
Administration Components
Hidden
Figure 2. Total Cost of Ownership for a Networked Desktop
0
5
10
15
20
25
SwitchedFDDI
Switched155 Mbps
ATM
Switched622 Mbps
ATM
Source: Dell’Oro Group, CY1997
Switched100 MbpsEthernet
Switched1000 Mbps
Ethernet
$ pe
r Mbp
s
Figure 3. The Price/Performance of Bandwidth
Source: The Gartner Group
the IEEE 802.3 Ethernet committee. The keyobjective of the 802.3z Gigabit Ethernet TaskForce was to develop a Gigabit Ethernet stan-dard that encompassed the following:• Allowed half- and full-duplex operation at
speeds of 1000 Mbps • Used the 802.3 Ethernet frame formats• Used the CSMA/CD access method with
support for one repeater per collisiondomain
• Addressed backward compatibility with10BASE-T and 100BASE-T technologies
Because the fundamental features of the802.3z specification have been stable duringthe last stages of the standardization process,network vendors have been able to build anddeliver quality, mature products to the mar-ketplace for many months. In addition,numerous interoperability demonstrationshave been sponsored by the Gigabit EthernetAlliance and other independent organizations,giving customers confidence in using GigabitEthernet products in their production net-works.
Gigabit Ethernet Cabling and Distance
Specifications
As shown in Figure 4, the physical (PHY)layer is a crucial part of the Gigabit Ethernetspecification. It provides the interface betweenthe media access control (MAC) layer and the
transceivers in Gigabit Ethernet hardware.The PHY layer performs encoding, decoding,carrier sense, and link monitor functions.
The IEEE 802.3z Gigabit Ethernet stan-dard ratified in June includes three physicallayer specifications, two for fiber optic media—1000BASE-SX and 1000BASE-LX—and onefor shielded copper media—1000BASE-CX.Another group, the IEEE 802.3ab Task Force,is defining the physical layer to run GigabitEthernet over the installed base of Category 5unshielded twisted pair (UTP) cabling. This1000BASE-T effort is scheduled for completionin March 1999. Figure 4 shows the variouslayers of Gigabit Ethernet and the distinctionbetween the 802.3z specification and the802.3ab specification. A fourth transceivertype, 1000BASE-LH, is a multivendor specifi-cation that defines optical transceivers thatsupport distances greater than the 1000BASE-LX specification.
Fiber cabling specifications. There are threespecifications for fiber cabling:• 1000BASE-SX (“S” for short wavelength)
defines optical transceivers or physical layerdevices for laser fiber cabling. The wave-length of SX lasers is 770 to 860 nanometers(nm) (commonly referred to as 850 nm).1000BASE-SX is based on the Fibre Channelsignaling specification and uses the 8B/10B
66
Media access control (MAC)full-duplex and/or half-duplex
Gigabit Media Independent Interface (GMII)(optional)
1000BASE-X PHY8B/10B auto-negotiation
1000BASE-TPCS
1000BASE-LXfiber optictransceiver
1000BASE-SXfiber optictransceiver
1000BASE-CXcopper
transceiver
1000BASE-TPMA
transceiver
Single-mode ormultimode fiber
Multimodefiber
IEEE 802.3z IEEE 802.3ab
Shieldedcopper cable
Unshieldedtwisted pair
Figure 4. Gigabit Ethernet Architecture Standard
Source: IEEE
physical coding sublayer (PCS). Targetedfor multimode fiber only, 1000BASE-SXtransceivers are less costly than those foundin products implementing the long wave-length specification.
• 1000BASE-LX (“L” for long wavelength)defines optical transceivers or physical layerdevices for laser fiber cabling. The wave-length of LX lasers is 1270 to 1355 nm(commonly referred to as 1350 nm).1000BASE-LX is also based on the FibreChannel signaling specification and uses FC8B/10B physical coding sublayer (PCS).1000BASE-LX is specified for use on eithermultimode or single-mode fiber.
Table 3 shows the Gigabit Ethernet dis-tance specifications for 1000BASE-SX and1000BASE-LX fiber optic media. Note thatthe distance Gigabit Ethernet can reachdepends on the bandwidth (measured in
MHz∗ km)—the greater the bandwidth ofthe fiber, the further the distance supported.It’s also important to note that IEEE speci-fies minimum rather than maximum ranges,and under average operating conditions, theminimum specified distance can beexceeded by a factor of three or four. How-ever, most network managers are conserva-tive when they design networks and use theIEEE specifications as the maximum dis-tances.
• 1000BASE-LH (“LH” for long haul) is amultivendor specification; each vendor has aset of transceivers covering different dis-tances. Although it is not an IEEE standard,many vendors are working to interoperatewith IEEE 1000BASE-LX equipment andare using the gigabit interface connector(GBIC) multivendor specification in orderto provide a common form factor and
77
Table 3. Gigabit Ethernet Distances for Fiber Optic Media
Fiber Diameter Bandwidth Minimum Range
Transceiver (microns) (MHz*km) (meters)
1000BASE-SX MM 62.5 160 2–220
MM 62.5 200 2–275
MM 50 400 2–500
MM 50 500 2–550
1000BASE-LX MM 62.5 500 2–550
MM 50 400 2–550
MM 50 500 2–550
SM 9 NA 2–5000
MM = multimode fiber
SM = single-mode fiber
Table 4. Gigabit Ethernet Distances for 1000BASE-LH Fiber Optic Media
Fiber Diameter Wavelength Minimum Range
Transceiver (microns) (nm) (meters)
1000BASE-LH SM 9 1310 1 km–49 km(Extended distance)
1000BASE-LH SM 9 1550 50 km–100 km(Extended distance)
greater flexibility. Table 4 shows specificwavelength distances recently qualified by3Com; this list will grow as more of theGBIC distances undergoing qualificationtesting are approved. The appearance ofthese long-distance devices has sparked theinterest of carriers in offering Gigabit Ether-net–based MAN (metropolitan area net-work) services. There are implementationcaveats, however. Network planners mustprovide sufficient cabling attenuation toavoid a 40 km transceiver overdriving a 10km transceiver on the receive end.
Copper cabling specifications. There are twocopper cabling specifications:• 1000BASE-CX (“C” for copper) defines
transceivers or PHY layer devices forshielded copper cabling. 1000BASE-CX isintended for short-haul copper connections(25 meters or less) within wiring closets. In1998, no vendor had announced plans todevelop 1000BASE-CX products. Thisphysical interface uses the Fibre Channel8B/10B physical coding sublayer (PCS), adifferent driver on shielded copper, and 150ohm balanced cable, and it supports wiringdistances up to 25 meters. (Note that IBMType 1 cable is specifically mentioned in theIEEE 802.3z standard as not recommended.)
• 1000BASE-T (“T” for twisted pair) is thedeveloping specification for Gigabit Ether-
net over four-pair Category 5 UTP coppercabling for distances up to 100 meters,enabling network managers to build net-works with diameters of 200 meters. Exten-sive signal processing will be required on the1000BASE-TX transceiver itself. The stan-dards effort is scheduled for completion inMarch 1999.
Implementing Gigabit Ethernet in the
Network
Assessing a new technology like Gigabit Ether-net involves many facets, including comparingit to other technologies and evaluating itsimpact on the existing network topology andnetwork equipment. The topics discussed hereare intended to help network managers con-sider both the advantages and limitations ofGigabit Ethernet.
Gigabit Ethernet and Link Aggregation
Link or port aggregation or trunking oftencomes up in discussions of Gigabit Ethernetmigration. Link aggregation is the ability tosupport multiple, point-to-point, parallelactive links between switches or between aswitch and a server. The advantages of linkaggregation are higher bandwidth, redundantlinks, and load sharing.
Link aggregation and Gigabit Ethernetare complementary technologies. If the busi-ness requirement is simply to add more band-
88
Single-mode (SM) fiber cable reduces theradius of the fiber cable core to between 5 and10 microns, which is very close to the order ofa wavelength, so that one single angle or“mode” of light energy passes along the fiber.Because there is a single transmission pathwith single-mode fiber, the light can travellong distances with little loss of signal. Single-mode fiber is commonly used betweenbuildings.
In contrast, multimode fiber has wider cores.Light is reflected along the core at multiple
angles. The light is propagated along multiplepaths, each path with a different length andhence a different time to traverse the fiber.These multiple angles or modes cause thesignal elements to spread out in time.Consequently, distortions occur that limit thedistance over which the integrity of the lightsignal can be maintained. Multimode fiber isthe predominant type of LAN fiber installedwithin buildings and is less expensive thansingle-mode fiber.
Multimode vs. Single-Mode Fiber Cabling
width between devices, and the devices can beupgraded to Gigabit Ethernet, Gigabit Ethernetis the right choice. If the business requirementincludes the need for resilient and redundantlinks with load balancing, then link aggregation(aggregating multiple links into one logicalconnection) is the right choice.
Gigabit Ethernet links can of coursethemselves be aggregated with link aggregationtechnology. The primary application of linkaggregation technology in the near future willbe to build resilient, redundant links betweenLayer 2 and Layer 3 Gigabit Ethernet switches.
Many vendor-specific link aggregationimplementations exist in the industry today.For example, 3Com supports link aggregationfor FDDI, Fast Ethernet, and Gigabit Ethernet.And in keeping with its standards-based solu-tion strategy, 3Com helped initiate the IEEE802.3ad standards effort for link aggregationin 1998.
Optimizing Servers to Handle Gigabit
Ethernet
While Gigabit Ethernet relieves the bottleneckat the server, server environments are not yetfully optimized to handle the entire availablegigabit bandwidth. The good news is thatserver systems and their software and hardwarecomponents are rapidly evolving to ensurethat the server system bandwidth capacity willhandle the Gigabit Ethernet data rate. In addi-tion, Gigabit Ethernet network interface cards(NICs) currently under design will overcomesome of the server system bottlenecks and beoptimized for the upcoming evolution in serverarchitecture. Critical server system improve-ments are described below.
PCI Bus Bandwidth
The PCI bus is the predominant bus in x86platforms, and is also available in some non-x86 systems. Although the original PCI buspresent in earlier server systems had insuffi-cient bandwidth to carry gigabit-speed I/Otraffic, in newer server systems the bus isquickly becoming wider and faster and is nota bottleneck for the Gigabit Ethernet networkconnection.
While Table 5 shows PCI buses support-ing gigabit rates today, the real bandwidth ofthe PCI bus is slightly lower than the maxi-mum bus bandwidth due to the PCI bus over-heads involved. At a minimum, the wider64-bit bus at 33 MHz is required. In full-duplex mode, Gigabit Ethernet provides 2Gbps bandwidth. Therefore, with the faster66 MHz bus speed due out in the next fewmonths, the PCI bus will no longer be thecritical bottleneck in the system.
In a 64-bit 33 MHz PCI server, the PCIbus with its overhead will be slower than thefull-duplex Gigabit Ethernet line rate. To solvethe slowdown, the Gigabit Ethernet NIC usesonboard NIC memory to buffer packets whilethe PCI bus is busy or catching up. In future64-bit 66 MHz PCI servers, the PCI bus willbe faster than the line rate and the faster PCIbus will need to be offloaded before the packetis put on the wire for transmission. The NIC’sonboard memory would then be used for stor-ing the transmit packets. NIC memory canthus have an important impact on perfor-mance. The memory is typically carved intoreceive and transmit buffers. The NIC’sonboard memory also serves to improve per-formance by storing and buffering packets on
99
Table 5. PCI Bus Bandwidth
Maximum Bus
Bus Width Bus Speed Bandwidth When Available
32 bit 33 MHz 1 Gbps Installed servers
64 bit 33 MHz 2 Gbps Now shipping
64 bit 66 MHz 4 Gbps 1999 servers*
* At least one x86 non-Intel chipset server system supports 66 MHz PCI already.
a heavily loaded bus with devices contendingfor access to the bus.
Flow control capability available on someGigabit Ethernet NICs also helps to preventbuffer overflows: flow control signals thesender to pause if the receive buffer is about tooverflow. This controlled slowdown in trans-mission provides better performance overall byeliminating costly timeouts and subsequentretransmissions.
Better techniques for efficiently utilizingthe PCI bus are also being engineered. Thesetechniques and innovations allow the PCINIC to better interact with the bus, andimprove performance and PCI bus utilizationin next-generation servers.
Network Operating System
The history of networking has always beenabout moving bottlenecks to some other partof the network. With the development ofGigabit Ethernet, Ethernet packets travel atspeeds of one billion bits per second. With thePCI bus developments described above, anI/O bus on the end station can support speedsof 1 billion bits per second or faster. The bot-tleneck has now moved from the network toinside the host itself and the challenges toincrease performance are not Gigabit Ethernetissues or even network interface card issues,but fundamental network operating system(NOS) and host issues.
The performance of the server connectiondepends heavily on the network operating sys-tem and underlying protocols. UNIX operat-ing systems appear better adapted to handlingGigabit Ethernet speeds, while the TCP/IPprotocol running under Microsoft NT 4.0 stillhas much room for improvement. TCP/IP is aconnection-oriented and complex protocolthat requires high CPU bandwidth to processpackets at gigabit per second rates.
Because it is a leading networking vendorand the Ethernet pioneer, 3Com is workingclosely with CPU and NOS vendors to pushthe envelope of host performance. With single-processor UNIX implementations, performancehas been measured in the range of 500 to 800Mbps. Industry testing with Windows NT 4.0and the latest Pentium CPU-based server sys-
tems running TCP/IP show network perfor-mance in the range of 600 Mbps. Under Win-dows NT 4.0, throughput will increase withnewer and more powerful Intel processor-based servers. The measured throughput onWindows NT single-processor Pentium sys-tems has increased significantly from earlierPentium servers to reach the current rate ofgreater than 600 Mbps. This throughput con-tinues to increase with advancements in serversystem and processor technology. With theadvent of Windows NT 5.0, these results willimprove even more dramatically.
Under Windows NT 4.0, when greaterCPU power is available to process the TCP/IPstack, server connection performance improves.In general, connection performance alsoimproves for multiprocessor systems. However,the network driver interface specification’s(NDIS’s) single-threading interface limits thebenefits of multi-CPU processing and providesless improvement and processor scalability.Microsoft plans to provide better-performanceNDIS-based implementations in the future.
CPU Utilization at Gigabit Speeds
At gigabit speeds, routine networking taskssuch as TCP/IP checksum calculations caneasily tie up the processor, resulting in 100percent CPU utilization. This leaves no pro-cessing power for running other applications.Therefore, well-designed Gigabit EthernetNICs offload such tasks from the host pro-cessor, performing them in the NIC hardware.These Gigabit Ethernet NICs also consolidateinterrupts, interrupting the CPU less fre-quently and for multiple tasks each time. Thisreduces the number of times the processor hasto save context to service interrupts, andresults in lower CPU utilization for betterapplication performance.
To overcome some of the bottleneckscaused by the host, a few vendors have sup-ported a proprietary solution known as“jumbo frames” to provide a better data pay-load-to-packet overhead ratio. A jumbo framehas a non-Ethernet standard maximum trans-mission unit (MTU), exceeding 1518 bytes.For example, one jumbo frame size supportedis that of 9 Kbytes. Although they provide
1100
some performance benefits, the major draw-back of jumbo frames is that the frame formatitself is proprietary. Any switch or NIC thatsupports jumbo frames cannot interoperatedirectly with other standards-based Ethernetswitches and NICs on the network.
Memory Subsystems
Memory subsystem performance affects over-all server performance and impacts networkconnection throughput. Memory with zero-wait states is typically better suited for servers
since frequent access to the large contiguousdata stores is usually required. While the ini-tial latency will typically be higher for suchmemory, with large contiguous data accesses,the server throughput could reach a high peaksustained level.
Cache memory holds data that is accessedoften or that is expected to be accessed next.This data is available for the NIC much fasterthan data off a disk storage subsystem. Cachememory and system memory help improveperformance by reducing the impact of the
1111
Gigabit Ethernet discussions invariably involvea comparison with Asynchronous TransferMode (ATM) technology. Originally developedby the telephony industry, ATM has been suc-cessfully deployed in campus network andWAN applications. It offers speeds rangingfrom 155 Mbps through 622 Mbps, and soon2.4 Gbps will be available. ATM was designedfor large, scalable, resilient network back-bones. One of its key objectives is to integratetelephony, data, voice, and video traffic ontoone consolidated network for LANs and WANs.
ATM’s explicit Quality of Service (QoS) capa-bility—from the transmitting end station tothe receiving end station over dedicatedvirtual circuits—and its ability to control jitterand limit latency across the network enable itto support real-time traffic. End stations canrequest a specific QoS level from the network,and ATM will deliver that service across acampus or wide area. Indeed, the ability to
deliver QoS and support real-time, interactivetraffic is one of ATM's greatest strengths.
Within the overall enterprise network, ATMand 10, 100, or 1000 Mbps Ethernet tech-nologies can coexist and complement eachother. The enterprise desktop is likely toremain Ethernet-based while the enterpriseWAN will increasingly be based on ATMtransport. From the ATM perspective, the issueof ATM versus Gigabit Ethernet becomes aquestion of how far into the backbone the ATM“cloud” penetrates. From the Ethernet per-spective, the question is how far toward theWAN Ethernet will extend. Where ATM andEthernet meet within the backbone dependson cost and the applications and servicesrequired, as well as on network size, topology,and redundancy requirements. The box belowsummarizes general guidelines on where touse ATM and Gigabit Ethernet technologies inyour enterprise network.
Partnering Gigabit Ethernet with ATM in Enterprise Networks
Use Gigabit Ethernet when:
• Simplicity and cost are high priorities• Seamless connectivity with Ethernet
desktops is critical• 802.1p/Q and RSVP can be leveraged for
traffic prioritization and multimedia
Use ATM when:
• One infrastructure for voice/video/data isdesired
• Meshed topologies for Layer 2 redundancyand load sharing are required
• Seamless LAN/MAN/WAN switching isimportant
Guidelines for Using Gigabit Ethernet and ATM
storage access latencies. While a large amountof cache memory is beneficial to performance,the amount of cache memory optimum forthe server varies based on the frequency andtype of cache accesses. Look for flexibility inthe server’s cache size configuration.
The system bus speed between the pro-cessor and external (L2) cache is also important.The system bus speed (not to be confusedwith the PCI I/O bus speed) has been at 66MHz until recently. In newer servers withprocessors supporting a 100 MHz system bus,the server throughput has improved markedly.
Storage Subsystems
Storage is typically the slowest component ofthe server. It usually uses rotating media tech-nology with mechanical limitations. Fast andWide SCSI, Fibre Channel, and disk arraysbring high performance to this area, but stor-age still remains the slowest link in the serverperformance chain. Fortunately, system mem-ory and cache can also be used to bring higherperformance to storage.
Gigabit Ethernet and Layer 3 Switching
Managing high-performance networks todayinvolves more than the deployment of band-width and capacity. Network managers mustalso build control into the network. Tradition-ally, CPU-based routers have been used in the network to manage and control trafficbetween subnets, isolate faults, control chattyprotocols, build firewalls, and so on. With thedevelopment of Gigabit Ethernet, interfacespeeds have reached speeds of one billion bitsper second or almost 1.5 million packets persecond with 64 byte packets. Traditionalrouters with their general-purpose CPUs androuter code residing in memory cannot keepup with these speeds.
Recent developments in integrated circuittechnology have enabled Layer 3 switches toattain wire-speed packet forwarding throughapplication-specific integrated circuits (ASICs)that embed the Layer 3 routing intelligence inthe switch hardware itself. This firmwareintelligence performs multiple, simultaneousanalysis and routing operations on packets.The Layer 3 switch is optimized for LAN
routing and LAN protocols such as IP, IPX,and AppleTalk.
The Layer 3 switch is able to forward androute traffic at wire speed and make intelligentdecisions about the type of traffic flowingthrough it. Layer 3 switching is actually inter-face independent: it can support 10 or 100Mbps Ethernet, Gigabit Ethernet, FDDI, andATM. Layer 3 switching technology is the keyto successful ultra-high-speed backbone con-nectivity. Like intelligent, high-performanceLayer 2 switches, Layer 3 switches supportflexible deployment options such as portaggregation and virtual LANs. They alsodeliver advanced levels of network manage-ment with extensive built-in Layer 2 RemoteMonitoring (RMON) capabilities, as well ashigher-level RMON2 monitoring capabilities.
In addition, Layer 3 switches can use theirpacket-by-packet filtering capability to sup-port Class of Service (CoS) and Quality ofService (QoS) based on the IEEE 802.1p(now 802.1D) and the 802.1Q standards aswell as the IETF Resource Reservation Proto-col (RSVP) bandwidth reservation scheme.Enterprise networks today need CoS and QoScapabilities to successfully converge differenttypes of application traffic—from traditionalIBM 3270 terminal and network backupapplications to newer transaction processing,streaming video, and LAN telephony applica-tions—onto one backbone infrastructure.
Gigabit Ethernet Migration Scenario
Gigabit Ethernet is best suited for uncloggingnetwork bottlenecks that occur in three mainareas:• Switch-to-switch connections• Connections to high-speed servers• Backbone connections
The following scenario details a networkmigration to Gigabit Ethernet. As shown inFigure 5, the initial building backbone in thenetwork scenario is 10 Mbps Ethernet. SeveralEthernet segments or workgroups are col-lapsed into a 10/100 Mbps switch, which inturn has several 10 Mbps Ethernet server con-nections. In the scenario:• Power users are experiencing bottlenecks
from their 10 Mbps links
1122
• Users on the shared segments are experienc-ing slow response times
• All new desktops are equipped with 10/100PCI cards
Figure 6 on page 14 implements the firstupgrade phase in three areas: (1) upgradingthe network backbone to 100 Mbps Fast Eth-ernet, (2) upgrading the power workgroup to100 Mbps Fast Ethernet connections betweenthe end stations and the switch, and (3)implementing 10 Mbps switching in otherworkgroups that need dedicated bandwidth.The result? Power users immediately acquire100 Mbps connection speeds, and other work-groups that need it enjoy dedicated 10 Mbpsbandwidth while the investment in existingswitches and NICs is preserved. The speed ofthe backbone also increases tenfold to accom-modate the overall increase in network band-width demand.
Figure 7 on page 14 shows upgrading theriser downlinks to Gigabit Ethernet to increasebackbone bandwidth even further. Theswitches in the wiring closets that supportpower users or large workgroups are upgradedwith Gigabit Ethernet downlink modules, thebasement switch is upgraded to 100/1000capability, and key servers are upgraded withGigabit Ethernet NICs. A key point to notice
is that the upgrades of the backbone to Giga-bit Ethernet support the extension of switch-ing to the edges. Both office workgroups nowsupport switched 10 or switched 10/100 Eth-ernet to the desktops.
Also in Figure 7, the Fast Ethernet back-bone switch that aggregates multiple 10/100switches is upgraded to a Gigabit Ethernetswitch, supporting multiple 100/1000switches. Gigabit full-duplex repeaters areinstalled as needed to aggregate servers orbuild Gigabit Ethernet support for work-groups that manipulate very large multimediaand graphics files. In addition, once the back-bone is upgraded to a Gigabit Ethernetswitch, high-performance server farms can beconnected directly to the backbone with Giga-bit Ethernet NICs, increasing throughput tothe servers for users with high-bandwidthapplication files or for bandwidth-intensivedata warehousing and backup operations.Overall, the network now supports a greaternumber of segments and more bandwidth persegment.
Summary
Gigabit Ethernet is an IEEE-standard technol-ogy that preserves Ethernet's simplicity, lever-ages its cost drivers, and takes advantage of the
1133
10/100 Mbps
switch
10 Mbps downlinks
10 Mbps server
connectionsShared 10 Mbps desktops
Shared and switched
10 Mbps Ethernet
“Power” workgroupSwitched 10 Mbps desktops
Office workgroup
Office workgroup
Data center
10 Mbps
Figure 5. Scaling Network Capacity: Starting Point
installed base of Ethernet devices. It also takesadvantage of the familiarity, knowledge, andexpertise network managers have with bothEthernet- and IP-based traffic and services.The deployment of Gigabit Ethernet in thenetwork increases the ability of network plan-ners and managers to implement moreswitched Ethernet and switched Fast Ethernet
to the horizontal wiring closets and serversand out to the desktops.
Gigabit Ethernet can be implementedincrementally into an enterprise network,replacing Fast Ethernet core switches andenabling the deployment of more switched 10and switched 10/100 Mbps Ethernet to theedge of the network. 1000BASE-SX supports
1144
10/100 Mbps
switch
100 Mbps downlinks
10/100 Mbps servers
Switched 10 Mbps desktops
Shared and switched
10 Mbps Ethernet
“Power” workgroupSwitched 10/100 Mbps desktops
Office workgroup
Data center
Office workgroup10 Mbps
100 Mbps
100/1000 Mbps
switch
Gigabit downlinks
1000 and 100 Mbps servers
Switched 10 Mbps desktops
Switched 10/100 Mbps
Ethernet
“Power” workgroupSwitched 10/100 Mbps desktops
Office workgroup
Data center
Office workgroup10 Mbps
100 Mbps
1000 Mbps
Figure 6. Scaling Network Capacity: Phase 1
Figure 7. Scaling Network Capacity: Phase 2
multimode fiber up to 550 meters, while1000BASE-LX supports multimode up to 550meters and single-mode fiber up to 5000meters. Products based on 1000BASE-SX and1000BASE-LX are shipping in volume today.The 1000BASE-T standard to support Giga-bit Ethernet operation on the installed base ofCategory 5 UTP copper cabling is targeted forcompletion by March 1999. Hardware andsoftware server system and server NIC devel-opment is well under way to ensure that theserver system bandwidth capacity will handleGigabit Ethernet data rates, an issue of realconcern in Gigabit Ethernet implementation.Layer 3 switching technology adds the criticalcomponent of “brain” to Gigabit Ethenet’s“brawn.”
3Com Corporation is the leader in Giga-bit Ethernet development and is committed to
the adoption of standards-based Gigabit Eth-ernet in the marketplace. The company is afounding member of the Gigabit EthernetAlliance, the main industry consortium com-mitted to accelerating the Gigabit Ethernetstandards process, end-user education, andmultivendor interoperability. 3Com is also afounding member of the Gigabit EthernetConsortium at the Interoperability Lab at theUniversity of New Hampshire, which devel-ops test suites and conducts group and indi-vidual product testing. 3Com offers theindustry’s most complete Gigabit Ethernetsolution for medium to large enterprises—from Gigabit Ethernet server NICs and full-duplex repeaters to workgroup and core Layer 3 Gigabit Ethernet switches.
1155
1 3Com Gigabit Ethernet solutions Web page,www.3com.com/technology/gigabit.html
2 Layer 3 Switching: An Introduction whitepaper, www.3com.com/technology/tech_net/white_papers/50066001.html
3 3Com Link Aggregation white paper,www.3com.com/technology/tech_net/white_papers/500666.html
4 Gigabit Ethernet Test Consortium Web site,www.iol.unh.edu/consortiums/ge/
5 IEEE 803.3z Task Force Web site,grouper.ieee.org/pub/802_main/802.3/gigabit/
6 Gigabit Ethernet Alliance Web site,www.gigabit-ethernet.org/technology/whitepapers
For Further Reference
3Com CorporationP.O. Box 581455400 Bayfront PlazaSanta Clara, CA 95052-8145Phone: 1 800 NET 3Comor 1 408 326 5000
Fax: 1 408 326 5001World Wide Web:www.3com.com
Asia Pacific Rim
Sydney, Australia Phone: 61 2 9937 5000Fax: 61 2 9956 6247
Melbourne, AustraliaPhone: 61 3 9934 8888Fax: 61 3 9934 8880
Beijing, ChinaPhone: 86 10 68492 568Fax: 86 10 68492 789
Shanghai, ChinaPhone: 86 21 6350 1581Fax: 86 21 6350 1531
Hong KongPhone: 852 2501 1111Fax: 852 2537 1149
IndiaPhone: 91 11 644 3974Fax: 91 11 623 3192
IndonesiaPhone: 62 21 572 2088Fax: 62 21 572 2089
Osaka, JapanPhone: 81 6 536 3303Fax: 81 6 536 3304
Tokyo, JapanPhone: 81 3 5977 3266Phone: 0120 313266(toll free from Japan)
Fax: 81 3 5977 3370
KoreaPhone: 82 2 3455 6300Fax: 82 2 319 4710
MalaysiaPhone: 60 3 715 1333Fax: 60 3 715 2333
New ZealandPhone: 64 9 366 9138Fax: 64 9 366 9139
PhilippinesPhone: 632 892 4476Fax: 632 811 5493
SingaporePhone: 65 538 9368Fax: 65 538 9369
TaiwanPhone: 886 2 2 377 5850Fax: 886 2 2 377 5860
ThailandPhone: 662 231 8151 5Fax: 662 231 8158
3Com Austria
Phone: 43 1 580 17 0Fax: 43 1 580 17 20
3Com Benelux B.V.
BelgiumPhone: 32 2 725 0202Fax: 32 2 720 1211
NetherlandsPhone: 31 346 58 62 11Fax: 31 346 58 62 22
3Com Canada
CalgaryPhone: 1 403 265 3266Fax: 1 403 265 3268
EdmontonPhone: 1 403 423 3266Fax: 1 403 423 2368
MontrealPhone: 1 514 683 3266Fax: 1 514 683 5122
OttawaPhone: 1 613 566 7055Fax: 1 613 233 9527
TorontoPhone: 1 416 498 3266Fax: 1 416 498 1262
VancouverPhone: 1 604 434 3266Fax: 1 604 434 3264
3Com Eastern Europe/CIS
BulgariaPhone: 359 2 962 5222Fax: 359 2 962 4322
Czech/Slovak RepublicsPhone: 420 2 21845 800Fax: 420 2 21845 811
HungaryPhone: 36 1 250 8341Fax: 36 1 250 8347
PolandPhone: 48 22 6451351Fax: 48 22 6451352
RussiaPhone: 7 095 258 09 40Fax: 7 095 258 09 41
3Com France
Phone: 33 1 69 86 68 00 Fax: 33 1 69 07 11 54 Carrier and Client Access
Phone: 33 1 41 97 46 00Fax: 33 1 49 07 03 43
3Com GmbH
Munich, Germany Phone: 49 89 627320Fax: 49 89 62732233
3Com Iberia
PortugalPhone: 351 1 3404505Fax: 351 1 3404575
SpainPhone: 34 1 509 69 00Fax: 34 1 307 79 82
3Com Latin America
U.S. HeadquartersPhone: 1 408 326 2093Fax: 1 408 764 5730
Miami, FloridaPhone: 1 305 461 8400Fax: 1 305 461 8401
ArgentinaPhone: 54 1 312 3266Fax: 54 1 314 3329
Brazil Phone: 55 11 246 5001Fax: 55 11 246 3444
Chile (serving Chile, Bolivia, andPeru)Phone: 562 240 6200Fax: 562 6240 6231
ColombiaPhone: 57 1 629 4847Fax: 57 1 629 4503
MexicoPhone: 52 5 520 7841/7847Fax: 52 5 520 7837
PeruPhone: 51 1 221 5399Fax: 51 1 221 5499
VenezuelaPhone: 58 2 953 8122Fax: 58 2 953 9686
3Com Mediterraneo
Milan, ItalyPhone: 39 02 253011Fax: 39 02 27304244
Rome, ItalyPhone: 39 6 5279941Fax: 39 6 52799423
3Com Middle East
Phone: 971 4 319533Fax: 971 4 316766
3Com Nordic AB
DenmarkPhone: 45 48 10 50 00Fax: 45 48 10 50 50
FinlandPhone: 358 9 435 420 67Fax: 358 9 455 51 66
NorwayPhone: 47 22 58 47 00Fax: 47 22 58 47 01
SwedenPhone: 46 8 587 05 600Fax: 46 8 587 05 601
3Com Southern Africa
Phone: 27 11 807 4397Fax: 27 11 803 7405
3Com Switzerland
Phone: 41 844 833 933Fax: 41 844 833 934
3Com UK Ltd.
EdinburghPhone: 44 131 240 2900Fax: 44 131 240 2903
IrelandPhone: 353 1 820 7077Fax: 353 1 820 7101
ManchesterPhone: 44 161 873 7717Fax: 44 161 873 8053
Winersh Phone: 44 1189 27 8200Fax: 44 118 969 5555
3Com Corporation enables individuals and organizations worldwide to communicate and shareinformation and resources anytime, anywhere. As one of the world’s preeminent suppliers of data,voice, and video communications technology, 3Com has delivered networking solutions to morethan 200 million customers worldwide. The company provides large enterprises, small and mediumenterprises, carriers and network service providers, and consumers with comprehensive, innovativeinformation access products and system solutions for building intelligent, reliable, and high-performance local and wide area networks.
To learn more about 3Com products and services, visit our World Wide Web site at www.3com.com. 3Com is a publicly traded corporation (Nasdaq: COMS).
© 1998 3Com Corporation. All rights reserved. 3Com, the 3Com logo, and SuperStack are registered trademarks of 3Com Corporation. More connected. is a trademark of 3Com Corporation. AppleTalk is atrademark of Apple Computer. Pentium is a trademark of Intel. Windows and Windows NT are trademarks of Microsoft. IPX is a trademark of Novell. Other brand or product names may be trademarks or regis -tered trademarks of their respective owners. All specifications are subject to change without notice.
Printed in U.S.A. on recycled paper 503003-001 10/98