InfiniBand ArchitectureTransforming I/O TechnologyAn IDC White PaperAnalysts: Jean S. Bozman and Vernon TurnerJanuary 2001InfiniBand Technology OverviewWhat is InfiniBand Architecture?InfiniBand architecture is a new I/O specification that will start to shipinawidevarietyofserverandinterconnectproductsduring2001.InfiniBand architecture is a well-described technology with a final speci-ficationthatwaspreparedbytheInfiniBandTradeAssociation (IBTA)inthefallof2000.Morethan215membercompaniesparticipateinthe IBTA and will work together to see that InfiniBand architecture isestablished as an industry standard.InfiniBandarchitecturespoint-to-pointlinkingtechnologywillbeusedasthebasisforanI/Ofabricthatwillincreasetheaggregatedata rate between servers and storage devices. This capability will allowserver vendors to replace outmoded system buses and to multiply totalI/Otrafficbyafactoroftwotofourtimesormorecomparedwithconventional system buses.More important, the I/O fabric of InfiniBand architecture will take on arolesimilartothatofthetraditional,mainframe-basedchannelarchi-tecture,whichusedpoint-to-pointcablingtechnologytomaximizeoverall I/O throughput by handling multiple I/O streams simultaneous-ly.ThatmeansthatI/Oneednolongerbeabottleneckinimprovingoverall data throughput for server systems.The Internet Will Drive InfiniBand Architecture Adoption IDCbelievesthattheInternetsN-tierarchitecturesarebeingdrivenby the demands of Internet-driven data processing. By arranging manysmall servers in Web server farms or in clusters, IT managers have theabilitytoaddmoreserversasneeded.Thistypeofpay-as-you-growbuild-outofcomputinginfrastructureisnecessarybecauseuserdemand for Internet data services ebbs and peaks unpredictably. OneofthemostsignificantobstaclestoovercomeintheN-tierarchitectures is that of interoperability created by multiple proprietaryarchitectures. By working from a universal standard, InfiniBand raises5 Speen Street Framingham, MA 01701 Phone (508) 872-8200 Fax (508) 935-4015the potential of seamless connectivity across the server landscape, giv-ingrisetoimprovedservicelevels,productdifferentiation,and consistent platform management.InfiniBandtechnologywasdesignedtomeettheemergingrequire-ments of the Internet age in the following ways:It acknowledges that there are very fast processors capable of 1GHzor more, which will push more data than most conventional systembuses can handle. The result is a technology mismatch in whichthe fast processor has no way to push multiplying data packets tothese nearby growing storage resources.It will accommodate Internet data types, such as streaming video,multimedia,andhigh-resolutiongraphics,andwilldrivemoredata traffic than traditional, business-oriented data processing.InfiniBandlinks,combinedintoanI/Ofabric,willhelpto alleviate potential bottlenecks inside servers.InfiniBandarchitecturewillimprovelinkspeedsbetweenserversand storage, including storage area networks (SANs).InfiniBand architecture can be used as an interconnect technologythatwilllinkindividualserversintoclustersforpurposesofhighavailability, scalability, and improved manageability.How InfiniBand Technology WorksInfiniBandtechnologyworksbyconnectinghost-channeladapters(HCAs)totargetchanneladapters(TCAs).TheHCAstendtobelocatedneartheserversCPUsandmemory,whiletheTCAstendtobelocatednearthesystemsstorageandperipherals.Aswitchislocated between the HCAs and the TCAs, directing data packets tothe correct TCA destination based on information that is bundled intothe data packets themselves (see Figure 1).The glue between the HCA and TCA is the InfiniBand switch, whichallowsthelinkstocreateauniformfabricenvironment.Oneofthekey points of this switch is that it will allow packets of information (ordata)tobemanagedbasedonvariables,suchasservicelevelagree-ments and a destination identifier.InfiniBand architecture has the potential to play a role in transformingor morphing computer architectures. Where once there were general-purposeservers,includingCPUmemoryandstorage,therecanbedenselypackedapplication-specificserversorbladesthatcanbechangedatwill.Atthesametime,storagecanbeseparatedfromthe 2 Copyright 2001 IDC. Reproduction without written permission is completely forbidden.External Publication of IDC Information and DataAny IDC information that is to be used in advertising, press releases, or promotional materials requiresprior written approval from the appropriate IDC Vice President or Country Manager. A draft of the proposed document should accompany any suchrequest. IDC reserves the right to deny approval of external usage for any reason.InfiniBand Architecture:Transforming I/O TechnologyPrinted onrecycledmaterialsA IDC 3 A IDCInfiniBand Architecture:Transforming I/O TechnologyCPU/memorycomponentsoftheseapplication-specificservers,andextremely large storage resources can be managed centrally by IT staffers. This transformation can be expected to start in server farms that areused as front-end tier 1 and tier 2 servers that handle Web serving andapplication serving (see Figure 2).Servervendorswillbeabletopacksuchserversclosertogetherby movingtheI/Osubsystemsandstorageandperipheralstoalocationoutsidethetraditionalserverbox.Thiswillpromotehigh-density server configurations in IT sites and in service provider (SP) data centers.Theresultcouldwellbeanevolutionfromtodayshighlydenserack-optimizedserversintoahighlydensearrayofhighlymodularserverappliancesorserverblades.Thesebladeswillbedesignedfora particularfunction,orsolution,suchascaching,email,security,orWebserving.Ifabladebreaksorfails,itcanberapidlyreplaced,andworkloads can be routed around it until it is repaired.Flexibleconfigurationsoftier1andtier2servers,linkedtogether(throughanInfiniBandfabric)byInfiniBandlinks,willpromotethereallocationofcomputingresourcesinfast-growing,Internetcomput-inginfrastructure.Further,InfiniBandlinkscanbeadaptedbyusingthree different link speeds. 1X, 4X, and 12X links are planned for ship-ment by system and interconnect vendors.Clusteringofserversinallthreetierstier1sWebservers,tier2sapplication servers, and tier 3s database servers can be enhanced byInfiniBandlinks.TheInfiniBandarchitectureI/Ofabric,madeupofTargetHost InterconnectCPUCPUTargetTCAHCAMemCntirSysMemTCANetwork or IBRouterSwitchRouterIB Link IB LinkIB LinkIB LinkIB LinkFigure 1The InfiniBand Architecture ModelSource: InfiniBand Trade Association 4 A IDCmultiple1X,4X,or12Xstrands,canbeusedasanewhigh-speedinterconnectinclusteredserverconfigurations.Theseclusterscanbedeployedinseveralways:Theycansupporthigh-availabilityfailover,scalability of workloads, or improved manageability of multiple serversfrom a single console. Finally, scalable server systems will likely use InfiniBand links insidetheserversystemitself.However,thedesigncycleformidrangeandhigh-endserversystemsisseveralyears.Verylikely,InfiniBandarchi-tecture adoption will be widespread in entry servers, in clusters, and inconnectionstoSANsbythetimethatInfiniBandarchitectureisamajorbuildingblockformidrangesymmetricalmultiprocessing(SMP) and cc:NUMA server systems.InfiniBand Architecture Value PropositionWhy Consider Another Interconnect?IDCbelievesInfiniBandfabricswillcoexistwithawidevarietyofexistingI/Ostandardsthatarealreadywidelydeployedinusersites.Often, multiple architectures do and can exist together and help makeeachotherbetter.ExistingarchitecturesincludePCI,Ethernet,andFibreChannel.IDCalsoexpectsInfiniBandfabricstocoexistwith follow-on I/O standards, including PCI-X, Gbit Ethernet, and 10GbitEthernet.Yet,InfiniBandarchitectureoffersanewapproachtoI/OefficiencybyreplacingthetraditionalsystembuswithanI/Ofabricthat supports parallel data transfers along multiple I/O links.InfiniBand Architecture:Transforming I/O TechnologyWeb servers/appliancesApplicationserver tierDatabase server tierROUTERFIREWALLInternet/IntranetFigure 2N-Tier ArchitectureSource: IDC and Dell, 2001 5 A IDCInfiniBand Architecture:Transforming I/O TechnologyHowever,theaspectsofInfiniBandarchitecturethatwilllikelycausemigrationawayfromthesewell-establishedI/Ostandardsarethe following:InfiniBandarchitectureoffloadsCPUcyclesforI/Oprocess-ing,itdeliversfastermemorypipesandhigheraggregatedata-transferrates,andtheInfiniBandfabricwillreducemanagementoverheadforthe server system.Unmet Needs that InfiniBand Architecture AddressesIn 2000, the server market saw the explosive introduction of applianceservers and rack-optimized servers into the entry server market (serverscostinglessthan$100,000).Manyofthegeneral-purposeserversandpedestal-basedserverswerebeingplacedintoracksineverynew Internetdatacenter.Earlierwediscussedthisphenomenonaspartofthemodularizationofthedatacenter.However,aswemigratetosmaller and denser servers (1U form factor or smaller), real estate issueson the server motherboard, heat and power dissipation, and consump-tion combined with sheer connectivity may inhibit the potential of theblade technology. InfiniBand fabrics have the potential to address all ofthese needs in a simple manner.InfiniBand Architectures Performance CharacteristicsInfiniBand architectures role will depend largely on the workload that itsserver,storage,andnetworkareattemptingtosupport.Forexample, iftheserverissimplyafront-enddevice,aWebhostingserver,orfile andprintappliance,thenthe1Xlinkshouldbesufficienttoprovideboththroughputandconnectivity.MostoftheInternettier1serverplatforms would be met by this link. The 4X link may manifest itself in the tier 2 environment. Here serversarelikelytobemainly2-wayand4-wayprocessorsthatneedtorunlarger workloads where some latency issues can be tolerated. However,manyoftheseworkl oadsbegi ntotakeonmi ssi on-cri ti cal ,service-level characteristics.Finally, the 12X link could be used for transaction processing and data-base management, where almost no latency is tolerated as features suchas two-phase commits are made to keep the data up-to-date. Given theseworkloadcharacteristics,the12Xlinkswouldappearinthehigher endof the tier 2 and tier 3 server architectures.Inaddition,IDCbelievesthatanydata-intensiveorheavytransactionprocessingthatisperformedoutsidetheboundsofalocaldatacenterenvironment would be ideal candidates for the 12X links. The additionalbandwidthandI/Oaggregationcouldbehandledbythistop-end switch.Cost ImplicationsThe impact on the cost of interconnect technology may be significant.As with any technology that becomes a standard and ultimately obtains 6 A IDCa ubiquitous role in the market place, InfiniBand architecture probablywilldrivedownthecostsofserverinterconnect,bridges,andswitches.IDC believes that vendors that can balance this decline with added valueintheirproducts(suchashigher-availabilitysolutions,software,andmanagement stacks) will maintain or increase their current margins.However,itisclearbythenumberofnewandemergingInfiniBandcompanies that the overall cost of the server technology will continueto decline. IDC can cite previous examples within the server industrywheretechnologydisruptionscreatedbettervaluefortheconsumer(e.g.,mainframetechnologymovingfromwater-cooledbipolartech-nology to air-cooled CMOS chipsets).Market Status and Potential: An Adoption PlanIDCs InfiniBand Architecture Server ForecastIDC has attempted to size the server market opportunity for InfiniBandarchitecture(seeTable1).Indoingso,itevaluatedthegreatestseg-mentsoftheservermarketwhereInfiniBandcanbemorereadilydeployed.Ithasidentifiedthattheentryservermarket(whichaccountsformorethan90%ofallserverunitshipments)asbeingmostacceptabletoInfiniBandtechnology.ItfurtherconstrainedthisforecastbyadvisingthereaderthatIDCsapplianceserverforecastisnot integrated into this existing forecast. (Note: IDC forecasts that in2004,theapplianceservermarketmayexceed2millionadditionalunits, or approximately 22% of the server market).InfiniBand Architecture:Transforming I/O TechnologyTable 1Worldwide InfiniBand-Capable Servers Forecast (20002004)Source: IDC, 2001Price Band 2000 2001 2002 2003 2004< $10K 0 39,744706,7531,651,3693,972,127 $1024.9K 0 4,362129,848290,189668,548 $2549.9K 0 49323,54572,144162,015 $5099.9K 0 828,58828,85068,914 Total 0 44,681868,7342,042,5534,871,604 Key Assumptions:Appliance server market is not included in the analysis.IA32, IA64, RISC, and CISC chip architectures are included.Messages in the Data:The greatest InfiniBand architecture opportunity lies in the