wp dc best prac infrastructure

When designing and laying out a data center, understanding best practices as well the pros and cons foreach type of data center is critical. The TIA 942 data center guidelines are very specific that horizon-

tal and vertical cabling should be run accommodating growth, so that these areas do not have to be revisited. It is also specific about equipment not being directly connected unless it is specifically required by the manufac-turer. This is in line with other standards documents such as ANSI/TIA/EIA 568-B that design for open systems architecture. So the question is raised: what is the best way to do this for a 10Gb/s environment?

I N F R A S T R U C T U R E D E S I G N

CONNECTING THE WORLD TO A HIGHER STANDARD

WW WW WW .. SS II EE MM OO NN .. CC OO MM

Best Practices for Data CenterInfrastructure DesignCabling Distances and Space Planning




There are considerations outside of the cable plant and number of connectors alone: usability, scalability,costs and the ability to perform Moves, Adds and Changes (MAC’s). Additionally, some limitations existbased on the category of the cabling system. Copper and fiber distances may vary with the type of cablingsystem selected. We will discuss some of those parameters and their potential impact on data center designs.

All copper channels are based on a worst case, 100 meter, 4 connector model. ISO/IEC 24764 (draft), TIA-942, ISO/IEC 11801 Ed2.0 and recommendations from electronics manufacturers suggest that the fixedhorizontal portion of the channel be a minimum of 15m (50 ft.). While some shorter lengths may be support-ed in other portions of the channels, there is a requirement in zone distribution and consolidation points forthis minimum distance. When moving to 10Gb/s electronics, the 15m minimum will likely exist for all hori-zontal cables due to recommendations from electronics manufacturers and that all models within IEEE arebased on a minimum 15m distance.

The 15m length is also dictated by signal strength issues as your signal is strongest in those first 15m. Thiscan create issues with two connectors in close proximity. By providing at least 15m to the first connectionpoint in the channel, you are allowing the attenuation to reduce the signal strength at the receiver or betweencomponents. In order to achieve the 15 m distance, two options exist: either provide space in the pathwayto take up the distance or create service loops under the floor. Service loops should not be a loop, but rathera loosely configured figure 8 for UTP systems. However, this configuration is not a requirement for F/FUTPor S/FTP systems due to their inherent resistance to interference. Bear in mind that the additional cable will consume more pathway space.

Copper distances for category 6A twisted pair cabling are limited to 100m for all channels. 10GBASE-T running on category 6/class E cabling will be limited to less than 37m depending upon the scope of poten-tial mitigation practices to control alien crosstalk. It should be noted that the purpose of TSB 155 is to pro-vide parameters for the qualification of existing Cat 6/Class E applications for use of 10GBASE-T, TSB 155should not be used for designing new installations.

Fiber channel lengths vary based on the grade and type of fiber and type of interface. Understanding theselimitations will assist in the design and layout of the data center space. If you are utilizing 10GBASE-CX4or Infiniband, you are distance limited to a maximum of 15m. The following chart summarizes the distancesfor all 10Gb/s applications and their associated cabling systems.

2



3


Application Media Classification Max. Distance Wavelength10GBASE-T Twisted Pair Copper Category 6/Class E UTP up to 55m*10GBASE-T Twisted Pair Copper Category 6A/Class EA UTP 100m

10GBASE-T Twisted Pair Copper Category 6A/Class EA F/UTP 100m

10GBASE-T Twisted Pair Copper Class F/Class FA 100m

10GBASE-CX4 Manufactured N/A 10-15m10GBASE-SX 62.5 MMF 160/500 28m 850nm 10GBASE-SX 62.5 MMF 200/500 28m 850nm 10GBASE-SX 50 MMF 500/500 86m 850nm 10GBASE-SX 50 MMF 2000/500 300m 850nm 10GBASE-LX SMF 10km 1310nm10GBASE-EX SMF 40km 1550nm10GBASE-LRM All MMF 220m 1300nm10GBASE-LX4 All MMF 300m 1310nm10GBASE-LX4 SMF 10km 1310nm

THE LAYOUT...WHERE AND HOW TO CONNECT

When designing a cabling infrastructure, too often cost is the deciding characteristic of the channel selected. However, once all elements are considered, a design with higher initial cost may have a lower overall cost of ownershipto a company that has a lot of MAC activity. The most important concern is that designers are familiar with all aspects ofthe different configurations available to make the best selection possible. A listing of cost, flexibility and performance islisted below.

Model Cost Flexibility Performance2-Connector Lowest Lowest Highest3-Connector with CP Medium Medium Medium3-Connector with CC Medium Medium Medium4-Connector Highest Highest Lowest

* As defined in 802.3an




SPACE PLANNING OPTIONS

The MDA (Main Distribution Area) is considered the core of the data center. Connectivity will be needed tosupport the HDA (Horizontal Distribution Area). Following TIA-942 recommendations and utilizing EDA(Equipment Distribution Areas) and ZDA (Zone Distribution Areas), we would like to present four designoptions for consideration.

OPTION ONE

Option One is to run all fibers and copper from the core horizontal distribution areas and equipment distri-bution areas to a central patching area equipped with patch panels. This provides one central area for patch-ing all channels.

There are several benefits to this design. First, all cabinets can remain locked. As patching is done in a cen-tral area – there is no need to enter a cabinet at any time unless there is an actual hardware change. Forindustries that are governed by compliance and security related issues, this may provide a greater benefitby reducing physical access to connections. Intelligent patching can be added to the patching field toincrease security by automatically monitoring and tracking moves, adds and changes in that environment.

4

MDACore & SAN

HDALAN

Switches

HDALAN

Switches

HDALAN

Switches

EDAServers

EDAServers

EDAServers

FiberPatching

Area

CopperPatching

Area

CopperPatching

Area

Provides Any to All connectivity.Patch cord changes in the

patching area can connect anydevice to any device.

Option 1

EDAServers

EDAServers

EDAServers

Another advantage is that all ports purchased foractive gear can be utilized. With the ability to useVLANs, networks can be segmented as needed.

In other scenarios, entire switch blades are likelydedicated to a cabinet of servers. However, if thereare insuffcient server NICs to utilize all ports, thenthe idle ports become costly and inefficient. For instance, if a 48 port blade was dedicated to acabinet at location XY12 but there was only 6servers with two connections each, 36 ports werepaid for but remain idle. By utilizing a centralpatching field, the additional 36 ports can be usedas needed elsewhere in the network, thereby lower-ing equipment and maintenance costs which are farmore expensive than the cable channels.

Note: Black lines are fiber Blue lines are Copper



OPTION TWO

Option Two is to place patch panels in server cabinets that correspond directly to their counterparts in theswitch cabinets. In this scenario, switch blades/ports will be dedicated to server cabinets. This may be eas-ier from a networking perspective, but may not provide the best usage of all ports in the active electronics.Extra ports can be used as spares or simply for future growth. However, if an enterprise is planning to imple-ment blade technology where server density may decrease per cabinet, this may not be a cost effectiveoption.

For the switch cabinets, the type of copper cabling chosen will be a significant factor due to the increasedUTP cable diameters required to support 10GBASE-T. In reality, cabinets and cabling (both copper and fiber)are changed far less frequently than the active electronics. But with the new category 6A UTP cable‘s max-imum diameter of .354 inches (9.1mm), pathways within the cabinets may not provide enough room to routecable and still provide the structural stability necessary. It is always recommended that fill percentage cal-culations be addressed with the cabinet manufacturer. Moving the patch panels to adjacent locations orimplementing a lower switch density may be required. While moving switches into open racks with adja-cent patch panels provides a solution, this is only recommended if proper access security processes exist andsome form of intelligent patching or other monitoring system is used so that network administrators can benotified immediately of any attempt to access switch ports.

5


MDACore & SAN

HDALAN

Switches

HDALAN

Switches

HDALAN

Switches

EDAServers

EDAServers

EDAServers

One to One patching for eachport. Least flexibility

Option 2

EDAServers

EDAServers

EDAServers





OPTION THREE

Option Three consists of providing consolidation points for connections. These can be either connectingblocks or patch panels. This allows for a zoned cabling approach, but may lead to higher moves, adds andchanges costs. It is also difficult to design within the parameters of a 4 connector channel when using Zonedistribution

The other disadvantage to the consolidation point model is that the changes take more time than swappinga patch cord if the pair count changes. Depending on the location of the consolidation point, there maybe additional risks from loss of static pressure under the floor when removing floor tiles ending up with morethan 4 connectors in a channel, or harming existing channels during changes.

6

MDACore & SAN

HDALAN

Switches

HDALAN

Switches

HDALAN

Switches

EDAServers

EDAServers

EDAServers

Consolidation Points (must be15m min. from horizontal patchpanels). Can be patched fromany CP to any server cabinet.

Option 3

EDAServers

EDAServers

EDAServers

CP CP CP




7


OPTION FOUR

A final option is to have all server cabinets and switch cabinetsin a row ,terminating to a single patching field for the row,rather than to a central location. Core connections from theMDA are brought into this patching field. This option can workwell in ISP or other environments where cross department/cus-tomer functionality is not desirable or tolerated. This optionprovides a bit of best of both worlds in that there will be somespare ports, but also the floor tiles will not have to be lifted toperform MAC work. While this is very similar to the firstoption, the segmentation can make it easier for network admin-istrators and physical plant technicians to coordinate efforts.Additionally this style of design provides for flexibility in theever changing environment of shrinking and expanding storage/networking requirements over time.

MDACore & SAN

HDALAN Switches

EDAServers

EDAServers

Copper /FiberPatching Area

For Row

Option 4

All Patching donewithin respective rows.

CONCLUSIONWhichever cabling choice or space option is made, the keystep is planning. Siemon has resources to assist in the layoutand planning or just as a second pair of eyes for any project.For more information, and additional resources go towww.siemon.com.

ABOUT THE AUTHOR

Carrie Higbie has been involved in the computing and networking for 25+ years in executive and consultantroles. She is Siemon’s Global Network Applications Manager supporting end-users and active electronics man-ufacturers. She publishes columns and speaks at industry events globally. Carrie is an expert on TechTarget’sSearchNetworking, SearchVoIP, and SearchDataCenters and authors columns for these and SearchCIO andSearchMobile forums and is on the board of advisors. She is on the Board of Directors and former Presidentof the BladeSystems Alliance. She participates in IEEE, the Ethernet Alliance and IDC Enterprise Expert Panels.She has one telecommunications patent and one pending.

Note: Black lines are fiber, Blue lines are Copper

W O R L D W I D E L O C A T I O N S



© 2

008

Siem

on W

P-In

frastr

uctu

reD

esig

n_Re

v.C

1/0

8

The AmericasSiemon – North America101 Siemon Company DriveWatertown, CT 06795-0400 USATel: (1) 866 474 1197Customer Service Direct:Tel: (1) 866 548 5814 (US)Tel: (1) 888 425 6165 (Canada)Fax: (1) 860 945 [email protected]

Siemon – VenezuelaCalle Veracruz,Torre Orinoco Piso 2, Oficina 2-CLas MercedesCaracas, VenezuelaTel: (58) 212 992 5884Fax: (58) 212 993 [email protected]

Siemon – CASACentral & South AmericaCalle.77 No.11-19,Oficina 601 Edificio Torre 77Bogota, ColombiaPhone: + 011-571-317-2121 Fax: [email protected]

Siemon – MexicoBlvd. Manuel Avila Camacho No. 2900-502Fracc. Los Pirules, TlalnepantlaEdo.de Mexico, C.P. 54040MexicoTel: (52) 55 5370 6100Fax: (52) 55 5370 [email protected]

Europe, Middle East, and Africa

Siemon – FranceParis Axe FranceZAC Paris Rive Gauche118-122 Avenue de France75013 Paris, FranceTel: (33) 1 46 46 11 85Fax: (33) 1 46 46 10 [email protected]

Siemon – GermanyMainzer Landstrasse 1660325 FrankfurtGermanyTel: (49) (0) 69 97168 184Fax: (49) (0) 69 97168 [email protected]

Siemon – ItalyVia Senigallia 18/220161 MilanoItalyTel: (39) 02 64 672 209Fax: (39) 02 64 672 [email protected]

Asia Pacific

Siemon – China (Chengdu)Rm. 1209-1210Western China Business TowerNo. 19,4 Section, Renminan RoadChengdu, Sichuan 610041, P.R. ChinaTel: (86) 28 6680 1100Fax: (86) 28 6680 1096

Siemon – Australia (Sydney)Unit 3A, 10 Rodborough RoadPO Box 6063Frenchs Forest NSW 2086Sydney, AustraliaTel: (02) 8977 7500Fax: (02) 8977 [email protected]

Siemon – Australia (Brisbane)Siemon - Australia (Brisbane)Unit 9, 128 Brookes StreetBrookes Terraces(Cnr Ann & Brookes Streets)Fortitude Valley QLD 4006Brisbane, AustraliaTel: (61) 07 3854 1200Fax: (61) 07 3854 [email protected]

Siemon – Southeast Asia46 East Coast Road#07-01/02 East GateSingapore 428766Tel: (65) 6345 9119 Fax: (65) 6345 [email protected]

Siemon – Australia (Melbourne)Siemon - Australia (Melbourne)Level 1, Suite 6161 Queens RoadMelbourne VIC 3004Melbourne, AustraliaTel: 03 9866 5277Fax: 03 9866 [email protected]

Siemon – China (Shanghai)Rm. 3407 - 3408,Hong Kong Square S.No. 283, Huai Hai Road Shanghai, 200021, P.R. ChinaTel: (86) 21 6390 6778Fax: (86) 21 6384 [email protected]

Siemon – China (Guangzhou)Rm. 1104,Middle Tower, Times Square28 Tianhebei RoadGuangzhou, 510620, P.R. ChinaTel: (86) 20 3882 0055Fax: (86) 20 3882 [email protected]

Siemon – China (Beijing)Suite 1108 SCITECH Tower22 Jianguomenwai AvenueBeijing 100004, P.R. ChinaTel: (86) 10 6559 8860Fax: (86) 10 6559 [email protected]

Siemon – UK36-48 Windsor StreetChertsey, SurreyKT16 8ASTel: (44) (0) 1932 571771Fax: (44) (0) 1932 [email protected]

Siemon — Japan10F Meguro G Bldg. 1-4-16 Meguro,Meguro-ku, Tokyo, 153-0063 JapanTel: 81 (3) 5437-1580Fax: 81 (3) [email protected]

Japan

Siemon – BrasilAv. Adolfo Pinheiro, 1001 -cj. 14504733-100 São Paulo/SP BrazilTel: (55) 11 5523 8122Fax: (55) 11 5521 [email protected]

Siemon – PeruCalle Martir Oloaya169 of 701Lima, Distrito MirafloresPeruTel: (511) 446 5188Fax: (511) 446 [email protected]

wp dc best prac infrastructure

Documents