don’t get left behind: cabling designs you can trust to...

#CPEXPO | CHANNELPARTNERSCONFERENCE.COM#DATACENTERWORLD | DATACENTERWORLD.COM

Dave FredricksData Center Infrastructure Architect

CABLExpress

Don’t Get Left Behind: Cabling Designs You Can

Trust to Support Speeds of 32G to 400G

#DATACENTERWORLD | DATACENTERWORLD.COM

• TIA Standards for data center cabling designs• IEEE 802.3ba/bm/bs Ethernet Standards for optics• Structured cabling designs for 40G to 400G Ethernet• Fibre Channel Gen6 and Gen7 optics• Structured cabling designs for 32G to 256G Fibre Channel• Final Considerations

Agenda


Telecommunications Industry Association (TIA)

TIA-942-B: released July 2017 named; Telecommunications InfrastructureStandard for Data Centers

TIA-568.3-D: released May 2016 named; Optical Fiber Cabling and Components Standard

TIA-606-C: released July 2017 named; Telecommunications Infrastructure and Equipment Administration Standard

*TIA Standards can be purchased at www.tiaonline.org


TIA-942-B


TIA-568.3-D

Three types of MPO/MTP polarity methods defined as Method A, B & C

Type-B MPO/MTP cable will connect together two parallel optics typically 40G or 100G QSFP type

Using flipped (Type-B) fiber optic trunks help support parallel and duplex optics


TIA-606-C

AE AF AG AH AI AJAC ADAB

09

10

11

12

13

AA

01

02

03

04

05

06

07

08

"X" COORDINATE

"Y" COORDINATE

AD02

AD03

AD04

AD05

AD07

AD08

AD09

AD10

AG04

AG05

AG06

AG08

AG09

AG10

AG11

AG13AD12

AG03COLD AISLE (CABINET FRO

NTS)

HOT AISLE (CABINET REAR)

HOT AISLE (CABINET REAR)

= cabinet corner used for grid location ID (right front facing the cabinet)

Cabinet location on data center floor

Port location in cabinet by rack unit (RU) number

Port location in rack unit, patch panel or enclosure


• Understand the TIA-942-B standard to use breaks in the connectivity links with patch panels, enclosures and port replication hardware. Patch panels give the ability to support different optics (next generation) while using existing horizontal or backbone cabling infrastructure

• Understand the different MPO/MTP polarity options in TIA-568.3-D to best plan for current and next generation optics

• Plan specific labeling schemes as shown by TIA-606-C to easily identify port locations throughout the data center space

TIA Considerations


IEEE 802.3ba/bm/bs

new

• IEEE 802.3ba released June 2010 introducing 40G Ethernet over 8 multimode fibers and 100G Ethernet over 20 multimode fibers

• IEEE 802.3bm released April 2015 updated several subset standards in 802.3ba to include 100G Ethernet over 8 multimode fibers as was 40G in previous standard

• IEEE 802.3bs released December 2017 jumped the speed from 100G to 200G and 400G Ethernet. Introduced new 16 fiber MPO/MTP connector for multimode and 8 fiber on singlemode glass *IEEE Standards can be purchased at www.ieee.org


40G & 100G Distance Limitations

IEEE 802.3baJune 2010

IEEE 802.3bmApril 2015


200G & 400G Distance Limitations

IEEE 802.3bsMultimode Glass

New 16 fiber MPO/MTP

IEEE 802.3bsSinglemode Glass

Standard 8/12 fiber MPO/MTP


• The introduction of the new multimode MTP-16 connector will create challenges to support the signal within existing MPO/MTP 8 and 12 fiber cabling systems

• Many believe that the MTP-16 will not replace the MPO/MTP 8 or 12 connector as the fiber connector of choice moving forward. The consensus looks to be that it will be used at the Spine or Leaf switch and then break out to run on existing and standard 8 and 12 fiber cabling infrastructures

• Reduce cost of 100G ports as breakout 4x 100G = 400G

400G Multimode Optics


MTP-16 to 2x MTP-8 or MTP-12 “Y” Cable

On the left side of the drawing would be an MTP-16 connector plugging into a 400G SR8 optic.

Then the “Y” cable would break the signal out into two 8 fiber links to work with 8 and 12 fiber cabling plants. No need to rip and replace existing horizontal or backbone cabling

xxxxxxxxxxxxx


• The cost of singlemode optics have decreased steadily over the past several years

• Singlemode fiber is typically less expensive than OM4 or OM5 multimode fiber

• The newer singlemode 100G PSM4 optic reduces the maximum distance link from 2 kilometers to 500 meters

• The singlemode PSM4 optic is closely priced now to multimode SR4 optics and should be considered on new structured cabling applications as it will work seamlessly for proposed 800G optics

Multimode verse Singlemode


• Plan to have a patch panel or break in the fiber link near the Core or Spine Switches. This allows easy migration to 40G, 100G, 200G, 400G or next generation 800G optics

• Calculate maximum dB link loss below -1.9 dB including inter-connects to use existing cabling for next generation optics

• Compare cost of multimode verse singlemode optics and cabling on new Spine and Leaf deployments. Consider new fiber links to support 100G Ethernet at a minimum

IEEE 802.3 Considerations


32 or 36 Port Line Card Replication


Cabling Designs to Support 40G to 400G


• Gen6 was released November 2015 and introduced speeds of 32G fibre channel over 2 fibers and 128GFCp fibre channel over 8 fibers

• Introduced the MPO/MTP parallel optic (128GFCp) to fibrechannel connectivity

• Gen6 hardware began to enter the data center space in large numbers in 2017

• Supports 1st generation of FC-NVMe

Fibre Channel Generation 6

*www.fibrechannel.org


• Released November 2018• Introduced 64GFC and 256GFCp• 64GFC to run on duplex or LC connections• 256GFCp to run on parallel or MPO/MTP connections• Fiber polarity same as Gen6 with distance limitation on OM4

multimode glass at 100 meters• Supports 2nd generation FC-NVMe

Fibre Channel Generation 7

*www.fibrechannel.org


Fibre Channel Gen6 32GFC & 128GFCp

*www.fibrechannel.orgOM4 and OM5 multimode glass rated at same distance of 100 meters for Gen7


• Plan to have a patch panel or break in the fiber link near the Director or Core Switches. This allows easy migration to any current Gen6 or Gen7 optic as a two fiber duplex LC or 8 fiber parallel MPO/MTP connector

• Calculate maximum dB link loss below -1.9 dB including inter-connects to use existing cabling for next generation optics

• Understand that the higher speeds from Gen6 and Gen7 along with a proper structured cabling system help reduce latency working with flash storage and NVMe

Fibre Channel Considerations


48 or 64 Port Blade Replication

10u replication enclosure with (8) 48 port MTP-LC cassettes

1u 64 port replication enclosure

H Series CABLExpress

16

81

9

1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16

16

81

9

1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16

1u enclosure with (2) 16 port MPO/MTP coupler panels to replicate 48 or 64 port blades using LC or MPO/MTP harnesses


Cabling Designs to Support 32GFC to 256GFCp

1 12 23 34 45 56 67 78 89 9

10 1011 1112 1213 1314 1415 1516 1617 1718 1819 1920 2021 2122 2223 2324 2425 2526 2627 2728 2829 2930 3031 3132 3233 3334 3435 3536 3637 3738 3839 3940 4041 4142 4243 4344 4445 45

SAN Cabinet #1

Core #1

Core #2

36/72 fiber MTP to MTP OM4 trunk

1 12 23 34 45 56 67 78 89 9

10 1011 1112 1213 1314 1415 1516 1617 1718 1819 1920 2021 2122 2223 2324 2425 2526 2627 2728 2829 2930 3031 3132 3233 3334 3435 3536 3637 3738 3839 3940 4041 4142 4243 4344 4445 45

MoR Patch Cabinet

H Series CABLExpress R

A B C D E F G H J K L M

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

Middle of Row (MoR) 6u patch enclosure

can support up to 288 LC ports with (12) 24

port MTP-LC cassettes

144 ports for Fabric A and 144 ports for

Fabric B





16

81

9

1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16

16

81

9

1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16

16

81

9

1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16

16

81

9

1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16

16

81

9

1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16

16

81

9

1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16

16

81

9

1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16

16

81

9

1 9 2 10 3 11 4 12 5 13 6 14 7 15 8 16

36, 72 or 144 fiber MTP to MTP OM4 trunk – Fabric A36, 72 or 144 fiber MTP to MTP OM4 trunk – Fabric B

1u enclosure has (2) 16 port MTP coupler panels to connect (2) FC32-64

blades

SAN X6 Core A

SAN X6 Core B

SAN Fabric A

SAN Fabric B

Maximum Link dB loss: .35 + .2 = -0.55 dB Typical Link dB loss: .23 + .15 = -0.38 dB


Cabling Designs to Support 32GFC to 256GFCp

1 12 23 34 45 56 67 78 89 9

10 1011 1112 1213 1314 1415 1516 1617 1718 1819 1920 2021 2122 2223 2324 2425 2526 2627 2728 2829 2930 3031 3132 3233 3334 3435 3536 3637 3738 3839 3940 4041 4142 4243 4344 4445 45

1 12 23 34 45 56 67 78 89 9

10 1011 1112 1213 1314 1415 1516 1617 1718 1819 1920 2021 2122 2223 2324 2425 2526 2627 2728 2829 2930 3031 3132 3233 3334 3435 3536 3637 3738 3839 3940 4041 4142 4243 4344 4445 45

1 12 23 34 45 56 67 78 89 9

10 1011 1112 1213 1314 1415 1516 1617 1718 1819 1920 2021 2122 2223 2324 2425 2526 2627 2728 2829 2930 3031 3132 3233 3334 3435 3536 3637 3738 3839 3940 4041 4142 4243 4344 4445 45

1 12 23 34 45 56 67 78 89 9

10 1011 1112 1213 1314 1415 1516 1617 1718 1819 1920 2021 2122 2223 2324 2425 2526 2627 2728 2829 2930 3031 3132 3233 3334 3435 3536 3637 3738 3839 3940 4041 4142 4243 4344 4445 45

SAN Cabinet #1 SAN Cabinet #2Patch Cabinet

Fabric A MoR Patch Cabinet

1 12 23 34 45 56 67 78 89 9

10 1011 1112 1213 1314 1415 1516 1617 1718 1819 1920 2021 2122 2223 2324 2425 2526 2627 2728 2829 2930 3031 3132 3233 3334 3435 3536 3637 3738 3839 3940 4041 4142 4243 4344 4445 45

Patch CabinetFabric B

Core #1 Core #2

SAN X6 Core A

SAN X6 Core B



12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

H S

erie

sC

AB

LExp

ress

H S

erie

sC

AB

LExp

ress

SAN Fabric A

SAN Fabric B









12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

12 24

131

1

13

2

14

3

15

4

16

5

17

6

18

7

19

8

20

9

21

10

22

11

23

12

24

Middle of Row (MoR) 6u patch enclosure

can support up to 288 LC ports with (12) 24

port MTP-LC cassettes

144 ports for Fabric A and 144 ports for

Fabric B

48/96 fiber MTP to MTP OM4 trunk

Cross-connectLC to LC

OM4 jumper

48 or 96 fiber MTP to MTP OM4 trunk – Fabric A48 or 96 fiber MTP to MTP OM4 trunk – Fabric B

(1) 10u enclosure has (8) 64 port LC

cassettes to replicate 512 ports from (1)

fully populated Brocade X6-8

directors with FC32-64 blades

Maximum Link dB loss: .35 + .35 + .35 = -1.05 dB Typical Link dB loss: .23 + .23 + .23 = -0.69 dB


• Understand published standards that apply to structured cabling from TIA, IEEE 802.3 and Fibre Channel.org

• Install a patch panel over Core, Spine or Directors to allow ability to connect optics that require an LC (duplex), MPO/MTP- 8 or 12 multimode fiber (parallel) or MPO/MTP-16 multimode fiber (parallel) connector

• Calculate link loss with decibels (dB) as maximum below -1.5 dB for inter-connects or connectors, total dB loss below -1.9 dB

• Plan to use cabling hardware that best replicates the switches, servers and compute equipment in the application as 1 to 1

• Consider singlemode fiber cabling infrastructure verse multimode

Final Considerations

don’t get left behind: cabling designs you can trust to...

Documents