us7630300

(12) United States Patent Warren et al.

US007630300B2

US 7,630,300 B2 *Dec. 8, 2009

(10) Patent N0.: (45) Date of Patent:

(54)

(75)

(73)

(21)

(22)

(65)

(63)

(60)

(51)

(52)

(58)

METHODS AND APPARATUS FOR TRUNKING IN FIBRE CHANNEL ARBITRATED LOOP SYSTEMS

Inventors: Bruce Gregory Warren, Poulsbo, WA (US); William Goodwin, Bothell, WA (US); Carl Mies, Snohomish, WA (US); Thomas Hammond-Doe], Everett, WA (US); Michael L. White, Oak Harbor, WA (US)

Assignee: Emulex Design & Manufacturing Corporation, Costa Mesa, CA (US)

Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 957 days.

This patent is subject to a terminal dis claimer.

Appl. N0.: 10/617,149

Filed: Jul. 10, 2003

Prior Publication Data

US 2004/0085972 A1 May 6, 2004

Related U.S. Application Data

Continuation-in-part of application No. 10/612,753, ?led on Jul. 1, 2003.

Provisional application No. 60/395,107, ?led on Jul. 10, 2002, provisional application No. 60/393,164, ?led on Jul. 2, 2002.

Int. Cl. H04L 12/28 (2006.01) H04J 14/00 (2006.01) U.S. Cl. ..................... .. 370/222; 370/351; 370/404;

370/406; 398/3; 398/59 Field of Classi?cation Search ............... .. 370/452,

370/447, 4604162, 411, 222, 351, 404, 406; 398/57, 70, 59, 3

See application ?le for complete search history.

Receive up

I Router

Lngm ms ‘

Loop witch SN 1: 1319

130

E 00p Switch SN#2 1320 132

Router '

Log1c 1317 I

w E‘ P" 1 v

(56) References Cited

U.S. PATENT DOCUMENTS

4,689,708 A 8/1987 Hager et a1.

(Continued) FOREIGN PATENT DOCUMENTS

0 858190 A2 8/1998

(Continued) OTHER PUBLICATIONS

“Increasing Intelligence Within the SAN Fabric”, Internet Citation: URLzhttp://www.br0cade.com/san/whiteipapers/pdf/ IncreasingIntelligienceWithinSan.pdf.

(Continued)

EP

Primary ExamineriAlpus H Hsu (74) Attorney, Agent, or FirmiMorrison & Foerster LLP

(57) Methods and apparatus for switching Fiber Channel Arbi trated Loop Systems is provided between a plurality of Fiber Channel Loop devices. In one aspect of the invention, the system switches based at least in part on arbitrated loop primitives. An exemplary interconnect system may include a ?rst port and a second port, both including port logic to monitor certain arbitrated loop primitives, a connectivity apparatus, a route determination apparatus including a rout ing table consisting of ALPA addresses and their associated ports, the route determination apparatus coupled to each port and the connectivity apparatus, where the connectivity appa ratus creates paths between the ports based on arbitrated loop primitives. In one embodiment, the connectivity apparatus is a crossbar switch. Examples of the arbitrated loop primitives that cause the switch to create paths between ports includes one or more of the following: ARB, OPN and CLS. In yet other aspects, the system ensures device access fairness through one or more techniques, including a rotating priority system, a counter to count the number of OPNs, especially sequential OPNs, and/or priority based on port type. Device Zoning may be implemented. In one implementation, the system includes trunking such that frames may be transferred on multiple ports.

ABSTRACT

14 Claims, 52 Drawing Sheets

Duplicate Cascade

US 7,630,300 B2

5,535,035 5,619,497 5,751,715 5,754,549 5,978,379 5,991,891 6,000,020 6,118,776 6,185,203 6,192,054 6,243,386 6,314,488 6,317,800 6,324,181 6,396,832 6,459,701 6,470,007 6,504,817 6,529,963 6,532,212 6,570,853 6,614,796 6,747,984 6,754,210 6,804,245 6,898,184 6,904,053 6,915,429 6,981,078 6,999,460 7,009,985 7,215,680 7,343,524 7,352,706 7,382,790 7,397,788

2002/0044561 2002/0044562 2002/0196773 2002/0196796 2003/0021223 2003/0021239 2003/0028663 2003/0095549 2003/0108058 2003/0189935 2004/0076428

U.S. PATENT DOCUMENTS

7/1996 4/1997 5/1998 5/1998

11/1999 11/1999 12/1999 9/2000 2/2001 2/2001 6/2001 11/2001 11/2001 11/2001 5/2002

10/2002 10/2002 1/2003 3/2003 3/2003 5/2003 9/2003 6/2004 6/2004 10/2004 5/2005 6/2005 7/2005 12/2005 2/2006 3/2006 5/2007 3/2008 4/2008 6/2008 7/2008 4/2002 4/2002 12/2002 12/2002 1/2003 1/2003 2/2003 5/2003 6/2003 10/2003 4/2004

DeFoster et al. ............. .. 398/3

Gallagher et al. .. 370/394 Chan et al. DeFoster et al. Chan et al. Hahn et al. Chin et al. ................ .. 711/162

Berman .... .. . 370/351

Berman Chan et al. Chan et al. ................ .. 370/403

Smith .......... .. 7 10/ 240

Westby et al. .. 710/40

Wong et al. 370/403 KranZler ...... .. 370/360

Henson et al. . . 370/405

Berman ....... .. . 370/351

Old?eld et al. . 370/217

Fredin et al. . . . . . . . . . . . .. 710/1

SoloWay et al. ........... .. 370/230

Johnson et al. Black et al. ............... .. 370/403

Hoglund et al. Ofek Mitchem et al. Anderson et al. Berman .................... .. 370/466

Allen et al. . . 713/170

Paul 710/100 Fiore ........ .. . 370/400

Black et al. ............... .. 370/403

Mullendore et al. KlotZ et al. KlotZ et al. Warren et al. Mies et al. Coffey ..................... .. 370/403

Killen, Jr. et al. Berman Ambe et al. Kashyap Mullendore et al. Mullendore et al. Berman .................... .. 370/392

Black et al. Warden et al. Green et al. ................. .. 398/9

2004/0081186 A1 2004/0085974 A1 2004/0085994 A1

FOREIGN PATENT DOCUMENTS

4/2004 Warren et al. 5/2004 Mies et al. 5/2004 Warren et al.

WO WO-98/36537 8/1998 WO WO-99/43002 8/1999 WO WO-99/48252 9/1999 WO WO-2005/008405 1/2005 WO WO-2005/008406 1/2005 WO WO-2005/017661 2/2005 WO WO-2005/017662 2/2005

OTHER PUBLICATIONS

Fibre Channel Fabric Generic Requirements, American National Standard, Aug. 7, 1996, pp. 1-23. “Optimizing The Performance And Management of 2 Gbit/ sec SAN Fabrics With ISL Trunking”, Internet Citation, Mar. 2002, Retrieved from the Internet: URLzhrrp://WWW.brocade.com/san/ Whiteipapers/ pdf/ I SLiTrunkingiWPiRS .pdf. Fibre Channel Arbitrated Loop (F C-AL-Z) Rev 7.0, ANSI T11.3 Project 960M FC-AL and project 1133-D FC-AL2, Apr. 1, 1999, Global Engineering, Englewood, CO. Fibre Channel Private Loop SCSI DirectA ttach (F C-PLDA) Rev 2.1 , ANSI T11.3 Project 1162-DT, Sep. 22, 1997, Global Engineering, Englewood, CO. “Increasing Intelligence Within the SAN Fabric”, Internet Citation: Jun. 2001, URL:http://WWW.brocade.com/san/Whiteipapers/pdf/ IncreasingIntelligienceWithinSan.pdf. International Search Report mailed Dec. 19, 2005 for PCT Applica tion No. PCT/US2004/022003, ?led Jul. 9, 2004. International Search Report mailed Mar. 25, 2005 for PCT Applica tion No. PCT/US2004/022131. Supplementary European Search Report mailed Feb. 13, 2007 for European Patent Application No. 047778378 ?led Sep. 7, 2004, four pages. International Search Report mailed Feb. 24, 2006, for PCT Applica tion No. PCT/US04/21982, ?led Jul. 9, 2004. International Search Report mailed Jan. 20, 2006, for PCT Applica tion No. PCT/US04/21983, ?led Jul. 9, 2004. Supplementary EP Search Report mailed Mar. 29, 2007 for EP Patent Application No. 047778295, ?led Jul. 9, 2004 (3 pages). Supplementary EP Search Report mailed Sep. 20, 2007 for EP Patent Application No. 047779137, ?led Jul. 9, 2004 (3 pages). Supplementary EP Search Report mailed Dec. 14, 2007 for EP Patent Application No. 047778287, ?led Jul. 9, 2004 (3 pages), ?led Jul. 9, 2004 (3 pages)

* cited by examiner

US. Patent Dec. 8, 2009 Sheet 1 0f 52 US 7,630,300 B2

Fibre Channel

Arbitrated Loop


42 41 4o

52 1

53 I H 4 3 2 1

70

82 81 so / / 83 84

5 6 7 8

/ I iA 54 \ 5s 57" 1 59*} * 6O

65 58 |

56 / 43 44 45 46

Fig. 2

US. Patent Dec. 8, 2009 Sheet 3 0152 US 7,630,300 B2

90 92 91

102 I

101

103 \I 111\ X/ 100

+/ 4 3 F 2 f 1

/ 120 113 112

14 115

5 6 7 8

104 \/ \ 105 107i‘, 109\\ +/ 110 1| @, 10

93

Fibre Channel Arbitrated Loop Hub

Fig. 3

\ 108

/ / 95

6

94 96


164 165 166

145 / 146 I 147 / 148 154 \ 1

153 \ ,

152 \

151 4 167

11 """""" “1i """""" “

181~ 173 172 171 150 / 149 \ ROUTER Q: SWITCH LOGIC _ 140

174 175 176 i 11 11

155 ____________________ __

156 5 157 """" '1 5

~ : 163 153 v r r

s 7 8 1 / /

/ / 144 143 142 141

160 161

Fig. 4 162


201

203 202

213 I

/ \

214 \ I/ 212 211 210

4 H

3 2 1 236 235 234 232

200

231 \ 230

5 5 7 8 237 238 239 233

216 I 215 \/\/ 220\ /221 218 — \

219

|

217 / 204 205 06 207 2

Fig. 5


253 252 251

i 263 i J

264 l/ 262 261 \\H/26O 4 3 2 1

293 292 291 290 282 250

/ 283 \

281 \ r 280

5 6 7 8 294 295 296 297

I T IT 255 \/\/ / 270\ /271 268 \ 259

I

267 / 254 255 255 257

Fig. 6


N .90

26 mom in \ com

P6

0 v 052 um 0 _ v 55

\ Em A| /v @QQ

wow 2m Sm

A \ 0 0mm

H tom 2mm 0 v

A \ ‘ H 89% k 1 Smog A v 20mg mHm \ HPHBE \ ?o?aw ww dd?vm Sm mom ( " Em mcw?om “tom 2m

-

2m " A v R550

mmtom 9mm. 0 0 mm m8

mBPEw A V

1b \ 13585 wow in

wow K m5

mom \


400

Port Logic 401 43:0

Port Receiver OS 402

Generator 404 with Parity M OS

/ 411 409 407 403 Decode 428‘

\Port 410 20/16 Bit Elasticity I A05> ' . \ Decoder 8;

Recewe > *-> - l v D Check Retlmmg . ata arity Buffer Panty

P 1 Check 427

y Logic ?~

Frequency T 426 Check 451 406 +

Router

Port Transmitter Switch 416 418 Logic

414 OS 4 1 2 \ 4 1 5 Generator

\ Port 413 16/20 Bit with Transmit ‘ Encoder M Panty

‘ Check U 419 Data . X 4

panty 420 421 I. , ‘y 425

Parlty ‘

417 chc_ck Decode 424 Log1c =

Port Control 422% Control _ 423 V

Fig. 8


513 . /

516/ Idle‘ 514% arb 12,12 515 512

\ 500 r Port 1 rx_data

\/

505 arb 12,12

511

501 504

509 506

502 503

508

Arb detected on

port rx_data

Fig 9a


522 arb 12,12 /

523

520 M Port 1

524

526 529

arb 12,12 527 530

528

531 532

opn 13, 12 /

521 533

Connection established. Open sourced

Fig 9b


ArbfO replaced with Arbff. Data Transfered. Close sourced.

Fig 9c


581 . /

583

585\\ 584 # Port 1 rx_data

588 589

590

592 593

Arbff detected on port rx_data. Disconnect. Source idles

Fig 9d


Port Transmitter

608 Port Receiver

603

opn 11, 12

/ 612 613

Arbs forwarded. Open sourced.

Fig 10a


opn ll, 12 661 \

652

651 “'_'

653

655 656

658 657

660 659

661

Open detected at port receiver. Port receiver holds open.

Provides ALPA to router and asserts opn_connect_req.

Fig 10b


Arb ff, ff Arb £0, £0 681 / 680 \

682

683 ———— Port 1

opn 11, 12 l 684

686

687 688

690 689

692

frame data

695

Fig 10c


700 \ idle Arb ff ff

701% 70% \ 703

704 ~—— Port 1

Arbff detected on port rx_data. Disconnect. Source idles.

Fig 10d


721/ Part1

Arb ll, 11 727 /

722 o o 723

Opn 12, 11 /

724

726

Arbs forwarded. Open Sourced.

Fig 11a


730

Arb 13, 13A Arb 13, 13 / \ 731 734

P9“ 1 —735

Cls ‘732

Frame data

Arb 13 forwarded. Data Transfered. Close sourced.

Fig 11b

us7630300

Documents