Storage Wide-Area Networks(SWANs)

Randy H. KatzComputer Science Division

Electrical Engineering and Computer Science DepartmentUniversity of California, Berkeley

Berkeley, CA 94720-1776

Storage Networks

• Primary goal is to share storage among computers in complex, heterogeneous environments, with PCs, workstations, file servers, and mainframes

• Storage can be:– Direct attached (e.g., host bus adapter/HBA)– Network attached (via a file server)– Channel attached (primarily fibre channel,

but also IBM SSA)

Storage Networks

Workstation

HI

Cache

Fibre Channel Loops

Dual Ported DisksAnd Controllers

HI HI HI

DI DI DI DI

Cache

Cache

Cache

CrossbarInterconnectMainframe

RemoteStorageManager

Fibre Channel

Or ESCON

Multiple HostInterconnections

E.g, Hitachi, EMC, IBM Storage Arrays

Storage Networks

• Major development in 1990s: storage networks– Native FC is a point-to-point or loop/string-oriented

(“arbitrated”) method of interconnection; 1 Gbps, extend up to 10 km

– Emergence of FC “fabrics”: FC switches arranged hierarchically to enable connectivity between any host and any storage device (e.g., Brocade Networks dominates this product space)

– Standardize FC protocol stack lives on top of such fabrics: device naming, transport, CoS, etc.

– SCSI-3 protocol over FC fabrics– Gigabit ethernet now emerging in SAN environment

Basic Attached Storage

• Device attached by SCSI HBA or channel interface• Host manages the file-to-block mapping

Host

OS

Disk Interface (DI)

AllocationTable

Disk, Cylinder,Track,

Sector

Network-Attached Storage (NAS)

aka Network File Service

• Mapping from File to Block done in network-attached File Server, not host

LAN

Host

Host

Host

NetworkFile

Server

OS

NetworkInterface

(NI)

NetworkInterface

(NI)

NetworkInterface

(NI)

File Name, Offset, Length

Network-Attached Secure Devices (NASD)

LAN

Host

Host

Host

NetworkFile

ServerOS

NetworkInterface

(NI)

NetworkInterface

(NI)

NetworkInterface (NI)

Network-attachedSecure Device (NASD)

OS

File Name, Offset, Length

Disk, Cylinder,Track, Sector

Gibson@CMU: Research project ondevice embedded protocol stack, authentication

Storage Platforms“Storage Virtualization”

DiskStorage

Subsystem

WorkStation

MainFrame

MainFrame

ChannelInterface

OSLUN,

Offset,Length

LUNToPHY

LUN = Logical UnitLogical disk mapping onto underlying physical disks on logical block to physicalblock basis

LAN

Host

Host

Host

NetworkInterface

(NI)

NetworkInterface

(NI)

NetworkInterface

(NI)

File Name, Offset, Length

FileServer

FileServer

FileServer

NAS

NAS distinguished by an exported Network File System interface over a standardLocal Area Network-based transport

OpticalDisk

StorageSubsystem

SAN

MainFrame

DiskStorage

Subsystem

TapeStorage

Subsystem

ChannelInterface

LAN

Host

Host

Host

NetworkInterface

(NI)

NetworkInterface

(NI)

NetworkInterface

(NI)

File Name, Offset, Length

FileServer

FileServer

FileServer

CI

LUN,Offset,Length

MainFrame

CI

CI

CI

LUN,Offset, Length

PHY Device,Cyl, Trk, Sector

NAS + SAN

SAN distinguished by a block-oriented interface;Usually implemented across a channel-oriented fabric

OpticalDisk

StorageSubsystem

SAN

MainFrame

DiskStorage

Subsystem

TapeStorage

Subsystem

ChannelInterface

LAN

Host

Host

Host

NetworkInterface

(NI)

NetworkInterface

(NI)

NetworkInterface

(NI)

File Name, Offset, Length

FileServer

FileServer

FileServer

CI

LUN,Offset,Length

Gateway

WAN

Gateway

LAN SAN

MainFrame

FS DSS

Remote SAN

CI

CI

CI

LUN,Offset, Length

PHY Device,Cyl, Trk, Sector

NAS + SAN + SWAN

Now extend the NAS or theSAN over a wide-areanetwork transport …NOTE: wide-areaSAN is new idea

Shared StorageReference Model

Application

Host

SAN

Device

File/Record Subsystem

Block Subsystem

Sto

rag

e D

om

ain

Serv

ice S

ubsy

stem

Dis

covery

, M

onit

ori

ng

Reso

urc

e M

gm

t, C

onfig

ura

tion

Secu

rity

, B

illin

gR

edundancy

Mgm

t, B

ack

-up

Hig

h A

vaila

bili

ty,

Fail-

over

Capaci

ty P

lannin

g

BlockAggregatio

n

SAN Reference Model

Application

Blo

ckFi

le FS

Host-based

SAN-based

Device-basedDA

SAN

Block-orientedSAN

SAN Reference ModelB

lock

File FS

Host

SAN

Device

LAN

NAS

Host Host

NASStorage

Application

SAN Reference ModelB

lock

File FS

Host

SAN

Device

LAN

NAS

Host Host

HeterogeneousStorage

Environment

Application

NASHead

SAN

DA

HostHosts/wraid

Seven Layer Stack

7

6

5

4

3

2

1

Application

Presentation

Session

Transport

Network

Data Link

Physical

IPNFS, CIFS

FTP, SNMP,TFTP, Telnet,FCP, SCSI-3

TCP, UDP

IP

LAN, MAN, WAN

Phy

Sw GigaENFS, CIFS

FTP, SNMP,TFTP, Telnet,FCP, SCSI-3

TCP, UDP

IP

Mac Client/Control

Phy

FCSCSI-3

VIIP

FC-4Protocol I/F

FC-3 Encrypt/Authentication

FC-2 Framing, FC,Class of ServiceFC-1 Encoding,

Link ControlFC-0 Phy

Fibre Channel Protocol Stack

• FC-0: gigabit physical layer• FC-1: data encoding and link layer control• FC-2: segmentation/reassembly of data

frames, flow control, class of service• FC-3: common services, e.g., encryption• FC-4: “upper layer protocol” upon which SCSI-

3 or IP can run• Contrast with Gigabit Ethernet

– Ethernet framing, VLAN tagging, frame prioritization (8 levels), link aggregation, 1.25 gbps

– IP + Gigabit Ethernet emerging for SANs

SCSI-3 Terminology

ApplicationClient

DeviceServer

Request

Response

Initiator Target

LUNs

Delivery Subsystem(e.g., Fibre Channel or

Serial SCSI over IP)

SCSIClient-Server

Model

WAN

FC over IP (FCIP)

• IETF IP Storage (IPS) working group– Recall FC fabrics developed in context of machine room/building-

scale interconnect (e.g., no congestion control!)• FC time outs in wide-area? Flow control interaction? QoS?• How does bridging actually work for FC e2e management?

Server

JBODServer

TapeStorage

Subsystem

FCOver

IP

FCSwitch

Server

JBODServer

TapeStorage

Subsystem

FCSwitch

FCOver

IP

Tunnel Session

IP Network

Internet FC Protocol (iFCP)

• Gateway to gateway protocol, sessions rather than tunnels• TCP for congestion control, error detection, recovery• Plug FC devices directly into iFCP switches• Session and naming semantics

FC_DeviceN_Port

F_PortiFCP layerFCP Portal

FC_DeviceN_Port

F_PortiFCP layerFCP Portal

FC Traffic

IP Network

FC DeviceAddress

IPAddressMapping

Control Data

iFCP gatewayregion

iFCP gatewayregion

iFCP Frames

iFCP Services

• 24-bit N-Port Address: <Domain, Area, Port>• iSNS: Internet Storage Name Service—discovery

and management protocol for IP storage networks (IPNSP)

• Protocol specification includes address translation feature to allow remote storage devices to be assigned a local, FC fabric compliant address– Local commands executed locally on the fabric– Remote commands executed on top of TCP connections

• Error Detection/Time Outs• Security

Other Protocols

• Metro Fibre Channel Protocol (mFCP)– FCP over IP using UDP rather than TCP (link

layer flow control and pacing)

• Internet SCSI (iSCSI)– IP to the storage device– Serial SCSI block data transfer over IP (SCSI

Access Method Command Set—SAM)– IPSec, command/data ordering, steering to

application memory

iSNS

• Discovery Process– Device registration– WWN or iSCSI names– Zoning/discovery domains

• iSNS objects– Portals– Storage Port– Storage Nodes

Storage Applications

• Data Centers– Server clustering– Storage centralization, consolidation,

management– LAN-free back-up

• IP Storage for Remote Applications– Remote back-up– Remote mirroring– Disaster recovery– Content distribution

Rhapsody Networks, Inc.

• Storage Application Director– “Alteon box for storage networks”– Peek into storage packets traversing fabric

(“deep frame classification”) and invoke code—for encryption/decryption, mirroring, LUN mappings, etc.

– Per port software processing and cut-through fabric routing

– Data copy engine, table lookup engine, in-transit I/O mods, data escrow/trap to software for complex errors or event processing

– Intelligent queue management