storage virtualization with ibm san volume controller · storage virtualization with ibm san volume...

IT-Symposium 2007 17.04.2007

www.hp-user-society.de 1

Storage Systems Division

DECUS IT-Symposium 2007 | 1G04 © 2007 IBM Corporation

Storage Virtualization withIBM SAN Volume Controller

Torsten RothenwaldtIBM European Storage Center


Storage Virtualization with IBM SAN Volume Controller | 1G04 © 2007 IBM Corporation2

Agenda

� Review different architectures of disk block virtualization.

� Explain implementation details of the IBM SAN Volume Controller as example of in-band appliance.

� Discuss current enterprise usage of disk/file virtualization.

IT-Symposium 2007 17.04.2007




Table of contents

Disk block virtualization

IBM SAN Volume Controller

Storage virtualization in enterprises today



Disk (Drive) Virtualization

� Physical data layout

Cylinder/Head/Sector addresses

� Media defects

� Virtual data layout

Logical Block Addresses (LBA)

� “Defect-free”

0000

0001

0002

0003

….

nnnn

IT-Symposium 2007 17.04.2007




Block-Level Virtualization

� Bound to a physical machine

� Fixed size and (pseudo-) geometry

� Limited performance

� Do break (occasionally)

� Anywhere (can be moved or replaced)

� Size as needed (can grow, shrink, or morph)

� Performance scaling

� Reliable as needed

Where does block-level virtualization reside?



Virtualization in the Storage Subsystem

+Heterogeneous hosts+Mature industry & products

�Performance �Stable & reliable�Perceived security

-Homogeneous storage-Management

�Device-specific�“per-box”

Virtualization

RAID subsystem

Per-host

Provisioning

IT-Symposium 2007 17.04.2007




Virtualization in the Host

+Flexibility (any storage)+File system coupling(online growth, re-layout, movement, snapshots,…)

-Server-centric management �of RAID subsystems�of volumes

-Complexity �shared data clusters

Virtualization

Per-host…still

Provisioning

Host

RAID subsystem



“Out-of-Band” Virtualization

+Data center-wide management+Shorter I/O path+Light-weight compared to full volume manager

-May be host-invasive �Host-specific

-Appliance availability-Secure protocols

Virtualization

Data center-wide

Provisioning

Host

RAID subsystem

“SAN appliance”�separate box�In-host

Infrastructure

IT-Symposium 2007 17.04.2007




Virtualization in the Network: “In-Band”

+Data center-wide management �Heterogeneous storage�Heterogeneous hosts

-Complexity �User (un)familiarity �Integration needs �Needs clustering

-Performance perceptions

Virtualization

Data center-wide

Provisioning

Host

RAID subsystem

SAN�Host access�Device access

Infrastructure



Virtualization Devices for In-Band

� Server-based device (appliance)

Virtualizes a variety of physical storage using different HBAs. One pool.

Implements complex storage solutions inexpensively

Adds another layer (managed separately)

Less interoperability issues (no fabric integration)

Appears as standard device.

� Switch-based device (fabric application)

Network optimized

High port counts

Expensive

Elaborated functions not available

Fabric integration critical

Standardization just beginning (ANSI T11 Fabric Application Interface Standard)

IT-Symposium 2007 17.04.2007




Table of contents






Example of a Block-Level In-Band Appliance:IBM SAN Volume Controller

� J. S. Glider, C. F. Fuente, W. J. Scales:The software architecture of a SAN storage control system.IBM Systems Journal, Vol. 42 (2003) Nr. 2, pp. 232-249

( http://www.research.ibm.com/journal/ )

� IBM Redbooks:

SG24-6423 IBM System Storage SAN Volume Controller

SG24-7275 Implementing the SVC in an OEM Environment

SG24-7371 Using the SVC for Business Continuity

( http://www.redbooks.ibm.com/ )

IT-Symposium 2007 17.04.2007




IBM SVC: Architecture

StorageStorageNetworkNetwork

NodeNode

Managed Disks

Virtual LUNs Virtual LUNs Virtual LUNs Virtual LUNs

NodeN ode NodeNode NodeNode NodeN ode N odeNodeNodeN ode NodeNode



IBM SVC: Zoning

Host Zone:-Hosts zoned only to SVC-See only Virtual Disks that SVC allows to see

Device Zone:-Devices zoned only to SV-See only SVC nodes as connected hosts

IT-Symposium 2007 17.04.2007




IBM SVC: Managed Disks

R5

VLUNR10 VLUN

R5

VLUN

R1

VLUN

R5

VLUN

R1

VLUN

RAID Controller RAID Controller

SANVC

R5

MD

R10

MD

R5

MD

R1

MD

R5

MD

R1

MD

� SVC does not perform RAID functions� Utilizes RAID capability of backend storage server� RAID-5, RAID-10, or RAID-1 recommended� Normally LUNs "surfaced" from storage systems are what the hosts on the SAN see as physical disks

� Disks surfaced by the storage systems are discovered by SVC and referred to as Managed Disks (mdisks)

� Spare capacity on mdisks can be reallocated transparently and dynamically



IBM SVC: Managed Disk Groups

R5R5

R5 Pool

...

Extent 1

Extent 2

Extent 3

Extent 4

Extent 5

Extent n

R1

R1 R1

R1 Pool

R10 R10R10

R10 Pool

R5

� Once the mdisks are available to SVC, the user assigns them to one or more pools called Managed Disk Groups

� These MDGs are addressed by the SVC in terms of extents:Extent size is determined at MDG creation time, default 16 MB, max 512 MB

Extent size determines maximum amount of storage SVC can manage

16-MB extents = 64 TB, 512-MB extents = 2 PB

Cannot migrate vdisks between MDGs with different extent sizes

Can migrate extents from mdisk to mdisk

IT-Symposium 2007 17.04.2007




IBM SVC: Virtual Disks

Extent 1a

Extent 1b

Extent 1c

Extent 1d

Extent 1e

Extent 1f

Extent 1g

Extent 2a

Extent 2b

Extent 2c

Extent 2d

Extent 2e

Extent 2f

Extent 2g

Extent 3a

Extent 3b

Extent 3c

Extent 3d

Extent 3e

Extent 3f

Extent 3g

Extent 1a

Extent 2a

Extent 3a

Extent 1b

Extent 2b

Extent 3b

Extent 1c

Extent 2c

Extent 3c

Create a striped

virtual disk

A host vdisk is a collection of Extents —each 16 MB - 512 MB

� From these extents, the user can build "virtual disks"� Various policies can be used to build them:

Striped — taking an extent in turn from each disk in the pool or a subset of the disks in the pool (Virtual Disk striped across multiple disks)

Sequential — using a single disk in the pool (Virtual Disk mapped sequentially to managed disk)

Image Mode — Virtual Disk = Physical LUN

SVC Cache — Enabled or Disabled

� Real physical capacity must be available to create a vdisk� Virtual disks can be expanded, reduced, or deleted� IO governing can be enabled to limit IO/s or MB/s



Cache Disabled Virtual Disks

� Can be used with Image Mode VDisks only to enable use of copy services of underlying disk subsystem

� Can be used with striped mode VDisksMay be useful to preserve cache resources

Example: Use for TSM/HSM disk pools

Example: Use for test/dev VDisks in mixed production environment

� Cache Disabled Virtual Disks are basically normal VDisks and can utilize SVC capabilities like migration and copy servicesCan use SVC copy services on the source VDisk

Cannot use SVC copy services on the target VDisk

� Provide a way for parallel direct access to the underlying disk subsystem

IT-Symposium 2007 17.04.2007




IBM SVC: IO Groups and Hosts

HostVDisk

WWPN1

WWPN2

WWPN3

WWPN4

IO Group

� Each virtual disk is assigned to a particular IO Group (node pair).

Every node in the cluster is aware of the virtual disk, but only owning IO Group services requests.

All IO is targeted at either of the nodes in the IO Group for purposes of caching and load balancing.

� It is these virtual disks that SVC presents to hosts on the SAN as targets of IO.The virtual disks are mapped to hosts (SDD for mutli-path operation).

Can be mapped to multiple hosts for use with clustering software.

The hosts see these as physical disks (in terms of the OS).

SVC knows hosts as groups of HBA WWPNs.



SVCCluster

I/O Group 1

VirtualDisk

LUN

ManagedDisk

mdisk0

10GB

mdisk1

10GB

mdisk2

10GB

mdisk3

10GB

mdisk6

20GB

mdisk5

20GB

mdisk4

20GB

FAStT

10GB

FAStT

10GB

FAStT

10GB

FAStT

10GB

ESS

20GB

ESS

20GB

ESS

20GB

ManagedDisk Groups

mdiskgrp0 [FAStT Group] - 40GB mdiskgrp1 [ESS Group] - 60GB

vdisk0

20GB

vdisk1

20GB

vdisk2

20GB

vdisk3

20GB

vdisk4

20GB

Virtual Disks Mapped to Hosts

SVC Combined Physical & Logical View

I/O Group 2

Virtual Disks are associated with particular I/O Groups

Managed Disk Groups are accessible by all I/O Groups in the Cluster.

IT-Symposium 2007 17.04.2007




IBM SVC: Clustering

�Cluster comprised of 2…8 nodes but administered as single image

�No Linux clustering software, failover/failback is function of SVC code

�One node automatically designated config/boss node for cluster

� Assigned cluster IP address

� Responsible for coordination of node transitions

� Automatic failover of config node and IP address

�Auto-restart of a node on failure and re-admission to cluster

�Cluster requires majority of nodes remain operating to ensure quorum

�Two nodes is a special case

� Quorum disk is elected as a tie-breaker

� Quorum disks are existing mdisks under SVC and use 1 extent

�A node stores a write in its own write cache and the write cache of its partner node before acknowledging to the host application

�On node failure, surviving node empties write cache and proceeds in write-through mode



RAIDcontroller 2

RAID controller 1

LUN 4

LUN 3

LUN 2

LUN 1

LUN 4

LUN 3

LUN 2

LUN 1

MD 4

MD 3

MD 2

MD 1

MD 8

MD 7

MD 6

MD 5

VD 4

VD 3

VD 2

VD 1

VD 7

VD 6

VD 5

IO Group

IBM SVC: Copy Services

RAIDcontroller 4

RAID controller 3

LUN 4

LUN 3

LUN 2

LUN 1

LUN 4

LUN 3

LUN 2

LUN 1

MD 4

MD 3

MD 2

MD 1

MD 8

MD 7

MD 6

MD 5

VD 4

VD 3

VD 2

VD 1

VD 7

VD 6

VD 5

IO Group

SAN Peer-to-Peer Remote Copy (PPRC) “outside the box”(Metro / Global Mirror)

SAN Data Migration “outside the box”SAN FlashCopy “outside the box”

IT-Symposium 2007 17.04.2007




IBM SVC Global Mirror

� Long distance asynchronous remote mirroring function

� Practically unlimited distances for business continuity

� Does not wait for secondary I/O before completing host I/O

Minimizes performance impact to applications

� Designed to maintain consistent secondary copy at all times

Once initial copy has completed

� Built on Metro Mirror code base

� Metro and Global Mirror delivered as single feature

Offers great implementation flexibility

No additional license chargefor existing MM users



Global Mirror: Overview

� Single remote copy partnership per cluster� Supports consistency groups that span I/O groups within cluster� Consistency group consists of MM or GM relationships not both

Consist-ency

Group

Cluster 1 Cluster 2Partnership

Vdisk

A

Vdisk

B

Vdisk

C

Vdisk

D

Relationship

Relationship

IT-Symposium 2007 17.04.2007




Global Mirror: Algorithm� I/O complete acknowledged to host at primary once following occurs:

Data is committed to cache in both nodes in I/O group

Log of I/O and its sequence number committed on both nodes of I/O group

� Remote I/O is transmitted concurrently with processing of local I/OLocal host I/O normally completes before remote I/O completes

� Extra messages passed between nodes in clusterUsed to detect concurrent host I/O among VDisks in same consistency group

� Concurrent I/O is assigned a shared “sequence number”

During high loads many I/Os will share sequence number to minimize overhead

� Remote I/Os are applied at secondary in “sequence number” order

Writes within a “sequence number” are non-dependant and written in any order once received at secondary

Many sequence number “batches” can be outstanding on the long distance link

At secondary site a single sequence number is applied at a time



Global Mirror: Link Tolerance

� Existing redundancy used to continue operation if node or link fails:

Recovery begins by re-driving writes that were in flight.

Log of I/O established during initial host write used to support recovery.

Recovery uses read to primary VDisk to capture data for replay of write

� Autonomic Link Tolerance feature to prevent performance problems leading to application outage:

Monitors write latency impact of GM and considers stopping a GM relationship or consistency group if impact exceeds a threshold

Link tolerance threshold allows ‘bursts’ of work to be tolerated

Defines period of time poor performance can be tolerated before GM stops

Settable from 1 minute to 1 day in 10 second increments (default 5 minutes)

Disabled by setting to ‘0’

IT-Symposium 2007 17.04.2007




Table of contents






Why To Use Storage Virtualization

� The management nightmare

Too many servers, operating systems, storage systems, management consoles, …

Too complex policies

Too large migration projects

� Availability requirements

� Storage resource utilization

� Storage performance

IT-Symposium 2007 17.04.2007




SANVolume Controller

SVC Version 4.1 Supported Environments

SAN with 4Gbps fabric

HitachiThunder

9200

95xxV

9520V

HPEVA

3000/5000

4000/6000

8000

IBMDS

DS4K / 6K

DS8000

DS4800

HitachiLightning

9980V

9970V

9910/9960

HPMA/EMA

8000

12000

16000

EMCSymm

8000

DMX

EMC/DellCLARiiON

FC4700

CX2/3/4/5/6/700

MicrosoftMSCS

MPIO, VSS, GDS

x64

IBMAIX

HACMP/XD

GPFS / VIO

SunSolaris 10VCS Clustering

SUN Cluster

HP/UX 11iV2TRU64

OpenVMSServiceGuard

Clustering

Linux(Intel/Power/zLinux)

RHEL/SUSE

W / LVM

RHEL AS 4

Ia32, x64, System p

IBMBladeCenterWin/Linux/VMWare/AIX

OPM/FCS/IBS

SAN

SANVolume Controller

Continuous Copy

Metro MirrorGlobal Mirror

VMWareWin / NW

guests

Point-in-time CopyFull volume

Copy on write

IBMESS,

FAStT

NovellNetWareClustering

Sun

9910/9960

9970/9980

HPXP

48, 128

512, 1024

. . .

1024Hosts

Cisco

McData

iSCSI to hostsVia Cisco IPS

HitachiTagma

StoreUSP, NSC55

IBMN series5200, 5500

NetAppFAS

3020, 3050

STKD173, 178, 220,

240, 280

FLX210, 240,

280, 380



Performance: SVC SPC-1 Benchmark

http://www.storageperformance.org/results/a00043-r1 _IBM_SPC1_executive-summary.pdf

IT-Symposium 2007 17.04.2007




Performance: SPC-2 Benchmark

� Newest addition to the Storage Performance Council Benchmarks

� Composite of three workloads to measure sequential performanceLarge file processing – scientific and large-scale financial processing

Large database queries – data mining and business intelligence

Video on demand – streaming movies to end users

� SVC posted the highest results in this industry-recognized storage performance testSVC (3.517 GB/s) + DS8300 (3.217 GB/s) lead the industry in this new benchmark



Enterprise Usage Today

� Widely accepted by customers:

Tape drive and library virtualization

Disk block level virtualization in LVM and storage subsystem

In-band file system virtualization (NAS)

Aggregated networks (iSCSI, long distance connections)

� Beginning of the lifecycle:

Disk block fabric-layer virtualization (in-band and out-of-band)

� Niche markets (this may change):

Virtual SANs

File and record virtualization

IT-Symposium 2007 17.04.2007




Why Do Customers Hesitate?

� Reliability

SAN and storage failures may affect the whole datacenter.

SAN and storage failures may destroy data.

Security implications not clear.

� Complexity

Lack of standards.

Products difficult to understand and evaluate.

Multi-vendor support.

� Market situation

Religious wars about architectures.

Too many different solutions.



Disclaimer InformationCopyright © 2007 by Inte rnatio na l Bus iness Machines Corporation.

No part o f this document ma y be reprod uced or transmitted in any form w ithout writte n permissio n from IBM Corporation.

All stateme nts regarding IBM 's future dire ctio n and intent are subje ct to cha nge or withdra wa l without notice, and

represent goals and objective s only.

Refere nce s in this document to curre nt or future IBM produc ts, progra ms, or services does not imp ly tha t IBM intend s to make

suc h such products, programs or services availab le in a ll countr ies in whic h IBM operates or does bus iness.

THE INFORMATION PRO VIDED IN THIS DO CUMEN T IS DISTRIBUTED "AS IS" WITHOUT ANY WARRAN TY,

EITHER EXPRESS O R IMPLIED. IBM EXP RESSLY DISCLAIMS AN Y WARRANTIES O F MERCHAN TABILITY,

FITNESS FO R A PARTICU LAR PURPOS E OR IN FRIN GEMEN T. IBM shall ha ve no responsib ility to upda te this

info rmatio n. IBM products are warra nted accord ing to the terms and cond itions of the agree me nts (e.g., IBM Custo mer

Agree me nt, Statement of Limited Warra nty, Interna tional Progra m License Agreement, etc.) unde r whic h they are provided.

The provisio n o f the informatio n contained herein is no t intended to, and does not, gra nt any r ight or lice nse under any IBM

pate nts or copyrights. Inquir ies regarding patent or copyright licenses sho uld be made, in writing, to :

IBM D irector of Lice nsing

IBM Corporatio n

North Castle Drive

Armonk, NY 10504-1785

U.S.A.

storage virtualization with ibm san volume controller · storage virtualization with ibm san volume...

Documents