redefining storage economics

Redefining Storage Economics

Redefining Storage EconomicsSan Diego VMUG July 25th, 201

Storage for Performance

• Coraid Introduction• Customer Case Study• Performance Overview

• Throughput• IOPS• RAID Math• Disk bound/controller bound

• Testing Tools & Tips• Product Demo• Q&A

Redefining Storage Economics

Technology Overview

Disrupting the SAN Industry

4

• Ethernet SAN technology: EtherDrive®

• 5-8x price performance advantage vs legacy

• Radically simplified SAN topology

In production at more than 1,400 companies and federal agencies

Optimized for VMware virtualization

Works with existing infrastructure

Storage 1.0 Storage 2.0 Storage 3.0

Companies

Business DriverNeed for shared storagePerformanceAvailability

Commoditization of Servers Virtualization

Commoditization of StorageCloudFlexibility

Architecture Mainframe servers to central controllers to mainframe storage

x86 servers to central controllers to mainframe storage x86 Servers to x86 Storage

Requirements High-PerformanceHigh-Availability

Varying PerformanceVarying AvailabilitySimplified management

On demand, easy to scale, low costStorage pools not silosVarying-Performance Capabilities

Key Innovations Fibre Channel, dual controllers, replication, mirroring

iSCSI, auto-tiering, caching, management capabilities

Ethernet connectivityControllerless architecturesDistributed processingCommodity hardware

Evolution of SAN

ELIMINATE RAISE

REDUCE CREATE

The Consumer is asking for …………..Evolving to Cloud

ProprietaryCustom HW

PerformanceScalability

Complexity

NewEconomic

Model

CustomerValue

• Transparent Scalability• Industry Standard Hardware• No “Controller Headaches”

• High Performance• 1GbE / 10GbE• 530MB - 1800MB / sec

• Low Budget Impact• $600 - $1250 / TB

The Future of Storage…

“Ethernet SANs are less complex, perform with better economics”Enterprise Strategy Group – Jan 2011

• Dynamic application to storage relationships

• Scale-out compute infrastructure

• Unpredictable, variable application access profiles

Storage Challenges

7

• Static connections

• Scale-up compute tied to scale up storage

• Predictable application access profiles

Legacy SAN Topology: Rigid, ExpensiveA Bottleneck: Dynamic Virtual WorkloadsB

Expensive HBAs, Static Workload

Complex Multipathing

Controlled Data Layout on Drives

Server Cluster with VMotion

Extremely Complex SAN Management

Chaotic Data Layout on Drives: Head Contention

ESG Research: The Evolution of Server Virtualization

Enterprise Strategy GroupThe Evolution of Server Virtualization

Bowker / Oltsik – November 2010

ESG Research: The Evolution of Server Virtualization

Impact on Customers:• “We have increased our use of SAN-based storage…”• “It has caused us to purchase from new storage vendors…”

ESG Research: The Evolution of Server Virtualization

11

Coraid EtherDrive: Scale-out Ethernet SAN

EtherDrive: Dynamic Virtual Workloads

Server Cluster with vMotion™ AoE EtherDrive

Storage Arrays

EtherDrive Benefits

Ethernet(1Gb / 10Gb)

MassivelyParallel

5-8x Price Performance Advantage• “Bare metal performance” • Off-the-shelf hardware

Operational Simplicity:• Eliminates complex topologies and

multipathing• Simple recovery – Zero Hour Support

Scale-out• No controller bottleneck• Grow in-line with business demand

Coraid = Ethernet-SAN

Connectivity Ethernet-SAN, not IP-SAN (iSCSI) or FC-SAN

Hardware x86-based, enterprise-level commodity shelves

Controller Per-shelf, not centralized

Maximum Capacity Per shelf: 108TBs, Per system: 65,000 shelves

Operating System CorOS distributed operating system

Shelf sizes 8, 16, 24, and 36-disk shelves

Supported Drives SATA, SAS, and SSD can be configured in the same chassis at the same time

Supported RAID levels 0, 1, 5, 6, 10, and JBOD (RAID-Z) in the same chassis

Availability Features Synchronous mirroring, snapshots, cloning, asynchronous replication

BackgroundDevelopment Timeline

QOS with Coraid : Modular Flexible SLA’s

85%+ SATA

SSD forReplicas

• Multi-tiered storage in a single shelf• Test small, grow in stages while preserving low $/desktop

EtherDrive™ Deployment Methods

• Ethernet cables can be connected directly to a host server or the connections can be made through a standard Ethernet switch

• A high availability SAN fabric should be created if possible by leveraging dual switches

Direct AttachedNote: Each SR/SRX port can be directly to a separate host

Networked

High Availability Fabric

Coraid – Filesystem Agnostic Blocks

NFS VMFS ZFS NTFS GPFS

Coraid EtherDrive

Coraid works well with:

• Virtualization • File sharing• Clustering• Single-namespace• OS operations• High performance computing• Etc.

• Layer 2 Protocol– Non-routable– SAN Fabric can span multiple Layer 2 Switches– Requires no IP Addressing Scheme

• Not Connection-Based– Physical Connection to SAN Fabric is only requirement– Simplified Storage Configuration

• Transmission Controls Built-In• iSCSI utilizes Layer 4 TCP for Transmission Control• Fibre Channel uses proprietary $$ hardware• Simple and Efficient

– 12 Page Specification• http://support.coraid.com/documents/AoEr11.txt

ATA over Ethernet - AoE Protocol

Flexible, High Performance, Scale Out

Storage

• High-density Ethernet SAN array, targeted at cloud and large enterprise environments

• SRX-Series supports up to:― 36 Drives (24/12 in 4U)― 108 TB with 3 TB drives― SATA/SAS/SSD - mix in same shelf― 10 Gb Ethernet― 1,800+ MB/sec throughput

• Starting at under $600/TB

18

Coraid EtherDrive SRX-Series

EtherDrive SRX Overview:

EtherDrive SRX Options:• SRX2800 – 2U, 16x 3.5” disks• SRX3200 – 4U, 24x 3.5” disks• SRX3500 – 2U, 24x 2.5” disks• SRX4200 – 4U, 36x 3.5” disks

Redefining Storage Economics

Customer Case Study

Redefining Storage Economics

Customer Case Study

• Cloud Application Provider• 21 ESXi 4.1 Servers• 8000 IOPs• 30 TB• Requirements:

– Cost sensitive– Cannot affect uptime– Flexible platform– GigE Connectivity

Redefining Storage Economics

Customer Case Study

• 3 SRX 3200 Chassis– 48 7.2k 2TB SATA disks– 32 15k 300GB SAS disks

• Able to leverage additional capacity for resting snap deltas• Non-disruptive migration to 10gigE SAN • Hot spare chassis• StressLinux/VI Client/SAN combo performance monitoring

Coraid Architecture

Star Topology Simplicity• Scale out architecture allows for SAN

to grow linearly in terms of performance and cost

• Additional shelves can be plugged directly into the same layer 2 network segment to expand the storage easily

LAN

SAN

Redefining Storage EconomicsAdding Compute Resources

• Expand compute as easily as storage• No path management required• Connecting Coraid HBA’s in standby servers immediately

attaches all shared storage, making the node instantly ready for the cluster

• Add compute and memory resources to cluster on-demand

Attaches to clustered storage with no configuration

Redefining Storage EconomicsAdding Storage Resources

• No single point of failure scale out• Start with single chassis

• Mirror critical volumes between chassis• Critical Datastores• SSD Replica Volumes• High performance pools

Mirror

Redefining Storage Economics

Performance Overview

Redefining Storage Economics

Throughput vs IOPs

• It’s not fast enough!– What defines performance – is it mb/s or IOPs?

• Determine if the workload is random or sequential– Seemingly sequential workloads can be random based on the amount of

access or shared VMs on the system

• Random environments will require attention to the IOPs of the storage system

• Throughput intensive environments will mostly focus on connectivity type – gigE, 10gigE, multipathing

Redefining Storage Economics

Throughput Options

• Marketing vs. Real Numbers– 3gbps, 6gbps, etc– The drives will spin at around 100mb/s MAX

• Add more spindles• Add more backend connections• Add more host based connections• Using testing to determine where the bottlenecks are• Choose a lightweight transport mechanism

– Coraid delivers 1,200mb/s – line rate over 10gigE

Redefining Storage Economics

IOPs Options

• IOPS are literally a measurement of IO per second• Mechanical limitation of the disk’s head to move across the platter

back and forth within a second for a random workload• All about adding spindles• Use the right tools to determine the IOP requirement• Determine how the RAID type will affect the available IOPs

Redefining Storage Economics

IOPs Options

• Disk type always dictates IOPs• More available moving spindles and heads, more available IOPs

Disk Type Average IOPsSATA 80SAS 200SSD 5000-20000

Redefining Storage Economics

RAID Math

Redefining Storage Economics

RAID Math

• Each RAID configuration will provide the disk activity with a different IOP penalty

• Penalties are applied to write activity, but not read activity

RAID Type IO Write PenaltyRAID 0 0RAID 1 2RAID 10 2RAID 5 4RAID 6 6

Redefining Storage Economics

RAID Math

• Front end IOPs: what the host actually sees• Back end IOPs: the total amount of IOPs available to the SAN

• Use the following formula to calculate the needed backend IOPs

(TOTAL IOps × % READ)+ ((TOTAL IOps × % WRITE) ×RAID Penalty) = Needed backend IOPs

Redefining Storage Economics

Monitoring Tools

Redefining Storage Economics

Testing Tools

• StressLinux (www.stresslinux.org)• ESXTop

• ESXPlot• VI Monitoring• SAN Based

Redefining Storage Economics

SAN BasedBefore Transfer:Port0:Total Packets Received: 22603770Total Packets Transmitted: 21702657 Port1:Total Packets Received: 22603783Total Packets Transmitted: 21702648 After Transfer:Port0:Total Packets Received: 22615833 ->12063Total Packets Transmitted: 21713926-> 11269 Port1:Total Packets Received: 22615843 -> 12060Total Packets Transmitted: 21713915 ->11267

~ # grep . /proc/ethdrv/ifstats*** ctlrindx=2*** EHBA-2-E-RJ45 00004100a6e40000 ***seen=00000081Ims=000000dd Icr=00000000 c->im=000000ddRdbal=43566400 Rdbah=00000000Tdbal=43568400 Tdbah=00000000 Rxdctl=02010000Packets Received (64 Bytes): 15734677 2137009Packets Received (512-1023 Bytes): 481932 89656Packets Received (1024-mtu Bytes): 6202225 649655Good Packets Received: 22418834 2876320Broadcast Packets Received: 388305 73600Good Packets Transmitted: 21702657 2737360Good Octets Received: 47876681928 5509227416Good Octets Transmitted: 84974576160 6481749920Total Octets Received: 47890007059 5511657455Total Octets Transmitted: 84974576160 6481749920Total Packets Received: 22603770 2911266Total Packets Transmitted: 21702657 2737360Packets Transmitted (64 Bytes): 6724309 745100Packets Transmitted (512-1023 Bytes): 144998 18260Packets Transmitted (1024-mtu Bytes): 14833350 1974000Broadcast Packets Transmitted: 21566 4085Interrupt Assertion: 29782386 4010570Interrupt Rx Pkt Timer: 22418834 2876320Interrupt Rx Abs Timer: 22135042 2871060Interrupt Tx Pkt Timer: 21702657 2737360Interrupt Tx Desc Low: 21702657 2737360nrd=256 rdfree=249 rxerr=0 nobufs=0rdh=146 rdt=139 drdh=146 drdt=139ntd=256 txavail=252 dropped=13tdh=80 tdt=83dtdh=80 dtdt=83rintr=20331826 tintr=10434138 lintr=0 intr=29515795link=1000*** EHBA-2-E-RJ45 00004100a7640000 ***seen=00000081Ims=000000dd Icr=00000000 c->im=000000ddRdbal=43b8ce00 Rdbah=00000000Tdbal=43b8ee00 Tdbah=00000000 Rxdctl=02010000Packets Received (64 Bytes): 15729773 2137011Packets Received (512-1023 Bytes): 487078 89733Packets Received (1024-mtu Bytes): 6201971 649574Good Packets Received: 22418822 2876318Broadcast Packets Received: 388305 73600Good Packets Transmitted: 21702648 2737358Good Octets Received: 47877720360 5508772216Good Octets Transmitted: 84978060984 6479224616Total Octets Received: 47891252560 5511202352Total Octets Transmitted: 84978060984 6479224616Total Packets Received: 22603783 2911265Total Packets Transmitted: 21702648 2737358Packets Transmitted (64 Bytes): 6729557 745077Packets Transmitted (512-1023 Bytes): 139606 18584Packets Transmitted (1024-mtu Bytes): 14833485 1973697Broadcast Packets Transmitted: 21566 4085Interrupt Assertion: 29760202 4007037Interrupt Rx Pkt Timer: 22418822 2876318Interrupt Rx Abs Timer: 22138946 2870738Interrupt Tx Pkt Timer: 21702648 2737358Interrupt Tx Desc Low: 21702648 2737358nrd=256 rdfree=230 rxerr=0 nobufs=0rdh=134 rdt=108 drdh=134 drdt=108ntd=256 txavail=252 dropped=7tdh=100 tdt=103dtdh=100 dtdt=103rintr=20325656 tintr=10436036 lintr=0 intr=29490268link=1000

Redefining Storage Economics

Demo

References

• Configuring Coraid EtherDrive SAN appliances and deploying with ESX/ESXi 3.5 and 4.x (Partner Support)

• http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1031322

• Vmware and Coraid Technology Alliance Partner• http://www.coraid.com/pdf/app_notes/VMW_1110_Coraid_TAP.pdf