1

Put Everything in Future (Disk) Controllers(it’s not “if”, it’s “when?”)

Jim Gray

http://www.research.Microsoft.com/~Gray

Acknowledgements:

Dave Patterson explained this to me a year ago

Kim Keeton

Erik Riedel

Catharine Van Ingen

Helped me sharpen these arguments

2

Remember Your Roots

3

Technology Drivers: Disks• Disks on track

• 100x in 10 years 2 TB 3.5” drive

• Shrink to 1” is 200GB

• Disk replaces tape?

• Disk is super computer!

Kilo

Mega

Giga

Tera

Peta

Exa

Zetta

Yotta

4

Data Gravity Processing Moves to Transducers

• Move Processing to data sources

• Move to where the power (and sheet metal) is

• Processor in

»Modem

»Display

»Microphones (speech recognition) & cameras (vision)

»Storage: Data storage and analysis

5

It’s Already True of PrintersPeripheral = CyberBrick

• You buy a printer

• You get a

»several network interfaces

»A Postscript engine • cpu, • memory, • software,• a spooler (soon)

»and… a print engine.

6

Tera Byte Backplane

• TODAY» Disk controller is 10 mips risc engine

with 2MB DRAM

» NIC is similar power

• SOON» Will become 100 mips systems

with 100 MB DRAM.

• They are nodes in a federation(can run Oracle on NT in disk controller).

• Advantages» Uniform programming model» Great tools» Security» economics (CyberBricks)» Move computation to data (minimize traffic)

All Device Controllers will be Cray 1’s

CentralProcessor &

Memory

7

Basic Argument for x-Disks• Future disk controller is a super-computer.

»1 bips processor»128 MB dram»100 GB disk plus one arm

• Connects to SAN via high-level protocols» RPC, HTTP, DCOM, Kerberos, Directory Services,…. »Commands are RPCs»management, security,….»Services file/web/db/… requests» Managed by general-purpose OS with good dev environment

• Move apps to disk to save data movement

»need programming environment in controller

8

The Slippery Slope

• If you add function to server

•Then you add more function to server

•Function gravitates to data.

Nothing = Sector Server

Everything = App Server

Something =

Fixed App Server

9

Why Not a Sector Server?(let’s get physical!)

• Good idea, that’s what we have today.

• But

»cache added for performance

»Sector remap added for fault tolerance

»error reporting and diagnostics added

»SCSI commends (reserve,.. are growing)

»Sharing problematic (space mgmt, security,…)

• Slipping down the slope to a 2-D block server

10

Why Not a 1-D Block Server?Put A LITTLE on the Disk Server• Tried and true design

»HSC - VAX cluster»EMC»IBM Sysplex (3980?)

• But look inside»Has a cache »Has space management»Has error reporting & management»Has RAID 0, 1, 2, 3, 4, 5, 10, 50,…»Has locking»Has remote replication»Has an OS»Security is problematic»Low-level interface moves too many bytes

11

Why Not a 2-D Block Server?Put A LITTLE on the Disk Server• Tried and true design

»Cedar -> NFS»file server, cache, space,..»Open file is many fewer msgs

• Grows to have»Directories + Naming»Authentication + access control»RAID 0, 1, 2, 3, 4, 5, 10, 50,…»Locking»Backup/restore/admin»Cooperative caching with client

• File Servers are a BIG hit: NetWare™»SNAP! is my favorite today

12

Why Not a File Server?Put a Little on the Disk Server

• Tried and true design

»Auspex, NetApp, ...

» Netware

• Yes, but look at NetWare

»File interface gives you app invocation interface

»Became an app server• Mail, DB, Web,….

»Netware had a primitive OS• Hard to program, so optimized wrong thing

13

Why Not Everything?

Allow Everything on Disk Server(thin client’s)

• Tried and true design

»Mainframes, Minis, ...

»Web servers,…

»Encapsulates data

»Minimizes data moves

»Scaleable

• It is where everyone ends up.

• All the arguments against are short-term.

14

The Slippery Slope

• If you add function to server

•Then you add more function to server

•Function gravitates to data.

Nothing = Sector Server

Everything = App Server

Something =

Fixed App Server

15

Disk = Node• has magnetic storage (100 GB?)

• has processor & DRAM

• has SAN attachment

• has execution environment

OS KernelSAN driver Disk driver

File System RPC, ...Services DBMS

Applications

16

Technology Drivers: System on a Chip

• Integrate Processing with memory on one chip»chip is 75% memory now»1MB cache >> 1960 supercomputers»256 Mb memory chip is 32 MB!»IRAM, CRAM, PIM,… projects abound

• Integrate Networking with processing on one chip»system bus is a kind of network»ATM, FiberChannel, Ethernet,.. Logic on chip.»Direct IO (no intermediate bus)

• Functionally specialized cards shrink to a chip.

17

Technology Drivers: What if Networking Was as Cheap As Disk IO?

• TCP/IP

»Unix/NT 100% cpu @ 40MBps

• Disk

»Unix/NT 8% cpu @ 40MBps

Why the Difference?Host Bus Adapter does

SCSI packetizing, checksum,…flow controlDMA

Host doesTCP/IP packetizing, checksum,…flow controlsmall buffers

18

Technology Drivers: The Promise of SAN/VIA:10x in 2 years

http://www.viarch.org/

• Today: »wires are 10 MBps (100 Mbps Ethernet)

»~20 MBps tcp/ip saturates 2 cpus

»round-trip latency is ~300 us

• In the lab»Wires are 10x faster Myrinet, Gbps Ethernet, ServerNet,…

» Fast user-level communication• tcp/ip ~ 100 MBps 10% of each processor

• round-trip latency is 15 us

19

Gbps Ethernet: 110 MBps

SAN: Standard Interconnect

PCI: 70 MBps

UW Scsi: 40 MBps

FW scsi: 20 MBps

scsi: 5 MBps

• LAN faster than memory bus?

• 1 GBps links in lab.

• 100$ port cost soon

• Port is computer

RIPFDDI

RIPATM

RIPSCI

RIPSCSI

RIPFC

RIP?

20

Technology Drivers:100 GBps Ethernet replaces SCSI

• Why I love SCSI

»Its fast (40MBps)

»The protocol uses little processor power

• Why I hate SCSI

»Wires must be short

»Cables are pricy

»pins bend

21

Functionally Specialized Cards• Storage

• Network

• Display

M MB DRAM

P mips processor

ASIC

ASIC

ASIC

Today:

P=50 mips

M= 2 MB

In a few years

P= 200 mips

M= 64 MB

22

Technology DriversPlug & Play Software

• RPC is standardizing: (DCOM, IIOP, HTTP)

»Gives huge TOOL LEVERAGE»Solves the hard problems for you:

• naming, • security, • directory service, • operations,...

• Commoditized programming environments » FreeBSD, Linix, Solaris,…+ tools» NetWare + tools» WinCE, WinNT,…+ tools» JavaOS + tools

• Apps gravitate to data.

• General purpose OS on dedicated ctlr can run apps.

23

Basic Argument for x-Disks• Future disk controller is a super-computer.

»1 bips processor»128 MB dram»100 GB disk plus one arm

• Connects to SAN via high-level protocols» RPC, HTTP, DCOM, Kerberos, Directory Services,…. »Commands are RPCs»management, security,….»Services file/web/db/… requests» Managed by general-purpose OS with good dev environment

• Move apps to disk to save data movement

»need programming environment in controller