Internet2 Distributed Storage Infrastructure Update

Micah BeckUniv. of Tennessee, Knoxville

Bert DempseyUniv. of North Carolina, Chapel Hill

Web Caching Workshop BOF31 March 1999, San Diego

http://dsi.internet2.edu

I2-DSI Participants

UT Knoxville / ICL Micah Beck Terry Moore Martin Swany Judi Talley

UNC Chapel Hill /SILS Bert Dempsey Paul Jones (MetaLab) Debra Weiss Zhiwei Xiao

GigaPOP and Campus Site Managers

UCAID/Internet2 Network Storage

Working Group Ted Hanss

Applications Director

NC Networking Initiative Digital Library Federation

A Word From Our Sponsors

Cisco DNS redirection Ellemtel engineering effort IBM large storage & DCE serversNovell storage & directory serversStarburst reliable multicast softwareStorageTek large storage serversSun design collaboration

Single Server Model

High performance locally

Unacceptable performance across commodity backbone

Relying on Wide Area QoS

High performance access with reserved bandwidth

Essential for real-time communication

Technically difficult, expensive, not generally available

I2-DSI Model: Replicated Services

Clients access nearby server

Everyone gets performance

Local resources implement a global service

I2-DSI Service Architecture

Replication

Rsynch+, Omnicast, AFS/DFS

Novell Replication

Resolution

Sonar DNS, Distributed

Director

Delegation

Cache prefetch

generalusers

replicated core

delegated server

local users

Internet Content Channels

A channel is a collection of content which can be transparently delivered to end user communities at a chosen (price,performance) point through a flexible, policy-based application of resources

Server Channel Examples

Replicated Web Servers APIs: Standard HTML, Active Server Pages Channels: Web sites

Streaming Media APIs: MPEG-2, proprietary file formats Channels: collections of multimedia presentations

Executable content APIs: Java byte code, Tcl, Perl Channels: CGI programs

Current Server Deployment

IBM Web Cache Manager

RS/6000 AIX Server1 GB RAM72 GB Disk / 900 GB TapeADSM Heirarchical Storage Mgt.

I2-DSI Server Operations

Project Operations Coordinator Judi Talley, University of Tennessee at Knoxville

Site Managers Dave Vernon, Indiana University David Lassner, University of Hawaii at Manoa Mark Johnson, NC Networking Initiative Michael Rechtenbaugh, EROS Data Center

Infrastructure Expansion

StorageTek 2 PC/Linux Servers 700GB disk, tape backup (hot!)

Novell 6 PC/NetWare Servers 100GB disk Smaller institutions or departments

InfrastructureDevelopment Efforts

Proximity ResolutionMartin Swany: SonarDNSGeoff Carpenter, German Goldszmidt: Narwhal (IBM)

Replication Mechanisms and Modeling Bert Dempsey & students:

Debra Weiss: Batch rsync + multicastZhiwei Xiao: Network metrics and modeling

Channel Representation and ServerLeif Abrahamsson, Christophe Achouiantz,

Patrik Johansson (Ellemtel)

I2-DSI Applications Workshop Chapel Hill, NC March 4 &5, 1999

10 applications Indiana: Digital music and

media library UNC-CH: Instructional

Management System San Jose State: Art history

images Vanderbilt: zoomable medical

images Viagenie: Network docs

database

Columbia: Earth sciences environment

UNC-CH: Virtual Laboratories

Ohio Supercomputer Center: High Volume Datasets

CalTech: Globally Interconnected Databases

Univ. of Kent: National Software Archive

Red Hat: pan-Linux source distribution

I2-DSI Applications Workshop Chapel Hill, NC March 4 &5, 1999

4 technologiesMinnesota: Scalable Video IBM Research: Multicast, Filter and StoreMoscow Ctr. for New Info. Tech. in Med. Ed.:

Semantic Text AnalysisIBM Research: Narwhal Resolution Proxy

http://dsi.internet2.edu/apps99.html

Special issue of the Journal of Network and Computer Applications (Academic Press)

Application Management Partner MetaLab.unc.edu

The site formerly known as SunSITE.unc.eduFearless Leader Paul Jones“A cool, tall glass of sweet tea on a hot day.” 2 M HTTP & 1/3 M FTP file transfers dailyCollections policy

teaching, research, or public service use technology in innovative and unique ways non-commercial or not-for-profit

Application Strategy

Chose initial applications Available or easily ported services Low update demands

Port to an I2-DSI server Our development effort is limited App developers can have access to the servers

Distribute to homogeneous coreDerive service abstractions

The Need for Channel Representation Standards

Origin Server

proxy

Replicated Server

Standard-based Web traffic

Replication of source files

Web clients

locally interpreted files

Replicated Server

replicated files

Replication Performance and Scalability Issues

Server placementServer resourcesServer description (metadata)Server Channel description (metadata)Object representationCharacterization of replication mechanismsChannel-to-server mapping (subscription)

NetStore ‘99 Workshop

Network Storage Technical Workshop Knoxville, TN, October 1999 http://dsi.internet2.edu/netstore99

Scope I2-DSI implementation I2-DSI applications Related networking projects Storage technology

Conclusions

A server platform is in placeInfrastructure development

Service abstractions (search, computation) Publication and replication protocols Portable representation and API Heterogeneous servers

Six months to show results from initial application development efforts