storage tank in data grid
Post on 06-Jan-2016
32 Views
Preview:
DESCRIPTION
TRANSCRIPT
Storage Tank in Data Grid
Shin, SangYong(syshin, #6468)
IBM Grid Computing
August 23, 2003
Storage Architecture Model
- app. data is in files
- files stored on block storage
- all managed by storage mgmt s/w
Application
File System
Block Virtualization
Storage Devices
Block subsystem
Sto
rage M
anag
emen
t
SAN
Block Virtualization
Today
SAN
Block Virtualization
Common view of block storageNo server impact on storage change
EmergingNo common view of block storageServer impact on storage change
- IBM block virtualization is Lodestone
High-end
Hos
t
Hos
t
Hos
t
Application
MidrangeDisk array
Shark, Brand X
...
FastT, Brand Y
RAID Brick
LVE = Lodestone Virtualization Engine
LVE
Disk array
FunctionsProviding Virtual DisksOnline- Dynamic Volume SizingAdvanced Copy FunctionsEconomic Disaster Recovery
Solutions.Different Level of PerformanceData Backup with low price diskNo Service Downtimeetc
LVE LVE LVE
Extending Lodestone for Grid
File Systems - Current Capabilities Vs Grid Requirements
SAN
GPFSHPC, Engineering, Digital MediaAccess from servers in a clusterConcurrent multiple I/OsAIX and Linux OS onlyNo access to other FS data
Storage TankCommercial, file sharing, DB servingAccess from servers on SANAll servers and OSesNo access to other FS data
Grid requirementsAccess from any machine, any OS, anywhereAccess to all file system data
Planned Approach:Allow remote access to our file systemsProvide multi-site supportIntegrate data from other sources
NFSv4 support for our file systems
We believe NFSV4 will be an important protocol for the grid
• has the necessary extensions for robust security and WAN access
• is the first NFS protocol to come through the standards process
• proposed standard in Dec. 2002; expected to be draft standard by 4Q03
Our plan is to provide NFSv4 support for our file systems (J2, GPFS and Storage
Tank)
• Best case will be late 2004
Storage Tank (ST) - a SAN file system
SN
ST agent
AIX
ST agent
Solaris
BackupData Data
data
Meta
-Meta data
ST agent
Linux
Capabilities:access to ST data through Globus GridFTP interfaceregister ST files in Globus Replica Location Serviceenabled to support OGSA services (e.g. replication)centralized, policy-based storage managementcross-platform file sharingperformance comparable to local file system with direct client-to-storage data path
ST agent
Win2K
ST Server
ST Server
ST Server
GridFTP, NFS
Prototypes: 2H02-1H03Customer: CERN
file attributes, file location info, control infoLAN
ST Clients
CERN Requirements
Data analysis of Large Hadron Collider (LHC) experiments• Basic unit of data is an LHC event• data represents physical collision between 2 protons• 1 to few MBs• stored within 1 GB files• event metadata stored in an RDBMS
Tiered structure• CERN is Tier 0• event data and metadata distributed to Tier 1 centers• physicists at Tier 2 centers analyze data at Tier 1 centers
2.4 PB of disk and 14 PB of tape by 2007
Grid access (AFS/DFS like), simple storage management
IP SANs, not FC
Our Proposal
Use Storage Tank for basic storage infrastructure
Use iSCSI disks• FAStT with iSCSI gateway or 200i
DB2 for event metadata
Research extensions• NAS head for Storage Tank• Grid access to Storage Tank• Object Store prototype for disks
Single namespace across multiple sites- Replication of files for good performance- Extended protocols for consistency across
replicas- Joint research w/ Johns Hopkins underway
Tank
NYC FargoSFO
Control Network (IP)
SANMeta-data
ServerCluster
Win2K
ST Agent
AIX
ST Agent
Solaris
ST Agent
Linux
ST Agent
data
Integrated ST/NASAppliance
Meta-dataServerCluster
Control Network (IP)
SANMeta-data
ServerCluster
Win2K
ST Agent
AIX
ST Agent
Solaris
ST Agent
Linux
ST Agent
data
ST ExtensionsPrototype: 1H04Customer: CERN, JHU
Branch office
Extend ST to Multiple Sites – Distributed Storage Tank
Ultimate Vision for Federated Grid File Systems
Access Server
Client
Client
. . .exporter exporter
Access Server
Client
Client
. . .exporter exporter
Proxy Server
Organization 1 Organization 2
file sources
Control Network (IP)
SANMeta-data
ServerCluster
Win2K
ST Agent
AIX
ST Agent
Solaris
ST Agent
Linux
ST Agent
dataControl Network (IP)
SANMeta-data
ServerCluster
Win2K
ST Agent
AIX
ST Agent
Solaris
ST Agent
Linux
ST Agent
data
Grid data repository
GridFTP
NFS
NAS data repository
Client
Client
Client
Client
NFS
NFS
Extend ST to access data from other file systems/sources
Storage Management in Grid Computing Environment
Applications
Lodestone Storage Tank Shark, Tape, etc.
CIM Provider Interface CIM Provider InterfaceCIM Provider Interface
Storage ManagementServices
CIM/XML
OGSA
OGSA-CIM Wrapper
CIM/XML
IBM storage management products
today (TSM, TSRM, ITSANM) and
planned products (Merlot) cover a
reasonable set of functions We are converging, with the industry,
on CIM/XML as the standard for
storage device management In support of grid, we expect:
• to convert our management
solutions to Web/OGSA services• to enhance functionality
We are just starting to focus on grid implications for storage management
Block subsystem
Summary of Data Grid
Extend ST & GPFS
Lodestone
Support OGSA upper interfaceSupport CIM lower interface
OGSA
CIM
Application
File System
Block Virtualization
Storage Devices
Block subsystem
Sto
rag
e M
an
ag
em
ent
top related