overview of the sdsc storage resource broker wayne schroeder (and other srb team members) may, 2004...
TRANSCRIPT
Overview of the SDSCStorage Resource Broker
Wayne Schroeder(and other SRB team members)
May, 2004
San Diego Supercomputer Center,University of California San Diego
SDSC/UCSD/NPACI
What is SRB? (1 of 3)
• The SDSC Storage Resource Broker (SRB) is client-server middleware that provides a uniform interface for connecting to heterogeneous data resources over a network and accessing unique or replicated data objects.
• SRB, in conjunction with the Metadata Catalog (MCAT), provides a way to access data sets and resources based on their logical names or attributes rather than their names and physical locations.
What is SRB? (2 of 3)
• The SDSC SRB system is a comprehensive distributed data management solution, with features to support the management, collaborative (and controlled) sharing, publication, and preservation of distributed data collections.
• The SRB also serves as middleware via a rich set of APIs available to higher-level applications and by providing a management layer on top of a wide variety of storage systems.
What is SRB? (3 of 3)
The SRB is an integrated solution which includes:– a logical namespace,– interfaces to a wide variety of storage systems, – high performance data movement (including parallel I/O), – fault-tolerance and fail-over, – WAN-aware performance enhancements (bulk operations), – storage-system-aware performance enhancements ('containers' to aggregate files),– metadata ingestion and queries (a MetaData Catalog (MCAT)), – user accounts, groups, access control, audit trails, GUI administration tool– data management features, replication– user tools (including a Windows GUI tool (inQ), a set of SRB Unix commands, and Web
(mySRB)), and APIs (including C, C++, Java, and Python).
SRB Scales Well (many millions of files, terabytes)
Supports Multiple Administrative Domains / MCATs (srbZones)
And includes SDSC Matrix: SRB-based data grid workflow management system to create, access and manage workflow process pipelines.
SRB Projects• Digital Libraries
– UCB, Umich, UCSB, Stanford,CDL– NSF NSDL - UCAR / DLESE
• NASA Information Power Grid• Astronomy
– National Virtual Observatory – 2MASS Project (2 Micron All Sky Survey)
• Particle Physics – Particle Physics Data Grid (DOE)– GriPhyN – SLAC Synchrotron Data Repository
• Medicine– Digital Embryo (NLM)
• Earth Systems Sciences– ESIPS– LTER
• Persistent Archives– NARA– LOC
• Neuro Science & Molecular Science– TeleScience/NCMIR, BIRN– SLAC, AfCS, …
Over 90 Tera Bytes in 16 million files
SRB Scalability
NPACI 6,050.00 2,317,368 367 8,350.00 2,903,386 372 8,570.00 2,946,526 375 8,822.00 2,995,432 377Digsky 46,100.00 5,719,025 68 46,100.00 5,719,025 68 46,100.00 5,719,025 68 42,786.00 6,076,982 69DigEmbryo 720.00 45,365 23 720.00 45,365 23 720.00 45,365 23 720.00 45,365 23HyperLter 215.00 5,097 27 215.00 5,097 28 215.00 5,097 28 215.00 5,097 28Hayden 7,078.00 59,399 142 7,130.00 59,781 158 7,830.00 59,983 158 7,835.00 60,001 168Portal 968.00 27,250 316 1,133.00 31,717 339 1,141.00 32,396 342 1,244.00 34,094 352SLAC 1,790.00 254,974 43 1,790.00 254,974 43 1,790.00 254,974 43 2,108.00 294,149 43NARA/Collection 52.80 79,195 51 53.10 79,112 55 53.90 79,118 55 67.00 82,031 56NSDL/SIO Exp 232.00 15,809 23 315.00 27,170 23 418.00 39,253 23 603.00 87,191 26TRA 90.60 2,385 25 92.00 2,378 26 92.00 2,387 26 92.00 2,387 26LDAS/SALK 498.00 9,858 60 737.00 12,898 66 767.00 12,906 66 824.00 13,016 66BIRN 121.00 237,283 138 273.00 675,531 145 382.00 2,048,132 156 389.00 1,084,749 167AfCS 95.30 18,762 20 99.00 19,714 20 102.00 20,260 20 107.00 21,295 21UCSDLib 1,084.00 138,415 29 1,085.00 138,421 29 1,085.00 138,421 29 1,085.00 138,421 29NSDL/CI 278.00 993,886 113 379.00 2,596,090 114 379.00 2,596,090 114 465.00 2,948,903 114SCEC 12.60 18,660 38 7,561.00 1,249,144 39 9,680.00 1,561,396 40 12,274.00 1,721,241 43TeraGrid 623.00 36,508 1,978 1,664.00 47,644 1,942 1,745.00 49,106 2,073 10,603.00 433,938 2,229
TOTAL 66,008.30 9,979,239 3,461 77,696.10 13,867,447 3,490 81,069.90 15,610,435 3,639 90,239.00 16,044,292 3,837
66 TB 9.97 million 3 thousand 77 TB 13.9 million 3 thousand 81 TB 15.6 million3 thousand 90 TB 16 million 3 thousand
Project Instance
As of 10/01/2003
Does not cover databases; covers only files stored in file systems and archival storage systems ** Does not cover data brokered by SRB spaces administered outside SDSC.
Count (files)Data_size (in
GB)Data_size (in
GB)Count (files)Users
Storage Resource Broker (SRB)Data brokered by SDSC instances of SRB**
Users
As of 7/24/2003
Count (files) Users
As of 9/12/2003
Does not cover shadow-
CommentsData_size (in
GB)Data_size (in
GB)Count (files)
Users
As of 11/14/2003
Case Study: SRB in BIRN
BIRN Toolkit
Mediator
Viewing/Visualization Queries/ResultsApplications Data Management
File System
MCAT
HPSS
Data M
od
elD
ata Access
Data G
ridC
ompu
tatio
nal G
rid
Collaboration
NM
IG
rid
Man
agem
ent
Globus
GridPort
Scheduler
Distributed Resources
Database
SRB
Database
SRB History
• A DataGrid since SRB 1.0, Production 1997• SDSC Started by General Atomics, 1985
– GA/UCSD Staff– On UCSD Campus– SRB by GA Employees
• Today, SDSC no longer GA, all UCSD– All staff UCSD employees
• GA Commercial SRB Version (Nirvana)– Based on SRB 1.1.8 (2001)– Nirvana and SDSC versions diverged– SDSC SRB free to academic organizations– License from Nirvana for commercial
SRB – A Data Grid Solution
• Storage Resource Broker– Collaborative client-server system that federates
distributed heterogeneous resources using uniform interfaces and metadata
– Provides a simple tool to integrate data and metadata handling – attribute-based access
– Blends browsing and searching
– Developed at SDSC - Operational for 5+ years;
- Under continual development since 1997;
- Customer-driven;
- Brokering over 90 TeraBytes in over 16 million files at SDSC
Using a Data Grid - Details
SRB
MCAT
DB
SRB
SRB
SRB
SRB SRB
•Data Grid has arbitrary number of servers•Complexity is hidden from users
SDSC Storage Resource Broker & Meta-data Catalog
SRBArchives
HPSS, ADSM,UniTree, DMF
DatabasesDB2, Oracle,
Sybase
File SystemsUnix, NT,Mac OSX
Application
C, C++, Linux I/O
Unix Shell
Dublin Core
Resource,User
User Defined
ApplicationMeta-data
RemoteProxies
DataCutter
Third-partycopy
Java, NTBrowsers
WebPrologPredicate
MCAT
HRM
SRBserver
SRB agent
SRBserver
Federated SRB Operation
MCAT
Read Application in Boston
SRB agent
1
2
3
4
6
5
5/6
Logical NameOr
Attribute Condition
1.Logical-to-Physical mapping2. Identification of Replicas3.Access & Audit Control
Peer-to-peer Brokering
Server(s) Spawning
Data Access
Parallel Data Access
R1 R2R2
San Diego
Durham
inQ Windows GUI
Virtual Hierarchical Collection Management
Attributes
• SRB metadata– Location, protocol– Unix semantics– Authorization, authentication– Latency management– Container aggregation
• Administrative– Dublin core, provenance– Annotations, comments
• Discipline specific attributes– Collection– User defined
Authentication Management
Grid Security Infrastructure (GSI) Encrypted Password GSS-API for Kerberos or DCE Collection-owned Data
Collection ID installed at each storage system Users authenticate themselves to the SRB SRB authenticates to local server Or GSI Delegation (Ananta Manandhar, CCLRC)
Logical File Name
One of the major functions of SRB is the mapping between a logical file name and itsphysical file. The mapped info of a logical filename includes:
Location of name in collection hierarchy Physical file location: host name and path Protocol: for fetching ‘local’ file Unix semantics for file manipulation Location in container Audit trail Access control list Locking status
Replica Management
Files can be replicated into any valid physical storage resource registered in SRB.
Each replica is managed by the same logical filename as the original one and a unique replication number. Each replica can have unique metadata.
1-to-many Replication: A logical resource can contain several physical storage resources.
Multiple replicas can be made to the same storage resource Many Modes of Replication:
Synchronous Replication; Sput to a logical resource Asynchronous Replication; Sput then later Sreplicate Out of Band Replication; Outside SRB, then register
Containers
• Physical Grouping of Objects• Similar to tar but has significant differences• Multiple Uses:
– To take advantage of resource characteristics– To aid access patterns– Move data sets together– Tie together logically different files– Automatic Archiving/Caching
• Chaining of Containers• Sharing of metadata• Containers for Collections
Proxy Operation
• Proxy operation - – server performs operations on behalf of client– performs operations where the data are located– subset and filter operations – datacutter– Metadata extraction and ingestion checks– srbExecCommand() API and Spcommand
utility -• request a specific server to execute a specific
command and stream the result to stdin• used by the NVO(national virtual observatory)
cutout service
SRB – More Features
• Client Support– Pure Java Client– Web Services - WSDL, Matrix workflow system– Web Support - MySRB Extensions– Pure Java Client & Browser– inQ Version 3.1 and more Windows Support
• Administrative Support– GUI-based Administration– More Features - Resource, User, Method Management– User-friendly Installation Procedures
Metadata Management
Metadata Insertion Through User Interfaces Bulk Metadata Insertion Template Based Metadata Extraction Metadata Search
system data user-defined metadata File Content Search: Key words are pre-extracted by a
template and saved as user-defined metadata.
Storage Resource Broker
• SRB wears many hats:– It is a distributed but unified file system– It is a database access interface– It is a digital library– It is a semantic web– It is a data grid system– It is an advanced archival system
Criticisms of SRB
• Not completely open source – But semi-open and available to academics
• Not standards-based – But internal protocols need not be
• Monolithic– Integrated– And well partitioned
Some SRB Weaknesses (my view)
• Difficult to explain and understand– SRB does so much, people tend to learn subsets and
are often unaware of useful features– Different groups are interested in different sets of
features– An “elevator speech” is either vague or incomplete
• Not completely open source• Collaborations difficult
– Need to expand
• Limited Staff– Feature-focused projects (+/-): docs, error messages
Some SRB Strengths
• Integrated solution– High performance– Highly functional– Relatively easy to enhance
• Middle-ware and Complete-ware• Customer driven• Sound architecture• Mature, but also being actively developed• Growing user base• Highly coordinated centralized team
TeamSRB, San Diego
• Reagan Moore (Program Director, DAKS)• Arcot Rajasekar (Director)• Michael Wan (Chief Architect)• Wayne Schroeder• George Kremenek• Bing Zhu• Sheau-Yen Chen• Charles Cowart• Arun Jagatheesan (GriPhyN)• Lucas Gilbert• Roman Olsachnowsky (BIRN)• Tim Warnock (BIRN)
Contacts
• For Additional Information:Web: http://www.npaci.edu/dice/srb
Mail: [email protected]
Mailing-list: [email protected]