tg 06 srb intro tutorial

8/14/2019 TG 06 SRB intro tutorial

1/26

By: Roman Olschanowsky

[email protected]

An Introduction to the


2/26

Outline

SDSC and History of SRB

Example Project

Introduction to SRB

Discussion on SRB basics

SRB Clients

Overview of a Data Grid

Infrastructure

Topology

Teragrid Demo

How to use your TG SRB account

How to access Digital Data Collections


3/26

ArchivalSystems

18 PB

15.6 TF

DataStarIBM Power4

4.4 TF

TeraGrid LinuxCluster (IA64)

1.4 PB

Storage AreaNetwork Disk Sun F15K

DiskServer

NetworkingVisualization

Storageand Compute

Resources

Human infrastructure:Experienced multi-

disciplinary staff support a

broad spectrum of nationalscience, engineering

and technology projects

Blue Gene/L

(Due 12/04)

2.8/5.7 TF

www.sdsc.edu
http://www.sdsc.edu/http://www.sdsc.edu/


4/26

Sites Using the SRBCiteSeer, Penn State

City Univ. of New YorkGeospatial Environment, UCSD

Drexel UniversityEOSDIS Distributed Active, NASA Goddard

Georgia TechKentucky State Libraries & Archives

Library of Congress

Los Alamos National Lab

NASA Ames

NASA Goddard Space Flight Center

NCSA Grid Computing

NIH (NCI Center for Bioinformatics)

Penn State University

Pittsburgh Supercomputing Center

Purdue University. Indiana

Stanford University

TACC, University of Texas

Texas A & M

UC Santa Cruz

UCLAUCSD Neuroscience

University of Maryland

University of Michigan, CAC department

University of New Mexico

University of Washington

University of Wisconsin

USC

Yale University

Academia Sinica, Taiwan

ASCC, Computing Centre, Taiwan

Australian National UniversityBedford Oceanography,Canada

Bioinformatics Institute, SingaporeCSIRO, Australia

Data Storage Institute, SingaporeEGEE, French National CenterGeoForschungsZentrum, Germany

James Cook University, Australia

KEK High Energy Physics, JapanMax Planck Institute, NetherlandsParallab, Norway

South Australian Advanced ComputingUIB (Parallab) , NorwayUniversity of Amsterdam

University of Cambridge, AstronomyUniversity of Cambridge, e-Science

University of EdinburghUniversity of Genoa, Italy

University of Hong KongUnivrsity of Manchester

University of OsloUniversity of SouthamptonYork Univ (UK)


5/26

SDSC SRB Projects (60 million, .5 PB )

Digital Libraries

UCB, Umich, UCSB, Stanford,CDL

NSF NSDL - UCAR / DLESE

NASA Information Power Grid

Astronomy

National Virtual Observatory

2MASS Project (2 Micron All Sky Survey)

Particle Physics

Particle Physics Data Grid (DOE)

GriPhyN

SLAC Synchrotron Data Repository

Medicine

Digital Embryo (NLM)

Earth Systems Sciences

ESIPS

LTER

Persistent Archives

NARA

LOC

Neuro Science & Molecular Science

TeleScience/NCMIR, BIRN

SLAC, AfCS,


6/26

Storage Resource Broker (SRB)

A distributed file system (Data

Grid)

Client-Server, Server-Server

architecture.

Abstracts physical

SRB provides the ability to

transparently share data acrossremote sites.

Heterogeneous Resources

Single sign on

Sin le lo ical file hierarchy


7/26

What we are familiar with


8/26

What we are not familiar with, yet


9/26

How do the file systems differ?

Logical Abstraction

Folders are NOT physicalFiles do NOT inherit physical location

Everything is potentially distributed

Access Control

Permissions are NOT rwxrwxrwxPermissions ARE on a object by objectbasis

Groups and permissions ARE more similarto NTFS

Domains

Geographical / logical grouping of users

Namespace scalability: john@harvardjohn@mit

Also doubles as groups


10/26

Interfaces to theStorage Resource Broker

inQ Windows Client

Scommands UNIX, DOS Command line

Client

Jargon Java API and GUIcomponents

mySRB Web Client

Matrix WSDL, Data Grid

WorkflowsC, C++ C and C++ API

Python Python API

Perl Perl API


11/26

Common Scommands (75 total)

Sinit

Senv

Spwd

Sls

Scd

Sget

SputSsh

Scp

Smv (logical)

Sphymove (physical)

Srm

Smkdir

Srmdir

Serror

Schmod

Sexit


12/26

mySRB


13/26

BIRN Portal (perl based)


14/26

NEEScentral Portal (php based)


15/26

Biomedical Informatics ResearchNetwork (BIRN)

Major collaboration with SDSC, several of theprojects Co-Investigators and Co-PIs are at

SDSC.

BIRNs purpose is to provide its consortiumof neuroscience laboratories the ability to

share, compute, and collaborate.

The Storage Resource Broker provides the

ability to transparently share data across

remote sites.


16/26

The BIRN SRB Data Grid


17/26

Doing this Manually


18/26

The BIRN Data Grid


19/26

The grid is in the details


20/26

File Replication

Sls

/home/Demo/SRB-Tutorial/files-2:

Doc.txt

Sls -l

/home/Demo/SRB-Tutorial/files-2:

romanoly 0 z-ucsd-ncmir-nas1 15 2003-07-09-05.15 Doc.txt

romanoly 1 z-jhu-cis-nas0 15 2003-07-09-05.16 Doc.txt

romanoly 2 z-stanford-lucas-nas 15 2003-07-09-05.16 Doc.txt

romanoly 3 z-umn-cmrr-nas0 15 2003-07-09-

05.16 Doc.txtromanoly 4 z-uci-bic-nas0 15 2003-07-

09-05.17 Doc.txt


21/26

Teragrid SRB

All Teragrid accounts are given a SDSC SRB Teragrid

accountThe username is the same as your SDSC UNIX account

name

Your SRB domain is teragrid

You must register your DN string with SDSCs grid-

mapfile or request a SRB password to activate your SRB

account. Instructions to do so are here:http://www.sdsc.edu/user_services/srb/account_form.html

Your ~/.srb/.MdasEnv file OR env variables

mdasCollectionHome '/home/.teragrid'

mdasDomainName 'teragrid'

srbUser ''

#AUTH_SCHEME 'ENCRYPT1'

AUTH_SCHEME 'GSI_AUTH'

srbHost 'srb.sdsc.edu'

srbPort '7321'

defaultResource 'sfs-tape-tgd'

SERVER_DN '/C=US/O=NPACI/OU=SDSC/CN=Storage ResourceBroker/USERID=srb'
http://www.sdsc.edu/user_services/srb/account_form.htmlhttp://www.sdsc.edu/user_services/srb/account_form.html


22/26

Scommand Features

Command line interface -> SCRIPTING

Available for all of the most popular UNIX

flavors and DOS

S-commands are the most flexible and powerful

of the clientsThey are the fastest, and most reliable

They are multithreaded for big gains in data

flow

They are great for scripts, perl wrappers,

batch jobs, etc

Installed man pages via man [Scommand]

man Sput


23/26

Scommand Notes

Shelp

Gives list of commands with brief summary

[Scommand] gives usage info

(usually) or try h flag

Sinit establishes a session

Senv displays connection information

Spwd display current working directory

Sexit ends session


24/26

Some Public SRB Collections

Southern California Earthquake Center

/home/public.teragrid/SCEC

Two Micron All Sky Survey

/home/public.teragrid/2MASS

The Palomar Digital Sky Survey

/home/public.teragrid/DPOSS


25/26

Watch me do a SRB demo


26/26

Thanks!

SRB handles large data and provides theability to share and collaborate on

distributed heterogeneous resources.

Questions?

Teragrid SRB userguide:

http://www.sdsc.edu/user_services/srb

SRB website:

www.sdsc.edu/srb

SRB email:

srb@sdsc edu
http://www.sdsc.edu/user_services/srbhttp://www.sdsc.edu/srbmailto:[email protected]:[email protected]://www.sdsc.edu/srbhttp://www.sdsc.edu/user_services/srb

tg 06 srb intro tutorial

Documents