chapter 4:- introduction to grid and its...
TRANSCRIPT
![Page 1: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/1.jpg)
Chapter 4:-
Introduction to Grid and its Evolution
Prepared By:- NITIN PANDYA
Assistant Professor
SVBIT.
![Page 2: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/2.jpg)
2
Overview
Background: What is the Grid?
Related technologies
Grid applications
Communities
Grid Tools
Case Studies
NITIN PANDYA
![Page 3: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/3.jpg)
3
What is a Grid? Many definitions exist in the literature
Early defs: Foster and Kesselman, 1998
“A computational grid is a hardware and software infrastructure that
provides dependable, consistent, pervasive, and inexpensive access to
high-end computational facilities”
Kleinrock 1969:
“We will probably see the spread of ‘computer utilities’, which, like present
electric and telephone utilities, will service individual homes and offices
across the country.”
NITIN PANDYA
![Page 4: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/4.jpg)
Grid computing (1)
“Coordinated resource sharing and problem solving in dynamic,
multi-institutional virtual organisations” (I. Foster)
4 NITIN PANDYA
![Page 5: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/5.jpg)
Grid computing (2)
Information grid
large access to distributed data (the Web)
Data grid
management and processing of very large distributed data sets
Computing grid
meta computer
5 NITIN PANDYA
![Page 6: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/6.jpg)
Parallelism vs grids: some recalls
Grids date back “only” 1996
Parallelism is older ! (first classification in 1972)
Motivations:
need more computing power (weather forecast, atomic simulation,
genomics…)
need more storage capacity (Petabytes and more)
in a word: improve performance ! 3 ways ...
Work harder --> Use faster hardware
Work smarter --> Optimize algorithms
Get help --> Use more computers !
6 NITIN PANDYA
![Page 7: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/7.jpg)
The performance ? Ideally it grows linearly
Speed-up:
if TS is the best time to process a problem sequentially,
then the parallel processing time should be TP=TS/P with P processors
speedup = TS/TP
the speedup is limited by Amdhal law: any parallel program has a purely sequential and a
parallelizable part TS= F + T//,
thus the speedup is limited: S = (F + T//) / (F + (T///P)) < P
Scale-up:
if TPS is the time to solve a problem of size S with P processors,
then TPS should also be the time to process a problem of size n*S with n*P processors
7 NITIN PANDYA
![Page 8: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/8.jpg)
8
Why do we need Grids?
Many large-scale problems cannot be solved by a single
computer
Globally distributed data and resources
NITIN PANDYA
![Page 9: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/9.jpg)
9
Background: Related technologies
Cluster computing
Peer-to-peer computing
Internet computing
NITIN PANDYA
![Page 10: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/10.jpg)
10
Cluster computing
Idea: put some PCs together and get them to communicate
Cheaper to build than a mainframe supercomputer
Different sizes of clusters
Scalable – can grow a cluster by adding more PCs
NITIN PANDYA
![Page 11: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/11.jpg)
11
Cluster Architecture
NITIN PANDYA
![Page 12: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/12.jpg)
12
Peer-to-Peer computing
Connect to other computers
Can access files from any computer on the network
Allows data sharing without going through central server
Decentralized approach also useful for Grid
NITIN PANDYA
![Page 13: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/13.jpg)
13
Peer to Peer architecture
NITIN PANDYA
![Page 14: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/14.jpg)
14
Internet computing
Idea: many idle PCs on the Internet
Can perform other computations while not being used
“Cycle scavenging” – rely on getting free time on other people’s
computers
Example: SETI@home
What are advantages/disadvantages of cycle scavenging?
NITIN PANDYA
![Page 15: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/15.jpg)
15
Some Grid Applications
Distributed supercomputing
High-throughput computing
On-demand computing
Data-intensive computing
Collaborative computing
NITIN PANDYA
![Page 16: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/16.jpg)
16
Grid Users
Many levels of users
Grid developers
Tool developers
Application developers
End users
System administrators
NITIN PANDYA
![Page 17: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/17.jpg)
17
Some Grid challenges
Data movement
Data replication
Resource management
Job submission
NITIN PANDYA
![Page 18: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/18.jpg)
Computational grid “Hardware and software infrastructure that provides dependable, consistent,
pervasive and inexpensive access to high-end computational capabilities” (I. Foster)
Performance criteria: security
reliability
computing power
latency
throughput
scalability
services
18 NITIN PANDYA
![Page 19: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/19.jpg)
Grid characteristics
Large scale
Heterogeneity
Multiple administration domain
Autonomy… and coordination
Dynamicity
Flexibility
Extensibility
Security
19 NITIN PANDYA
![Page 20: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/20.jpg)
Levels of cooperation in a computing grid
End system (computer, disk, sensor…) multithreading, local I/O
Cluster synchronous communications, DSM, parallel I/O
parallel processing
Intranet/Organization heterogeneity, distributed admin, distributed FS and databases
load balancing
access control
Internet/Grid global supervision
brokers, negotiation, cooperation…
20 NITIN PANDYA
![Page 21: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/21.jpg)
Basic services Authentication/Authorization/Traceability
Activity control (monitoring)
Resource discovery
Resource brokering
Scheduling
Job submission, data access/migration and execution
Accounting
21 NITIN PANDYA
![Page 22: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/22.jpg)
Layered Grid Architecture
(By Analogy to Internet Architecture)
Application
Fabric “Controlling things locally”: Access to, & control of, resources
Connectivity “Talking to things”: communication (Internet protocols) & security
Resource “Sharing single resources”: negotiating access, controlling use
Collective “Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services
Internet
Transport
Application
Link
Inte
rnet P
roto
col A
rchite
ctu
re
From I. Foster 22 NITIN PANDYA
![Page 23: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/23.jpg)
Resources
Description
Advertising
Cataloging
Matching
Claiming
Reserving
Checkpointing
24 NITIN PANDYA
![Page 24: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/24.jpg)
Resource management (1)
Services and protocols depend on the infrastructure
Some parameters
stability of the infrastructure (same set of resources or not)
freshness of the resource availability information
reservation facilities
multiple resource or single resource brokering
Example of request: I need from 10 to 100 CE each with at least 512 MB
RAM and a computing power of 150 Mflops
25 NITIN PANDYA
![Page 25: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/25.jpg)
Resource management and scheduling (1)
Levels of scheduling
job scheduling (global level ; perf: throughput)
resource scheduling (perf: fairness, utilization)
application scheduling (perf: response time, speedup, produced data…)
Mapping/Scheduling process
resource discovery and selection
assignment of tasks to computing resources
data distribution
task scheduling on the computing resources
(communication scheduling)
26 NITIN PANDYA
![Page 26: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/26.jpg)
Resource management and scheduling (2)
Individual perfs are not necessarily consistent with the global
(system) perf !
Grid problems
predictions are not definitive: dynamicity !
Heterogeneous platforms
Checkpointing and migration
27 NITIN PANDYA
![Page 27: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/27.jpg)
GRAM GRAM GRAM
LSF Condor NQE
Application
RSL
Simple ground RSL
Information Service
Local resource managers
RSL specialization
Broker
Ground RSL
Co-allocator
Queries
& Info
A Resource Management System Example (Globus)
NQE: Network Queuing Env.
(batch management; developed
by Cray Research
LSF: Load Sharing Facility
(task scheduling and load balancing;
Developed by Platform Computing)
Resource Specification Language
28 NITIN PANDYA
![Page 28: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/28.jpg)
Resource information (1)
What is to be stored ?
virtual organizations, people, computing resources, software packages,
communication resources, event producers, devices…
what about data ???
A key issue in such dynamics environments
A first approach : (distributed) directory (LDAP)
easy to use
tree structure
distribution
static
mostly read ; not efficient updating
hierarchical
poor procedural language 29 NITIN PANDYA
![Page 29: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/29.jpg)
Resource information (2)
Goal:
dynamicity
complex relationships
frequent updates
complex queries
A second approach: (relational) database
30 NITIN PANDYA
![Page 30: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/30.jpg)
Programming on the grid: potential
programming models
Message passing (PVM, MPI)
Distributed Shared Memory
Data Parallelism (HPF, HPC++)
Task Parallelism (Condor)
Client/server - RPC
Agents
Integration system (Corba, DCOM, RMI)
31 NITIN PANDYA
![Page 31: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/31.jpg)
Program execution: issues Parallelize the program with the right job structure, communication patterns/procedures,
algorithms
Discover the available resources
Select the suitable resources
Allocate or reserve these resources
Migrate the data
Initiate computations
Monitor the executions ; checkpoints ?
React to changes
Collect results
32 NITIN PANDYA
![Page 32: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/32.jpg)
Data management
It was long forgotten !!!
Though it is a key issue !
Issues:
indexing
retrieval
replication
caching
traceability
(auditing)
And security !!!
33 NITIN PANDYA
![Page 33: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/33.jpg)
34
Some Grid-Related Projects
Globus
Condor
Nimrod-G
NITIN PANDYA
![Page 34: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/34.jpg)
35
Globus Grid Toolkit Open source toolkit for building Grid systems and applications
Enabling technology for the Grid
Share computing power, databases, and other tools securely online
Facilities for:
Resource monitoring
Resource discovery
Resource management
Security
File management
NITIN PANDYA
![Page 35: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/35.jpg)
36
Data Management in Globus Toolkit
Data movement
GridFTP
Reliable File Transfer (RFT)
Data replication
Replica Location Service (RLS)
Data Replication Service (DRS)
NITIN PANDYA
![Page 36: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/36.jpg)
37
GridFTP High performance, secure, reliable data transfer protocol
Optimized for wide area networks
Superset of Internet FTP protocol
Features:
Multiple data channels for parallel transfers
Partial file transfers
Third party transfers
Reusable data channels
Command pipelining
NITIN PANDYA
![Page 37: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/37.jpg)
38
More GridFTP features
Auto tuning of parameters
Striping
Transfer data in parallel among multiple senders and receivers
instead of just one
Extended block mode
Send data in blocks
Know block size and offset
Data can arrive out of order
Allows multiple streams
NITIN PANDYA
![Page 38: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/38.jpg)
39
Striping Architecture
Use “Striped” servers
NITIN PANDYA
![Page 39: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/39.jpg)
40
Limitations of GridFTP
Not a web service protocol (does not employ SOAP, WSDL,
etc.)
Requires client to maintain open socket connection
throughout transfer
Inconvenient for long transfers
Cannot recover from client failures
NITIN PANDYA
![Page 40: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/40.jpg)
41
GridFTP
NITIN PANDYA
![Page 41: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/41.jpg)
42
Reliable File Transfer (RFT)
Web service with “job-scheduler” functionality for data movement
User provides source and destination URLs
Service writes job description to a database and moves files
Service methods for querying transfer status
NITIN PANDYA
![Page 42: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/42.jpg)
43
RFT
NITIN PANDYA
![Page 43: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/43.jpg)
44
Replica Location Service (RLS)
Registry to keep track of where replicas exist on physical storage
system
Users or services register files in RLS when files created
Distributed registry
May consist of multiple servers at different sites
Increase scale
Fault tolerance
NITIN PANDYA
![Page 44: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/44.jpg)
45
Replica Location Service (RLS) Logical file name – unique identifier for contents of file
Physical file name – location of copy of file on storage system
User can provide logical name and ask for replicas
Or query to find logical name associated with physical file location
NITIN PANDYA
![Page 45: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/45.jpg)
46
Data Replication Service (DRS) Pull-based replication capability
Implemented as a web service
Higher-level data management service built on top of RFT and RLS
Goal: ensure that a specified set of files exists on a storage site
First, query RLS to locate desired files
Next, creates transfer request using RFT
Finally, new replicas are registered with RLS
NITIN PANDYA
![Page 46: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/46.jpg)
47
Condor
Original goal: high-throughput computing
Harvest wasted CPU power from other machines
Can also be used on a dedicated cluster
Condor-G – Condor interface to Globus resources
NITIN PANDYA
![Page 47: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/47.jpg)
48
Earth System Grid
Provide climate studies scientists with access to large datasets
Data generated by computational models – requires massive
computational power
Most scientists work with subsets of the data
Requires access to local copies of data
NITIN PANDYA
![Page 48: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/48.jpg)
49
ESG Infrastructure
Archival storage systems and disk storage systems at several sites
Storage resource managers and GridFTP servers to provide access to
storage systems
Metadata catalog services
Replica location services
Web portal user interface
NITIN PANDYA
![Page 49: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/49.jpg)
50
Earth System Grid
NITIN PANDYA
![Page 50: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/50.jpg)
51
Earth System Grid Interface
NITIN PANDYA
![Page 51: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/51.jpg)
52
Laser Interferometer Gravitational
Wave Observatory (LIGO)
Instruments at two sites to detect gravitational waves
Each experiment run produces millions of files
Scientists at other sites want these datasets on local storage
LIGO deploys RLS servers at each site to register local mappings and
collect info about mappings at other sites
NITIN PANDYA
![Page 52: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/52.jpg)
53
Large Scale Data Replication for LIGO
Goal: detection of gravitational waves
Three interferometers at two sites
Generate 1 TB of data daily
Need to replicate this data across 9 sites to make it available
to scientists
Scientists need to learn where data items are, and how to
access them
NITIN PANDYA
![Page 53: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/53.jpg)
54
LIGO
NITIN PANDYA
![Page 54: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/54.jpg)
55
LIGO Solution
Lightweight data replicator (LDR)
Uses parallel data streams, tunable TCP windows, and tunable
write/read buffers
Tracks where copies of specific files can be found
Stores descriptive information (metadata) in a database
Can select files based on description rather than filename
NITIN PANDYA
![Page 55: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/55.jpg)
56
TeraGrid
NSF high-performance computing facility
Nine distributed sites, each with different capability , e.g.,
computation power, archiving facilities, visualization software
Applications may require more than one site
Data sizes on the order of gigabytes or terabytes
NITIN PANDYA
![Page 56: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/56.jpg)
57
TeraGrid
NITIN PANDYA
![Page 57: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/57.jpg)
58
TeraGrid
Solution: Use GridFTP and RFT with front end command line
tool (tgcp)
Benefits of system:
Simple user interface
High performance data transfer capability
Ability to recover from both client and server software failures
Extensible configuration
NITIN PANDYA
![Page 58: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/58.jpg)
59
TGCP Details
Idea: hide low level GridFTP commands from users
Copy file smallfile.dat in a working directory to another system: tgcp smallfile.dat tg-login.sdsc.teragrid.org:/users/ux454332
GridFTP command:
globus-url-copy -p 8 -tcp-bs 1198372 \
gsiftp://tg-gridftprr.uc.teragrid.org:2811/home/navarro/smallfile.dat \
gsiftp://tg-login.sdsc.teragrid.org:2811/users/ux454332/smallfile.dat
NITIN PANDYA
![Page 59: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/59.jpg)
60
The reality
We have spent a lot of time talking about “The Grid”
There is “the Web” and “the Internet”
Is there a single Grid?
NITIN PANDYA
![Page 60: Chapter 4:- Introduction to Grid and its Evolutionsvbitce2010.weebly.com/uploads/8/4/4/5/8445046/chapter04.pdf · Chapter 4:- Introduction to Grid and its Evolution Prepared By:-](https://reader034.vdocuments.us/reader034/viewer/2022051407/5af906de7f8b9a2d5d8c6104/html5/thumbnails/60.jpg)
61
The reality
Many types of Grids exist
Private vs. public
Regional vs. Global
All-purpose vs. particular scientific problem
NITIN PANDYA