cgw 04, stripped replication for the grid environment as a web service1 stripped replication for the...
TRANSCRIPT
CGW 04, Stripped replication for the grid environment as a web service 1
Stripped replication for the Grid environment as a web service
Marek Ciglan, Ondrej Habala, Ladislav HluchýInstitute of informatics Slovak Academy of Sciences
CGW 04, Stripped replication for the grid environment as a web service
2
Stripped Replication for Grids
Overview
• Replication in Grid environment• Principles of stripped replication (SR)
method• Optimization of stripped replication• Prototype Implementation as a Web
Service• Experimental Results• Future Work
CGW 04, Stripped replication for the grid environment as a web service
3
Stripped Replication for Grids
Replication in Grid environment
• Creation of multiple copies of single data source across Grid infrastructure
• Replication increases data availability• RLS - Replica Location Service• Grid monitoring services – network
monitoring
CGW 04, Stripped replication for the grid environment as a web service
4
Stripped Replication for Grids
Replication in Grid environment
Storage Element 1
File 1
Storage Element 2
Storage Element 3
CGW 04, Stripped replication for the grid environment as a web service
5
Stripped Replication for Grids
Replication in Grid environment
Storage Element 1
File 1
Storage Element 2
Storage Element 3
File 1
CGW 04, Stripped replication for the grid environment as a web service
6
Stripped Replication for Grids
Replication in Grid environment
Storage Element 1
File 1
Storage Element 2
Storage Element 3
File 1
CGW 04, Stripped replication for the grid environment as a web service
7
Stripped Replication for Grids
Replication in Grid environment
Storage Element 1
File 1
Storage Element 2
Storage Element 3
File 1
File 1
CGW 04, Stripped replication for the grid environment as a web service
8
Stripped Replication for Grids
Stripped Replication - Principles
• Transfer from multiple Grid sites, in parallel• Transfer only a portion of file from each
Storage Element (SE)• Different file portions (stripes) are obtained
from different SEs• Parallel transfer increases replication speed• If SR is not managed properly, process
could be time consuming• Optimization of SR management is required
CGW 04, Stripped replication for the grid environment as a web service
9
Stripped Replication for Grids
Stripped Replication - Optimization
Replicated data source
Replica 1
Replica 2 Replica 3
CGW 04, Stripped replication for the grid environment as a web service
10
Stripped Replication for Grids
Stripped Replication - Optimization
Replicated data source
Replica 1
Replica 2 Replica 3
Replica 1
Replica 2 Replica 3
CGW 04, Stripped replication for the grid environment as a web service
11
Stripped Replication for Grids
Stripped Replication - Optimization
Replica 1
Replica 2 Replica 3
CGW 04, Stripped replication for the grid environment as a web service
12
Stripped Replication for Grids
Stripped Replication - Optimization
Replica 1
Replica 2 Replica 3
Replica 1
Replica 2 Replica 3
CGW 04, Stripped replication for the grid environment as a web service
13
Stripped Replication for Grids
Stripped Replication - Optimization
Replica 1
Replica 2 Replica 3
Replica 1
Replica 2 Replica 3
Replica 1
Replica 2 Replica 3
CGW 04, Stripped replication for the grid environment as a web service
14
Stripped Replication for Grids
SR Prototype Implementation
• Java programming language• CoG 1.2 API (GridFTP interface)• Integrated with EDG Replica Location
Service• EDG RLS API (RLS interface)• File Chunks – basic data units for transfer• Implemented as a Web Service
( motivation :OGSA, WSRF)
CGW 04, Stripped replication for the grid environment as a web service
15
Stripped Replication for Grids
Service WorkflowStripped Replication Service
LFNGetGUID
CGW 04, Stripped replication for the grid environment as a web service
16
Stripped Replication for Grids
Service Workflow
LFNGetGUID
ReplicaMetadataCatalog
Stripped Replication Service
CGW 04, Stripped replication for the grid environment as a web service
17
Stripped Replication for Grids
Service Workflow
LFNGetGUID
ReplicaMetadataCatalog
Stripped Replication Service
GetPFNs
CGW 04, Stripped replication for the grid environment as a web service
18
Stripped Replication for Grids
Service Workflow
LFNGetGUID
ReplicaMetadataCatalog
GetPFNs
LocalReplicaCatalog
Stripped Replication Service
CGW 04, Stripped replication for the grid environment as a web service
19
Stripped Replication for Grids
Service Workflow
LFNGetGUID
ReplicaMetadataCatalog
GetPFNs
LocalReplicaCatalog
StrippedReplicationAlgorithm
Stripped Replication Service
CGW 04, Stripped replication for the grid environment as a web service
20
Stripped Replication for Grids
Service Workflow
LFNGetGUID
ReplicaMetadataCatalog
GetPFNs
LocalReplicaCatalog
StrippedReplicationAlgorithm
GridFTPSite 1
GridFTPSite N
. .
.
. .
.
Stripped Replication Service
CGW 04, Stripped replication for the grid environment as a web service
21
Stripped Replication for Grids
Service Workflow
LFNGetGUID
ReplicaMetadataCatalog
GetPFNs
LocalReplicaCatalog
StrippedReplicationAlgorithm
GridFTPSite 1
GridFTPSite N
. .
.
. .
.
Register Replica
Stripped Replication Service
CGW 04, Stripped replication for the grid environment as a web service
22
Stripped Replication for Grids
Properties of Stripped Replication
• Parallel transfer from multiple sites increases replication process speed
• Proposed optimization does not use network monitoring services
• SR adapts to varying nature of network load
• SR optimally distributes network load
CGW 04, Stripped replication for the grid environment as a web service
23
Stripped Replication for Grids
Experimental Results
• Motivation test case– File size 223.9Mb – Best replica transfer with standard replication tool
(EDG rm) - 713 sec– Stripped replication (2 replicas) – 405 sec (43 %)– Stripped replication (3 replicas) – 209 sec (71 %)
• Average time saving– 2 replicas – 37% time saving– 3 replica – 55% time saving
CGW 04, Stripped replication for the grid environment as a web service
24
Stripped Replication for Grids
Future Work
• Implementation refinement– Add logging functionality– Refine error states handling
• Evaluation of SR integration in Grid projects
CGW 04, Stripped replication for the grid environment as a web service
25
Stripped Replication for Grids
Thank you for your attention !