karolina sarnowska, university of virginia andrew grimshaw, university of virginia mark morgan,...
TRANSCRIPT
![Page 1: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/1.jpg)
Standards-based Access to LFCs
Karolina Sarnowska, University of VirginiaAndrew Grimshaw, University of Virginia
Mark Morgan, University of VirginiaAkos Frohner, CERN
Erwin Laure, KTH
![Page 2: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/2.jpg)
2
Outline
IntroductionDevelopment InteroperabilityPerformanceConclusion
![Page 3: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/3.jpg)
3
INTRODUCTIONInteroperability and Grid Standards
![Page 4: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/4.jpg)
4
Goal & Motivation
Create a standards-based interface for accessing data resources in EGEE grids
Read data from LFCs via standardized interfaceEnable interoperabilityValidate specifications
![Page 5: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/5.jpg)
5
LFC: The LCG File Catalog
Catalog of data files on storage elementsEach logical file entry can correspond to
multiple physical files (replicas)Provides mapping between logical file names
and storage URLs LFC/grid /atlas /alice /cms
SESEStorage
ElementsEGEE Grid
![Page 6: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/6.jpg)
6
InfrastructureWS-Addressing, Resource Namespace Service (RNS), WS-Naming
ComputeJSDL, OGSA-BES, HPC Basic Profile
DataRNS, OGSA-ByteIO, gridFTP, WS-DAI
APIsSAGA, DRMAA, GridRPC
SecurityWSI-BSP, WS-Security, WS-Trust, WS-Federation, WSSecurityPolicy,
WS-SecureNaming, WS-Secure Communication
OGF Standards
![Page 7: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/7.jpg)
7
Step One: Identify Resource
Need means of referring to resources
What’s in a name? human-readable maps to unique resource
![Page 8: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/8.jpg)
8
The RNS Specification
RNS: Resource Namespace Service
Describes standard way of mapping names to grid endpoints
RNS 1.0 Operations: Add, List, Remove, Query, Move
![Page 9: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/9.jpg)
9
Standards-based Naming for Accessing Resources in an LFC
The SNARL Service:An Implementation of RNS
SNARLService
(1) RNS Requests
(3) RNS Responses
LFC/grid /atlas /alice /cms /dteamGrid
Client (2) LFC API Calls
SESE
EGEE GridAnother Grid
StorageElements
![Page 10: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/10.jpg)
10
Step Two: Access Resource Data
Need means of accessing data associated with resources
![Page 11: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/11.jpg)
11
The ByteIO Specification
Describes standard way of handling transfer of data associated with grid entities
POSIX-like interfaceRandom, session-less or stateful session
accessOperations: Read, Write, Append,
TruncAppend, SeekRead, SeekWrite
![Page 12: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/12.jpg)
The SABLE Service:An Implementation of ByteIO
Standards-based Access to Bytes of LFC Entries
12
(3) LCG & GFAL API Calls
SABLEService
(1) ByteIO Requests
(3) ByteIO Responses
LFC/grid /atlas /alice /cms /dteamClient SE
SE
EGEE GridAnother Grid
StorageElements(2) GFAL API
Calls
![Page 13: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/13.jpg)
13
DEVELOPMENTCreating web services with Axis2/C
![Page 14: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/14.jpg)
14
Development Process
1. Setup web services framework with Axis2/C
2. Create skeleton services with WSDL2C
3. Create clients for testing
4. Implement RNS and ByteIO Operations using gLite APIs
![Page 15: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/15.jpg)
15
Why Apache Axis2/C?
Web services engine implemented in C
Handles SOAP message processing
Built-in support includes: WS-Addressing WS-Policy WS-SecurityPolicy
![Page 16: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/16.jpg)
16
Service Skeleton Generation with WSDL2C Tool
1. Specify web service in WSDL file
2. Use tool to generate C web service skeleton
Resulting Service Skeleton Size:SNARL: 89k LOCSABLE : 77k LOC
![Page 17: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/17.jpg)
17
Test Client Creation
Example: RNS Add Request<soap:Envelope> <soap:Header> <wsa:Action>http://schemas.ggf.org/rns/2006/03/rns/add</wsa:Action> </soap:Header> <soap:Body> <rns:add xmlns:rns=‘http://schemas.ggf.org/rns/2006/05/rns’> <rns:entry_name> foo </rns:entry_name> </rns:add> </soap:Body></soap:Envelope>
![Page 18: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/18.jpg)
18
Implementing RNS Ops
Translating RNS operations to API calls
Should replicas be visible to users?
RNS Operation LFC API CallAdd lfc_creatList lfc_readdirRemove lfc_unlinkQuery lfc_statMove lfc_unlink + lfc_creat
![Page 19: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/19.jpg)
19
Revealing Logical Files vs. Replicas
LFC•/grid/foo
SNARLService
(1) RNS List
(3) RNS List Response
GridClient
(2) LFC API Calls
SE•fooReplicaA
SE•fooReplicaB
![Page 20: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/20.jpg)
20
Implementing ByteIO OpsTranslating ByteIO operations to API calls
Should writes be allowed?
ByteIO Operation API CallAll gfal_openRead gfal_lseek + gfal_readWrite/Append gfal_lseek + gfal_writeTruncAppend gfal_creat + Append
![Page 21: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/21.jpg)
21
INTEROPERABILITY TESTS
![Page 22: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/22.jpg)
22
Spec AdoptionProject/Spec RNS ByteIO
Genesis II X X
Unicore 6 X
Fujitsu USMT X
Microsoft CCS
Globus
OMII-UK X
GridSAM
Crown
Platform
OGSA-DAI X X
NAREGI
gFarm X
![Page 23: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/23.jpg)
23
Interoperability Test Setup
1. Using RNS list request, choose LFC entry2. Using ByteIO read request, read data
resource associated with LFC entry
![Page 24: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/24.jpg)
24
Step One:List LFC Entries via SNARL
(3) RNS List Response
LFC/grid /atlas /alice /cms /dteam
GridClient
(2) LFC API Calls
SESE
(1) RNS ListRequest
SNARLService
StorageElements
EGEE GridGenesisII Grid
![Page 25: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/25.jpg)
25
Step Two: Read Resource Data via SABLE
(3) ByteIO Read Response
LFC/grid /atlas /alice /cms /dteam
GridClient
SESE
(1) ByteIO Read Request
SABLEService
StorageElements(2) GFAL API
Calls
EGEE GridGenesisII Grid
![Page 26: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/26.jpg)
26
PERFORMANCE
![Page 27: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/27.jpg)
27
Setup: Machine Specs
Scientific CERN Linux 4 Intel Pentium 4 2.8 GHz CPU1 GB memory
![Page 28: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/28.jpg)
28
SNARL Performance
Dir Size:lfc-ls
CmdlineLFC API
Axis2 Client
Genesis II Client
Small(10 entries) 0.42s 1.49s 1.94s 4.43s
Medium(100 entries) 0.47s 11.73s 15.92s 20.27s
Large (1000 entries) 0.71s 2m 0.37s 2m 1.11s 2m 5.05s
![Page 29: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/29.jpg)
29
SABLE Performance
File Size:lcg-cp
CmdlineGFAL API
Axis2 Client
Genesis II Client
Small(1 KB) 2.54s 2.60s 2.55s 3.49s
Medium(1 MB) 2.83s 3.61s 2.93s 3.64s
![Page 30: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/30.jpg)
30
CONCLUSIONS
![Page 31: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/31.jpg)
31
Future Directions
Add security context Implement MIME, DIME, MTOMExplore use as interface between LFC and
other toolsUpdate for RNS 1.1 Implement StreamableByteIO
![Page 32: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/32.jpg)
32
Conclusion
Implementation of RNS and ByteIO specification for existing grid system is possible
RNS and ByteIO provide an interface that enables the sharing of grid data
![Page 33: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/33.jpg)
33
Thank You!
CERN & DM TeamUVA Genesis II TeamOpenLab Program at CERN
![Page 34: Karolina Sarnowska, University of Virginia Andrew Grimshaw, University of Virginia Mark Morgan, University of Virginia Akos Frohner, CERN Erwin Laure,](https://reader035.vdocuments.us/reader035/viewer/2022062315/5697bfa31a28abf838c96701/html5/thumbnails/34.jpg)
34
QUESTIONS?
1. "Open Grid Forum," http://www.ogf.org.2. M. Pereira, O. Tatebe, L. Luan, and T. Anderson, "Resource
Namespace Service Specification,” http://www.ggf.org/documents/GFD.101.pdf, 2006.
3. M. Morgan, “ByteIO Specification,” http://www.ggf.org/documents/GFD.87.pdf, 2006.
4. M. M. Morgan and A. S. Grimshaw, "Genesis II – Standards Based Grid Computing," Seventh IEEE International Symposium on Cluster Computing and the Grid, 2007.
5. “Apache Axis2/C,” http://ws.apache.org/axis2c