Download - Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl
![Page 1: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/1.jpg)
Grid Services at NERSC
Shreyas CholiaOpen Software and Programming Group, NERSC
[email protected] User Group Meeting
September 17, 2007
![Page 2: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/2.jpg)
NERSC User Group Meeting, September 17, 2007 2
Introduction to Grids
Q: What is a Grid?
A: A Grid is a system that:1) coordinates resources that are not
subject to centralized control …
2) … using standard, open, general-purpose protocols and interfaces
3) … to deliver nontrivial qualities of service.
- Ian Foster, 2002, http://www-fp.mcs.anl.gov/~foster/Articles/WhatIsTheGrid.pdf
![Page 3: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/3.jpg)
NERSC User Group Meeting, September 17, 2007 3
Introduction to Grids
• Grid computing provides the ability to share and aggregate heterogeneous, distributed computational capabilities and deliver them as a service.
• Uniform set of software interfaces to access non-uniform and physically distributed compute and storage resources.
• Resources are part of a larger whole:– Compute Grid– Data Grid
![Page 4: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/4.jpg)
NERSC User Group Meeting, September 17, 2007 4
Software Overview
• Globus Toolkit 4.0.3– GridFTP for data transfer across the
Grid.– GRAM for remote job submission and
batch access.– GSI and X509 certificates for security.
• Using OSG Software Stack based on the Virtual Data Toolkit (VDT).
![Page 5: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/5.jpg)
NERSC User Group Meeting, September 17, 2007 5
Core Grid Services at NERSC
• GRAM Gatekeeper– Allows you to run / submit, monitor and query
jobs on any NERSC machine remotely.– Interfaces with various backend batch systems
(PBS, Loadleveler, SGE) while presenting a uniform interface to the user (Globus RSL).
• GridFTP– Provides high performance data movement
between NERSC and remote site / client system.
– Supports striped, third party and parallel transfers.
![Page 6: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/6.jpg)
NERSC User Group Meeting, September 17, 2007 6
Certificate Services
• All Grid services at NERSC use X509 Grid certificates for authentication and access.
• NERSC Users can acquire a grid certificate from the DOEGrids site (https://pki1.doegrids.org/ca/).
• User must register their certificate with NERSC in NIM.
• Once the certificate has been approved in NIM, user can access any of the NERSC Grid Services with this.
![Page 7: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/7.jpg)
NERSC User Group Meeting, September 17, 2007 7
Certificate Services
• MyProxy Server– Allows users to store their certificates
in a secure fashion, and generate short term proxy certificates that can be accessed from anywhere.
– Users do not have to leave long term certificates lying around on all their client systems.
![Page 8: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/8.jpg)
NERSC User Group Meeting, September 17, 2007 8
System Deployment
PDSF
Cln
GF
HPSS
GK
GF
Cln
Jacquard
Cln
Davinci/NGF
Cln
Bassi
GK
GF
GridFTP Gatekeeper
GK
GF
GK
GF
GK
GF Cln
SGE PBS PBS LoadLev
Cln Globus Client
MPI Jobs
![Page 9: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/9.jpg)
NERSC User Group Meeting, September 17, 2007 9
Grid Clients
• Job Submission Tools– globus-job-submit, globus-job-query– Condor-G
• GridFTP Clients– globus-url-copy– uberftp
• Certificate Management Tools– grid-proxy-init– myproxy-init, myproxy-logon– grid-cert-info
![Page 10: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/10.jpg)
NERSC User Group Meeting, September 17, 2007 10
Open Science Grid
• Distributed computing infrastructure for large-scale scientific research, built and operated by a consortium of universities, national laboratories, scientific collaborations and software developers.
• Researchers from many fields, including astrophysics, bioinformatics, computer science, medical imaging, nanotechnology and physics, use the OSG infrastructure to advance their research.
![Page 11: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/11.jpg)
NERSC User Group Meeting, September 17, 2007 11
NERSC and the OSG
• OSG users have access to compute/storage resources including NERSC.
• NERSC resources are made available to OSG users via Grid Interfaces.
• NERSC users may be able to access the OSG by making a request to the OSG Engagement team.
![Page 12: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/12.jpg)
NERSC User Group Meeting, September 17, 2007 12
Coming Soon …
• Grid Services on Franklin• SRM Access to NERSC Systems
– NGF– HPSS
• NERSC Online CA– Simplify the process of acquiring cert– User will be given a short lived cert
based on LDAP password
![Page 13: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/13.jpg)
NERSC User Group Meeting, September 17, 2007 13
Sample Users
• STAR
• NREL
• Planck
• OSG - WRF
• OSG - LIGO
• ESG
• Incite 8 and 13
![Page 14: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/14.jpg)
NERSC User Group Meeting, September 17, 2007 14
What it means to YOU!
• You can submit, manage and coordinate jobs across NERSC systems (and across other grid sites) using a common interface directly from a client machine.
• Especially useful for managing workflows across multiple systems / sites.
• Job control can be integrated into software running on your desktop.
• The ability to move data across sites using high performance parallel transfers.
![Page 15: Grid Services at NERSC Shreyas Cholia Open Software and Programming Group, NERSC scholia@lbl](https://reader036.vdocuments.us/reader036/viewer/2022062520/56815b13550346895dc8be61/html5/thumbnails/15.jpg)
NERSC User Group Meeting, September 17, 2007 15
Example
HPSS
Bassi
NGF
DaVinci
RemoteWeb Server
MyProxy
• Get Certificate• Submit Job to Bassi• GridFTP to HPSS• Do analysis on DaVinci• Move results to web server