c. loomis – title – date – 1 european datagrid charles loomis seillac june 26, 2002 outline...
Post on 19-Dec-2015
220 Views
Preview:
TRANSCRIPT
C. Loomis – Title – Date – 1
European DataGrid
Charles Loomis
Seillac
June 26, 2002
Outline
EDG Introduction & Goals
EDG Architecture
Example Job Submission
Future Developments
Further Information
C. Loomis – Title – Date – 2
European DataGrid (EDG)
European DataGrid
EU-funded, 3-year project (2001-3)
Goals:—develop grid middleware
—deploy onto working testbed
—demonstrate grid technology with working applications
Strong application component unique!
EDG Organization
WP1 Workload Mgt.
WP2 Data Mgt.
WP3 Info. & Monitoring Sys.
WP4 Fabric Mgt.
WP5 Storage Mgt.
WP6 Testbed
WP7 Networking
WP8 HEP Apps.
WP9 Biomedical Apps.
WP10
Earth Ob. Apps.
WP11
Dissemination
WP12
Project Mgt.
6 Partners; 21 Associates
C. Loomis – Title – Date – 3
Computing Grid Electrical Grid
Accessing computing resources as simple as plugging in a toaster!
Fixed resources (CPU)
Mobile resource (Data)
Semi-mobile (Tape)
Transparency (I.e. standards)
Simplifies sharing & trading of resources.—Does not imply that all resources are free.
Should be applicable from desktop to supercomputers.
Security (Owners retain control!)
System administrators control machine access.
Who can read your data? Software enforces access control.
CPU Cycles
Network Bandwidth
Disk/tape Storage
Data/Databases
Software
C. Loomis – Title – Date – 4
Transparency for Physicists
Single Login:
Access all available local & remote resources w/o a dedicated account.
Consistent User Environment:
Same working environment everywhere--same software tools; same configuration.
Brokering:
System finds “optimal” resources for task automatically.
Resource discovery, moving job to data, moving data to job.
High-Level Selection of Resources:
Ex. choose storage by size/lifetime/reliability/cost/transfer speed; not disk/tape/shared file system/location.
Security:
Does not impede authorized access to resources.
Data and computations are (reasonably) safe from tampering.
C. Loomis – Title – Date – 5
Interoperability & Interdependency
Standards:
Global Grid Forum (like W3C)
Or from dominant implementation
Plethora of Projects:
Sharing of code & expertise.
Competing for people, funding, etc.
C. Loomis – Title – Date – 6
EDG Architecture
Global Batch System:
Centralized Architecture.
Heavy infrastructure.
Other choices Peer-to-peer
Computing Element
Storage Element
Site X
Information Systems
submit
submit
publishstate
query
retrieve
retrieve
broker choosesoptimal site for job
Resource Broker
User Interface
C. Loomis – Title – Date – 7
Hidden Infrastructure
Certification Authorities
Attest to your identity.
Virtual Organizations
Maintain membership lists.Used for group authorization.
/C=FR/O=CNRS/OU=LAL/CN=Charles Loomis/Email=loomis@lal.in2p3.fr
Replica Catalog
Location & DirectoryInfo. for data.
Maintained by:
ALICEATLASBaBarCMSLHCbBiomedical Apps.Earth Observation
C. Loomis – Title – Date – 8
EDG Testbeds
Production Testbed (1.1.4)
For applications to use & stress software in “semi-production” environment.
10 sites (6 countries)
Development Testbed (1.2b5)
To facilitate testing and integration of new middleware.
4 sites (4 countries)
Site Location
Catania Catania (I)
CC-IN2P3 Lyon (F)
CERN* Geneva (CH)
CNAF* Bologna (I)
Imperial College
London (UK)
LIP** Lisbon (PT)
MSU Moscow (Russia)
NIKHEF* Amsterdam (NL)
Padova Padova (I)
RAL* Rutherford (UK)
Torino Torino (I)
C. Loomis – Title – Date – 9
Example
Prerequisites
write script
job description
Job Submission
grid-proxy-init
dg-job-submit
dg-job-status
dg-job-get-output
#!/bin/sh
# Get matched physical file name.
filename = `./GetPFN.pl $EDG_WL_RB_BROKERINFO`
# Analyze this file.
count_evts $filename
Find data file and count events.
C. Loomis – Title – Date – 10
Example
Prerequisites
write script
job description
Job Submission
grid-proxy-init
dg-job-submit
dg-job-status
dg-job-get-output
# Executable to run and the arguments.Executable = "ChecksumRC.sh";Arguments = "none";
# Define file names for standard output and error.StdOutput = "std.out";StdError = "std.err";
# Files to transfer as input and output.InputSandbox = {"ChecksumRC.sh","GetPFN.pl”,”count_evts”};OutputSandbox = {"std.out","std.err"};
# Desired file and protocol.InputData = {"LF:utve_v_1_71008101_12902_0609.lv0"};DataAccessProtocol = {"file"};
# My experiment's "Data Catalog"ReplicaCatalog = "ldap://grid-vo.nikhef.nl:10389/rc=EarthObReplicaCatalog,dc=eu-datagrid,dc=org";
Describe the requirements for the job.
C. Loomis – Title – Date – 11
Example
Prerequisites
write script
job description
Job Submission
grid-proxy-init
dg-job-submit
dg-job-status
dg-job-get-output
> grid-proxy-init
Your identity: /C=FR/O=CNRS/OU=LAL/CN=Charles Loomis/Email=loomis@lal.in2p3.fr
Enter GRID pass phrase for this identity: *************
Creating proxy .......................................... Done
Your proxy is valid until Sun Jun 23 06:13:51 2002
Create a proxy for authorization like logging in.
C. Loomis – Title – Date – 12
Example
Prerequisites
write script
job description
Job Submission
grid-proxy-init
dg-job-submit
dg-job-status
dg-job-get-output
> dg-job-submit ChecksumRC.jdl
Connecting to host testbed011.cern.ch, port 7771Logging to host testbed011.cern.ch, port 15830
************************************************************************************ JOB SUBMIT OUTCOME The job has been successfully submitted to the Resource Broker. Use dg-job-status command to check job current status. Your job identifier (dg_jobId) is:
https://testbed011.cern.ch:7846/137.138.181.253/162422289427581?testbed011.cern.ch:7771
************************************************************************************
Submit job. Broker will automatically find appropriate place.
C. Loomis – Title – Date – 13
Example
Prerequisites
write script
job description
Job Submission
grid-proxy-init
dg-job-submit
dg-job-status
dg-job-get-output
>dg-job-status <job id>
…
dg_JobId = https://testbed011.cern.ch:7846/137.138.181.253/…
Status = OutputReady
Last Update Time (UTC) = Sat Jun 22 16:29:30 2002
Job Destination = gppce05.gridpp.rl.ac.uk:2119/jobmanager-pbs-S
Status Reason = terminated
Job Owner = /C=FR/O=CNRS/OU=LAL/CN=Charles Loomis/…
Status Enter Time (UTC) = Sat Jun 22 16:29:30 2002
Get the status of the submitted job.
C. Loomis – Title – Date – 14
Example
Prerequisites
write script
job description
Job Submission
grid-proxy-init
dg-job-submit
dg-job-status
dg-job-get-output
>dg-job-get-output <job id>
…
Output sandbox files for the job: - https://testbed011.cern.ch:7846/137.138.181.253/… have been successfully retrieved and stored in the directory: /tmp/162422289427581
Pull the output back to home machine.
>cat /tmp/162422289427581/std.out
File: /flatfiles/eo/utve_v_1_71008101_12902_0609.lv0Contains: 1034 events
C. Loomis – Title – Date – 15
Future Developments
EDG Plans:more advanced data management (I.e. real “Storage Element”)
job splitting, checkpointing; interactive jobs
more robust, consistent security model
OGSA (Open Grid Services Architecture)Proposal by Globus/IBM based on web-services
Adds concepts like notification and soft-state registration.
Standardizes the protocol and interface descriptions for Grid services.
Cross-fertilization of ProjectsLAL is ideally situated for this. (EDG, LCG, XtremWeb, CGP2P)
C. Loomis – Title – Date – 16
Conclusions
EDG has interesting (to a physicist) software & working testbed:
Current release:—Basic features available.
—Somewhat painful: reliability, low-level mucking around
Fall release:—Expect big change in functionality, especially data management.
—Concerted effort on reliability should make testbed less frustrating.
About the time to get involved…
Long-term success:
When you work routinely in a “Grid” environment on your desktop.
Fully automatic: resource discovery, optimization, and task dispatch.
Not there yet, but going in right direction.
C. Loomis – Title – Date – 17
Getting Involved & Further Surfing
Talk to grid contact:
ATLAS (Laura Perini)
ALICE (Piergiorgio Cerello)
CMS (Andrea Sciaba)
LHCb (Eric van Herwijnin)
others (see me)
Register:
“User Registration” @ WP6 website
Warnings:
Dev. project Frequent changes.
No guarantees on availability.
Schedule:
EDG 1.2 (shortly)
EDG 2.0 (fall 2002)
Interesting Web Sites
EDG (Main Website)
http://www.eu-datagrid.org
EDG (WP6) http://marianne.in2p3.fr
CrossGrid http://www.crossgrid.org
DataTag http://www.datatag.org
GridPP http://www.gridpp.ac.uk
LCG http://lhcgrid.web.cern.ch/LHCgrid
NorduGrid http://www.nordugrid.org
Condor http://www.cs.wisc.edu/condor
Globus http://www.globus.org
GriPhyN http://www.griphyn.org
iVDGL http://www.ivdgl.org
PPDG http://www.ppdg.net
OGSA http://www.globus.org/ogsa
Global Grid Forum
http://www.gridforum.org
top related