![Page 1: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/1.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1
General components of Grid middleware and User Interfaces
Roberto Barbera
EGEE is funded by the European Union under contract IST-2003-508833
![Page 2: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/2.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 2
Overview
• The multi-layer model of middleware services • The Command Line user interface• The GENIUS Portal• Summary and conclusions
![Page 3: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/3.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 3
Middleware architecture
Collective Services
Information &
Monitoring
Replica Manager
Grid Scheduler
Local Application Local Database
Underlying Grid Services
Computing Element Services
Authorization Authentication and Accounting
Replica Catalog
Storage Element Services
SQL Database Services
Fabric services
ConfigurationManagement
Node Installation &Management
Monitoringand
Fault Tolerance
Resource Management
Fabric StorageManagement
Grid
Fabric
Local Computing
Grid Grid Application Layer
Data Management
Job Management
Metadata Management
Service Index
APPLICATIONS
GLOBUSCondorG(via VDT)
M / W
![Page 4: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/4.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 4
Middleware interfaces
Collective Services
Information & Monitoring
Replica Manager
Grid Scheduler
Local Application Local Database
Underlying Grid Services
Computing Element Services
Authorization Authentication and Accounting
Replica Catalog
Storage Element Services
SQL Database Services
Fabric services
ConfigurationManagement
Node Installation &Management
Monitoringand
Fault ToleranceResource
ManagementFabric StorageManagement
Grid Application Layer
Data ManagementJob
ManagementMetadata
ManagementObject to File
Mapping
Service Index
Computing Elements
SystemManagers
Scientists
OperatingSystems
File Systems
StorageElementsMass Storage
SystemsHPSS, Castor
User Accounts
Certificate Authorities
ApplicationDevelopers
Batch SystemsPBS, LSF, etc.
![Page 5: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/5.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 5
Job workflowReplicaReplicaCatalogueCatalogue
UIJDL
Logging &Logging &Book-keepingBook-keeping
ResourceResourceBrokerBroker
Job SubmissionJob SubmissionServiceService
StorageStorageElementElement
ComputingComputingElementElement
Information Information ServiceService
Job Status
DataSets info
Author.&Authen.
Job Submit
Event
Job Q
uery Job S
tatu
s
Input “sandbox”
Input “sandbox” + Broker InfoGlobus RSL
Output “sandbox”
Output “sandbox”
Job Status
Publish
grid
-pro
xy-in
it
Exp
ande
d JD
L
SE & CE info
![Page 6: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/6.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 6
The Command Line user interface
• The user interacts with Grid via a Workload Management System (WMS)
• The Goal of WMS is the distributed scheduling and resource management in a Grid environment.
• What does it allow Grid users to do?• To submit their jobs• To execute them on the “best resources”
• The WMS tries to optimize the usage of resources• To get information about their status• To retrieve their output
![Page 7: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/7.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 7
Job Description Language (JDL)
• The supported attributes are grouped in two categories:• Job Attributes
• Define the job itself• Resources
• Taken into account by the RB for carrying out the matchmaking algorithm (to choose the “best” resource where to submit the job)
• Computing Resource– Used to build expressions of Requirements and/or Rank
attributes by the user– Have to be prefixed with “other.”
• Data and Storage resources – Input data to process, SE where to store output data, protocols
spoken by application when accessing SEs
![Page 8: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/8.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 8
JDL: relevant attributes
• JobType • Normal (simple, sequential job), Interactive, MPICH, Checkpointable • Or combination of them
• Executable (mandatory)• The command name
• Arguments (optional)• Job command line arguments
• StdInput, StdOutput, StdError (optional)• Standard input/output/error of the job
• Environment• List of environment settings
• InputSandbox (optional)• List of files on the UI local disk needed by the job for running• The listed files will automatically staged to the remote resource
• OutputSandbox (optional)• List of files, generated by the job, which have to be retrieved
![Page 9: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/9.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 9
JDL: relevant attributes
• Requirements• Job requirements on computing resources • Specified using attributes of resources published in the Information Service• If not specified, default value defined in UI configuration file is considered
• Default: other.GlueCEStateStatus == "Production" (the resource has to be able to accept jobs and dispatch them on WNs)
• Rank• Expresses preference (how to rank resources that have already met the
Requirements expression)• Specified using attributes of resources published in the Information Service• If not specified, default value defined in the UI configuration file is
considered • Default: - other.GlueCEStateEstimatedResponseTime (the lowest estimated
traversal time)• Default: other.GlueCEStateFreeCPUs (the highest number of free CPUs) for
parallel jobs (see later)
![Page 10: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/10.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 10
JDL: relevant attributes
• InputData• Refers to data used as input by the job: these data are published
in the Replica Location Service (RLS) and stored in the SEs)• LFNs and/or GUIDs
• DataAccessProtocol (mandatory if InputData has been specified)
• The protocol or the list of protocols which the application is able to speak with for accessing InputData on a given SE
• OutputSE• The Uniform Resource Identifier of the output SE• RB uses it to choose a CE that is compatible with the job and is
close to SE
![Page 11: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/11.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 11
Job submission
• edg-job-submit [–r <res_id>] [-c <config file>] [-vo <VO>] [-o <output file>] <job.jdl>
• -r the job is submitted directly to the computing element identified by <res_id>
• -c the configuration file <config file> is pointed by the UI instead of the standard configuration file
• -vo the Virtual Organization (if user is not happy with the one specified in the UI configuration file)
• -o the generated edg_jobId is written in the <output file>• Useful for other commands, e.g.:• edg-job-status –i <input file> (or edg_jobId)
-i the status information about edg_jobId contained in the <input file> are displayed
![Page 12: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/12.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 12
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
![Page 13: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/13.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 13
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
submitted
Job Statu
s
UI: allows users to access the functionalitiesof the WMS(via command line, GUI, C++ and Java APIs)
![Page 14: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/14.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 14
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
edg-job-submit myjob.jdlMyjob.jdl
JobType = “Normal”;Executable = "$(CMS)/exe/sum.exe";InputSandbox = {"/home/user/WP1testC","/home/file*”, "/home/user/DATA/*"};OutputSandbox = {“sim.err”, “test.out”, “sim.log"};Requirements = other. GlueHostOperatingSystemName == “linux" && other. GlueHostOperatingSystemRelease == "Red Hat 6.2“ && other.GlueCEPolicyMaxWallClockTime > 10000;Rank = other.GlueCEStateFreeCPUs;
submitted
Job Statu
s
Job Description Language(JDL) to specify job characteristics and requirements
![Page 15: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/15.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 15
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
RBstorage
Input Sandboxfiles
Jobwaiting
submitted
Job Status
NS: network daemon responsible for acceptingincoming requests
![Page 16: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/16.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 16
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
RBstorage
waiting
submitted
Job Status
WM: responsible to takethe appropriate actions to satisfy the request
Job
![Page 17: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/17.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 17
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
RBstorage
waiting
submitted
Job Status
Match-Maker/Broker
Where must thisjob be executed ?
![Page 18: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/18.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 18
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
RBstorage
waiting
submitted
Job Status
Match-Maker/ Broker
Matchmaker: responsible to find the “best” CE where to submit a job
![Page 19: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/19.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 19
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
RBstorage
waiting
submitted
Job Status
Match-Maker/ Broker
Where are (which SEs) the needed data ?
What is thestatus of the
Grid ?
![Page 20: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/20.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 20
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
RBstorage
waiting
submitted
Job Status
Match-Maker/Broker
CE choice
![Page 21: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/21.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 21
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
RBstorage
waiting
submitted
Job Status
JobAdapter
JA: responsible for the final “touches” to the job before performing submission(e.g. creation of wrapper script, etc.)
![Page 22: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/22.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 22
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
RBstorage
Job Status
JC: responsible for theactual job managementoperations (done via CondorG)
Job
submitted
waiting
ready
![Page 23: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/23.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 23
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
CE characts& status
SE characts& status
RBstorage
Job Status
Job
InputSandboxfiles
submitted
waiting
ready
scheduled
![Page 24: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/24.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 24
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
RBstorage
Job Status
InputSandbox
submitted
waiting
ready
scheduled
running
“Grid enabled”data transfers/
accesses
Job
![Page 25: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/25.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 25
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
RBstorage
Job Status
OutputSandboxfiles
submitted
waiting
ready
scheduled
running
done
![Page 26: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/26.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 26
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
RBstorage
Job Status
OutputSandbox
submitted
waiting
ready
scheduled
running
done
edg-job-get-output <dg-job-id>
![Page 27: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/27.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 27
Job submission
UI
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ReplicaLocationServer
Inform.Service
ComputingElement
StorageElement
RB node
RBstorage
Job Status
OutputSandboxfiles
submitted
waiting
ready
scheduled
running
done
cleared
![Page 28: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/28.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 28
Job monitoring
UI
Log Monitor
Logging &Bookkeeping
NetworkServer
Job Contr.-
CondorG
WorkloadManager
ComputingElement
RB node
LM: parses CondorG logfile (where CondorG logsinfo about jobs) and notifies LB
LB: receives and stores job events; processes corresponding job status
Log ofjob events
edg-job-status <dg-job-id>edg-job-get-logging-info <dg-job-id>
Job status
![Page 29: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/29.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 29
Other (most relevant) UI commands
•edg-job-list-match• Lists resources matching a job description• Performs the matchmaking without submitting the job
• edg-job-cancel• Cancels a given job
• edg-job-status• Displays the status of the job
• edg-job-get-output• Returns the job-output (the OutputSandbox files) to the user
• edg-job-get-logging-info• Displays logging information about submitted jobs (all the events “pushed” by
the various components of the WMS)• Very useful for debug purposes
![Page 30: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/30.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 30
Data Management Tools
• Tools for• Locating data• Copying data• Managing and replicating data• Meta Data management
• On EDG you have• Replica Location Service (RLS)• Replica Metadata Service (RMC)• Replica Optimisation Service (ROS)• Replica Manager (RM)
RMRLS
ROS
RMC
![Page 31: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/31.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 31
StorageElement
Replication Services: Basic Functionality
Replica ManagerReplica Location
Service
Replica Metadata Catalog
StorageElement
Files have replicas stored at many Grid sites on Storage Elements.
Each file has a unique Grid ID.Locations corresponding to the GUID are kept in the Replica Location Service.
Users may assign aliases to the GUIDs. These are kept in the Replica Metadata Catalog.
The Replica Manager provides atomicity for file operations, assuring consistency of SE and catalog contents.
![Page 32: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/32.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 32
StorageElement
Higher Level Replication Services
Replica ManagerReplica Location
Service
Replica Optimization Service
Replica Metadata Catalog
SEMonitor Network MonitorStorage
Element
The Replica Manager may call on the Replica Optimization service to find the best replica among many based on network and SE monitoring.
Hooks for user-defined pre- and post-processing for replication operations are available.
![Page 33: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/33.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 33
Naming Conventions
• Logical File Name (LFN) • An alias created by a user to refer to some item of data e.g.
“lfn:cms/20030203/run2/track1”
• Site URL (SURL) (or Physical File Name (PFN))
• The location of an actual piece of data on a storage system e.g. “srm://pcrd24.cern.ch/flatfiles/cms/output10_1”
• Globally Unique Identifier (GUID)
• A non-human readable unique identifier for an item of data e.g. “guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6”
Logical File Name 1
Logical File Name 2
Logical File Name n
GUIDPhysical File SURL n
Physical File SURL 1
![Page 34: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/34.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 34
Replica Metadata Catalog (RMC) vs. Replica Location Service (RLS)
• RMC:• Stores LFN-GUID mappings
• RLS:• Stores GUID-SURL mappings
Logical File Name 1
Logical File Name 2
Logical File Name n
GUIDPhysical File SURL n
Physical File SURL 1
RMC RLS
RMRLS
ROS
RMC
![Page 35: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/35.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 35
Replica Location Service (RLS)
• The Replica Location Service is a system that maintains and provides access to information about the physical location of copies of data files.
• It is a distributed service that stores mappings between globally unique identifiers of datafiles and the physical identifiers of all existing replicas of these datafiles.
• Design is a joint collaboration between Globus and EDG-WP2
RMRLS
ROS
RMC
![Page 36: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/36.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 36
The Replica Manager Interface –Management Commands
• copyAndRegisterFile args: source, dest, lfn, protocol, streams
• Copy a file into grid-aware storage and register the copy in the Replica Catalog as an atomic operation.
• replicateFile args: source/lfn, dest, protocol, streams
• Replicate a file between grid-aware stores and register the replica in the Replica Catalog as an atomic operation.
• deleteFile args: source/seHost, all • Delete a file from storage and unregister it.
• Example
edg-rm --vo=tutor copyAndRegisterFilefile:/home/bob/analysis/data5.dat -d lxshare0384.cern.ch
![Page 37: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/37.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 37
The Replica Manager Interface –Catalog Commands (1)
• registerFile args: source, lfn• Register a file in the Replica Catalog that is already stored
on a Storage Element.
• unregisterFile args: source, guid• Unregister a file from the Replica Catalog.
• listReplicas args: lfn/surl/guid• List all replicas of a file.
• registerGUID args: surl, guid• Register an SURL with a known GUID in the Replica
Catalog.
• listGUID args: lfn/surl• Print the GUID associated with an LFN or SURL.
![Page 38: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/38.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 38
The Replica Manager Interface –Catalog Commands (2)
• addAlias args: guid, lfn• Add a new alias to GUID mapping
• removeAlias args: guid, lfn• Remove an alias LFN from a known GUID.
• printInfo()• Print the information needed by the Replica
Manager to screen or to a file.• getVersion()
• Get the versions of the replica manager client.
![Page 39: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/39.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 39
The Replica Manager Interface –Optimization Commands
• listBestFile args: lfn/guid, seHost• Return the 'best' replica for a given logical file identifier.
• getBestFile args: lfn/guid, seHost, protocol, streams
• Return the storage file name (SFN) of the best file in terms of network latencies.
• getAccessCost args: lfn/guid[], ce[], protocol[]
• Calculates the expected cost of accessing all the files specified by logicalName from each Computing Element host specified by ceHosts.
![Page 40: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/40.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 40
The Replica Manager Interface –File Transfer Commands
• copyFile args: soure, dest • Copy a file to a non-grid destination.
• listDirectory args: dir • List the directory contents on an SRM or a GridFTP server.
![Page 41: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/41.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 41
The “transparent” grid access
localdisk
StorageStorageElementElement
StorageStorageElementElement
StorageStorageElementElement
ReplicaReplicaCatalogueCatalogue
UIdisk
ResourceResourceBrokerBroker
ComputeComputeElementElement
ComputeComputeElementElement
VOVOcolleaguecolleague
VOVOcolleaguecolleague
![Page 42: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/42.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 42
A web portal: why and how ?
• It can be accessed from everywhere and by “everything” (desktop, laptop, PDA, phone).
• It can keep the same user interface independently of the underlying middleware.
• It must be redundantly “secure” at all levels: 1) secure for web transactions, 2) secure for user credentials, 3) secure for user authentication, 4) secure at VO level.
• All available grid services must be incorporated in a logic way, just “one mouse click away”.
• Its layout must be easily understandable and user friendly.
![Page 43: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/43.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 43
Title 1
OS & Net services
Basic Services
High level GRID middleware
ALICE ATLAS CMS LHCbApplications’specific layer Other apps
GLOBUS
toolkit
DataGRID architectur
e
GENIUS web portal
GENIUS®
(Grid Enabled web eNvironment for site Independent User job Submission)
[https://genius.ct.infn.it] INFN/NICE collaboration
![Page 44: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/44.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 44
GENIUS: how it works
Roberto Barbera
Apache
EnginFrame
GENIUShttps+java/xml+rfb
WEB Browser
EDGUI
Local
WS
the GridEDG+GSI
3-tier model
![Page 45: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/45.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 45
Grid authentication with MyProxy
EDGUI
Local
WS
MyProxy
Server
GENIUS
Server(EDG
UI)
grid-proxy-initmyproxy-init
any grid service
myproxy-get-delegation
output
the Grid
execution
WEB Browser
![Page 46: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/46.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 46
GENIUS home page
![Page 47: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/47.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 47
GENIUS file services
![Page 48: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/48.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 48
GENIUS authentication with MyProxy
![Page 49: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/49.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 49
GENIUS grid settings
![Page 50: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/50.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 50
GENIUS VO selection
![Page 51: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/51.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 51
GENIUS graphic job editor (work in collaboration with DATAMAT)
![Page 52: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/52.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 52
GENIUS graphic workflow editor (integration with TRIANA from GridLab)
![Page 53: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/53.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 53
GENIUS single job submission
![Page 54: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/54.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 54
GENIUS single job queue
![Page 55: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/55.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 55
GENIUS multi job submission
![Page 56: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/56.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 56
GENIUS multi job queue
![Page 57: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/57.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 57
GENIUS data spooler (1/2)
![Page 58: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/58.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 58
GENIUS data spooler (2/2)
![Page 59: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/59.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 59
GENIUS RLS browsing
![Page 60: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/60.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 60
GENIUS data publication
![Page 61: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/61.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 61
GENIUS data replication
![Page 62: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/62.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 62
GENIUS information services
![Page 63: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/63.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 63
GENIUS monitoring services
![Page 64: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/64.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 64
GENIUS interactive services
![Page 65: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/65.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 65
GENIUS P2P services
![Page 66: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/66.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 66
GENIUS for PDA (1/2)
Home Page
Services Remote filebrowsing Remote file
inspection
![Page 67: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/67.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 67
GENIUS for PDA (2/2)
GridICEintegration
RC browsingJob monitoring
Job submission
![Page 68: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/68.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 68
GENIUS for cell phone
![Page 69: Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 1 General components of Grid middleware and User Interfaces Roberto](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b0a7f8b9ab05998a38e/html5/thumbnails/69.jpg)
Induction: General components of Grid middleware and User Interfaces –April 26-28, 2004 - 69
Summary and conclusions
Let’s go to the practical