sam - sequential data access via metadata schema metadata functionality workshop glasgow university...
TRANSCRIPT
SAM - Sequential Data Access via Metadata
Schema Metadata
Functionality Workshop
Glasgow University April 26-28,2004
Sam Services
Experiment Specific• Runs• Luminosity• Streams & Triggers• Events (D0)
Core• Files• Processes• Cache/
ResourceMgt (Stations)
Job Handling• Job Request (MC)• Batch Processing • General Support
Tables
Metadata
Relational DB
Metadata Query
Metadata Query Language (Dimensions)
• Is a metadata service. • Associates keyword value pairs to their
representations on the experiment independent databases.
• Allows definition of metadata within the query service. The constraints per dimension are needed to discover the relevant physics metadata.
• Is strongly enhanced by the dynamic parameter definition mechanisms associated with the job request services.
Files• Are the heart and soul of Sam, they are
the complete file metadata catalog.
• Also, maintains volume information for enstore (Should move to that specific SE).
FilesJob
Request
LuminosityStream&Triggers
Events
Batch ProcessingProcesses
StationsGeneralSupport
Runs
Processes• Process metadata stores
– application, version– status of file processing (requested, delivered,
crashed, ok)
FilesJob
Request
LuminosityStream&Triggers
Events
Batch ProcessingProcesses
StationsGeneralSupport
Runs
Stations• Caching
– All locations, sizes, for a file on all hardware
– State: locked, available– Algorithm (policy)
• Resource Management– Admins (control resources)– Cache quota by group– Station rules (cache space,
project limits)
FilesJob
Request
LuminosityStream&Triggers
Events
Batch ProcessingProcesses
StationsGeneralSupport
Runs
Production Job Requests(MC & Batch)
• Processes monte carlo & farm requests.• Proposal to unify submission services.• File metadata predefined by request metadata.
FilesJob
Request
LuminosityStream&Triggers
Events
Batch ProcessingProcesses
StationsGeneralSupport
Runs
General Support
• Authentication– Who am I?– What group am I in?– What can I do?– Grid Subjects
• Fabric Definition– Nodes– Operating systems– Hardware
FilesJob
Request
LuminosityStream&Triggers
Events
Batch ProcessingProcesses
StationsGeneralSupport
Runs
Runs• Tracks run numbers from online (key to
Experiment-specific Online metadata).
• Maintains association between events and files.
FilesJob
Request
LuminosityStream&Triggers
Events
Batch ProcessingProcesses
StationsGeneralSupport
Runs
LuminosityStreams & Triggers
• Two sections in sam dealing with luminosity, and streams & triggers.
FilesJob
Request
LuminosityStream&Triggers
Events
Batch ProcessingProcesses
StationsGeneralSupport
Runs
Events
• Tracking of events within a file
• Metadata for each event
• Huge volumes of data (D0: 17M evts/wk)
FilesJob
Request
LuminosityStream&Triggers
Events
Batch ProcessingProcesses
StationsGeneralSupport
Runs
