sam - sequential data access via metadata schema metadata functionality workshop glasgow university...

SAM - Sequential Data Access via Metadata

Schema Metadata

Functionality Workshop

Glasgow University April 26-28,2004

Sam Services

Experiment Specific• Runs• Luminosity• Streams & Triggers• Events (D0)

Core• Files• Processes• Cache/

ResourceMgt (Stations)

Job Handling• Job Request (MC)• Batch Processing • General Support

Tables

Metadata

Relational DB

Metadata Query

Metadata Query Language (Dimensions)

• Is a metadata service. • Associates keyword value pairs to their

representations on the experiment independent databases.

• Allows definition of metadata within the query service. The constraints per dimension are needed to discover the relevant physics metadata. 

• Is strongly enhanced by the dynamic parameter definition mechanisms associated with the job request services.  

Files• Are the heart and soul of Sam, they are

the complete file metadata catalog.

• Also, maintains volume information for enstore (Should move to that specific SE).

FilesJob

Request

LuminosityStream&Triggers

Events

Batch ProcessingProcesses

StationsGeneralSupport

Runs

Processes• Process metadata stores

– application, version– status of file processing (requested, delivered,

crashed, ok)

FilesJob

Request

LuminosityStream&Triggers

Events

Batch ProcessingProcesses

StationsGeneralSupport

Runs

Stations• Caching

– All locations, sizes, for a file on all hardware

– State: locked, available– Algorithm (policy)

• Resource Management– Admins (control resources)– Cache quota by group– Station rules (cache space,

project limits)

FilesJob

Request

LuminosityStream&Triggers

Events

Batch ProcessingProcesses

StationsGeneralSupport

Runs

Production Job Requests(MC & Batch)

• Processes monte carlo & farm requests.• Proposal to unify submission services.• File metadata predefined by request metadata.

FilesJob

Request

LuminosityStream&Triggers

Events

Batch ProcessingProcesses

StationsGeneralSupport

Runs

General Support

• Authentication– Who am I?– What group am I in?– What can I do?– Grid Subjects

• Fabric Definition– Nodes– Operating systems– Hardware

FilesJob

Request

LuminosityStream&Triggers

Events

Batch ProcessingProcesses

StationsGeneralSupport

Runs

Runs• Tracks run numbers from online (key to

Experiment-specific Online metadata).

• Maintains association between events and files.

FilesJob

Request

LuminosityStream&Triggers

Events

Batch ProcessingProcesses

StationsGeneralSupport

Runs

LuminosityStreams & Triggers

• Two sections in sam dealing with luminosity, and streams & triggers.

FilesJob

Request

LuminosityStream&Triggers

Events

Batch ProcessingProcesses

StationsGeneralSupport

Runs

Events

• Tracking of events within a file

• Metadata for each event

• Huge volumes of data (D0: 17M evts/wk)

FilesJob

Request

LuminosityStream&Triggers

Events

Batch ProcessingProcesses

StationsGeneralSupport

Runs