metadata progress gridpp18 20 march 2007 mike kenyon

Metadata Progress

GridPP1820 March 2007Mike Kenyon

Metadata Team

• Glasgow– Mike Kenyon– Caitriana Nicholson– Tom Doherty– Paul Millar

• CERN/Glasgow– Helen McGlone

ATLAS Event-level Metadata

• Navigation from Tag to events in AOD/ESD/RAW format

• Produced during AOD production– Oracle DBs at Tier0, Tier1– MySQL (probably) at Tier2

• ATLAS are implementing a Tag database to allow efficient identification and selection of events for analysis

• First-level cuts will be possible via queries on relational databases

Tag Navigator Toolhttps://twiki.cern.ch/twiki/bin/view/Atlas/TagNavigat

orTool

• Need to integrate Tag DB with Distributed Data Management (DQ2) and Distributed Analysis (Ganga) tools

• Tag Navigator Tool (TNT) developed to plug that gap

• Aims to take user from query on Tag DB to analysis using results of query

TNT Process Summary

• Runs a query on the database and outputs a local Root collection

• Divides this into a number of sub-collections

• Submits user jobs to LCG, one per sub-collection

• Output files can be registered as new DQ2 dataset

TNT Process Summary

Ganga Plugin

• TNT has also been implemented as a plugin to Ganga

• Ganga already allowed use of Tag files – TNT plugin is complementary to this

• Makes use of existing Ganga functionality for job submission, handling etc

• Will be available in Ganga release 4.3

Future Plans

• Continue development of TNT and GangaTnt in response to user feedback

• Performance testing of Tags in TNT• Scalability and performance

measurements on Tag DB• Continue integration of Tags with AMI

AMI-VOMS solution1. The grid user authenticates

themselves to AMI using a grid certificate.

2. AMI allows the user to launch a Java web start application to create a VOMS proxy.

3. The user chooses the x509 certificate to be sent to the VOMS server

4. The VOMS server sends back a VOMS proxy certificate to the Java application.

5. The VOMS proxy file is uploaded to the server, which hosts a proxy delegation service, via a secure web service attachment.

6. The user’s VO, group and role can be extracted from the proxy and used for authorisation purposes.

SourceForge project: Acacia

12

3

4

5

67

DQ2/AMI integration• Use of delegated VOMS proxy certificate - Contact

application on behalf of user, e.g. DQ2.• Register a dataset simultaneously in AMI and DQ2• Other dq2 commands possible

Tags/AMI integration• Preliminary work on prototype database to allow

physics predicate cut on a collection and catalogue the results within AMI

pyAMI

• AMI Web Service Client for Python• AMI tutorial available at:

http://ami3.in2p3.fr:8080/opencms/opencms/AMI/www/Tutorial/pyAMI.html

• Future development and maintenance• Providing AMI command wrappers for groups/tools such as

POOL Conditions, Muon and Tag collector• Code optimisation – replace SAX with DOM parsing of AMI

output• Reformatting output – interface to format XML

The present: MonAMI v0.6

• “Universal sensor framework”– Capable of monitoring many different services– Can send data to any number of information

systems– Plugin structure

• v0.6 deployed at all ScotGrid sites & Grenoble

• Configuration details available at– http://www.gridpp.ac.uk/wiki/MonAMI

Upcoming: MonAMI v0.7

• v0.7– Support for 64-bit (AMD64) architecture– Improved documentation– Major improvements to R-GMA support

• Robust against temporary R-GMA failures• Support for TLS-based secure upload of data• Support for proxy-certificate based authentication• “Drop in” deployment

– Support for event monitoring (e.g. DPM/GridFTP)

• Released pending solution of minor SL4-related issues

The Future: MonAMI v0.8

• Planned features– New python plugin API– Web & RSS feeds– PostGreSQL monitoring