mardi 30 mars 2010

Lavoisier : a way to integrate heteregeneous monitoring systems.

Cyril L’Orphelin

IN2P3/CNRS Computing Centre, Lyon, France

HEPiX Spring 2010 Workshop 2

OutlinesOutlines

- Motivations

- Main benefits of using Lavoisier

- Concrete example : the Operations Dashboard

- Future developments

HEPiX Spring 2010 Workshop 330/03/10 3

EGEE / LCG information providersEGEE / LCG information providers

CIC-DB- VO Information - VO Contacts- User Contacts

SQLSQLVO repository (static)

GOC-DB - Site Information- Site Contacts- Downtimes

SQLSQLSite repository (static)

GGUS - Create ticket- Update ticket- View ticket

SOAPTicketing System

SAM- Test results- Status per nodes

X-SQLJob Submission Framework

BDII LDAPDynamic information per nodes , per sites ...

LDAP Database for sites

VOMS HTTP + Script - VOMS certificates.- VOMS users

VO User repository (dynamic)

- GIIS status per site- Storage Information- Jobs Information

GSTAT HTTPGIIS browser

HTTP

Monitoring Framework Nagios Notifications about problem on services

AMQ

HEPiX Spring 2010 Workshop 4

Multiple heterogeneous sources and technologies :

No common language

No direct cross queries

• For a web application , it means lot of computations and treatment and a dependency to the technologies proposed

The information might be : Not usable Not available Too long to be queried

Motivations - ContextMotivations - Context

HEPiX Spring 2010 Workshop 530/03/10 5

Our solution : LavoisierOur solution : Lavoisier

An answer developped at CC-IN2P3: the Lavoisier service

A good mechanism to build unified views from heterogeneous data sources

http://grid.in2p3.fr/lavoisier

"Nothing is lost, nothing is created, all is transformed" Antoine Laurent Lavoisier (1743 - 1794)

HEPiX Spring 2010 Workshop 6

Access data views through SOAP, REST/JSON or REST/XMLo get data data viewo cross-query data views using standard languages: XSLT (and soon: XPath, XQuery)

Accesses to dataAccesses to data

HEPiX Spring 2010 Workshop 7

Separation of roles (changes related to a role have no impact on other roles)o the plug-in developer adds support for new technologies by developing adapters.o the service administrator configures the adaters and cache management according to the

characteristics of the data views and data sources, and according to the constraints of the use-cases.

o the user queries the data views.

Separation of rolesSeparation of roles

HEPiX Spring 2010 Workshop 8

Data cache management o select cache type

in-memory (simple or hierarchical, parsed once) on-disk (simple or hierarchical) no-cache (directly invoke the data source)

o combine cache refresh triggering rules: whenCreated, whenNotified, whenTime, whenMaxUnrefreshedTimeElapsed, whenMaxUnrefreshedTimeElapsedAndDataAccessed, whenUpdated, whenRefreshed, whenRefreshedAndPublished

Extensibility

o Reusable adapters are provided for commonly used technologies: HTTP, LDAP, RDBMS, XSLT...

o Support for new technlogies can be added by developing adapters

Validation of generated XML data o check XML syntax only o check XML syntax and conformity to provided XML Schema

Administration o get data views index and properties o get data views status o on-the-fly reconfiguration (detect views to be reconfigured)

Other main featuresOther main features

HEPiX Spring 2010 Workshop 9

Lavoisier Console

HEPiX Spring 2010 Workshop 10

The Operations dashboard

Heterogeneous data XML HTML

HEPiX Spring 2010 Workshop 11

The Synoptic view

https://operations-portal.in2p3.fr/

HEPiX Spring 2010 Workshop 12

Access to details

HEPiX Spring 2010 Workshop 13

Future developments (Lavoisier 1.x)

RDBMS adaptor (on-going development)user see the entire database as a huge XML document

structure described by Hibernate or Doctrine ORM mapping file

query with XPath / response as XML

but of course this huge XML document is never builtXPath query is translated to SQL query

only the result set is converted to XML data

XQuery and XPath as alternatives to XSLTavailable through SOAP and REST operations

available as adapters (=> cacheable processing result)

HEPiX Spring 2010 Workshop 14

Future developments (Lavoisier 2.0)

Refactoring of core enginebased on JMS to increase flexibility for notifications

within a Lavoisier instance (through in-memory messages)

between Lavoisier instances (though OpenWire protocol)

improve extensibility with adapter interfaces for…

Data protocol connectors

Data format transformers

Cache content managers 4 hard-coded managers

Cache refresh triggers 8 hard-coded triggers

Cache refresh scheduler No scheduler (overload risk)

Adapters do both (less reusable)

Lavoisier 2.0Lavoisier 2.0 Lavoisier 1.xLavoisier 1.x

HEPiX Spring 2010 Workshop 15

Links

Lavoisier : • http://grid.in2p3.fr/lavoisier/

Dashboard :• https://operations-portal.in2p3.fr (you need to be

registered in GOC DB )

• https://forge.in2p3.fr/projects/show/opsportaluser