crest project description · andrea formica: svom dev meeting cppm conditions data outline •...
TRANSCRIPT
Crest project description
Andrea Formica IRFU
Conditions dataAndrea Formica: Svom dev meeting CPPM
Outline
• Crest project‣ Which use cases are addressed
‣ It belong to the set of commons service we are starting to develop for SVOM community
• Data model‣ Description and functionalities overview
• Architecture‣ Toward a full REST access for conditions
2
Conditions dataAndrea Formica: Svom dev meeting CPPM
Conditions data in Crest
3
• Attempting a definition‣ Conditions/Calibration data are non-event data accessed
pipelines
• Concrete examples‣ The above definition may cover:
- Calibrations data from instrument center
- An additional step using CALDB is needed in order to deliver these data to pipelines if they use existing software
- Part of the configuration data needed for pipelines
Conditions dataAndrea Formica: Svom dev meeting CPPM
Crest data model
4
ATLAS_PHYS_COND.GLOBAL_TAG_MAP*PF GLOBAL_TAG_NAME VARCHAR2 (100 CHAR)*P LABEL VARCHAR2 (100 CHAR)*P RECORD VARCHAR2 (100 CHAR)*F TAG_NAME VARCHAR2 (100 CHAR)
GLOBAL_TAG_MAP_PK (GLOBAL_TAG_NAME, LABEL, RECORD)
FK_2LB1J0I8MW5YFR7OCPYSVRV56 (GLOBAL_TAG_NAME)FK_76CJJ2HVL9AKLLLG6E3YDNT91 (TAG_NAME)
ATLAS_PHYS_COND.IOV*P INSERTION_TIME TIMESTAMP*P SINCE NUMBER (38)*PF TAG_NAME VARCHAR2 (256 CHAR)*F PAYLOAD_HASH VARCHAR2 (64 CHAR)
IOV_PK (TAG_NAME, SINCE, INSERTION_TIME)
FK_EMC1JLDXMYST41O982A1SPF6 (TAG_NAME)FK_IOV_PAYLOAD (PAYLOAD_HASH)
IOV_PK (TAG_NAME, SINCE, INSERTION_TIME)
ATLAS_PHYS_COND.PAYLOAD*P HASH VARCHAR2 (64 CHAR)* DATA BLOB* INSERTION_TIME TIMESTAMP* OBJECT_TYPE VARCHAR2 (100 CHAR)* STREAMER_INFO BLOB* VERSION VARCHAR2 (20 CHAR)
PAYLOAD_PK (HASH)
ATLAS_PHYS_COND.TAG*P NAME VARCHAR2 (256 CHAR)* DESCRIPTION VARCHAR2 (4000 CHAR)* END_OF_VALIDITY NUMBER (38)* INSERTION_TIME TIMESTAMP* LAST_VALIDATED_TIME NUMBER (38)* MODIFICATION_TIME TIMESTAMP* OBJECT_TYPE VARCHAR2 (256 CHAR)* SYNCHRONIZATION VARCHAR2 (20 CHAR)* TIME_TYPE VARCHAR2 (16 CHAR)
TAG_PK (NAME)
ATLAS_PHYS_COND.GLOBAL_TAG*P NAME VARCHAR2 (100 CHAR)* DESCRIPTION VARCHAR2 (4000 CHAR)* INSERTION_TIME TIMESTAMP* RELEASE VARCHAR2 (100 CHAR)* SCENARIO VARCHAR2 (100 CHAR)* SNAPSHOT_TIME TIMESTAMP* TYPE CHAR (1 CHAR)* VALIDITY NUMBER (38)* WORKFLOW VARCHAR2 (100 CHAR)
GLOBAL_TAG_PK (NAME)
5 tables: global tags, tags, mappings, iovs, payloadsAll conditions stored in the same schema
Simple ManyToMany mapping between tags and global tags
Conditions dataAndrea Formica: Svom dev meeting CPPM
Crest data model
5
ATLAS_PHYS_COND.GLOBAL_TAG_MAP*PF GLOBAL_TAG_NAME VARCHAR2 (100 CHAR)*P LABEL VARCHAR2 (100 CHAR)*P RECORD VARCHAR2 (100 CHAR)*F TAG_NAME VARCHAR2 (100 CHAR)
GLOBAL_TAG_MAP_PK (GLOBAL_TAG_NAME, LABEL, RECORD)
FK_2LB1J0I8MW5YFR7OCPYSVRV56 (GLOBAL_TAG_NAME)FK_76CJJ2HVL9AKLLLG6E3YDNT91 (TAG_NAME)
ATLAS_PHYS_COND.IOV*P INSERTION_TIME TIMESTAMP*P SINCE NUMBER (38)*PF TAG_NAME VARCHAR2 (256 CHAR)*F PAYLOAD_HASH VARCHAR2 (64 CHAR)
IOV_PK (TAG_NAME, SINCE, INSERTION_TIME)
FK_EMC1JLDXMYST41O982A1SPF6 (TAG_NAME)FK_IOV_PAYLOAD (PAYLOAD_HASH)
IOV_PK (TAG_NAME, SINCE, INSERTION_TIME)
ATLAS_PHYS_COND.PAYLOAD*P HASH VARCHAR2 (64 CHAR)* DATA BLOB* INSERTION_TIME TIMESTAMP* OBJECT_TYPE VARCHAR2 (100 CHAR)* STREAMER_INFO BLOB* VERSION VARCHAR2 (20 CHAR)
PAYLOAD_PK (HASH)
ATLAS_PHYS_COND.TAG*P NAME VARCHAR2 (256 CHAR)* DESCRIPTION VARCHAR2 (4000 CHAR)* END_OF_VALIDITY NUMBER (38)* INSERTION_TIME TIMESTAMP* LAST_VALIDATED_TIME NUMBER (38)* MODIFICATION_TIME TIMESTAMP* OBJECT_TYPE VARCHAR2 (256 CHAR)* SYNCHRONIZATION VARCHAR2 (20 CHAR)* TIME_TYPE VARCHAR2 (16 CHAR)
TAG_PK (NAME)
ATLAS_PHYS_COND.GLOBAL_TAG*P NAME VARCHAR2 (100 CHAR)* DESCRIPTION VARCHAR2 (4000 CHAR)* INSERTION_TIME TIMESTAMP* RELEASE VARCHAR2 (100 CHAR)* SCENARIO VARCHAR2 (100 CHAR)* SNAPSHOT_TIME TIMESTAMP* TYPE CHAR (1 CHAR)* VALIDITY NUMBER (38)* WORKFLOW VARCHAR2 (100 CHAR)
GLOBAL_TAG_PK (NAME)
IOVs are metadata as wellThey can be queried separately from payload
Simple OneToMany mapping between tags and iovs
Conditions dataAndrea Formica: Svom dev meeting CPPM
Crest data model
6
ATLAS_PHYS_COND.GLOBAL_TAG_MAP*PF GLOBAL_TAG_NAME VARCHAR2 (100 CHAR)*P LABEL VARCHAR2 (100 CHAR)*P RECORD VARCHAR2 (100 CHAR)*F TAG_NAME VARCHAR2 (100 CHAR)
GLOBAL_TAG_MAP_PK (GLOBAL_TAG_NAME, LABEL, RECORD)
FK_2LB1J0I8MW5YFR7OCPYSVRV56 (GLOBAL_TAG_NAME)FK_76CJJ2HVL9AKLLLG6E3YDNT91 (TAG_NAME)
ATLAS_PHYS_COND.IOV*P INSERTION_TIME TIMESTAMP*P SINCE NUMBER (38)*PF TAG_NAME VARCHAR2 (256 CHAR)*F PAYLOAD_HASH VARCHAR2 (64 CHAR)
IOV_PK (TAG_NAME, SINCE, INSERTION_TIME)
FK_EMC1JLDXMYST41O982A1SPF6 (TAG_NAME)FK_IOV_PAYLOAD (PAYLOAD_HASH)
IOV_PK (TAG_NAME, SINCE, INSERTION_TIME)
ATLAS_PHYS_COND.PAYLOAD*P HASH VARCHAR2 (64 CHAR)* DATA BLOB* INSERTION_TIME TIMESTAMP* OBJECT_TYPE VARCHAR2 (100 CHAR)* STREAMER_INFO BLOB* VERSION VARCHAR2 (20 CHAR)
PAYLOAD_PK (HASH)
ATLAS_PHYS_COND.TAG*P NAME VARCHAR2 (256 CHAR)* DESCRIPTION VARCHAR2 (4000 CHAR)* END_OF_VALIDITY NUMBER (38)* INSERTION_TIME TIMESTAMP* LAST_VALIDATED_TIME NUMBER (38)* MODIFICATION_TIME TIMESTAMP* OBJECT_TYPE VARCHAR2 (256 CHAR)* SYNCHRONIZATION VARCHAR2 (20 CHAR)* TIME_TYPE VARCHAR2 (16 CHAR)
TAG_PK (NAME)
ATLAS_PHYS_COND.GLOBAL_TAG*P NAME VARCHAR2 (100 CHAR)* DESCRIPTION VARCHAR2 (4000 CHAR)* INSERTION_TIME TIMESTAMP* RELEASE VARCHAR2 (100 CHAR)* SCENARIO VARCHAR2 (100 CHAR)* SNAPSHOT_TIME TIMESTAMP* TYPE CHAR (1 CHAR)* VALIDITY NUMBER (38)* WORKFLOW VARCHAR2 (100 CHAR)
GLOBAL_TAG_PK (NAME)
IOVs request for global tag with snapshot timeSnapshot time can be used to say that a global tag needs to access IOVs inserted “before” that time. Useful for reprocessing campaign.
Can access older version of IOVs
Conditions dataAndrea Formica: Svom dev meeting CPPM
Crest data model
7
ATLAS_PHYS_COND.GLOBAL_TAG_MAP*PF GLOBAL_TAG_NAME VARCHAR2 (100 CHAR)*P LABEL VARCHAR2 (100 CHAR)*P RECORD VARCHAR2 (100 CHAR)*F TAG_NAME VARCHAR2 (100 CHAR)
GLOBAL_TAG_MAP_PK (GLOBAL_TAG_NAME, LABEL, RECORD)
FK_2LB1J0I8MW5YFR7OCPYSVRV56 (GLOBAL_TAG_NAME)FK_76CJJ2HVL9AKLLLG6E3YDNT91 (TAG_NAME)
ATLAS_PHYS_COND.IOV*P INSERTION_TIME TIMESTAMP*P SINCE NUMBER (38)*PF TAG_NAME VARCHAR2 (256 CHAR)*F PAYLOAD_HASH VARCHAR2 (64 CHAR)
IOV_PK (TAG_NAME, SINCE, INSERTION_TIME)
FK_EMC1JLDXMYST41O982A1SPF6 (TAG_NAME)FK_IOV_PAYLOAD (PAYLOAD_HASH)
IOV_PK (TAG_NAME, SINCE, INSERTION_TIME)
ATLAS_PHYS_COND.PAYLOAD*P HASH VARCHAR2 (64 CHAR)* DATA BLOB* INSERTION_TIME TIMESTAMP* OBJECT_TYPE VARCHAR2 (100 CHAR)* STREAMER_INFO BLOB* VERSION VARCHAR2 (20 CHAR)
PAYLOAD_PK (HASH)
ATLAS_PHYS_COND.TAG*P NAME VARCHAR2 (256 CHAR)* DESCRIPTION VARCHAR2 (4000 CHAR)* END_OF_VALIDITY NUMBER (38)* INSERTION_TIME TIMESTAMP* LAST_VALIDATED_TIME NUMBER (38)* MODIFICATION_TIME TIMESTAMP* OBJECT_TYPE VARCHAR2 (256 CHAR)* SYNCHRONIZATION VARCHAR2 (20 CHAR)* TIME_TYPE VARCHAR2 (16 CHAR)
TAG_PK (NAME)
ATLAS_PHYS_COND.GLOBAL_TAG*P NAME VARCHAR2 (100 CHAR)* DESCRIPTION VARCHAR2 (4000 CHAR)* INSERTION_TIME TIMESTAMP* RELEASE VARCHAR2 (100 CHAR)* SCENARIO VARCHAR2 (100 CHAR)* SNAPSHOT_TIME TIMESTAMP* TYPE CHAR (1 CHAR)* VALIDITY NUMBER (38)* WORKFLOW VARCHAR2 (100 CHAR)
GLOBAL_TAG_PK (NAME)
PAYLOADs are Blobs (files)They can be accessed by individual queries
Simple OneToMany mapping between iovs and payloads PAYLOADs hide conditions complexity (channels,…)
Conditions dataAndrea Formica: Svom dev meeting CPPM
Data model in words
8
• Conditions data‣ Payload : values are consumed as an aggregated set,
typically an header and some parameters container(s)
• Conditions metadata‣ IOV: time information, based on 1 time column (time, run
number, …), valid until the next entry in timean IOV point to 1 payload (via an sha256 hash)
‣ Tag: label to identify a specific set of IOVs
‣ Global Tag: consistent set of tags, involved in a given data-flow. A tag can be associated to many global tags.
Conditions dataAndrea Formica: Svom dev meeting CPPM
Crest architecture
• Designing the global architecture of the system:‣ Take into account the caching layer
‣ Keep the different components well decoupled
‣ Reuse tools and libraries in Open Source community
• The proposed architecture is a multi-tier model, with the middleware as a simple REST web application‣ cacheable, language independent
9
Conditions dataAndrea Formica: Svom dev meeting CPPM
Crest architecture
• Components‣ Database, Backend services , Frontend controllers, clients
10
Crest DB
JPA JDBC
HTTPJAX-RS
DataPkg
DataRepositories
WebControllers
Oracle
Sqlite
Postgres
h2
EXPERT(Python, Java,
C++, Javascript)
Conditions dataAndrea Formica: Svom dev meeting CPPM
Crest DB server• As Frontier, the CrestDB server exposes functionalities via REST
‣ But SQL is not involved in the dialogue between client and server, instead the internal resources are accessible via URLs
‣ All HTTP verbs can be used: POST/PUT (to create/update resources), GET and DELETE… - GET <server>/tags/A_TAG_NAME => retrieve resource with id A_TAG_NAME - POST <server>/tags {request body} => create new resource (parameters are taken from body ) - PUT <server>/tags/A_TAG_NAME {request body} => update resource A_TAG_NAME - DELETE <server>/tags/A_TAG_NAME => delete resource A_TAG_NAME
‣ Request and Response bodies are formatted in JSON
‣ Header of the requests can be used (e.g. for formatting the output, deal with caching related parameter etc.)
• Prototype Implementation‣ Based on standard Java technologies (JEE, Spring) and specifications (JAX-RS, JPA)
‣ Can be deployed in the same Tomcat server as Frontier
11
Conditions dataAndrea Formica: Svom dev meeting CPPM
Lexicon• JEE
‣ Java Enterprise Edition: is a set of specifications, extending Java SE[1] with specifications for enterprise features such as distributed computing and web services.[2] Java EE applications are run on referencing runtimes, that can be microservices or application servers, which handle transactions, security, scalability, concurrency and management of the components it is deploying.
• Spring‣ is an application framework and inversion of control container for the Java platform. The framework's core
features can be used by any Java application, but there are extensions for building web applications on top of the Java EE (Enterprise Edition) platform.
• JAX-RS (reference implementation: Oracle Jersey)‣ Java API for Restful Web Services : provides support in creating web services according to
the Representational State Transfer (REST) architectural pattern.[1] JAX-RS uses annotations, introduced in Java SE 5, to simplify the development and deployment of web service clients and endpoints.
• JPA (reference implementation: RedHat Hibernate)‣ Java Persistence API: specification that describes the management of relational data in
applications using Java Platform, Standard Edition and Java Platform, Enterprise Edition.
12
Conditions dataAndrea Formica: Svom dev meeting CPPM
Spring, Hibernate, JAX-RS examples
• Spring repositories
• Hibernate entities
• JAX-RS Jersey endpoints
13
Conditions dataAndrea Formica: Svom dev meeting CPPM
Crest API• The Crest API is a set of REST methods, allowing
external clients to interact with the resources represented in the data model.‣ Set of methods are today focused on storage of file into the
system using a TAG name and an IOV, and also on retrieval capabilities (search by IOV range etc.)
• Crest API is written using OpenAPI specifications‣ A JSON file describe the URLs (a.k.a. endpoints) and their
parameters, and the Request and Response bodies
‣ Using this file(s) we can generate clients and server stubs in several frameworks, via the swagger-codegen library
14
Conditions dataAndrea Formica: Svom dev meeting CPPM
Swagger and OpenAPI
• Simplified view of the swagger generation‣ Endpoint definition, component definition
‣ Client and server generation
15
Components(objets)
Endpoints(urls)
JSON - OpenAPI
Server stubs(Python)
Server stubs(Java)
Client(Python)
Client (Java)
Conditions dataAndrea Formica: Svom dev meeting CPPM
Building the prototype• Focus on business code‣ Reduce the amount of boiler plate by using existing
frameworks
16
HTTP Jerseyservlet
/tags
/iovs
/globaltags
Tags controller
Iovs controller
Globaltags controller
Functions to handle tags
Functions to handle iovs
Functions to handle gtags
Request / Responsebody is JSON
DB
Conditions dataAndrea Formica: Svom dev meeting CPPM
Building the prototype
• Generate what you can…
17
HTTP Jerseyservlet
/tags
/iovs
/globaltags
Tags controller
Iovs controller
Globaltags controller
Functions to handle tags
Functions to handle iovs
Functions to handle gtags
Request / Responsebody is JSON
DB
Conditions dataAndrea Formica: Svom dev meeting CPPM
Some URL examples• Retrieve resources
‣ GET method, retrieve tags filter by name: <host>/tags?by=name:ECL&size=1000&sort=objectType:DESC
‣ GET method, select iovs in a range for a given tag:<host>/iovs/selectIovs?tagname=[some tag]&since=0&until=INF
‣ GET method, retrieve one payload:<host>/payloads/[some hash]
• Insert a tag‣ POST method, body is a TAG data transfer object in JSON
<host>/tags. Req body: { ‘name’: ‘ECL_MY_TAG’, ‘objectType’: ‘some type’…}
• Full API documentation‣ To be stored in software.svom.fr (in particular in swagger-ui)
18
Conditions dataAndrea Formica: Svom dev meeting CPPM
Prototype implementation• Prototype Implementation‣ Based on standard Java technologies (JEE, Spring) and
specifications (JAX-RS, JPA)
‣ Can be deployed in the same Tomcat server as Frontier
‣ Can run as a micro-service (embedded Tomcat, Wildfly, Undertow, Jetty, …all these are servlet containers)
‣ Can build on any system with Java8 (no other dependencies needed, all downloaded automatically)
‣ Multi-DB platform support (thanks to JPA): tested on Oracle, SQLite, H2, Postgresql
19
Conditions dataAndrea Formica: Svom dev meeting CPPM
Continuous Integration
• Crest DB server repository in git‣ https://gitlab.cern.ch/formica/swagger_crestdb
- Exists also in drf-gitlab, but there I cannot yet run the pipeline because it requires Java for compiling
‣ use gitlab-ci to trigger a pipeline when code is committed - Perform unit tests - Create a docker container - Run the container in svomtest.svom.fr
- use an embedded undertow web server (RedHat Wildfly)
20
Conditions databaseA.Formica: Svom dev meeting CPPM
On going tests
• Use Crest server to test uploads and downloads‣ The server deployed on svomtest.svom.fr has been used
by Chinese colleagues to test network performances for upload and download - Tests are still on going - Chinese colleagues want to deploy as well the server in
China for completeness