seismology and epos integrated core services€¦ · seismology and epos integrated core services....

Seismology and EPOS Integrated Core Services

Daniele Bailo (INGV) and the EPOS IT team

ORFEUS meeting – September 22, 2015 - Bucarest

What is EPOS? EPOS is a long-term plan for the integration

of research infrastructures for solid Earth Science in Europe

EPOS integrates the existing (and future) advanced European facilities into a single, distributed,sustainable infrastructuretaking full advantage of new e-science opportunities

EPOS Architecture

Governance

Financial Work

Technical Implementation

Communication

Harm

onization

EPOS IP project Timeline

Implementation Validation Pre-operation

Communities and services

Communities in numbersMAP OF:- Seismic/GPS stations- Laboratories-- etc….

Diversity in data type and formats

http://www.epos-eu.org/ride/

Research InfrastructureLIst

• 244 Research Infrastructures

• 138 Institutions• 22 countries• 2272 GPS receivers• 4939 seismic stations• 464 TB Seismic data• 1.095 PB Storage

capacity (seismology)• 1.240 PB Storage

capacity (GNSS DC)• 828 instruments in 118

Laboratories

http://www.epos-eu.org/ride/

The challenge of integration

Storage / curation challenge

Yesterday Today

Data exchange challenge

Yesterday Today

Format / standard challenge

Yesterday Today

quakeML

SEED

ASCII filesBinary filesLocal metadataNo metadata

TelegraficFormat

IASPEI Seismic Format (ISF)SAC

SEED

stationML

Dost, B., Zednik, J.,et al. (2012): Seismic Data Formats, Archival and Exchange. - In: Bormann, P. (Ed.), New Manual of Seismological Observatory Practice 2 (NMSOP-2), Potsdam : Deutsches GeoForschungsZentrum GFZ, p. 1-19.

Future challenges: do you know …• Who (name and surname) and how is

using your data?• Can you measure the impact on society?• Which is the “value” of your data?• Can I use it for commercial purposes?• Can I make computation without

downloading waveforms?• Will the data / software survive you?

Future challenges: do you know …• How to increase the usage of your data?• How to increase the citation of your

data/DC/publications?• How to improve system performance?• How to bridge with other disciplines (e.g.

seismic data from volcanologist)?

…the answer is …….

INTEGRATION

Advantages of integration• USERS: access to an environment where

heterogeneous data and tools available:– Multidisciplinary science made simple [1]– More papers, more citations– Data and tool availability

• DATA PROVIDERS:– Great diffusion of data (EU funds attraction?)– Track usage of data and services– steering capacity– EU open data policy– do not reinvent the wheel

[1]: [Chiaraluce, L. (2012). Unravelling the complexity of Apenninic extensional fault systems : A review of the 2009 L ’ Aquila earthquake Journal of Structural Geology. ]

HOW TO achieve (technical) INTEGRATION for solid Earth

data/services?A) By means of discussions, conferences,

meetingsB) Promoting interoperability of systemsC) By setting up a new international

standardization bodyD) Promoting participation to European projects

Interoperability“Promoting interoperability of systems”

… What does it mean?

1.Connect several systems (bricks)2.By means of interfaces (APIs or

web services)3.No need to know system interal

specifics (black box concept)

Interoperability: example

FDSN station FDSN dataselect FDSN events GSAC services

Seismic Waveforms Seism Events GPS wavef.

THIN LAYER (or mediator, or EPOS integrated services or…)

Interoperability implies usage of standards:• For metadata• For data formats• For communication protocols

FuturePlatform

Scientist!

Latest Earthquakes

Shakemaps

Hystorical Earthquakes

Documents & other resources

Sicily North East Coast2013.08.15 - Download

Sicily sea2013.05.14 - Download

Civil protection: Sicily hazards2013.02.01 - Link

Hystory of Sicily Hearthquakes2013.02.01 - Link

Lipari Islands M:2.22013.08.23 - Info

Sicily North East Coast M:2.12013.08.21 - Info

Sicily North East Coast M:2.32013.08.18 - Info

Siracusa Aea - Imax: 11MCS1693.01.11 - Infos

Sicily North East Coast

Accelerometric DataMI05 07-04-09 17:47 5.5AQV 06-04-09 01:32 6.3AQG 06-04-09 01:32 6.3AQA 06-04-09 01:32 6.3AQK 06-04-09 01:32 6.3PCB 09-04-09 19:38 5.2AQV 07-04-09 21:34 4.5AQM 06-04-09 02:37 5.1AQU 06-04-09 01:32 6.3PCB 09-04-09 00:53 5.4GSA 07-04-09 17:47 5.5PCB 13-04-09 21:14 5.0

SAR Data

Remove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | ShowRemove | Show

Bulk Actions

Accelerometric WaveformsFilter

Hi-PassLow-PassBand-Pass

ENVISAT-1-09083009242556-1600.ASA_IM__0PENVISAT-1-09072609242628-1600.ASA_IM__0PENVISAT-1-09062109242691-1600.ASA_IM__0P

Accelerometric Data:

SAR Data:

• Integrate use of SAR, GPS, Accelerometric Data, etc.• Use different codes and languages (python, fortran, any other…)• Perform heavy processing online (use of HPC resources)• Compare results (e.g. focal mechanism catalogues) • Compare different data• Save data in personal area•…and more… (even download the data)

Scientist at work

Then… what can the scientist do?

Generic Features1. Download

2. Visualization

3. Processing

4. Modeling

GPS DATADOWNLOAD

SEISMIC DATA DOWNLOAD

EPOS demonstrator

Collaboration with VERCE http://epos.cineca.it

http://epos.cineca.it

IT Architecture & Design

Architecture

Thematic Core Services(TCS)

Integrated Core Services(ICS)

Interoperability layer

TCS generic architecture

TCS system Data/metadata catalogue

National network

Nationalrepository

API / web service

Local HPC

EPOS ICS

PrinciplesCo-development (ICS and TCS)

• The development of the ICS depends on the requirements and provided data, services, resources and users of the TCS. The TCS are at different stages of maturity. For some – with little infrastructure to date - the adoption of the EPOS architecture is straightforward. For others – with several years of infrastructural investments behind - a jointly agreed evolutionary plan to converge to interoperability with ICS will be adopted. The EPOS approach in this context is neither top-down nor bottom up: the main idea is to use the general architecture and follow a cooperative approach in the designing and development of the software to build the compatibility layer, which is the place where harmonisations and communications are achieved.

PrinciplesDo not reinvent the wheel

• a. Reuse local technologiesThe co-development philosophy maximises re-use of existing software services, data availability and resources. (e.g. VERCE, EUDAT)

• b. Do not build a supercomputer: build ICS-DEPOS will have its own computing resources mainly to provide the ‘uniform view’ over EPOS entities. For the massive processing EPOS will provide – subject to authorisation – access to appropriate computing facilities including HPC (High Performance Computing) machines for modelling and simulation, and HTC (High Throughput Computing) clusters for data intensive applications such as data mining. These facilities are known collectively as ICS-D (Integrated Cores Services – Distributed)

• c. EU Initiatives Harmonization: EGI, ENVRI, EUDAT, RDA…

Principles

Microservices approach• The Microservices architecture approach envisages small

atomic services dedicated to the execution of a specific class of tasks, which have high reliability. EPOS will either take existing software services and ‘wrap’ them for EPOS use or build new services complying with EPOS architectural standards. The aim is to have atomic services with defined interfaces which can be composed together to form a software stack able to address unpredictable user request.

Principles

Clear long-term technical goals, but iterative short-term approach

• The overall architecture of EPOS is clearly defined and agreed through EPOS-PP. However, its implementation in EPOS-IP would require a step-by-step approach to build a reliable system environment to meet the requirements of end-users and their communities

Principles – lessons learnt

• Working in phase (ICS and TCS)• Communicating across different domain (IT,

scientists, lawyers ..etc)• Collaborating and building communities

Thank You

www.epos-eu.org

WebSite

www.epos-eu.org/ride

R.I.D.E.

Newsletter

www.epos-eu.org/newsletter

Epos Social

[email protected]

IT Architecture & Design

Architecture

Thematic Core Services(TCS)

Integrated Core Services(ICS)

Interoperability layer

ICS - Integrated Core ServicesICS-C and ICS-D

ICS-C ICS-D• ICS-Central Hub• Orchestrates external

resources (ICS-D)• EPOS member states

competition for hosting it

• ICS-Distributed services• Provide IT resources not

built by EPOS ICS-C• Computational resources• Visualization resources• Others…

EPOS GUI

METADATAcatalogue

EPOS APIs

System manager

Dataset

Services

Software

TCS API

AAI system

Workflow Engine

ICS-D

ICS-C

TCS

NationalLayer

IT Seismic net FR Seismic

net

Dataset

Services

Software

TCS API

PT GPS net NO GPS

net

…Other TCSs

Elements of EPOS ecosystem

Complete ICT environment for research

Researchers, research managers, innovators, media

Processing Model

User Model

Data Model

Resource Model

User interaction with the system:AAAI, interaction workflow, multlinguality

Describes functions of processing environment etc.

Describes research dataand ancillary info (instituions)

Representing ICTresources

Metadata model- 3-layers- Manage the complexity

(see previous slide)- CERIF (formal syntax,

declared semantics)Functions-Discovery, selection-Impact assessment-Manage interoperabilityIssues (selection of)- heterogeneity- Many standards to be

mapped from TCS- Ontologies- Complex work

Discovery (DC) and (CKAN, eGMS)

Contextual (CERIF metadata model)

Detailed (community specific)

Web portal, Spatio-Temporal Search

domain specific – data(TCS or Naational Ris)

Contextual (CERIF metadata model)

(http://www.eurocris.org/)Common European Research Information Format

EPOS Interface

Human interface (GUI)Website or portal

Machine interface- API or Web service - which execute scripts

or queries- Returns data/metadata

in a given standard

EPOS APIs

CERIF XML APIs

Metadata avaialble in different formats:• RDF export (“old” ENVRI)• OAI-PMH, CKAN, opensearch… (EUDAT)• other standards

OAI-PMHRDF …

EPOS Interface: interoperability

Ontologies

Machine readable andMachine understandable

How does it work?1. Connects concepts2. Needs vocabulary

Issues• Many ontologies exist• Vocabulary Mapping • CERIF supports it

Michelini

CNT

Is Director of

INGV

Is section of Gresta

Is president of

Sailing

Has hobby

Trieste

Is Born

Italy

Located in

Boat

use

sea

use

AAI system (federeated & distributed)

Purpose - Authenticate users- Authorize users - Transparent access to

TCS and ICS-D data & services

Issues- Delegation- Many system,

sometimes non interoperable

- EPOS-EGI-CC- AARC

- X509- EPOS IdP- TCS need guidelines- Accept all auth.

mechanisms

Workflow engines

Purpose - Tracks data

transformation- Allows versioning- Allows reproducibility

Comments- Interoperability among

various workflow engines- VERCE contribution- E-FLOW by EGI- CERIF ??

How to register/cite data or publications?PID system

Purpose - DO / publication can be

uniquely referenced- Assign a PID at data

creation times

Issues- Need for a simple

mechanism to implement it

- EUDAT can help- Not at ICS level (only for

TCS?)

TCS envisaged architecture

TCS system Data/metadata catalogue

National network

Nationalrepository

API / web service

Local HPC

EPOS ICS

ICS-TCS common work

Metadata• Definition of community, standard-

based metadata

Match & Map:• Match/Mapping of community

metadata within the Metadata Catalogue

APIs• Definition of APIs for data discovery

and for data access/retrieval