international virtual observatoryopenuniverse.asi.it/documents/presentations/ivoa... · metadata,...

International Virtual Observatory

AllianceThe Standards Organization

for Data Interoperability in Astronomy

G. FabbianoHarvard-Smithsonian Astrophysical Observatory

Chair of the IVOA Executive Committee

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 1

What is the Virtual Observatory?

A multi-wavelength digital sky that can be searched,

visualized, and analyzed in new and innovative ways

Space, Ground, and Theory data

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 2

What is the Virtual Observatory?The latest stage of good data practices in

astronomy

• Each data center / archive with different file structure,

metadata, table organization

• The data is there, but it takes work to access it, especially if

several data sets are used together

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 3

What is the Virtual Observatory?The latest stage of good data practices in

astronomy

• Each data center / archive with different file structure,

metadata, table organization

• The data is there, but it takes work to access it, especially if

several data sets are used together

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

• FITS provided a first standardization

4

What is the Virtual Observatory?The latest stage of good data practices in

astronomy

• Each data center / archive with different file structure,

metadata, table organization

• The data is there, but it takes work to access it, especially if

several data sets are used together

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

• FITS provided a first standardization

• The Virtual Observatory is the natural

progression towards interoperability of data,

services and tools

5

What is the Virtual Observatory?

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

Data management & Access

InteroperabilityBig Data

6

What is the Virtual Observatory?

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

Data management & Access

InteroperabilityBig Data

7

What is the Virtual Observatory?

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

Data management & Access

InteroperabilityBig Data

8

What is the Virtual Observatory?

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

Data management & Access

InteroperabilityBig Data

9

What is the Virtual Observatory?

Paradigm for supporting interdisciplinary and collaborative

research, education, outreach and citizen science in

astronomy, exploiting the full power of growing and

emerging data sets

• The VO is not a fixed system, but rather a way of doing

things, a framework

– For data centers to provide co-operating data services,

– For software providers to offer a variety of compatible analysis

and visualization tools and user interfaces

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 10

IVOA StandardsAt the core of the VO is a set of standards developedwithin the International Virtual Observatory Alliance(IVOA) to support good data management and

interoperability

• Data and metadata for observed and simulated data– Data Model; Uniform Content Descriptors

• Data exchange methods – Data Access Layer; VO Query Language

• Lists and characteristics of available services – IVOA registry

• Application messaging protocol– SAMP

– VO-enabled tools and services can interface seamlessly with VO-enabled archives worldwide

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 11

IVOA StandardsAt the core of the VO is a set of standards developedwithin the International Virtual Observatory Alliance(IVOA) to support good data management and

interoperability

• Data and metadata for observed and simulated data– Data Model; Uniform Content Descriptors

• Data exchange methods – Data Access Layer; VO Query Language

• Lists and characteristics of available services – IVOA registry

• Application messaging protocol– SAMP

– VO-enabled tools and services can interface seamlessly with VO-enabled archives worldwide

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 12

New data systems with VO at their core are

being developed world-wide

e.g., CADC, Gaia, Euclid,

Australian Sky Mapper Database….

IVOA Todayhttp://www.ivoa.net

• Created in 2002, the International Virtual Observatory Alliance (IVOA) today has 23 member projects– 2 well attended interoperability working meetings per year

• The IVOA has no direct funding– Affiliated projects seek funding from their national agencies

– Members have come and gone, some projects have persisted and some others have stopped

• Hard lessons learnt on management and developing standards– Management framework and roadmaps

– User participation is important

• IVOA has survived because VO is a good idea

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 13

IVOA Todayhttp://www.ivoa.net

• Created in 2002, the International Virtual Observatory Alliance (IVOA) today has 23 member projects– 2 well attended interoperability working meetings per year

• The IVOA has no direct funding– Affiliated projects seek funding from their national agencies

– Members have come and gone, some projects have persisted and some others have stopped

• Hard lessons learnt on management and developing standards– Management framework and roadmaps

– User participation is important

• IVOA has survived because VO is a good idea

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 14

IVOA Todayhttp://www.ivoa.net

• Created in 2002, the International Virtual Observatory Alliance (IVOA) today has 23 member projects– 2 well attended interoperability working meetings per year

• The IVOA has no direct funding– Affiliated projects seek funding from their national agencies

– Members have come and gone, some projects have persisted and some others have stopped

• Hard lessons learnt on managing the development of standards– Management framework and roadmaps

– User participation is important

• IVOA has survived because VO is a good idea

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 15

IVOA Todayhttp://www.ivoa.net

• Created in 2002, the International Virtual Observatory Alliance (IVOA) today has 23 member projects– 2 well attended interoperability working meetings per year

• The IVOA has no direct funding– Affiliated projects seek funding from their national agencies

– Members have come and gone, some projects have persisted and some others have stopped

• Hard lessons learnt on management and developing standards– Management framework and roadmaps

– User participation is important

• IVOA has survived because VO is a good idea

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 16

IVOA Organization Chart

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

ExecutiveCommittee

Technical Coordination Group

DataModel

Data AccessLayer

Applications

SemanticsRegistryGrid & Web Services

Knowledge Discovery in Databases

OperationsTime Domain

TheoryEducationData Curation& Preservation

Committee on Science Priorities

Standards & Processes Committee

Working Groups Interest Groups

17

IVOA Successes

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

Stable Architecture Stable Processes

Documents accessible via www.ivoa.net

and NASA ADS (by authors)

18

IVOA Successes –Archives in the

VO• Major astronomical data collections accessible through the VO

– CDS, CADC, MAST, IRSA, ESA, Chandra, …

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 19

IVOA Successes – VO in Data Centers

• VO-ready infrastructure built-in astronomical data centers

– CADC, Gaia, Euclid, …

– Data (file & database) access and User Work Space (VOSpace)

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 20

IVOA Successes – Reliable Operations

• Reliability of VO Services Operations

– Hundreds of VO services in the IVOA

Registries…

• Compliance of VO services ?

• VO Services validators

– IVOA Operations Interest Group monitors

operationsChair: Tom McGlynn, HEASARC (NASA-GSFC)

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 21

IVOA Successes – Seamless Science

A new paradigm for research

Integrated archive – literature - analysis sessions, through: • IVOA-compliant user interfaces

• Query protocols

• SAMP – VO interoperable applications

• ADS-archive two–way links (e.g. Chandra)

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 22

VO interoperable applications

• Topcat, Aladin, VOSpec, SPLAT-

VO, Iris, DS9 …

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 23

Chandra Source Catalog

Thread

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

VO

Python scripting

VO Data

TopCat

CSCView

DS9 24

Chandra Source Catalog

Thread

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 25

IVOA Successes – Education, Outreach &

3rd-party applications

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

• EuroVo for Education

– Suite of IVOA – compliant tools

and use cases– http://vo-for-

education.oats.inaf.it/eng_download.html

• WorldWide Telescope

(Microsoft)

– VO inspired and VO compliant

– Now under AAS

26

New Project Engagement in IVOA

• Standards development is not a top-down activity

• Project input and participation are vital

• New science projects are welcome to participate in IVOA standards definition and development

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 27

New Project Engagement in IVOA

• The IVOA Committee on Science Priorities seeks input for science drivers – identifies fast-track standards

• Organizes focus session with projects – e.g. Cape Town IVOA meeting, May 2016

“Interoperability of data from major astronomical projects”

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

Presenter Project: Multi-l, Space and Ground

R. Taylor LSST

M. Allen et al. ASTERICS: E-ELT, SKA, CTA, LOFAR, …

M. Zhu FAST

J. Chapman ASKAP

B. Merin Euclid, Gaia

T. Donaldson JWST

28

The VO is not a Killer Application

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu

Dr. Strange

29

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 30

The VO is an Evolving Ecosystem

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 31

The VO is an Evolving Ecosystem

IVOA Organization

• Review/Approve standards to ensure processes are followed

• Prioritize work with input from CSP and TCG (see below)

• IVOA Chair, IVOA vice-Chair (rotating, 18 months term)

• Representatives from all national VO projects

Executive Committee

• Identify research needs that can benefit from VO related tools and services

• Take action within the context of the IVOA to assist in placing such tools and services into the research community

Committee on Science Priorities

• Develop IVOA interoperability standards to support science cases

• TCG Chair / Vice Chair coordinate activities and maintain roadmap

• 6 Working Groups defining interoperability standards (Applications, Data Access Layer, Data Model, Grid and Web Services, Registry and Semantics)

• Each WG has Chair and Vice Chair

• 6 Interest Groups (Data Curation and Preservation, Education, Theory, Time Domain, Operations, Knowledge Discovery in Databases)

• Standard and Processes Committee

Technical Coordination Group

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 32

IVOA Working Groupswhere the standards are developed

• Applications– Interoperability between Astronomy data, services and software empowers

astronomers to combine these for scientific discovery and analysis

• Data Access Layer– Define and formulate VO standards for remote data access

• Client data analysis software use these services to access data via the VO framework

• Data providers implement these services to publish data to the VO

• Data Model– Framework for the description of metadata attached to observed or simulated data

– Logical relationships between metadata and architecture to handle them • How does an astronomer wants to retrieve, process and interpret astronomical data?

• Grid & Web Services– Define the use of Grid technologies and web services within the VO context

– Investigate, specify, and implement required standards

• Registry– Provides the mechanism for users and applications to discover and select resources

• data and services

• Semantics– IVOA Standard Vocabulary

– Exploration of Ontologies

Rome, 4/11/17 Pepi Fabbiano - gfabbiano@cfa.harvard.edu 33