The astronomical Virtual The astronomical Virtual

ObservatoryFrançoise Genova

OGF-Europe Community Outreach Seminar

Digital Repositories - Interoperablility Using Grid Technologies

OGF23 - 5 June 2008, Barcelona, Spain

• Sustainability is a central issue

• Talk centered on challenges & lessons learnt from building the astronomy data services and the Virtual Observatory to give seamless access to them

• Brief description of the astronomical Virtual Observatory

• Different points of view on requirements and lessons learnt

– Users

– Data Centres

– VObs project

– Funding bodies

The astronomical Virtual

Observatory (1)

• Heterogeneous, distributed ‘data’ services: archives of observations, value-added data bases, tools, bibliographic data (including e-journals), simulation data

• Most data freely accessible after proprietary period

• The Virtual Observatory concept

Seamless and transparent query of data centres

New analysis and visualisation tools

A standard structure for data centres to publish

their data and services

The Astronomical Virtual

Observatory (2)

• A thin interoperability layer

• A data and service grid, incrementally made available for community usage

• An international endeavour from the beginning

• International Virtual Observatory Alliance

– An alliance of very diverse national projects, each depending of its own national context

– Defines a common roadmap and discipline-specific interoperability standards

Europe (incl. ESA,

ESO + 6 partners)

China

India

Canada

SpainItaly

Armenia

FranceFranceGermany

Hungary

Japan

Korea

USA

Russia

UK (AstroGrid)Australia

The IVOA Working Groups

• Standardization process adapted from W3C

• Working Groups– Grid & Web Services (assesses usage of Grid technologies)

– Resource registry (OAI compliant but much more information for retrieving resources)

– Data modelling

– Query language

– Data Access layer

– Semantics

• Interest Groups: Applications; Theory; Data Curation & Preservation, OGF Astro-RG

Data available at selected point are highlighted in tree

Genova, The astrophysical Virtual

Observatory

Field of view outlines are plotted automatically

Image metadata – provenance and characterisation

AVO prototype based on CDS Aladin

Data Centre AlliancePopulates the VOwith data and servicesFP6 Coordination Action tohelp data providers andgather feedback

The VObs components

Euro-VO

Best effort alliance

8 partners

Technology CentreDistributed• FP6 VO-TECHInfrastuctureDesign Study• ESA-VO (et al.)

Facility CentreSupport to usersRegistry

FP7 I3 EuroVO-AIDA

Scientists (1)

• They want to do their science

• They want to be able to reuse data

• But they also want to get new instrument and direct

support to their science programssupport to their science programs

• Not so easy to put some money on data conservation

and distribution

• The Digital Repositories have to be very useful for

scientists to convince the community (and evaluation

committees)

Scientists (2)

• In astronomy, the usefulness of scientific repositories has been known from a long time

• IUE (1978-1996): five times more publications from data retrieved in the archive than from the selected observing teams

• HST: three times more publications and citations

• A significant and growing fraction of science is multi-wavelength astronomy, hence the VObs – Interoperability increases significantly the impact of the repositorieswavelength astronomy, hence the VObs – Interoperability increases significantly the impact of the repositories

• The data services and tools are part of everyday life of scientists (160 000 queries/day on CDS services in2007)

• But we have to continue to continuously convince the community

• And thus be SCIENCE DRIVEN ( using new technologies when useful) is an absolute requirement

Scientists (3)

• Feedback from scientists

– They do not want to know about the standards and the VObs implementation has to be transparent

– They need a proper description of data

– Most want science ready data (especially for data they are not specialists about)specialists about)

– All want good information about provenance (instrument, observation conditions, data processing)

• A huge work of data specialists to feed the repositories whith data which will be useful and confidently reused

• Discipline specific interoperability standards defined with science usage in mind

Data centres (1)

• Heterogeneous, distributed data

• Oviously, archives of observational data

(which are public after a one year proprietary

period)period)

• But also added-value databases, bibligraphic

data (NASA bibliographic database, electronic

journals), simulation data, …

Data centres (2)

• A huge diversity in size, objectives and context

from international agencies providing archives of their

large telescopes to small teams in scientific laboratories

willing to provide a service focussed on solving a specific

question (‘sharing of knowledge’)question (‘sharing of knowledge’)

• Census of European data centres (EuroVO-DCA)

– More than 70 answers

– Many international connections (data, services)

Data Centres (3)

• Have to listen to the community needs

• QUALITY is a must, and on the long term routine is the worst enemy

• Adaptable, agile strategy: astronomy, and the technical and political contexts change, sometimes technical and political contexts change, sometimes very quickly. Integrate science needs and IT progress.

• ‘Data’ is better curated by people who know about it (observatory archives for data, data centres for value-added services)

Virtual Observatory projects (1)

• Interoperability standards

– A bottom-up approach

– The definition of the ‘proper’ (‘usable’ and ‘useful’)

standards is a huge task, which works when the proper

people from the community are mobilized (archive people from the community are mobilized (archive

providers, data centre staff) – ‘enough’ complexity to start

with, not too much to remain useable by implementers

and understandable by astronomers: a delicate balance

– Take into account feedback from implementation by data

centres and usage by scientists

Virtual Observatory projects (2)

• International cooperation/agreement on interoperability standards is a must

• European coordination/cooperation has been and will remain critical – improves global and will remain critical – improves global results and increases global impact

• R&D/evolution of standards must continue (feedback, scientific and technical evolution, progressive inclusion of more complexity)

• VO-TECH Design Study

(Astrogrid UK, CNRS, ESO, INAF)

• ESA-VO

• EuroVO-DCA

• Architecture (interoperability)

• Tools

• Intelligent rresource discovery

• Data mining and visualisation

• Connection with Grid• Connection with Grid

• Interoperability standards

• Web 2.0, Semantics, data mining

Funding bodies (1)

• Optimize the return of large projects by facilitating the usage and reuse of data

• VObs is a strong tool for world-wide community integration

• OECD recommendation (2004): : “Agencies should encourage broadening of existing VO[bs] collaborations into a fully representative global activity”

• Astronet (ERA-NET) European strategy exercise recognizes the • Astronet (ERA-NET) European strategy exercise recognizes the importance of the VObs and will make recommendations to EC and to the national agencies

• Part of research infrastructure

• New projects MUST include provision for data pipelines, data archiving, data distribution & VObs compliance (and it seems that they understand it)

Funding bodies (2)

• Funding of data centres has to be consolidated in

many cases: European and national agencies,

support from local authorities

• VObs: support needed at different levels for the

operational phase

– Support to national projects

– IVOA is a best effort alliance, medium term commitment of

the national projects needed

– European coordination: how to get support on the

medium term?