talk structuretalk structure

• who are we ?

• what is a VO ?

• what are the challenges ?

• what is an e-project ?

Andy Lawrence Garching June 2002

Andy Lawrence Garching June 2002

AstroGrid http://www.astrogrid.org.uk

AstroGrid http://www.astrogrid.org.uk

Andy Lawrence Garching June 2002

Andy Lawrence Garching June 2002

BelfastCambridgeEdinburghJodrellLeicesterMSSLRAL

AstroGrid http://www.astrogrid.org.uk

AstroGrid http://www.astrogrid.org.uk

Andy Lawrence Garching June 2002

Andy Lawrence Garching June 2002

OpticalInfraredX-rayRadioSolarSpace Plasma

peoplepeople

• PL = Andy Lawrence

• PS = Nic Walton

• PM = Tony Linde

slogansslogans

• the archive is the sky (stolen from US-NVO)

• everybody can be a power user

• shift the results not the data

• a supercomputer on your desk

goalsgoals

• A working datagrid for key UK databases

– machines, middleware and user interface

• tools for simultaneous browsing– plus advanced visualisation, links to spectra etc.

• tools for on-line data analysis– complex queries, statistics, model fitting, cluster analysis etc

• system for uploading code

• resource discovery method

the projectthe project

• PPARC funded project Sept 2001-2004– approx 6Meuro

• Grid technology development programme– part of UK e-Science programme

– links to GridPP, MyGrid (Bio), OGSA (Globus)

• Stepping stone to Virtual Observatory– partner in AVO

– working links with US-VO

methodologymethodology

• using Unified Process / UML

• architecture driven

• use case centric

• iterative

• open project approach– open collaboration interactive web sites

– code sharing intended

– open source coding under consideration

statusstatus

• elaboration phase two-thirds finished– science problems / use cases / architecture

– technology assessment reports

– collaborative web pages set up

• construction phase begins Sept 2002

• continuing R&D stream– contribution to AVO

– collaboration with OGSA project

AstroGrid & AVOAstroGrid & AVO

• AstroGrid partner in AVO – subset of work credited as AVO work

• CDS = V0.2

• demos early 2003 = V0.5

• AstroGrid late 2004 = V1– fully working but functions limited as necessary

• AVO Phase B 2004+ = V2

collectivisation and democratisation collectivisation and democratisation

• thirty year trend towards communal organisation– facility class (common-user) instruments

– central development of data reduction s/w

– calibrated archives with simple tools

– information services (Vizier, ADS, NED)

– large consortium projects (MACHO, 2dF, SDSS, UKIDSS, VISTA...)

• next steps– inter-operable archives (joint queries)

– communal exploration and analysis tools (data mining)

– automated resource discovery (registry)

the Virtual Observatory conceptthe Virtual Observatory concept

• Aim to make all archives speak the same language

– all searchable and analysable by the same tools

– all data sources accessible through a uniform interface

– all data held in distributed databases that appear as one

– archives form the Digital Sky

– eventual interface to real observatories

the archive is the sky

• shared managed distributed resources– documents + data + software + storage + cycles + expertise

• network : ability to pass messages

• web : transparent document system

• computational grid : transparent CPU

• datagrid : transparent data access and services

• information grid, knowledge grid ... ?

• Virtual Organisations ?

the Grid conceptthe Grid concept

a supercomputer on your desktop

everybody canbe a power user

obstacles to overcomeobstacles to overcome

• sociology

• internet technology

• i/o bottleneck

• network bottleneck

obstacles to overcome (1)obstacles to overcome (1)

• sociology– need agreed formats for data, metadata, provenance

– need standardised semantics ("ontology")

• internet technology– need protocols for publishing and exchanging data

– need registry for publishing service availability and semantics

– need method of transmitting authentication/authorisation

– need methods for managing distributed resources

obstacles to overcome (2)obstacles to overcome (2)

• i/o bottleneck– need database supercomputers

– need innovative search and analysis algorithms

• network bottleneck– data centres must provide analysis service

– facility class analysis code needed

shift the results not the data

grid geometry neededgrid geometry needed

• not P2P (like Napster) • not a hierarchy (like LHC grid)

• service providers + users (like most commerce)– some unplanned open use

– some registered use

– variety of access rights

two riverstwo rivers

• academic / Globus – remote log on

– identity/authentication/authorisation

– resource management

• commercial / W3C– exchange of data (B2B)

– service description and publication

– "Web services" = XML + SOAP + WSDL (cf GLU)

Globus problemsGlobus problems

• only half works

• data transfer primitive– flat files only

• not a services-user structure

Web service problemsWeb service problems

• one-to-one

• bulky

• no general auth/auth solution

• no accepted service registry solution

• no ontology solution

solutions ?solutions ?

• Grid Services : OGSA project– Globus + IBM + more

• Database access : OGSA-DAI project– UK e-science programme

– MyGrid and AstroGrid = early adopters

• need Astro-registry solution• need Astro-ontology service (cf UCDs)• need Astro-QL

e-projecte-project

• open, interactive, collaborative web sites– reading open to all– registered users can post/edit items

• AstroGrid News

• AstroGrid Forum

• AstroGrid Wiki– shared documents– shared code– collaborative development

FIN

Web DB access todayWeb DB access today

browser front end

user

CGIrequest

html

web page DB

engine

SQL

data

Web serviceWeb service

application web service

user

XMLrequest

XMLdata

DBengine

SQL

nativedata

anything