Earth Science GRID on Demand & e-collaboration

The ESRIN GRID team - luigi.fusco@esa.int

“Grid and e-collaboration for the Space Community”

ESA-ESRIN, 2 February 2005

Summary

Earth Science and GRIDEarth science and e-collaborationPresent infrastructure at ESRINMedium term visionExample of applications

GRID for Earth Science means: – Single stop shop for data access– Number-crunching, e.g EO data reprocessing – e-collaboration, e.g. access and sharing of

data sources, tools, means, models, algorithms …

Earth Science and Grid

May 2004 June 2004July 2004 August 2004 September 2004 October 2004 MERIS global composite

> 1 Terabyte data processed – 1 Km res.

e-collaboration technologies

Earth Science community analysis The Internet and improved

communication Web-Services GridSemantic Technologies and Ontologies Sensor Web Wireless TechnologiesWorkflow Management TechnologiesVirtual Reality or Augmented RealityStandards (OGC, W3C, …)

Part

of

op

era

tional

ESA

EO

envir

onm

ent

SO

AP

SO

AP

MUIS ESA Catalogue

AMS ESA Data

Archive

ESA

EO

op

era

tional envir

onm

ent

htt

p:/

/ww

w.e

oport

al.org

WCS WFS / WMS

Catalogue (CSS)

Reference infrastructure

GLOBUS+?

Computing

Elements

Storage Elemen

ts

Local Grid

DATAGRID

Computing

Elements

Storage Elemen

tsEuropean Grid

Client Applications

(JAVA / Linux / Windows)

Web Portal

Client Applications

OG

C W

eb

Serv

ices

GRID Engine

LCG + ?

Computing

Elements

Storage

Elements

ESA EO GRID

MEGALAB

LCG + ?

Computing

Elements

Storage Elemen

ts

EGEE

Computing

Elements

Storage Elemen

tsEuropean GridESA EO planned

operational environment

geant

GRID & EO Web Services @ ESA

ESA actions and 2y vision

Mid term vision – 2 years–Target: mid –2007–EO missions Ground Segment operations•Satellites data routine acquisition, processing, archiving, dissemination

–Earth Science User environment•Science and commercial applications

EO Grid infrastructure G/S vision

Transfer to operation of present GRID capabilities:– Support large scale reprocessing requirements

• Demonstrated already for L1 L2 GOMOS• Planned for GOMOS / MERIS in 2005• New EO processors based on Linux

– Enable new classes of large-scale applications (e.g. processing of global MERIS Level 3)

– Innovative solution for the data access and exchange

– Increased exploitation of available resources GRID interconnection of ESA and other

operational facilities– Reference infrastructure for new missions G/S

studies– Take advantage of EC e-infrastructure initiative

Earth Science User Segment Vision

Set up of the EO Science Grid User Segment – Include support to science data processing /

analysis based on Envisat tools (e.g. BEAM) – Platform for e-collaboration in Earth Science

(e.g. multi data source CAL/VAL…) Goal 1: Move algorithms and analysis tools to

data– data assimilation … integration in models…– Support applications within the ENVISAT AO, e.g.

• MERIS data assimilation for chlorophyll forecast… Goal 2: extend approach to wide science user

community– Wish to define a dedicated AO for next year– Contribute to the definition of GMES User

segment

Status of applications in Grid environment

Ready for operations– On Demand: Ozone GOME validation, Global and regional

MERIS L3 products (MGVI, Chlorophyll …)– Validation of MGVI with Science users

Under development in house– Preparation of ENVISAT (GOMOS, MERIS ) reprocessing– ASAR Global Monitoring Antarctica (sea ice, icebergs, …)– AATSR fire detection and SST; RA2 Lake and river altimetry – Direct access to NASA archives (CEOS)

Under development with external partners– e-collaboration for ozone (GOMOS, SCIAMACHY, MIPAS, …)– Objective analysis SST and chlorophyll in the Med (AATSR,

MODIS, MERIS…)– MERIS applications: case 2 waters, data assimilation for

chlorophyll forecast, …– CNR – integration of MSG for fire detection; – Uni Tor Vergata – reprocessing of ERS-2 SAR for oil spill

statistics

Portal Grid Services

Job selection, chaining, launching and live status

Direct result visualization in WMS (using open source quickWMS client)

Join personal data folder as a WMS

Chlorophyll BEAM processing

May 2004 June 2004 July 2004

Input Data:

198.81Gb in 862 files

Output Files:

700 Kb (JPEG)

28 Mb (TIFF)

36 Mb (DIM-MAP)

Input Data:

158.31Gb in 1643 files

Output Files:

700 Kb (JPEG)

28 Mb (TIFF)

36 Mb (DIM-MAP)

Input Data:

178.92 Gb in 1828 files

Output Files:

700 Kb (JPEG)

28 Mb (TIFF)

36 Mb (DIM-MAP)

Task chaining in the “Science Grid on Demand” environment. Based on L2 products

COLLECT FUNCTION

Searches and retrieves the

requested files from the storage

elements

BINNING FUNCTIONProjects the Envisat

files to a given datum and projection and according to the

requested geographical bounding box

AGGREGATION FUNCTION

Assembles the resulting files

from the binning processes

Earth Observation Visualization FunctionGeneric functions for image composition

and format conversion (JPEG/TIFF/…)

DATA REGISTRY FUNCTION

Registries the data files and the

corresponding metadata in the defined storage

element

RETRIVE FUNCTIONGeneric function for file transfer

(FTP, SFTP, gridFTP,…)

Chlorophyll processing on GRIDBEAM code

is embedded

here

Alternate MERIS MGVI algorithms

Arithmetic Mean

Algorithm

fapar (JRC)

Algorithm

Testing different temporal

aggregation methods for MERIS MGVI

MERIS ASAR Global mode mapping

May 04June 04

July 04

Extension of Sea Ice Pack

Displacement of icebergs

Participation to external projects

GRID – EC funded:– Participation to EGEE – NA4– DILIGENT– Digital Libraries in GRID environ’t

• started in Sept 04• ESA in charge of Environmental User scenario• Very interesting technology for Long Term Data

and Knowledge Preservation

e-collaboration – EC funded:– BRAINBRIDGE – to KO in Mar 05

• define road map for e-collaboration initiative in FP7

Infrastructure – Regional funding:– MEGALAB

• GRID infrastructure across Frascati-Tor Vergata-Rome area

Grazie!