infso-ri-508833 enabling grids for e-science project gridification: the unosat experience patricia...

INFSO-RI-508833

Enabling Grids for E-sciencE

www.eu-egee.org

Project Gridification: the UNOSAT experience

Patricia Méndez Lorenzo

CERN (IT-PSS/ED)

CERN, 1st March 2006

EGEE USER FORUM

CERN, 1st March 2006 Patricia Méndez Lorenzo


INFSO-RI-508833

Outlook

◘ UNOSAT Presentation

◘ Collaboration with CERN

◘ UNOSAT and GRID

◘ Production of UNOSAT inside the Grid environment

◘ Future Plans



INFSO-RI-508833

UNOSAT Presentation

UNOSAT is a United Nations Initiative

◘ Objectives➸ Provide the humanitarian community with access to

satellite imagery and Geographic Information System services

► Reduce disasters and plan sustainable development

➸ Ensure cost-effective and timely products

◘ Core Services➸ Humanitarian Mapping➸ Image Processing

VEGETATION – 1 Km

IKONOS – 1m



INFSO-RI-508833

UNOSAT Service Chain

Data suppliers

Ground station

USER

UNOSATCentral Unit

WWW

a) Detailed satellite maps of the area are required by UNOSAT Satellite responds the requirement with radar and optic techniques

b) Satellites send raw data back to Earth

c) Advanced data processing performed

d) Data relayed by UNOSAT

e) Additional processing anddownload in the web pageof UNOSAT



INFSO-RI-508833

Relief Projects of UNOSAT

◘ Case Study: Indian Ocean Tsunami Relief and Development◘ 29th Dec 2004: First Map distributed online to field users

➸ 14th Jan 2005: Imagery Bank online:► 100 Tsunami-related maps (pre and post)► 670 raw satellite images

➸ January: 200,000 tsunami maps downloaded in total

◘ After the Tsunami catastrophe

➸ Pakistan

➸ Filipinas

◘ In both cases maps are hosted at CERN

◘ Projects founded by European

partners as GMES (Global Monitoring Environment and Security)



INFSO-RI-508833

First step: UNOSAT and CERN

◘ UNOSAT is CERN partner since 2002

◘ CERN supports them with network facilities, with computer

infrastructure and with human (support) resources

◘ Asian Tsunami Example: ➸ Central Web Services at CERN under considerable strain

➸ Availability of the UNOSAT data at any time

◘ UNOSAT provides the users with a web interface able to find

the files of the images by clicking on the earth images

➸ Attractive method, easy to use

◘ Something similar to do with the GRID ➸ One of our goals inside the UNOSAT-GRID Collaboration



INFSO-RI-508833

One step further: GRID

◘ Potential Bottlenecks: ➸ UNOSAT beginning to suffer from limited capacity and processing power ➸ Multiple satellites being launched ➸ Larger and larger storage capacity needed

◘ In summer 2005 we have provided a whole structure at CERN for

UNOSAT➸ UNOSAT Virtual Organization (VO)➸ 3.5TB in CASTOR➸ Computing Elements, Resource Brokers ➸ Collaboration with ARDA group➸ AFS area of 5GB

◘ We have run some UNOSAT tests (images compression) inside the

GRID environment (quite successful)

◘ The framework developed for in principle for Geant4 (See Alberto

Ribon’s presentation [49]) has been adapted for UNOSAT needs

We have provided the whole GRID infrastructureAt CERN



INFSO-RI-508833

UNOSAT Production inside GRID

◘ UNOSAT provided us with a set of images for testing

◘ Associated to each image a metadata file was included

File name, directory path, geographical coordinates

◘ Steps:

STORAGE LEVEL

➸ Copy and registration of the images in Castor@CERN

► Use of the LFC Catalog ➸ Parse the metadata files to extract the different metadata ➸ Use of the AMGA tool to parse metadata to location of the files

COMPUTING LEVEL

➸ Use of compression tools to compress images inside LCG resources ➸ Use of the general submission tool adapted to UNOSAT needs



INFSO-RI-508833

A GRID Metadata Catalogue

◘ LFC Catalogue➸ Mapping of LFN to PFN

◘ UNOSAT requires➸ User will give as input data certain coordinates

➸ As output, he wants the PFN for downloading

◘ The ARDA Group assists us setting up the AMGA tool for UNOSAT

Oracle DB

ARDA APP

LFC

CASTOR

SRMMetadata

(x,y,z)LFN PFN



INFSO-RI-508833

Tools developed for new Gridifications

Generation of a general framework consisting of 2 major tools:

1. Tool to perform the automatic job submission 2. Tool to retrieve and handle the corresponding output

1. Automatic job submission◘ Given an user’s jdl this tool performs the following actions: ➸ It lists all sites able to run the jdl provided by the user ➸ It creates automatically a jdl file based on that provided

by the user ➸ It submits the just created jdl containing the user

application(s) ➸ Moreover it creates a subdirectory (defined by the user)

containing a list of the sites where the jobs have been submitted, the corresponding jdls and the jobs IDs



INFSO-RI-508833


◘ Additional Features: ➸ The user can define the queues where the jobs are submitted. These queues are checked to see whether it fixes the job requirements. ➸ Requested LFN files can be included. The corresponding

TURLs are searched and included in a file passed in the InputSandbox to the WN

◘ Applications ➸ This tool has been used for the 1st and the 2nd phases of the production: software installation and production

◘ Usage: ./submitter_general -vo geant4 -jdl jdlexample -jobfile G4_PROD -data /grid/geant4/production_software



INFSO-RI-508833


2. Retrieve and handle of the outputs

➸ The 2nd tool checks the status of the jobs from the job IDs included in the directory given by the user

◘ Usage: ./get_output -jobfile G4_PROD -dest G4_PROD/outputs

◘ OutputThe job run in ramses.dcic.ups.es:2119/jobmanager-torque-

dteam is in status: ScheduledThe job run in grid01.phy.ncu.edu.tw:2119/jobmanager-torque-

dteam is in status: runningThe job run in scaic10.scai.frauhofer.de:2119/jobmanager-

torque-dteam is in status: over



INFSO-RI-508833


◘ Additional Features:

➸ It is possible to

visualize the

outputs on the web

➸ A html report is

provided

showing the files

decided by the user



INFSO-RI-508833

Future Plans with UNOSAT

◘ Collaboration between UNOSAT, ARDA and GD➸ 1(2) ARDA and 2 UNOSAT Students

➸ Still many discussions needed

➸ Support from other sites foreseen

(x,y,z)GRID WORLD

Application

◘ User can get the info in his laptop too

◘ Fundamental AMGA

infso-ri-508833 enabling grids for e-science project gridification: the unosat experience patricia...

Documents