Steve Fisher, Shantenu Jha*, Andre Merzky, O Weidner, A Wilson

http://saga.cct.lsu.edu

EGEE-III INFSO-RI-222667

Providing uniform and standardised access to European grid middleware with SAGA

SAGA: In a nutshell

There exists a lack of programmatic approaches that:• Provide general-purpose, basic &common grid functionality for

applications and thus hide underlying complexity, varying semantics..

• The building blocks upon which to construct “consistent” higher-levels of functionality and abstractions

• Meets the need for a Broad Spectrum of Application: • Simple scripts, Gateways, Smart Applications and

Production Grade Tooling, Workflow…

Simple, integrated, stable, uniform and high-level interface• Simple and Stable: 80:20 restricted scope and Standard• Integrated: Similar semantics & style across• Uniform: Same interface for different distributed systems

UMD: Components of gLite, ARC, UNICORE & globus

SAGA: In a thousand words..

gLite CREAM Overview

CREAM = Computing Resource Execution And Management

Lightweight service for job management operation at the Computing Element (CE) level

CREAM can be used via:• gLite Workload Management System• the ICE service• a generic client (Java/C++ + command line

interfaces available)• The Simple API for Grid Applications

(SAGA).

gLite CREAM SAGA Adaptor

SAGA provides a prototype job adaptor to access gLite CREAM CEs

Implemented with the CREAM C++ API Currently provides the following features:

• job description (saga-jd to JDL mapping)• job control (run, submit, suspend, cancel, etc...)• listing of jobs associated with a CE• re-connection to running jobs• Encapsulates (hides) proxy delegation

SAGA/CREAM Python Example

SAGA/CREAM C++ Example

SAGA/CREAM Getting Started

The adaptor code is available via SVN:https://svn.cct.lsu.edu/repos/saga-adaptors/glite/

Due to the limitations of the CREAM/gLite build-system the adaptor only supports platforms that are supported by gLite (Scientific Linux, etc…)

But will be release with glite in near future Please send comments and/or questions to:

oweidner@cct.lsu.edu

Movie time…

SAGA BES Adaptor

SAGA BES adaptor based on bes++• C reference implementation for HPC-BP

by Chris Smith (Platform) et.al.• has been used for BES-HPCBP interop• uses gSoap• provides ’besclient’ and ’besserver’

• code is somewhat messy, as usual with gsoap based code

SAGA BES Adaptor (cont.)

C++ wrapper for bes++ client lib was added simplifies adaptor code to (job::run)void job_cpi_impl::sync_run (saga::impl::void_t & ret) { if ( state_ != saga::job::New ) { SAGA_ADAPTOR_THROW ("can run only 'New' jobs", saga::IncorrectState); }

job_epr_ = bes_.run (jsdl_); jobid_ = "[" + endpoint_url_.get_string () + "]-[" + job_epr_->str + "]";

{ adaptor_data_type adata (this); // scoped lock state_ = adata->get_saga_state (bes_.get_state (job_epr_)); }}

• svn co https://svn.cct.lsu.edu/repos/saga-adaptors/ogf/trunk• license: GPL (due to gSoap)

SAGA BES Adaptor (cont.)

SAGA example submit to BES#include <saga/saga.hpp>

int main () { saga::session s; saga::context c; c.set_attribute (saga::attributes::context_username, “ogf”); c.set_attribute (saga::attributes::context_userpass, “ogf”); s.add_context (c); saga::job::service js (s, ep.url); saga::job::job j = js.run_job_job (“/bin/sleep 10”); while ( j.get_state () == saga::job::Running ) { cout << "Running" << endl; ::sleep (1); } if (j.get_state () == saga::job::Done ) cout << "Done" << endl; else cout << "Failed" << endl; return j.get_state ();}

SAGA BES Adaptor (cont.)

funtional:• endpoint verification• certificate and username/pass auth• SAGA job description -> JSDL• JSDL -> SAGA job description• job submission• BES job state -> SAGA state• terminate job

not yet implemented• list backend jobs• reconnect to job based on job-id

SAGA BES Adaptor (cont.)

Interop Tests: bes++: besserver with SAGA backend ;-)

fully functional Unicore: https://zam1161v01.zam.kfa-juelich.de:8002/

fully functional GridSam: https://gridsam-endpoint.oerc.ox.ac.uk:18443/

demo certificate expired ARC: http://knowarc1.grid.niif.hu:50000/arex

demo CA expired

Planned Tests:• NAREGI, CREAM, Genesis-II, PBS-Pro, MS HPC Server• Please provide us with interop endpoints! :-)

GANGA OVERVIEW

SAGA-GANGA Integration

DIANE INTEGRATION

Diane without SAGA Diane with SAGA

DIANE is an execution manager with support for pilot-jobs + worker agents(IDEAS Redux)

Master

Agents scheduling

Heterogeneous resourcesallocation (Ganga + Ganga/SAGA)

Lattice-QCD Applications on heterogeneous resources

Ganga/gLite

Ganga/SAGA (to TeraGrid)

Ganga/SAGA (to *)

Payload distribution

Application-aware (and resource-aware) scheduling

Federating resources! EGEE Conference (Apr’10) (Not in this demo:

cloud resources, additional Grid infrastructures…)

RENKEI Project Aims

SAGA-Engine

gLiteNAREGISRBiRODS

Adpt Adpt Adpt

C++ InterfacePython Binding

Service & Applications Svc Apps Apps

Cloud LRMSLSF/PBS/SGE/…

Middleware-independent service & application

RNSYet Another FC

service based on OGF standard

SAGA adaptors

SAGA framework

This activity is funded by MEXT as a part of RENKEI project which develops seamless linkage of resources in the Grids and the local one for e-Science.

KEK

Osaka Univ.Tsukuba

Univ.

HEPLibrary

SAGA

BigJob: Infrastructure Independent Pilot-Job

BigJob: Infrastructure Independent Pilot-Job (Each sub-job is a MPI-based MD)

Service Discovery

This API (GFD.144) identifies URLs of services Hides the GLUE-I ↔ GLUE-II transition from the user saga-sd --sf "Site IN ('INFN-CNAF', 'RAL-LCG2')" --

sa• Return details of all services you are authorized to use at

INFN-CNAF and RAL-LCG2

gLite adapter being added to UI and WN patches this week

NAREGI just starting work on an adapter• Hope to see it at next OGF in Chicago

Information System Navigator

ISN exposes the schema to allow navigation over any ER style schema

gLite adapter supports GLUE-I now. GLUE-II will be tested when data are available.

Available as experimental API in LSU SAGA implementation with CLI and gLite adapter

GUI portal to SD and ISN being tested. Will be made generally available soon.

SAGA: Available Adaptors Job Adaptors

• Fork (localhost), SSH, Condor, Globus GRAM2, OMII GridSAM, Amazon EC2, Platform LSF

File Adaptors• Local FS, Globus GridFTP, Hadoop Distributed Filesystem (HDFS),

CloudStore KFS, OpenCloud Sector-Sphere

Replica Adaptors• PostgreSQL/SQLite3, Globus RLS

Advert Adaptors• PostgreSQL/SQLite3, Hadoop H-Base, Hypertable

Other Adaptors• Default RPC / Stream / SD

SAGA-based Tools and ProjectsAdvantage of Standards

JSAGA from IN2P3 (Lyon)• http://grid.in2p3.fr/jsaga/index.html• gLite adaptors exist

JAVASAGA (Amsterdam)• Has a wide range of adaptors• JAVASAGA gets released by gLite (next few weeks)

NAREGI/KEK (Active)• http://www.ogf.org/OGF27/materials/1767/OGF27_SAGA_KEK.pdf

DEISA/DESHL• http://www.fz-juelich.de/nic-series/volume38/pringle.pdf )• http://deisa-jra7.forge.nesc.ac.uk/ and

http://www.ogf.org/OGF19/materials/501/SAGA-DEISA.ppt

XtreemOS• http://saga.cct.lsu.edu/index.php?option=com_content&task=view&id=95

&Itemid=174

Summary..

Power of well-defined, agreed upon interfaces• Tools, Gateways and Applications

Advantages across levels (close to the middleware, upto direct application level), and scales (single EGEE/EGI site, to largest TG resources)

SAGA provides application-level interoperability across different middleware and distributed resource types

Interoperability first step towards effective utilisation of DCI• IDEAS• http://www.cct.lsu.edu/~sjha/select_publications

SAGA Team and DPA Team and the UK-EPSRC (UK EPSRC: DPA, OMII-UK , OMII-UK PAL), NSF (HPCOPS, Cybertools) and LA-BOR

People:

SAGA D&D: Hartmut Kaiser, Ole Weidner, Andre Merzky, Andre Luckow, Joohyun Kim, Lukasz Lacinski, João Abecasis, Chris Miceli, Bety Rodriguez-Milla

SAGA Users: Andre Luckow, Yaakoub el-Khamra, Kate Stamou, Cybertools (Abhinav Thota, Jeff, N. Kim), Owain Kenway

Google SoC: Michael Miceli, Saurabh Sehgal, Miklos Erdelyi

Collaborators and Contributors: Steve Fisher & Group, Sylvain Renaud (JSAGA), Go Iwai & Yoshiyuki Watase (KEK)

DPA: Dan Katz, Murray Cole, Manish Parashar, Omer Rana, Jon Weissman

Acknowledgements