diet a cloud middleware for seed4c
DESCRIPTION
DIET a Cloud Middleware for Seed4C. Eddy Caron Join work with Jonathan Rouzaud-Cornabas, Frédéric Desprez, Rajesh Palanichamy and the DIET Team Ecole Normale Supérieure de Lyon AVALON Research Team SysFera. DIET’s Goals. http://graal.ens-lyon.fr/DIET/. Our goals - PowerPoint PPT PresentationTRANSCRIPT
Eddy CaronJoin work with Jonathan Rouzaud-Cornabas, Frédéric Desprez,
Rajesh Palanichamy and the DIET Team
Ecole Normale Supérieure de LyonAVALON Research Team
SysFera
DIET’s GoalsDIET’s Goals
Our goals To develop a toolbox for the deployment of environments using the
Application Service Provider (ASP) paradigm with different applications Use as much as possible public domain and standard software To obtain a high performance and scalable environment Implement and validate our more theoretical results
Scheduling for heterogeneous platforms, data (re)distribution and replication, performance evaluation, algorithmic for heterogeneous and distributed platforms, …
Based on CORBA and our own software developments FAST for performance evaluation, LogService for monitoring, VizDIET for the visualization, GoDIET for the deployment
Several applications in different fields (simulation, bioinformatics, …) Release 2.8 available on the web since november ACI Grid ASP, RNTL GASP, ANR LEGO CIGC-05-11, ANR Gwendia,
Celtic-plus Project SEED4C
http://graal.ens-lyon.fr/DIET/
6/03/12 DIET a Cloud Middleware for Seed4C
RPC and Grid-Computing: Grid-RPCRPC and Grid-Computing: Grid-RPC
• One simple idea – Implementing the RPC programming model over the grid – Using resources accessible through the network– Mixed parallelism model (data-parallel model at the server
level and task parallelism between the servers)
• Features needed– Load-balancing (resource localization and performance
evaluation, scheduling), – IDL, – Data and replica management, – Security, – Fault-tolerance, – Interoperability with other systems,– …
Design of a standard interface – within the OGF (Grid-RPC and SAGA WG)– Existing implementations: NetSolve/GridSolve, Ninf, DIET,
OmniRPC
6/03/12 DIET a Cloud Middleware for Seed4C
RPC and Grid Computing: Grid-RPCRPC and Grid Computing: Grid-RPC
AGENT(s)
S1 S2 S3 S4
A, B, C
Answer (C)
S2 !
Request
Op(C, A, B)
Client
6/03/12 DIET a Cloud Middleware for Seed4C
Client and server interfaceClient and server interface Client side
So easy … Multi-interface (C, C++, Fortran,
Java, Python, Scilab, Web, etc.) Grid-RPC compliant
Server side Install and submit new server to
agent (LA) Problem and parameter description Client IDL transfer from server Dynamic services
new service new version security update outdated service Etc.
6/03/12 DIET a Cloud Middleware for Seed4C
Architecture overviewArchitecture overview
6/03/12 DIET a Cloud Middleware for Seed4C
Workflow ManagementWorkflow Management
Workflow representation Direct Acyclic Graph (DAG)
Each vertex is a task Each directed edge represents
communication between tasks
Goals Build and execute workflows Use different heuristics to solve
scheduling problems Extensibility to address multi-
workflows submission and large grid platform
Manage heterogeneity and variability of environment
ANR Gwendia Language definition (MOTEUR & MADAG) Comparison on Grid’5000 vs EGI
Idle time Data transfert
Execution time
EGI (Glite) 32.857s 132.143 s 274.643 s
Grid’5000 (DIET)
0.214s 3.371 s 540.614 s6/03/12 DIET a Cloud Middleware for Seed4C
DIET Scheduling: Plug-in Schedulers DIET Scheduling: Plug-in Schedulers SeD level
Performance estimation function Estimation Metric Vector - dynamic collection of performance
estimation values Performance measures available through DIET
FAST-NWS performance metrics Time elapsed since the last execution CoRI (Collector of Resource Information)
Developer defined values
Aggregation Methods Defining mechanism to sort SeD responses: associated with the
service and defined at SeD level Tunable comparison/aggregation routines for scheduling Priority Scheduler
Performs pairwise server estimation comparisons returning a sorted list of server responses;
Can minimize or maximize based on SeD estimations and taking into consideration the order in which the request for those performance estimations was specified at SeD level.
6/03/12 DIET a Cloud Middleware for Seed4C
Data ManagementData Management
DAGDA Joining task scheduling and data
managementStandardized through GridRPC OGF
WG.Data Arrangement for Grid and
Distributed Applications Explicit data replication: Using
the API. Implicit data replication. Data replacement algorithm:
LRU, LFU AND FIFO Transfer optimization by selecting
the more convenient source. Storage resources usage management. Data status backup/restoration.
6/03/12 DIET a Cloud Middleware for Seed4C
Join work with Gaël LeMahec (UPJV/MIS)
Parallel & sequential jobstransparent for the usersystem dependent submission
SeDBatch
Many batch systemsBatch schedulers behaviourInternal scheduling process
Monitoring & Performance prediction Simulation (Simbatch)
Parallel and batch submissionsParallel and batch submissions
SeDSeD
OAR
OGELSF
PBS
Loadleveler
SeDBatch
SeD//
NFS
6/03/12 DIET a Cloud Middleware for Seed4C
SLURM
DIET Cloud: PrototypeDIET Cloud: Prototype
Inside the CloudDIET platform is virtualized
inside the cloud. (as Xen image for example)
Very flexible and scalable as DIET nodes can be launched
Scheduling is more complex
DIET as a Cloud managerEucalyptus interfaceEucalyptus is treated as a new Batch SystemProvide a new implementation for the BatchSystem
abstract class
6/03/12 DIET a Cloud Middleware for Seed4C
DIET Cloud: New ArchitectureDIET Cloud: New Architecture
6/03/12 DIET a Cloud Middleware for Seed4C
Building a nation wide experimental platform for Grid & P2P researches (like a particle accelerator for the computer
scientists) 9 geographically distributed sites hosting clusters with 256 CPUs to
1K CPUs) All sites are connected by RENATER (French Res. and Edu. Net.) Design and develop a system/middleware environment for safely test and
repeat experiments Use the platform for Grid experiments in real life conditions 4 main features:
A high security for Grid’5000 and the Internet, despite the deep reconfiguration feature
Single sign-on High-performance LRMS: OAR A user toolkit to reconfigure the nodes and monitor experiment: Kadeploy
Grid’5000
Grid’5000Grid’5000
DIET deployment over a maximum of processors 1 MA, 8 LA, 540 SeDs 1120 clients on 140 machines DGEMM requests (2000x2000 matrices) Simple round-robin scheduling
6/03/12 DIET a Cloud Middleware for Seed4C
Conclusions
http://www.grid5000.org/
Grid-RPC Interesting approach for several applications Simple, flexible, and efficient Many interesting research issues (scheduling, data management,
resource discovery and reservation, deployment, fault-tolerance, …)
DIET Scalable, open-source, and multi-application platform Concentration on several issues like resource discovery,
scheduling (distributed scheduling and plugin schedulers), deployment (GoDIET), performance evaluation (CoRI), monitoring (LogService and VizDIET), data management and replication (DAGDA), Cloud support
Large scale validation on the Grid’5000 platform A middleware designed and tunable for different applications
6/03/12 DIET a Cloud Middleware for Seed4C
ResultsResults
A complete Middleware for heterogeneous infrastructureDIET is light to useDedicated to many applicationsDesigned for Grid and Cloud Efficient even in comparison to commercial toolsDIET is high tunability middlewareUsed in production
The DIET Team
SysFera Compagny (16 persons today) http://www.sysfera.com
6/03/12 DIET a Cloud Middleware for Seed4C
http://graal.ens-lyon.fr/DIET