Project Overview

and perspectives

Andrea Manieri

Engineering Ingegneria Informatica SpA

Former VENUS-C Project Director

www.venus-c.eu

22

What scientists want

ONEAccount.Connect.

Interop.Toolkit

VMsRepo

CDMIProxy

PMES-COMPSs

PMES-GW

AzureAccount.Connect.

Account.And

Billing

EMOTIVEAccount.Connect.

TRE.

VENUS-C API

3

User Community in VENUS-C

3

>=5

3-4

1-2

4

Open Call Mgmt

• 60 proposals received

• 2 parallel evaluations

• 15 selected as pilots

• 5 more experiments

• 15 contracts subject to Italian Laws

• Monitoring of activities

– 7 internal monthly progress reports

– Monitoring Effort consumption

– Four deliverables per pilot

– Final Demo (available from web portal)

Conceptual Architecture

5

Client app / Portal

Execution Environment

Re

sea

rch

er

Traffic RedundancyElimination

Accounting, Billing and Monitoring

Job Management

SDK

Data

Management

SDK

Programming

Model

Enactment

Cloud Data

Management

Interface

Cloud

Infrastructure

Management

Storage

ServicesLocal

Storage

Venus-C API andComponent Interaction

Accounting, Billing and Monitoring

Execution Environment

HD

Local

Storage

Re

sea

rch

er

Storage

Services

Client app / Portal

Data

Management

SDK

Usage Tracker

Cloud Data

Management

Interface

TRE

Programming

Model

Enactment

Job Management

SDK

Cloud

Infrastructure

Management

Accounting

PortalConnector

SOAP WS–*

Sec. Scaling

Sec. Notification

Sec. Job Mngt.

Application Code

VENUS-C Services & SDKs

Accounting Storage Service

SNIA CDMI

OCCI

OGF

RUS/UROGF

BES /JSDL

6

Platform Release

• Rapid release cycles:– Incremental releases (V1.0 V1.5, V2.0)

– Central public download site: http://resources.venus-c.eu

– Feedback infrastructure http://venuscfeedback.codeplex.com

• Common Open Source license (Apache 2.0)

7

Usage of the subsystems:

Different Execution Models

Cloud-sideCloud-side

8

Submission

services

GW /

COMPSs

Submission

services

GW /

COMPSs

Cloud

Storage

Processin

g Workers

Processin

g WorkersAccounting

service

Accounting

service

CDMI

Service

CDMI

Service

Client-side Client-side

Stand-alone client

or ASP

Stand-alone client

or ASP

Local

Storage

Data Mgnt

CDMI Client

Data Mgnt

CDMI Client

Worker

for

Enactor

Worker

for

Enactor

Processing

Processing

Processing

splitter reducer

• High-Throughput Computing through

individual jobs executed by multiple

users or Parameter Sweep.

• Data flow, through the execution of

jobs that have different stages

(communicated by files), which may

have different concurrency level.

• Coordination of jobs expressed as

fine-grain workflows orchestrated

by an enactor.Staging and Scaling

up / down resources

when needed, job

grouping.

Staging and Scaling

up / down resources

when needed, job

grouping.

Agnostic

Interface to

data.

Agnostic

Interface to

data.

Synchronization

of jobs

Synchronization

of jobs

Execution Models

9

T5.2 Building T5.2 Building

Information Information

ManagementManagement

T5.5 BioinformaticsT5.5 Bioinformatics

T5.7 Drug T5.7 Drug

DiscoveryDiscovery

T5.4 Civil T5.4 Civil

Protection and Protection and

EmergenciesEmergencies

T5.3 Data for T5.3 Data for

Science Science ––

AquaMapsAquaMaps

T5.1 T5.1 Structural Structural

Analysis for Civil Analysis for Civil

EngineeringEngineering

Success of the user

community

• The 7 scenarios will be demonstrated during the review.

• A VENUS-C pilots showcase event

celebrated in Pisa 27 June.

• 15 prototypes addressing new

requirements

– Matlab front-end1, Use of “R”,

Customised VMs2 and integration of

desktop computing3.

– 6 deployed in Linux, 8 in Azure and 1 in both.

• Many interesting examples of how Clouds

and VENUS-C help them solve their

problems.

10

Earthquake Propagation

Simulation Portal

11

• Developed by the

Aristotle University of

Thessaloniki.

• When an earthquake

happens, it can be

used to simulate the

propagation of seismic

waves and its impact.

• Automatically it captures

the data from the seismic

registers offering in

nearly real-time, information about the areas affected.

• It will be ineffective to have a cluster of 100 nodes

dedicated for the processing of events that occur rarely.

Virtual Docking in a mixed Volunteer and cloud computing

infrastructure

12

• One of the most computationally

intensive tasks in Drug Design is the

identification of ligands reacting against

specific targets.

• Hundreds of thousands of molecules in

different configurations must be tested.

• Virtual docking is feasible through

widely common tools such as Autodock.

• The University of Westminster, partner of EDGI,

had a tool developed for voluntary computing that

benefits from the elastic provisioning of cloud

resources from VENUS-C to guarantee throughput.

• In this case, 180K molecules were analysed in

VENUS-C with respect to the 38 family of the

manosidase, selecting 9 candidates.

Cloud for Radiotheraphy planning

brings a new exploitation model

13

• Radiotherapy planning based in

Monte-Carlo methods is a highly

accurate model for the estimation

of the doses

– Especially in IMRT.

• Well-known problem in research,

already adapted to Grids and clusters.

– However, research resources cannot be used for exploitation and daily practice in hospital environments.

– Moreover, usage ratios makes the infrastructure costs for this purpose unaffordable.

• By the use of VENUS-C, CESGA is validating the feasibility of

exploiting this result.

Benefits from the VENUS-C

Platform

• Scenarios and pilots concur on: Reduced response time;

Increased problem size and Improved Business Opportunities

– Speed-ups of up to 94x with 100 cores.

– Data increase of up to 25x.

– Users are interested in two different offering models

• “free”, accessing a reduced pool of local resources.

• “subscribers only” accessing a larger amount of resources from public Clouds.

• Finally, 4 Pilots have found that

VENUS-C enabled them to open

new research lines

– Social trends analysis through

cloud computing.

– A repository of ICU vital signs for

studying early predictors.

14

Pilots

Scen.

Conclusion

• Users carried out an in-depth evaluation of the VENUS-C Platform.

– More than 1.5 Million of CPU hours in total (more than 1.3 Billion SPecInt2k

hours in EGI terms), 30 TB of cumulative data stored and 80 TB of data transfers.

• Application developers give a good score to the subsystems released

in April (from 3.92 to 4.37)

– User requirements completeness, ease of applications adaptation and

interoperability got the higher marks.

• The cooperation among the user community, developers and

infrastructure providers were very fruitful.

• VENUS-C demonstrates that public cloud infrastructures (i.e.

Windows Azure) are practical for scientific research, and that the use

of VENUS-C subsystems improves user experience.

– Open-source, private infrastructures have also been tested, with similar

conclusions regarding user experience.

15

Thanks!

• www.venus-c.eu

• Acknowledgment:

– Goetz Brasche, European Microsoft Innovation

Center,

– Ignacio Blanquer, Univ. Politecnica de Valencia

16