e-clouds a platform and marketplace to access and publish scientific applications as a service

e-Clouds: A Platform and Marketplace to Access and Publish

Scientific Applications as a Service

1er Foro Internacional de Bioinformática

Ciencia y Tecnología para el Futuro del País - 2014

Mario José Villamizar Cano [email protected]

Project Assistant – Cloud Computing Advisor Universidad de los Andes

Bogotá, Colombia

mailto:[email protected]

THE PROBLEM

RELATED WORK

SOLUTION REQUIREMENTS

SOLUTION ARCHITECTURE

IMPLEMENTATION

FUTURE WORK


Ciencia y Tecnología para el Futuro del País

THE PROBLEM

When Small and Medium research groups have projects that requires large processing capabilities they regularly have to use dedicated

cluster and/or grid computing infrastructures. However …



THE PROBLEMS WITH CLUSTER AND GRID

1) The total cost of ownership

(TCO).

2) Complex processes related to IT

management.

3) There are different models to

parallelize applications.

4) Researchers require large

computing capabilities during

peak periods.



THE CLOUD COMPUTING MODEL - SaaS

Users pay a subscription to access the service (not to buy the license or to buy and maintain hardware)



THE CLOUD COMPUTING MODEL



THE CLOUD COMPUTING MODEL - IaaS

VM Types vCPUs ECUs Memory Storage Price/Hour USD

m1.small 1 1 1,70 1 x 160 0,044

m1.medium 1 2 3,75 1 x 410 0,087

m1.large 2 4 7,50 2 x 420 0,175

m1.xlarge 4 8 15,00 4 x 420 0,350

General Purpose Instances

VM Types vCPUs ECUs Memory Storage Price/Hour USD

c3.large 2 7 3,75 2 x 16 SSD 0,105

c3.xlarge 4 14 7,50 2 x 40 SSD 0,210

c3.2xlarge 8 28 15,00 2 x 80 SSD 0,420

c3.4xlarge 16 55 30,00 2 x 160 SSD 0,840

c3.8xlarge 32 108 60,00 2 x 320 SSD 1,680

Compute Optimized Instances



THE CLOUD COMPUTING MODEL



THE PROBLEM

Research groups of different sizes are adopting IaaS solutions. The adoption of cloud computing is a complex process, where non-IT researchers have to face similar or more complex processes as those found in cluster or grid infrastructures.



THE PROBLEM

At enterprise level, the Software as a Service (SaaS) model allows that small businesses can easily adopt complex and ready-to-use applications. Problems associated to the management and the configuration of the underlying (PaaS, IaaS) infrastructure are abstracted.



https://www.zoho.com/

RELATED WORK – e-Clouds GOAL

e-Clouds: A SaaS marketplace where researches can access to on-demand and ready-to-use scientific

applications via a Web Browser using a pay-per-use model.

We believe that cloud computing may provide more opportunities for a large number of small and medium research groups.



RELATED WORK

Some commercial scientific SaaS marketplaces such as Cyclone and Cloud Numbers are offering scientific SaaS application on public IaaS.



RELATED WORK




In most in-house HPC/HTC environments, clusters are the core element used to execute scientific applications.




A multi-tenant SaaS solution for scientific computing should allow that

researchers ca access to different ready-to-use-applications, execute

and monitor jobs, and the movement of data from and to the cloud; we

call these three elements the workspace.

In-house Cluster

End-user interfaces

Batch Applications

Distributed File Systems

External Libraries

Cluster middleware

Operating Systems

Virtualization

Servers

Storage

Networking

Public Scientific SaaS (e-Clouds)

End-user interfaces

Batch Applications

Cloud Storage Systems

External Libraries

Cluster middleware

Operating Systems

Virtualization

Servers

Storage

Networking


Ciencia y Tecnología para el Futuro del País M

an

ag

ed

by

res

ea

rch

ers

/ g

rou

ps

Ma

na

ged

by

res

ea

rch

ers

/ g

rou

ps


Blob Storage Scalable storage systems that can

grow to any size

App Management On-demand installation of hundreds

of scientific applications

Scheduling Servers can be turned on or

turned off as needed.




e-Clouds is based on a new architecture that abstract the above problems, allowing researchers to focus their efforts in result analysis and experimental designs, and not on technical IT issues.



Application management

Data management

Job execution and

monitoring

Pricing Security Workflow

based execution

Private/Public IaaS




e-Clouds provides mechanisms that allow users to act as application configurators, facilitate the installation and configuration of new applications: cluster creation on demand.

Queue based communications and different components of e-Clouds scale automatically and allows e-Clouds to support the execution of large workloads.

Costs Efficiency.

Traditional directory structure based on scalable blob systems.




Researchers/users:

1. Choose the application.

2. Configure the execution.

3. Monitor the jobs/executions.

4. Download the results.

5. Pay only for what they use.

e-Clouds is in charge of

interacting with the public/private IaaS in all of the steps.

IMPLEMENTATION



Amazon Web Services (AWS) Ruby on Rails / Heroku.

PostgreSQL. EC2, S3. e-Clouds Agent (Ruby).

Script application in R of Von Humboldt Institute.



IMPLEMENTATION

WHEN TO USE CLOUD SOLUTIONS SUCH AS e-Clouds

Cases where Private and Commercial Clouds may be Cost Effective (Magellan R.): Unknown Demand. In the case of a new project or a new application where the potential demand is still poorly understood. Sporadic Demand. One of the more common cases for using commercial cloud offerings is when the demand is highly variable, especially if there are also time sensitive requirements for the service. Facility Constrained. Some sites are severely infrastructure limited.



FUTURE WORK Usability tests with more researchers. Large scale performance tests.

Auto scaling and queues mechanisms. More Bag of Tasks applications. e-Clouds as an Open Source project. Shared workspaces. Different public IaaS. Types of applications (MPI, GPUs). Stronger security mechanisms.



WE ARE LOOKING FOR BETA USERS/RESEARCHERS



Más información en: eclouds.uniandes.edu.co

THANKS FOR YOUR ATTENTION!

Mario Villamizar - Universidad de los Andes

[email protected] - @mariocloud


Ciencia y Tecnología para el Futuro del País – 2014

e-clouds a platform and marketplace to access and publish scientific applications as a service

Science

foro internacional

adoption of cloud computing

large computing capabilities

service saas model

large number of small

medium research groups

scientific applications

complex processes