orchestration of applications on multiple clouds with compss · 2016-09-11 · orchestration of...
TRANSCRIPT
![Page 1: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/1.jpg)
www.bsc.es
Orchestration of Applications on MultipleClouds with COMPSs
CLOUD COMPUTING 2013 - Valencia, May 28 2013
Clouds with COMPSs
Daniele LezziJavier Alvarez, Rosa M. BadiaJorge Ejarque, Francesc
Lordan, Roger Rafanell, Raul Sirvent, Enric Tejedor
![Page 2: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/2.jpg)
Outline
Overview of COMPSs/ServiceSs
Overview of interoperability approaches withCOMPSs/ServiceSs
Interoperability to cloud middleware through connectors
Use cases & projects
![Page 3: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/3.jpg)
• StarSs
StarSsCellSs
SMPSs
GPUSs
GridSs
ClearSpeedSs
ClusterSs
OmpSs
ClusterSs
COMPSs
• Programmability/Portability– “Same” source code runs on “any” machine
The StarSs programming model
• StarSs
– Sequential C/Fortran/Java +annotations
– Task based
– Simple linear address space
– Support for SMP, GPUs, Cluster, Gridsand Clouds
– “Same” source code runs on “any” machine
– Incremental parallelization/restructure
– Focus in the problem, not in the hardware
• Performance– Intelligent Runtime
– Automatically extracts and exploitsparallelism
– Locality awareness
– Matches computations to specificresources on each type of targetplatform
Open Sourcehttp://compss.sourceforge.net
![Page 4: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/4.jpg)
initialize(f1);
for (int i = 0; i < 2; i++) {
genRandom(f2);
add(f1, f2);
}
print(f2);
Annotatedinterface
T1 T3
Grids
COMPSs Infrastructure
User code
Custom Loader
Javassist
print(f2);T2 T4
GridsClustersClouds
Files
![Page 5: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/5.jpg)
Overview of interoperability approaches withCOMPSs/ServiceSs
Interoperability through web services: ServiceSs– “Tasks” in ServiceSs can be WS
– Whole applications can be exposed as WS
Interoperability throughhigh-level standards– Venus C execution platform
implements OGSA-BEimplements OGSA-BEstandard
– Enables transparent executionof applications
Interoperability to cloud middleware through connectors
![Page 6: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/6.jpg)
COMPSs'Run+me'
Cloud'
Azure'
Job' Storage' Accoun+ng'Adaptors'CDMI' S3'
OpenNebula'
Introduction to COMPSs
OCCI' EC2'
Azure'
Usage'Records'
GAT' GRAM' SSH'
Azure'gLite'
FTP'
gridFTP'
![Page 7: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/7.jpg)
Interoperability to cloud middleware through connectors
The runtime communicates with the Cloud by means of Cloudconnectors
The connectors implement the interaction of the runtime with a givenCloud provider
Connectors abstract theruntime from the particular APIof each provider
This design facilitates theThis design facilitates theaddition of new connectors forother providers.
![Page 8: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/8.jpg)
Middleware interoperability in COMPSs
Task Scheduler– Assigns tasks to VMs or physical resources
– Each VM or resource has its own task queue
Scheduling Optimizer– Checks status of workers
– Can decide• To perform load balancing
• Create/destroy new VMs
Resource ManagerResource Manager– Manages all cloud middleware related features
– Holds information about all workers and about cloud providers
– Scheduler Optimizer sends to the RM requirements about new VM characteristics• i.e., VM that can run 3 tasks of type T1 and 2 tasks of type T2
– Resource Manager, evaluates the cloud providers and chooses the best option• More economic
• The decision can be to open a new private or public VM
– For each Cloud provider, a data structure stores the different available instances(with its features) and the connector that should be used
![Page 9: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/9.jpg)
Middleware interoperability in COMPSs
Cloud Connector– Interface that enables
• Create VM
• Destroy VM
• Cost?
• Time to create?
– When we want to add a new Cloud Provider, we just need toimplement this interfaceimplement this interface
• A special case is an implementation that supports the OCCI standard
– Two type of threads in the Connector• Creation thread
– To create and contextualize a VM
• Deletion thread– To destroy a machine
– Before destroying the VM, TS waits until all tasks assigned finish and FTMmoves remaining files to the master
![Page 10: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/10.jpg)
![Page 11: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/11.jpg)
Federatedclouds
ServiceProvider
InfrastructureProvider
Internalinfra-
structure
Service Provider
Burstedinternalclouds
OPTIMISdeployment
scenarios
InfrastructureInfrastructure
InfrastructureProvider
ServiceProvider
Multi-clouds
ProviderInfrastructure
ProviderInfrastructure
Provider
InfrastructureProvider
InfrastructureProvider
InfrastructureProvider
Broker
![Page 12: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/12.jpg)
The VENUS-C Platform
Goal: “Create a sustainableinfrastructure that enables userapplications to leverage cloudcomputing principles”
e-Science as a Service– 7 Scenarios
– 15 Open-Call Pilots
– 5 Open-Call Experimentscomputing principles”
Funded by EuropeanCommission as FP7 ResearchInfrastructures Projects
Interactive web and trainingchannel: http://www.venus-c.eu/
– 5 Open-Call Experiments
June ‘10 - May ‘12(support until May ‘13)
Free of charge access to Azure
![Page 13: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/13.jpg)
ExecutionEnvironments EMIC Generic Worker BSC COMPSsVENUS-
Arc
hit
rave
Scenario /Algorithm C
OLB
CN
R
AEG
EA
N
UP
V.B
io
Co
SB
I
UN
EW
C++Programming
LanguageC++ Java .NET C++ .NET Java
Parametersweep
Type ofworkload
Batch HTC Data flow WorkflowMap /
ReduceCEP
Users
The VENUS-C Platform
MSFT ENG KTH BSCCloud
Provider Customer
Environments EMIC Generic Worker BSC COMPSsVENUS-C
WindowsAzure
OpenNebula EMOTIVE
Windows LinuxOperating
System
CloudTechnology
IaaSPaaSCloud
Paradigm
…
Windows
BSC
Su
per
Co
mp
ute
r(n
ot
inth
ecl
ou
d)
On
Pre
mis
es
(no
tin
the
clo
ud
)
Infra-structure
![Page 14: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/14.jpg)
EU-BrazilOpenBio
EU-Brazil OpenBio
Combining Biodiversity Science and the Open AccessMovement to deploy a joint European and Brazilian e-Infrastructure of open access resources supporting the
needs of the
biodiversity scientific community.
EU-Brazil Open Data and Cloud Computing e-Infrastructure for
Biodiversity
Further EU-Brazilcollaboration in support
of the biodiversity area &infrastructures
Computingresources& SW platforms EU & Brazilian biodiversity
scientific communitiesData and resource managers &Open Access communityEuropean & Brazilian policy andfunding bodies
Who will benefit fromEUBrazilOpenBio?
biodiversity scientific community.
Two biodiversity usecases
![Page 15: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/15.jpg)
EU-BrazilOpenBio
Biodiversity VRE
BiodiversityUse Case GUI
Use Case API Instances Legacy Clients
InformationSystem
Workspace andStorage Service
ResultVisualisation
Service
SpeciesDiscovery
Service
Use CaseEnactment
Service
Experiment Orchestrator Service
VENUS-CHTCondor OMWS
VENUS-C Cloud
COMPSsWorkflow
Orchestrator
InternalInternalStorage
Condor
InternalInternalStorage
WorkflowOrchestrator
Public clouds(EC2, Azure)
Private clouds(OpenNebula, EMOTIVE)
OMWS Cluster
OMWSServer
![Page 16: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/16.jpg)
Interoperable execution of workflows in EGI Cloud
• EGI Federated Cloud:interoperable integration ofvirtualised resources fromdifferent resource providers toprovide an integrated federatedvirtualised resourcesinfrastructure for exploitation byEGI’s user community.
• Interoperability based on• Interoperability based onstandards
• Different communities samearchitecture.
• COMPSs enables the executionof Taverna workflows thanks tointeroperability features
![Page 17: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/17.jpg)
Evaluation: Elasticity and Bursting
![Page 18: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/18.jpg)
Evaluation: Performance and Scalability
![Page 19: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/19.jpg)
Evaluation: Performance
Hybrid configuration:workload unbalance.When the number ofresources allows a good loadbalancing, the speedup curverecovers (see 32+12)
Execution time (a) and speedup (b) values depending on the number of processors
The speedup keeps aquasi-linear gain.
private private+public(outsourcing)
![Page 20: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/20.jpg)
Conclusions
COMPSs/ServiceSs abstract application developers from theunderlying infrastructure
Provides a PaaS interoperable with different Cloud providers
ServiceSs applications can be offered as SaaS
Interoperability offered at different levels
![Page 21: Orchestration of Applications on Multiple Clouds with COMPSs · 2016-09-11 · Orchestration of Applications on Multiple Clouds with COMPSs CLOUD COMPUTING 2013 - Valencia, May 28](https://reader033.vdocuments.us/reader033/viewer/2022042323/5f0d51277e708231d439bf65/html5/thumbnails/21.jpg)
Thanks for your attention
www.bsc.es/compsswww.bsc.es/compss