indigo-paas antonacci 04072016 · e-research summer hackfest - 04 july 2016, catania indigo paas:...

INDIGO PAAS

!Marica Antonacci

on behalf of WP5 team!!

[email protected] !

INFN-Bari

RIA-653549

e-Research Summer Hackfest - 04 July 2016, Catania

Outline

▪INDIGOPaaSoverview▪PaaScomponents

▪Usagescenarios:•automatedIaaS

•deploymentofLong-RunningService

•executionofuserapplicationincontainer

2


INDIGOPaaSOverview

3


INDIGOadvancedPaaSPlatform

INDIGOwill implement an advanced PaaS layer allowing scienBficcommuniBes to exploit, in a powerful and high-level way, severalheterogeneous compuBng and data e-infrastructure such as: IaaSCloud, Helix Nebula, EGI Grid, EGI Federated Cloud, PRACE, HPC,EUDAT,etc.

4

! !!!!!!!!!!!!!!!!!!!!!!!INDIGO'DataCloud!receives!founding!from!the!European!Union’s!Horizon!2020!research!and!innovation!programme!under!grant!agreement!RIA!653549

OpenStack and OpenManage both manage resource allocation among multiple user groups only through a static partitioning of resources. This means that resource usage is usually far from optimal: a specific user community can't exceed its assigned amount of resources even if there are unused resources allocated to other groups.!INFN, in the context of the INDIGO project, is addressing this issue.For what concerns OpenStack this is being done through the development of the Synergy service. !Its main capabilities are:!• Fair-share based scheduling algorithm to optimize resource usage.!• Queuing mechanism for handling the user requests that can't be immediately fulfilled.!The very same problem is being addressed also for OpenNebula. A new component, that will either replace or complement the existing OpenNebula scheduler (based on a simple FIFO logic), is being designed.!

INDIGO&DataCloud!is!a!project!financed!through!the!European!Commission!Horizon!2020!program,!running!from!April!2015!to!September!2017.!!

Its!Consortium!is!composed!of!260European0partners,!including!universities,!research!institutions,!resource!providers!and!commercial!companies.!!

The!primary! goal! of! the! INDIGO!project! is! to!develop0 an0open0 source0Cloud0platform! for! computing! and!data! tailored! to!multi&disciplinary0 scientific0communities,!and!deployable!on!hybrid0Cloud0infrastructures.!!INDIGO'DataCloud!is!coordinated0by0the0Italian0National0Institute0for0Nuclear0Physics0(INFN);!this!poster!summarizes!the!main!technical!activities!carried!

on!by!the!INFN!sites!participating!to!the!project!(INFN!CNAF,!Bari,!Padova,!Torino,!Catania,!Pisa).

Addressing scheduling issues

•OpenStack and OpenManage both manage resource allocation among multiple user groups only through a static

partitioning of resources. This means that resource usage is usually far from optimal: a specific user

community can't exceed its assigned amount of resources even if there are unused resources allocated to

other groups. •INFN, in the context of the Indigo project, is addressing this issue.

For what concerns OpenStack this is being done through the development of the Synergy service. Its two main

capabilities are:•Fair-share based scheduling algorithm to optimize resource usage.Queuing mechanism for handling the user requests that can't be immediately fulfilled.

The very same problem is being addressed also for OpenNebula. A new component, that will either replace or complement the existing OpenNebula scheduler (based on a simple FIFO logic)., is being designed.

Indigo is also finding some solutions for data centers willing to provide access to their resources both trough a Batch System interface and through Cloud technologies. Tools are being implemented to dynamically switch the role of one or more nodes, moving them from the Batch to Cloud partition (or vice versa).

Addressing!Scheduling!Issues

INFN!contributions!to!the!INDIGO&DataCloud!project

Addressing scheduling issues

•OpenStack and OpenManage both manage resource allocation among multiple user groups only through a static

partitioning of resources. This means that resource usage is usually far from optimal: a specific user

community can't exceed its assigned amount of resources even if there are unused resources allocated to

other groups. •INFN, in the context of the Indigo project, is addressing this issue.

For what concerns OpenStack this is being done through the development of the Synergy service. Its two main

capabilities are:•Fair-share based scheduling algorithm to optimize resource usage.Queuing mechanism for handling the user requests that can't be immediately fulfilled.

The very same problem is being addressed also for OpenNebula. A new component, that will either replace or complement the existing OpenNebula scheduler (based on a simple FIFO logic)., is being designed.

Indigo is also finding some solutions for data centers willing to provide access to their resources both trough a Batch System interface and through Cloud technologies. Tools are being implemented to dynamically switch the role of one or more nodes, moving them from the Batch to Cloud partition (or vice versa).

INDIGO is also finding some solutions for data centers willing to provide access to their resources both trough a Batch System interface and through Cloud technologies. !Tools are being implemented to dynamically switch the role of one or more nodes, moving them from the Batch to Cloud partition (or vice versa).

Deployment of a Managed Service/Application using INDIGO

Orchestrator

Service

REST

Brokering/Policy

Service

QoS/SLA

Service

Managed Services/

Applications (MSA)

Service

Monitoring

Service

Data

Management

Services

Infrastructure

Manager

TOSCA

IAM Service

GridHPC Clusters

SAGA

POSIX /

WebDavSAGA

POSIX /

WebDav

EUDAT

CDMI

GUI-based

Portlets

Repository

TOSCA-

compliant

Templates

1. Access

2. Authenticate

3. Select /

Customize

5. Status

6. Prioritize

7. Agree

4. Deploy

External Cloud

Cloud APIs CDMI

Partner IaaS

OpenNebula

IM

REST CDMI

OpenStack

HEAT

... 10. Deliver WNs

Mesos

8. State

9.a Provision

Worker Nodes

Master Master

Worker

...

9.b.1 Delegate

Deploy

Worker

App/

Service/

Job

Mesos Cluster

11. Deploy

9.b.2

Provision &

Configure

Contemporary distributed computing infrastructures (DCIs) are not easily and securely accessible by common users: they are typically hard to integrate due to interoperability problems resulting from the use of different authentication mechanisms, identity negotiation protocols and access control policies.!INDIGO wants to provide the services and tools needed to enable a secure composition of resources from multiple providers in support of scientific applications. A common AAI architecture has been defined that satisfies the following requirements:!• provides a layer where identities coming from different sources can be managed in a uniform way;!• defines how attributes linked to these identities are represented and understood at the various levels of

the INDIGO stack;!• defines a cryptographically strong mechanism to enforce authorization and the tools to define,

propagate, compose and enforce authorization policies;!• defines how controlled delegation of privileges can happen;!• is mainly targeted at supporting authentication/authorization for services exposed via RESTful APIs!

INFN leads the Task 5.2, Security and Authorization, which will develop the INDIGO Identity and Access Management service.!The IAM service provides a layer where identities, enrollment, group membership and other attributes and authorization policies on distributed resources can be managed in an homogeneous way, supporting the federated authentication mechanisms (SAML, OpenID connect) supported by the INDIGO AAI. !The IAM service provides user identity and policy information to services so that consistent authorization decisions can be enforced across distributed services.

Addressing!Security!&!Authorization

INDIGO0advanced0PaaS0PlatformINDIGO will implement an advanced PaaS layer allowing scientific communities to exploit, in a powerful and high-level way, several heterogeneous computing and data e-infrastructure such as: IaaS Cloud, Helix Nebula, EGI Grid, EGI Federated Cloud, PRACE, HPC, EUDAT, etc. It will be possible to process large amounts of data and to exploit the efficient storage and preservation technologies and infrastructure already available in the European e-infrastructure, with the appropriate mechanisms to ensure security and privacy.

Transparency

Error Management

Elasticity

SLA Management

Geographical!Deployment!of!Services!and!Applications

INFN leads the Task 5.3, High-level geographical application/service deployment, that will implement a service in charge of scheduling, spawning, executing and monitoring applications and services on a distributed infrastructure.!The core of this component will consist of an elastic Mesos cluster with slave nodes dynamically provisioned and distributed on the IaaS sites. !!!!!!!!!!!!!!Batch clusters, based on HTCondor, will be also provisioned on-demand and will be deployed exploiting the available computational resources across the underlying infrastructures.

In case of Long Running Services, the Geo-Deployment Service will use the Mesos Framework called Marathon to ensure that the services are always up and running. Marathon is able to deal with the restart of the services, their migration in the event of problems, the dependency among them, load-balancing, etc.!The Geo-Deployment Service will use the Mesos Framework called Chronos to execute a given application with the definition of input/output/dependency, or simple workflow.!

Portals!and!user!interfaces

Work Package 6 (WP6) addresses the complex challenge of guaranteeing a simple and effective final user experience, both for software developers and for researchers running the applications. This objective requires different activities, starting with the development of APIs to access the PaaS framework, so that its features can be exploited by Portals, Desktop Applications and also by Mobile Apps.!INFN is strongly involved in the developments of WP6 components and its main contributions are in the Future Gateway framework which will provide a full software stack spanning from the submission engine to the end user portal. !!!!!!!!!!!!Dynamic instantiation of the components (e.g. publication service, big data cluster, WfMS instance, CLI apps) will be performed through specific JSAGA adaptors (e.g. IDC) interacting with the INDIGO PaaS layer through the TOSCA specification.

•Kepler!will!be!adopted!to!run!the!workflows!related!to!the!scientific!experiments!for!the!climate!change!

community;!!

• the!Future0Gateway0Engine,!jointly!with!JSAGA!and!the!related!adaptors!will!act!as!the!Core!Engine;!!

•Ophidia! will! be! exploited! as! big! data! analytics!system;!!

• a!generic!HTTP!service!will!be!used!as!Publication0service;!• ESGF! nodes! will! provide! the! needed! Fabric! layer!components.

@indigodatacloudwww.indigo'datacloud.eu info@indigo'datacloud.eu

INFN contributions focus on the authentication and authorization issues, the storage solutions integration and the implementation of the geo-deployment service at the PaaS level. Moreover, INFN contributes to the developments in the IaaS Cloud area (scheduling and storage components) and to the implementation of the user interfaces layer.


INDIGOPaaS:Keyfeatures

▪ ImprovedcapabilitiesinthegeographicalexploitationofCloudresources.

▪StandardinterfacetoaccessPaaSservices.• INDIGOwillusetheTOSCAstandard

▪SupportfordatarequirementsinCloudresourceallocations.

• Resourcescanbeallocatedwheredataisstored.

▪ IntegrateduseofresourcescomingfrombothpublicandprivateCloudinfrastructures

5



▪DistributeddatafederationssupportinglegacyapplicationsaswellashighlevelcapabilitiesfordistributedQoSandDataLifecycleManagement.

• ThisincludesforexampleremotePosixaccesstodata.

▪Transparentclient-sideimport/exportofdistributedClouddata.

• Thissupportsdropbox-likemechanismsforimportingandexportingdatafrom/totheCloud.ThatdatacanthenbeeasilyingestedbyCloudapplicationsthroughtheINDIGOunifieddatatools.

▪Supportfordistributeddatacachingmechanismsandintegrationwithexistingstorageinfrastructures.

6



▪Deployment,monitoringandautomaticscalabilityofexistingapplications.

▪ Integratedsupportforhigh-performanceBigDataanalytics.

▪Supportfordynamicandelasticclustersofresources.

• batchsystemson-demand(suchasHTCondororTorque)

• extensibleapplicationplatforms(suchasApacheMesos)capableofsupportingbothapplicationexecutionandinstantiationoflong-runningservices.

7


INDIGOPaaSArchitecture

8

Onedata

IAM

Orchestrator (TOSCA-compliant)

Monitoring

QoS/SLA Management

CloudProviderRanker

REST API micro-service

micro-service

micro-service

micro-service

micro-service

Managed Service/Application Deployment

micro-service

Mesos

Marathon Chronos

micro-service

Cloud/Grid/HPC Platforms

Storage

REST POSIX/WebDav

Compute/Network

IM/Heat SAGA

WP4 interfaces

WP4 interfaces

TOSCA

Dynafedmicro-service

FTSmicro-service

Data Management Services

Custom Framework

Accounting

Information System

Repositories

Kubernetes

PaaS micro-services Orchestration


HostingthePaaSCoreServices

10

▪ThePaaSCoreisprovidedasasetofservicesthatexposeRESTinterfacesandinteractamongthemviaHTTP.!▪ThePaaSCoreServiceshavetobe:• Deployed•Managed• Upgraded•Monitored• Scaled• Self-healed!▪TheplatformtoprovidesuchfunctionalitiesisKubernetes.• http://kubernetes.io!▪Anopen-sourcesystemformanagingcontainerizedapplicationsacrossmultiplehostsinacluster


IAMService

▪The IdentityandAccessManagement(IAM)serviceprovidesalayer where identities, enrollment, group membership andother attributes and authorization policies on distributedresourcescanbemanagedinanhomogeneousway,supportingthe federated authentication mechanisms (SAML, OpenIDconnect)supportedbytheINDIGOAAI.

▪The IAM service provides user identity andpolicy informationto services so that consistent authorization decisions can beenforcedacrossdistributedservices.

!!

11


IAMArchitecture

12

▪Mainfunctionalities:

• Authentication

• Sessionmanagement

• Enrollment

• Attributeandidentitymanagement

• Userandgroupprovisioninganddeprovisioning

• Policydefinition,distributionandenforcement


OrchestratorService

13

▪TheOrchestratorcoordinatesthedeploymentprocessovertheIaaSplatforms

▪TheOrchestratorcollectsalltheinformationneededtodeployaserviceconsumingothersPaaSµServicesAPIs:

•MonitoringService:getthecapabilitiesoftheunderlyingIaaSplatformsandtheirresourceavailability;

• QoS/SLAService:gettheprioritizedlistofSLAperuser/group

• CloudProviderRanker(RuleEngine)Service:sortthelistsitesonthebasisofrulesdefinedperuser/group/use-case;

• DataManagementService:getthestatusofthedatafilesandstorageresourcesneededbytheservice/application

▪TheorchestratordelegatesthedeploymenttoIM,HEATorMesosbasedontheTOSCAtemplateandthelistofsites.

▪Cross-sitedeploymentswillalsobepossible.


OrchestratorService

14

RESTAPI

Adapter(IaaS)

Openstack4j

RDBMSDAO

Recipe(HEAT)Na>veAPI

Na>veAPI

ORCHESTRATORENGINE

MonitoringConnector

MonitoringPillar

RESTAPI

Na>veAPI

Na>veAPI+HeatTranslator

MesosConnectors(forChronos,Marathon)

TOSCAparser

BasedonAlien4Cloud

InfrastructureManager

InfrastructureManager

Na>veAPI

Na>veAPI

OtherINDIGOcomponentsfor:•  SLA•  CloudSiteselec>on•  DataManagementServices•  etc

▪BuiltonJBPM6.1(long-runningworkflow)

▪ExposesRESTfulAPIs▪SupportstheTOSCASimpleProfileinYAMLVersion1.0specification.


CloudProviderRankerService

▪ProvidesinformationthatwillbeconsumedbytheOrchestratorinordertoproperlycoordinatethedeploymentoftherequiredresourceonthesites.

• AWEBServiceprovidingRESTAPIstorankCloudProvidersdescribedbyaJSONblobcontaining:

- TotalVirtualCPUs,totalVirtualRAM

- TotalVirtualEphemeralDisk(spaceforinstances)

- TotalVirtualDisk(blockstorage,e.g.Cinder)

- InuseVCPU,inuseVRAM,inuseVDISK,inuseVEphDisk

▪RankingalgorithmimplementedusingthelargelydiffusedDroolsRuleEngineruntimeframework

15


MonitoringService

▪TheMonitoringserviceprovidesacomprehensiveRESTAPItogatherinformationabout

• thePaaSCoreServices

- UsingHeapsterintheKubernetescluster.

• thecustomizedvirtualinfrastructures

- UsingZabbixagentsdeployedinsideVMs/containers

• thestateofthesites(leveragingEGIFedCloudapproach)

!

▪MonitoredinformationhastobeexposedviaaRESTAPItobeconsumedbyotherservices

16


QoS/SLAService

▪AllowsthehandshakebetweenauserandasiteonagivenSLA▪ProvidestheOrchestrator/RankerwiththeusefulinformationfortakingthedecisionontasksschedulingaccordingtotheagreedandvalidSLAs

▪DescribestheQoSthataspecificuser/grouphasbothoveragivensiteorgenerallyinthePaaSasawhole;thisincludesapriorityofagivenusers,thecapabilitytoaccesstodifferentQoSateachsite(Gold,Silver,Bronzeservices)

17


AccountingService

▪AccountsforresourceusageontheINDIGOPaaSandprovidesthatdatatootherINDIGO-DataCloudservices

• QoS/SLAservicewilluseinformationgatheredbytheAccountingservice(andthemonitoringpillar)tomonitorSLAviolation

▪Usagedataisextractedfromthesystemwheretheresourcesareusedandsenttoacentralrepository

▪Therepositoryaggregatesthedatafromacrosstheinfrastructuretoproducetotalsbasedonanumberoffields-suchasuser,site,month,year,etc.

18


CMDBService

▪ IndigoConfigurationManagementDB(CMDB)

• NeedtoknowwhatareINDIGOprovidersandservices

• needtoregisterspecificdata(notcoveredbyGocDB)

▪ IdeacomesfromgoodpracticesofITServiceManagement(ITIL,FitSM)

19

SLAManager

CMDB

Customer Userscollectsresourcefor

managesandcontrolsresourcesusing

Orchestrator

requestsaservice(s)(viaothercomponents)

Monitoring

checksavailableop9ons(SLAs)forausertoget

service(s)fromprovider(s)

providesdataaboutrealisa9onofSLAs

otherPaaScomponents

providesdetailsaboutservicesandproviders

providesdetailsaboutservicesandproviders

managesandcontrolsresourcesusing

registersaproviderandservices


InfrastructureManager(IM)Service

20

▪ TheIMisaserviceforthewholeorchestrationofvirtualinfrastructuresandapplicationsdeployedonthem,includingresourceprovisioning,deployment,configuration,re-configurationandtermination

▪ AconfigurationmanagerbasedonAnsibleconfigurestheVMsdeployedbythecloudconnectorandinstallsthenecessarysoftware.

▪ IMsupportsAPIsfromalargenumberofvirtualplatforms,makinguserapplicationscloud-agnostic.

▪ IMhasbeenextendedinINDIGOtosupporttheTOSCASimpleProfileinYAMLVersion1.0forinfrastructuredescription.

▪ Documentation:http://www.grycap.upv.es/im

http://www.grycap.upv.es/im


AutomaticScalingService

▪ExtendsEC3CLUESaddingtheinterfacesrequiredtointeractwiththeINDIGOOrchestrator

▪Documentation:http://www.grycap.upv.es/ec3

▪ Implementstheelasticityrulesconsideringthestateofthevirtualcluster.

▪Thevirtualclusterwilldeployadditionalworkernodesasrequired,andintegratethemontheLRMSwithoutuserintervention,inordertocopewithincreasedworkloadofjobs.Workernodeswillbeterminatedwhentheyarenolongerrequired.

▪Pluginsareavailablefor:SLURM,Torque/PBS,HTCondor,Mesos

21

http://www.grycap.upv.es/ec3


ManagedServices/Application(MSA)DeploymentService

▪ Thisserviceisinchargeofscheduling,spawning,executingandmonitoringapplicationsandservicesonadistributedinfrastructure.

▪ ThecoreofthiscomponentconsistsofanelasticApacheMesosclusterwithslavenodesdynamicallyprovisionedanddistributedontheIaaSsites.

▪ ApacheMesosprovidesefficientresourceisolationandsharingacrossdistributedapplications(frameworks).

22


DataServices:Unifieddataaccess

▪Unifiedvisionofgeographicallydistributeddataset▪Dataaffinity▪Computationjobsstartedonresourcesclosetodata.

▪Federateddataaccess• InteroperabilityandOpenData

▪OptimizationandDataonthefly

• whendataisnotstaged

23


INDIGOPaaSUsageScenarios

24


ScenarioI:DeploymentofVirtualInfrastructures

25

Deployment of Customized Virtual Infrastructures using INDIGO-DataCloud

Orchestrator Service

Brokering/Policy Service

QoS/SLA Service

Managed Services/Applications (MSA)

Service

Monitoring Service

Data Management

Services

Infrastructure Manager

TOSCA

IAM Service

GridHPC Clusters

SAGAPOSIX / WebDavSAGA

POSIX / WebDav

EUDAT

CDMI

GUI-based Portlets

Repository

TOSCA-compliant Templates

1. Access

2. Authenticate

3. Select /Customize

9.a Delegate Deploy

5. Status

6. Prioritize

7. Agree

8.a Transfer (Optional)

8.b. Transfer (Optional)

4. Deploy

9.b.1 Delegate Deploy

External Cloud

Cloud APIs CDMI

Partner IaaS

OpenNebula

IM

CDMI

OpenStack

HEAT

TOSCA

9.b.2 Provision & Configure

...

Virtual Infrastructure for User Application/Service

10. Deliver VMs

...

Figure 5: Deployment of a customized virtual infrastructure: When a cus-tomized virtual infrastructure deployment is requested (scenario A), the Or-chestrator manages the instantiation and configuration of the required resources(e.g. virtual machines) on the selected IaaS infrastructure using the REST APIsexposed by the IaaS orchestrator (i.e. Heat or IM) of the INDIGO sites or del-egating the interaction with external clouds to a dedicated instance of the IM.

14


▪ADockercontainerisinstantiatedautomaticallyafterasimplerequestonthewebportalfromanend-user.

• ThiswillexploitaTOSCATemplatethroughtheWP5orchestrator

▪ThecontainerhasapublicIPaddressandtheuser(ortheportal)cangetaccessdirectlytoit.

▪Userscouldmount(atleastinteractively)alocalorremoteposixfile-systemthroughOnedata

▪TheapplicationintheDockercontainerisabletosimplyreadthefilesprovidedviawebbrowserbytheenduserandtowriteposixfilesthatareavailabletousersviawebbrowsers.

▪ThesameDockercontainercouldbeusedtoexecutealargelistofapplicationsinabatch-likebehaviour.

26

Use-case:InteractiveusageofaDockercontainerwithssh

e-Research Summer Hackfest - 04 July 2016, Catania 27

3

FutureGatewayAPIServer

Orchestrator

OneDock nova-docker

WP6

WP5

WP4

TOSCADocumentsandDockerfilesperUseCase OtherPaaS

CoreServices

CloudSite

DockerContainerPublicIP

SSHdINDIGO-DataCloud

DockerHubOrganizaLon

Provider

Champion+JRA

User

1.a.1)build,push

1.a.2)Dockerfile(commit)

1.b)AutomatedBuild

3)DeployTOSCA

2)StageData

5)Mount

4)Access

App

IM

InteractiveusageofaDockercontainerwithssh


▪Astand-aloneGalaxyserverisinstalledautomaticallyinavirtualmachine.

▪Ablockdeviceusedforhostingthereferencedataisinstantiated▪Allthesteps(installingGalaxy,installingapplications,downloadingreferencedata)areautomaticallyexecutedbyparametricscripts.

▪AlltheneededIaaSresourcesareorchestratedbyTOSCATemplates

!

▪NOTE:Theprocedurewouldbesimilarforcontainers.

28

Use-case:InteractiveusageofaPortalwithinaVirtualMachine


InteractiveusageofaPortalwithinaVirtualMachine

29


Orchestrator

IM

OpenNebula

WP6

WP5

WP4

TOSCADocumentsReferencingArGfactsperUseCase OtherPaaS

CoreServices

CloudSite

VMPublicIP

Provider

Champion+JRA

User

1)CreateArGfactstodeploysoMware(Ansibleroles)

3)DeployTOSCAwithVanillaVM

2)StageData

5)Mount

6)AccessWebPortal

Galaxy

OpenStack

Heat4)Install/Configure

IM


▪Thewebportalisinstantiated,installedandconfiguredautomaticallyexploitingscriptsandTemplates.

▪Aremoteposixshareisautomaticallymountedonthewebportal

▪Thesameposixshareisautomaticallymountedalsoonworkernodes

▪End-userscanseeandaccessthesamefilesviasimplewebbrowsersorsimilar.

▪AbatchsystemisdynamicallyandautomaticallyconfiguredviaTOSCATemplates

▪Theportalisautomaticallyconfiguredinordertoexecutejobonthebatchcluster

▪Thebatchclusterisautomaticallyscaledup&downlookingatthejobloadonthebatchsystem.

30

Awebportalthatexploitsabatchsystemtorunapplications

e-Research Summer Hackfest - 04 July 2016, Catania 31


WP6

WP5

Front-EndPublicIP

Provider

User2)DeployTOSCAwithVanillaVM/Container

1)StageData

5)Mount

6)AccessWebPortal

Galaxy

4)Install/Configure

WNWNWN …

VirtualElasOcCluster

Orchestrator

IM

OpenNebula

WP4

OtherPaaSCoreServices

CloudSite

OpenStack

HeatClues

IM

Awebportalthatexploitsabatchsystemtorunapplications


Galaxy:elasticcluster

32

PaaSOrchestrator

CloudSite

VM

Galaxy

CLUES

VM

Job

VM

Job

…

Elas=cVirtualCluster

1.DeployGalaxyTOSCATemplate

2.ProvisionFront-End

4.AccessGalaxyportal

6.Spawnaddi=onalVMs(horizontalelas=city)

SLURM …

5.SubmitjobstoLRMS

7.Executejobs

8.TerminateVMswhenunused

3.DeployVM&Configure

TOSCATemplates


ScenarioII:DeploymentofaManagedService/application

33

Deployment of a Managed Service/Application using INDIGO-DataCloud

Orchestrator Service

Brokering/Policy Service

QoS/SLA Service

Managed Services/Applications (MSA)

Service

Monitoring Service

Data Management

ServicesInfrastructure

Manager

TOSCAIAM Service

GridHPC Clusters

SAGAPOSIX / WebDavSAGA

POSIX / WebDav

EUDAT

CDMI

GUI-based Portlets

Repository

TOSCA-compliant Templates

1. Access

2. Authenticate

3. Select /Customize

5. Status

6. Prioritize

7. Agree

4. Deploy

External Cloud

Cloud APIs CDMI

Partner IaaS

OpenNebula

IM

CDMI

OpenStack

HEAT

... 10. Deliver WNs

Mesos

9.a.1 Deploy Job/Service

9.a.2 Provision Worker Nodes

Master Master

Worker

...

9.b.1 Delegate Deploy

Worker

App/Service/

Job

Mesos Cluster

11. Deploy

9.b.2 Provision & Configure

TOSCA

8.a Manage Transfer

(Optional)

8.b. Manage Transfer (Optional)

Figure 6: Deployment of a managed service/application: When a managedPaaS service deployment is requested (scenario B), the Orchestrator interactswith the Managed Service/Application (MSA) Deployment Service in order tosupervise its deployment on the elastic Mesos cluster that will host the userapplication/service.

15


ManagedServicesDeploymentandApplicaBonsExecuBonthroughMesos

• Mesosisabletomanageclusterresources(cpu,mem)providingisola3onandsharingacrossdistributedapplicaBons(frameworks)

!• MarathonandChronosaretwopowerfulframeworksthatcanbedeployed

ontopofaMesosCluster.!

• INDIGOPaaSuses:• Marathontodeploy,monitorandscaleLong-Runningservices,ensuring

thattheyarealwaysupandrunning.• ChronostorunuserapplicaVons(jobs),takingcareoffetchinginputdata,

handlingdependenciesamongjobs,reschedulingfailedjobs.

34


MesosinINDIGO

▪AutomaticdeploymentthroughAnsiblerecipesembeddedinTOSCAandHOTtemplates

•Alltheservicesrunindockercontainers;▪High-availabilityoftheclustercomponents:

•LeaderelectionamongmasternodesmanagedbyZookeeper;

•HALoad-balancing;▪ServicediscoverythroughConsulthatprovidesalsoDNSfunctionalityandhealthchecks;

•servicesareautomaticallyregisteredinConsulassoonastheyaredeployedonthecluster

▪Theexternalaccesstothedeployedservicesisensuredthroughload-balancersinHA(uniqueentrypoint:clusterVirtualIP)

▪Clusterelasticityandapplicationauto-scalingthroughCLUESplugin35


INDIGOMesosCluster

36

MesosMaster

MesosWorker

MesosWorker

Marathon

Chronos

scalein

scaleout

Elastic Mesos Cluster


INDIGOPAASTutorial

▪ IntroductoryConcepts• TOSCA

• Ansible

• Docker

• OrchestratorAPIs

• INDIGOTOSCAcustomtypesandtemplates

▪Demos

37