2013 05-fite-club-working-models-cloud-growing-up

Working Models ...or how cloud is growing up and thinking about applications

Alex Heneveld / @ahtweetinFite Club, 22 May 2013

CloudIt’s not just about VM’s...

It’s about storage, networking, maas, ...

It’s not just about VM’s...

Cloud

It’s not just about VM’s...

Ultimately it’s about applications.

Cloud

Declarative — AWS CloudFormation and OpenStack HeatProcedural — Apache Whirr and Cloudsoft BrooklynOpenStack Summit — Red Hat, RAX, IBM, OASIS TOSCA and CAMPDSLs in Progress — OpenStack Heat and CAMP PDP

Ultimately it’s about applications.

AWS CloudFormation

{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "WordPress is web software …", "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair for SSH access", "Type" : "String" }, ... }, "Mappings" : { "AWSInstanceType2Arch" : { "m1.small" : { "Arch" : "64" }, "m1.large" : { "Arch" : "64" }, ... }, "AWSRegionArch2AMI" : { "us-east-1" : { "32" : "ami-31814f58", "64" : "ami-1b814f72" }, ... } }, "Resources" : { "WebServer": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "gcc-c++" : [], "make" : [],

...

{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "WordPress is web software …", "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair for SSH access", "Type" : "String" }, ... }, "Mappings" : { "AWSInstanceType2Arch" : { "m1.small" : { "Arch" : "64" }, "m1.large" : { "Arch" : "64" }, ... }, "AWSRegionArch2AMI" : { "us-east-1" : { "32" : "ami-31814f58", "64" : "ami-1b814f72" }, ... } }, "Resources" : { "WebServer": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "gcc-c++" : [], "make" : [],

...

"Resources" : { "WebServer": { "Type": "AWS::EC2::Instance"

AWS CloudFormation


Parameters

Mappings(config metadata)

WaitConditions &WaitHandles

(orchestration)

Outputs

Functions

AWS CloudFormation


AWS CloudFormation

"Type": "AWS::AutoScaling::AutoScalingGroup"

"Type": "AWS::EC2::EIP"


"Type": "AWS::SQS::Queue"

"Type": "AWS::CloudFormation::Stack"

"Type": "AWS::S3::Bucket"

AWS CloudFormation

"Type": "AWS::AutoScaling::AutoScalingGroup"

"Type": "AWS::EC2::EIP"


"Type": "AWS::SQS::Queue"

"Type": "AWS::CloudFormation::Stack"

"Type": "AWS::S3::Bucket"

AWS CloudFormation

Composition!

— Limited to AWS cloud

— Limited to AWS concepts

— Cumbersome to write

— Limited dependency injection

Nice, but:

AWS CloudFormation

— CloudFormation clone

— Targetting OpenStack resources and AWS compatibility

— Adds YAML support (comments and easier)

— Multi-cloud support using Apache Deltacloud

OpenStack Heat

(AWS CloudFormation){ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "WordPress is web software …", "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair for SSH access", "Type" : "String" }, ... }, "Mappings" : { "AWSInstanceType2Arch" : { "m1.small" : { "Arch" : "64" }, "m1.large" : { "Arch" : "64" }, ... }, "AWSRegionArch2AMI" : { "us-east-1" : { "32" : "ami-31814f58", "64" : "ami-1b814f72" }, ... } }, "Resources" : { "WebServer": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "gcc-c++" : [], "make" : [],

...

OpenStack Heat

OpenStack Heat

HeatTemplateFormatVersion: 2012-12-12 Description: WordPress is web software ... Parameters: KeyName: Description: Name of an existing EC2 KeyPair for SSH access Type: String ... Mappings: AWSInstanceType2Arch: m1.small: {Arch: '32'} m1.large: {Arch: '64'} ... DistroArch2AMI: F18: {'32': F18-i386-cfntools, '64': F18-x86_64-cfntools}, ... Resources: WebServer: Type: AWS::EC2::Instance Metadata: AWS::CloudFormation::Init: config: packages: yum: gcc-c++: [] make: []

...

— CloudFormation clone

— Targetting OpenStack resources and AWS compatibility

— Adds YAML support (comments and easier)

— Multi-cloud support using Apache Deltacloud

OpenStack Heat

Available NOW in Grizzly!

OpenStack Heat

— Still limited to selected clouds

— Still limited in concepts

— Only slightly less cumbersome to write

— Very limited dependency injection

But...

Procedural Approach

10 INPUT "What app do you like? ", A$20 INPUT "What cloud do you like? ", C$30 GOSUB 10040 END

100 REM deploy C$ to A$110 RETURN

Apache Whirr

provision

install

configure

manage

— Apache top level project

— Deploy multi-node applications— Hadoop, Hama, Mahout, web,

Cassandra & many more

— Define topologies declaratively— Add new roles, configuration,

constraints and wiring procedurally

$ whirr launch-cluster \ --cluster-name=hbase \ --instance-templates='1 hbase-master,6 hbase-regionserver'

— Define topologies declaratively

Apache Whirr

— Add new roles, configuration,constraints and wiring procedurally

spec = new ClusterSpec();spec.setProvider("cloudservers-uk");spec.setIdentity(apikey);spec.setCredential(secret);spec.setClusterName("hbase");spec.setInstanceTemplates(ImmutableList.of( new InstanceTemplate(1, "hbase-master"), new InstanceTemplate(6, "hbase-regionserver")));

cluster = new ClusterController().launchCluster(spec);

Apache Whirr

— Limited parameterisation

— Defining new roles requires coding

— Strict phases limit applicability

But...

— Composable to one dimension

— Portable locations support

— Pluggable with Chef, Puppet, and more

Apache Whirr

Brooklyn & Cloudsoft AMP

• Brooklyn = Apache-licensed OSS

• AMP = Enterprise + Support

• Deployment and Management

• Hierarchical Autonomic Policies

Brooklyn

Contention ManagementCustom PaaSBig Data

Region #1

Presentation

GeographicDNS Load

Balancer

Tomcat

Processing (Monterey)

Data (Gemfire)

Tomcat

Tomcat

Region #2

Presentation

LoadBalancer

Tomcat


Data (Gemfire)

Tomcat

TomcatProvisioning

Monitoring

Management

Brooklyn: Example, Physical Architecture

Data (Gemfire)Processing (Monterey)

Region #1

Presentation

GeographicDNS

LoadBalancer

Tomcat

Processing (Monterey) Data (Gemfire)

Tomcat

Tomcat

Region #2

Presentation

LoadBalancer

Tomcat


Tomcat

Tomcat

Provisioning Monitoring Management

WebappFabric

TomcatNodes

NginxNode

LoadBalancedWebAppCluster

TomcatNodes

NginxNode


WebappGeoDnsFabric MontereyFabric

MontereyCluster

MontereyNodes

MontereyNodes

MontereySegments

GemfireFabric

MontereyCluster

GemfireCluster

GemfireClusterGeoscaling

Brooklynand friends

GemfireNodes

GemfireNodes

Brooklyn: Example, Logical Architecture


Autonomic Hierarchical Management

GeographicDNS

Presentation

LoadBalancer

Tomcat Tomcat

WebappFabric


Cluster Cluster Cluster Cluster

MyApplication

Policy

follow-the-{sun,moon,X}

Policy

resizer

Policy

restart-2x

Policy

Brooklynand friends

WebappGeoDnsFabric MontereyFabric GemfireFabric

Brooklyn


Typed Blueprintsreplace groups with PaaS tools andhave the same sensors and policies

GeographicDNS

Presentation

WebappFabric

PaaS

Cluster Cluster Service Cluster

MyApplication

Brooklynand friends

WebappGeoDnsFabric MontereyFabric GemfireFabric

Brooklyn

Policy

follow-the-{sun,moon,X}

Policy

resizer

Policy

restart-2x

Policy

Data (Gemfire)

Data (Gemfire)


Presentation

GeographicDNS

LoadBalancer

Tomcat


Tomcat

Tomcat

Presentation

LoadBalancer

Tomcat


Tomcat

Tomcat


WebappFabric

TomcatNodes

NginxNode


TomcatNodes

NginxNode



MontereyCluster

MontereyNodes

MontereyNodes

MontereySegments

GemfireFabric

MontereyCluster

GemfireCluster


GemfireNodes

GemfireNodes fixed IP

&c

Brooklyn: Run in Many Locations

Data (Gemfire)

Data (Gemfire)


Presentation

GeographicDNS

LoadBalancer

Tomcat


Tomcat

Tomcat

Presentation

LoadBalancer

Tomcat


Tomcat

Tomcat


WebappFabric

TomcatNodes

NginxNode


TomcatNodes

NginxNode



MontereyCluster

MontereyNodes

MontereyNodes

MontereySegments

GemfireFabric

MontereyCluster

GemfireCluster


GemfireNodes

GemfireNodes

Policies

Policies

WARfile

code schema

credentials

GeoDNSconfig actor

descriptor

Regions

Brooklyn: finally { Simplify }

public class MyWebCluster extends AbstractApplication implements MyWebClusterConstants { public void init() { MySqlNode mysql = addChild(EntitySpecs.spec(MySqlNode.class) .configure("creationScriptUrl", DB_SETUP_SQL_URL)); ControlledDynamicWebAppCluster web = addChild(EntitySpecs.spec( ControlledDynamicWebAppCluster.class) .configure(WebAppService.HTTP_PORT, PortRanges.fromString("8080+")) .configure(JavaWebAppService.ROOT_WAR, WAR_PATH) .configure(javaSysProp("brooklyn.example.db.url"), formatString("jdbc:%s%s?user=%s\\&password=%s", attributeWhenReady(mysql, MySqlNode.MYSQL_URL), DB_TABLE, DB_USERNAME, DB_PASSWORD)) ); }}

Brooklyn: Nested Elastic Blueprints

public class MyWebCluster extends AbstractApplication implements MyWebClusterConstants { public void init() { MySqlNode mysql = ...; ControlledDynamicWebAppCluster web = ...;

web.addEnricher(HttpLatencyDetector.builder().url(ROOT_URL). rollup(10, SECONDS).build());

web.getCluster().addPolicy(AutoScalerPolicy.builder(). metric(REQUESTS_PER_SECOND_IN_WINDOW_PER_NODE). metricRange(10, 100).sizeRange(2, 5).build());

addEnricher(SensorPropagatingEnricher.newInstanceListeningTo(web, ROOT_URL, REQUESTS_PER_SECOND_IN_WINDOW, REQUEST_LATENCY_IN_SECONDS_IN_WINDOW)); }

}

Brooklyn: Policies, Enrichers and KPI’s

http://web.addEnricher

http://web.addEnricher

http://web.getCluster

http://web.getCluster

Brooklyn

— Defining blueprints requires coding

But...

— Easily parameterisable blueprints

— Composable and substitutable

— Portable and powerful locations support

— Pluggable with VM images, Heat, Chef, more

Applications

— Components (IaaS-resources)

— DeclarativeInfrastructure

— Procedural

— Extensible (new resources)

— Composable (stacks)

— Management

— Portable (hybrid, metal)

Heat in Grizzly

Man

agem

ent

Com

posa

ble

Decl

arat

ive

Com

pone

nts

Exte

nsib

le

Port

able

thanks largely to Red Hat

Portland Design Summit — Heat

Man

agem

ent

Com

posa

ble

Decl

arat

ive

Port

able

Com

pone

nts

Exte

nsib

le


Man

agem

ent

Com

posa

ble

Decl

arat

ive

Port

able

Com

pone

nts

Exte

nsib

lelots of new blood (interest and ideas)


Com

posa

ble

Decl

arat

ive

Com

pone

nts

Man

agem

ent

Port

able

Ease

-of-

Use

Exte

nsib

leingredients from Brooklyn, Whirr, OoO and others


Com

posa

ble

Decl

arat

ive

Com

pone

nts

Man

agem

ent

Port

able

Exte

nsib

le

Auto

-wiri

nga special ingredient from Rackspace

Ease

-of-

Use


Com

posa

ble

Decl

arat

ive

Com

pone

nts

Man

agem

ent

Exte

nsib

le

Auto

-wiri

ng

Rela

tions

hips

Ease

-of-

Use

Port

able

a special ingredient from IBM

Man

agem

ent

Com

posa

ble

Decl

arat

ive

Com

pone

nts

Auto

-wiri

ng

Rela

tions

hips

Exte

nsib

le

Ease

-of-

Use

Port

able


Heat in Havana

Man

agem

ent

Com

posa

ble

Decl

arat

ive

Com

pone

nts

Auto

-wiri

ng

Rela

tions

hips

Exte

nsib

le

Ease

-of-

Use

Port

able

ConcurrentDistributedResilientFunctions / DI

Heat in Havana

Man

agem

ent

Com

posa

ble

Decl

arat

ive

Com

pone

nts

Auto

-wiri

ng

Rela

tions

hips

Exte

nsib

le

Ease

-of-

Use

Port

able

ConcurrentDistributedResilientFunctions / DIStandards ?

OasisTOSCA

http://www.oasis-open.org/committees/tosca/

Topology and Orchestration Specification for Cloud Applications



Modeling Topologies with TOSCA Service Topologies are described using the TOSCA “Meta-model”:

Artifacts Describe Installables and Executables required to

instantiate and manage a service. Currently, they include:

Implementation Artifacts: – Executables or Plans that implement a Node’s

or Relationship’s Operations (e.g. a Bash script) Deployment Artifacts:

– Installables of the components (e.g. a TAR file)

A service’s Topology Model is included in a TOSCA Service Template which is packaged and shared, along with all dependent artifacts, as a TOSCA Cloud Service Archive (CSAR)

Service Templates Group the nodes and relationships that make up

a service’s topology – Allowing modeling of sub-topologies

Service Templates “look like nodes” enabling: Composition of applications from one or more

service templates Substitution of abstract Node types with

available service templates of the same type

Nodes Represent Components of an application or service

and their Properties. Example nodes include: – Infrastructure: Compute, Network, Storage, etc. – Platform: OS, VM, DB, Web Server, etc. – Granular: functional Libraries, Modules, etc.

Include Operations which are the management functions for the node – e.g. deploy(), start(), stop(), connect(), etc.

Export their dependencies on other nodes as Requirement and Capabilities

Relationships Represent the logical Relationships between

nodes – e.g. “hostedOn”, “connectsTo”, etc.

Describes the valid Source and Target nodes they are designed to couple – e.g. source “web application” node is designed

to “connectTo” a target “database” node Have their own Properties and Constraints

Slide origin: IBM

TOSCA: Modelling Topologies

<Definitions id="PayrollDefinitions" targetNamespace="http://www.example.com/ste" xmlns:pay="http://www.example.com/ste/Types"> <Import namespace="http://www.example.com/ste/Types" location="http://.../PayrollTypes.tosca" importType=" http://docs.oasis-open.org/tosca/ns/2011/12"/> <ServiceTemplate id="Payroll" name="Payroll Service Template"> <TopologyTemplate ID="PayrollTemplate"> <NodeTemplate id="Payroll Application" type="pay:ApplicationNodeType"> <DeploymentArtifacts> <DeploymentArtifact name="PayrollEAR" type="http://..../CSARref"> EARs/Payroll.ear </DeploymentArtifact> </DeploymentArtifacts> </NodeTemplate> <NodeTemplate id="Application Server" type="pay:ApplicationServerNodeType"> <DeploymentArtifacts> <DeploymentArtifact name="ApplicationServerImage" type="http://.../AMIref"> ami-edf2cf99 </DeploymentArtifact> </DeploymentArtifacts> </NodeTemplate> <RelationshipTemplate id="deployed_on" type="pay:deployed_on"> <SourceElement ref="Payroll Application"/> <TargetElement ref="Application Server"/> </RelationshipTemplate> </TopologyTemplate> </ServiceTemplate></Definitions>

TOSCA: Sample Definition

http://www.example.com/ste

http://www.example.com/ste

http://www.example.com/ste/Types




http://www.example.com/ste/Types/PayrollTypes.tosca

http://www.example.com/ste/Types/PayrollTypes.tosca

http://docs.oasis-open.org/tosca/ns/2011/12

http://docs.oasis-open.org/tosca/ns/2011/12

http://www.example.com/ns/tosca/2011/12/DeploymentArtifactTypes/CSARref

http://www.example.com/ns/tosca/2011/12/DeploymentArtifactTypes/CSARref

http://www.example.com/ns/tosca/2011/12/DeploymentArtifactTypes/AMIref

http://www.example.com/ns/tosca/2011/12/DeploymentArtifactTypes/AMIref

— Fails the “ease-of-use” test

— Does not standardise an API

But...

— Powerful and broad modelling coverage

— Workflow, locations, substitution

— Relatively mature

TOSCA

OasisCAMP

www.cloudspecs.org

Cloud Application Management Platform

http://www.oasis-open.org/committees/camp/

http://www.cloudspecs.org

http://www.cloudspecs.org



CAMP: Objectives and Non-Objectives

— Software

— an Orchestration Spec

— a Modelling System

But CAMP is NOT

— a REST API

— for Deployment and Management

— of Applications

It provides a flexible way to use many platforms.Interoperability.

CAMP: Key Concepts

Clients

REST

Platform

PlatformComponent

Template

RequirementType

PlatformComponent

TemplatePlatformComponent

Template

RequirementType

App ComponentType

App ComponentType

Clients Platform

PlatformComponent

Template

RequirementType

PlatformComponent


Template

RequirementType

App ComponentType

App ComponentType

PDP = Platform Deployment Plan

(blueprint)

— Deployment PlanCAMP: Key Concepts

DSL & REST

App ComponentType


(blueprint)

CAMP: Key Concepts

Clients

REST

Platform

PlatformComponent

Template

RequirementType

PlatformComponent


Template

RequirementType

App ComponentType

— Creating Templates

POST

App ComponentTemplate

Clients

REST

Platform

PlatformComponent

Template

RequirementType

PlatformComponent


Template

RequirementType

App ComponentType

App ComponentType

— ResolutionCAMP: Key Concepts

AssemblyTemplate(blueprint)


(blueprint)POST

CAMP: Key Concepts

Clients

REST

Platform

PlatformComponent

Template

RequirementType

PlatformComponent


Template

RequirementType

App ComponentType

App ComponentType


RequirementType

PlatformComponent

Template

RequirementType

App ComponentType

— Resolution


POST


CAMP: Key Concepts

Clients

REST

Platform

PlatformComponent

Template

RequirementType

PlatformComponent


Template

RequirementType

App ComponentType

App ComponentType


— Resolution

200

AssemblyTemplate

POST


CAMP: Key Concepts

Clients

REST

Platform

PlatformComponent

Template

RequirementType

PlatformComponent


Template

RequirementType

App ComponentType

App ComponentType


— Deployment

Assembly(blueprint) Platform

Component

PlatformComponent

App Component


CAMP: Key Concepts

Clients

REST

Platform

PlatformComponent

Template

RequirementType

RequirementType

App ComponentType

App ComponentType


— Deployment

POST

PlatformComponent

Template

PlatformComponent

Template


Component

PlatformComponent

App Component

CAMP: Key Concepts

Clients

REST

Platform

— Deployment


Component

PlatformComponent

App Component


PlatformComponent

Template


PlatformComponent

Template

POST

200

Assembly

CAMP

— depends on orchestrations supporting this

— wants standardised requirement and component types

— still needs a nice DSL

But...

— all very general

— designed to map on to a wide range of orchestrationsPaaS, TOSCA, Brooklyn, Heat

— OSS code in progressgithub.com/brooklyncentral/camp-server

CAMP: PDP/Blueprint DSL (work in progress)

Parameters

Components

Requirements

Functionsparameters & sensors

Outputs

*

*

CAMP: PDP/Blueprint DSL (work in progress)

Parameters

Functionsparameters & sensors

Outputs

*

*

components: my_app: type: WAR content: { href: hello.war } requirements: hosted_on: javaVersion: [1.7,)

Mappings(config metadata)

WaitConditions &WaitHandles

(orchestration)

Parameters

components: my_instance: type: server ram: 2048m requires: connects_to: target: my_database

Heat: New HOT DSL (also work in progress)

Functions

Outputs

HeatOrchestration

Template

Coming inHavana

Wrap-Up

Declarative — AWS CloudFormation and OpenStack HeatProcedural — Apache Whirr and Cloudsoft BrooklynOpenStack Summit — Red Hat, RAX, IBM, OASIS TOSCA and CAMPDSLs in Progress — OpenStack Heat and CAMP PDP

It’s about the apps.Come get involved!

Thank [email protected]

@ahtweetin

2013 05-fite-club-working-models-cloud-growing-up

Technology

aws compatibility

declarative aws cloudformation

aws concepts cumbersome

instanceaws cloudformation

existing ec2 keypair

easier multicloud support

web software

progress openstack heat