cloud foundry | how it works

Cloud FoundryHow it works

@jacopenKazuto KusamaNTT Communications

Developer of Cloudn PaaS

Introduction

Planned topic was originally,

• Briefly discuss about Cloud Foundry architecture overview

• Talk about the featured changed in V2

But ! Overview slides nearly become 100, gave up to deep dive into it.

So, this material is for beginner

Deep dive is next time！

$ lsGemfile README.md dora.rb instances.rb logging_service.rb spec stress_testers.rbGemfile.lock config.ru get_instance_cookie_jars.sh log_utils.rb scripts stress vendor

https://github.com/cloudfoundry/cf-acceptance-tests/tree/master/assets/dora

Deploy Sinatra application to Cloud Foundry

[23:08] jacopen@cape ~/Project/dora ✘╹◡╹✘ (master) cf push doraCreating app dora in org jacopen / space jacopen-space as admin...OK

Creating route dora.107.22.72.200.xip.io...OK

Binding dora.107.22.72.200.xip.io to dora...OK

Uploading dora...Uploading from: /Users/jacopen/Project/dora1.2M, 36 filesOK

Starting app dora in org jacopen / space jacopen-space as admin...OK（中略）1 of 1 instances running

App started

Showing health and status for app dora in org jacopen / space jacopen-space as admin...OK

requested state: startedinstances: 1/1usage: 256M x 1 instancesurls: dora.107.22.72.200.xip.io

state since cpu memory disk#0 running 2014-04-21 11:09:25 PM 0.0% 72.2M of 256M 0 of 1G

ok

[23:09] jacopen@cape ~/Project/dora ✘╹◡╹✘ (master) cf appsGetting apps in org jacopen / space jacopen-space as admin...OK

name requested state instances memory disk urlsdora started 1/1 256M 1G dora.107.22.72.200.xip.io

cf push

What happened inside CF during this time

Let’s reveal Cloud Foundry black box

Agenda

3 chapters to understand mechanism of Cloud Foundry

• Inspect Cloud Foundry internal from external interface

• Understand role of Cloud Foundry components

• Understand communications between Cloud Foundry components

Not included this time

• Explanation of Warden container and Buildpack• Explanation of services such as MySQL or

Postgress.• Explanation of related services such as

loggregator

Let’s begin

Inspect Cloud Foundry internal from external interface1

CF_TRACE=true[23:49] jacopen@cape ~/Project/dora ✘╹◡╹✘ export CF_TRACE=true[23:49] jacopen@cape ~/Project/dora ✘╹◡╹✘ cf push dora

REQUEST:GET /v2/spaces/dbf6ed0a-3108-45d8-9c78-a512d5071358/apps?q=name%3Adora&inline-relations-depth=1 HTTP/1.1Host: api.107.22.72.200.xip.ioAccept: application/jsonAuthorization: [PRIVATE DATA HIDDEN]Content-Type: application/jsonUser-Agent: go-cli 6.0.0-90db382 / darwin

RESPONSE:HTTP/1.1 200 OKContent-Length: 107Content-Type: application/json;charset=utf-8Date: Mon, 21 Apr 2014 14:49:13 GMTServer: nginxX-Content-Type-Options: nosniffX-Vcap-Request-Id: 18dcd4aefdd000506e49c4b5cf739aaa::65195823-dac9-410a-855a-4ef5ba249198

{ "total_results": 0, "total_pages": 0, "prev_url": null, "next_url": null, "resources": [

]}Creating app dora in org jacopen / space jacopen-space as admin...

CF_TRACE=true : Key to inspect

CF_TRACE=true

Enable to trace requests behind cf command.

Take a look atcf push

GET /v2/spaces/dbf6ed0a-3108-45d8-9c78-a512d5071358/apps?q=name%3Adora&inline-relations-depth=1 HTTP/1.1HTTP/1.1 200 OK

POST /v2/apps?async=true HTTP/1.1{"name":"dora","space_guid":"dbf6ed0a-3108-45d8-9c78-a512d5071358"}HTTP/1.1 201 Created

GET /v2/shared_domains HTTP/1.1HTTP/1.1 200 OK

GET /v2/domains?inline-relations-depth=1&q=name%3A107.22.72.200.xip.io HTTP/1.1HTTP/1.1 200 OK

GET /v2/routes?inline-relations-depth=1&q=host%3Adora%3Bdomain_guid%3A524ba14d-4d3f-495f-a2c7-a5ed25a70e81 HTTP/1.1HTTP/1.1 200 OK

POST /v2/routes?async=true&inline-relations-depth=1 HTTP/1.1{"host":"dora","domain_guid":"524ba14d-4d3f-495f-a2c7-a5ed25a70e81","space_guid":"dbf6ed0a-3108-45d8-9c78-a512d5071358"}HTTP/1.1 201 Created

PUT /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/routes/ed912294-be29-4013-8b1c-735851acee24 HTTP/1.1HTTP/1.1 201 Created

PUT /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/bits?async=true HTTP/1.1Content-Type: multipart/form-data;HTTP/1.1 201 Created

GET /v2/jobs/6dd46fe7-40b0-41d1-94aa-dd2ee80fd3b2 HTTP/1.1HTTP/1.1 200 OK(queued)

GET /v2/jobs/6dd46fe7-40b0-41d1-94aa-dd2ee80fd3b2 HTTP/1.1HTTP/1.1 200 OK(finished)

CONNECTING TO WEBSOCKET: wss://loggregator.107.22.72.200.xip.io:443/tail/?app=0e948568-aa90-47e9-a128-9ff3cc680600

PUT /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600?async=true&inline-relations-depth=1 HTTP/1.1{"state":"STARTED"}

GET /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/instances HTTP/1.1HTTP/1.1 400 Bad Request{"code":170002,"description":"App has not finished staging","error_code":"CF-NotStaged"}


GET /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/instances HTTP/1.1HTTP/1.1 200 OK{"0":{"state":"RUNNING","since":1398092703.1739626,"debug_ip":null,"debug_port":null,"console_ip":null,"console_port":null}}

GET /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/stats HTTP/1.1HTTP/1.1 200 OK{"0":{"state":"RUNNING","stats":{"name":"dora","uris":["dora.107.22.72.200.xip.io"],"host":"10.244.0.26","port":61026,"uptime":3,"mem_quota":268435456,"disk_quota":1073741824,"fds_quota":16384,"usage":{"time":"2014-04-21 15:05:04 +0000","cpu":0,"mem":75673600,"disk":0}}}}



GET /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/summary HTTP/1.1HTTP/1.1 200 OK

GET /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/instances HTTP/1.1HTTP/1.1 200 OK

GET /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/stats HTTP/1.1HTTP/1.1 200 OK

Communicating something to api.107.22.72.200.xip.io

Things found out

cf push

Cloud Foundry provides API

Something which provides API

cf push

cf push deploy using various APIs

GET xxxx

PUT xxxx

POST xxxx

GET xxxx

Something which runs application


Let’s check access point

API

api.107.22.72.200.xip.io

Application

dora.107.22.72.200.xip.io

Uhm？

$ nslookup api.107.22.72.200.xip.ioServer: 192.168.11.1Address: 192.168.11.1#53

Non-authoritative answer:api.107.22.72.200.xip.io canonical name = api.1jkk1uz.xip.io.Name: api.1jkk1uz.xip.ioAddress: 107.22.72.200

$ nslookup dora.107.22.72.200.xip.ioServer: 192.168.11.1Address: 192.168.11.1#53

Non-authoritative answer:dora.107.22.72.200.xip.io canonical name = dora.1jkk1uz.xip.io.Name: dora.1jkk1uz.xip.ioAddress: 107.22.72.200

IP addresses are same for both API and App

cf push

Something to dispatch access based on URL

Som

eth

ing

to d

isp

atc

h a

ccess

Browser



cf scale

$ cf scale -i 3 -m 256M doraScaling app dora in org jacopen / space jacopen-space as admin...OK

cf scale command changes “number of active

instances” and/or “memory size”.

Som

eth

ing

to d

isp

atc

h a

ccess

cf push

Browser



Something to dispatch access based on URL

What happens when application die ?

dora/sigterm/KILL

Kill itself using sigterm/KILL

Port number has been changed.

Something to monitor application health status

Monitor Health

Som

eth

ing

to d

isp

atc

h a

ccess


Result

External view of Cloud Foundry

Somethingto run App

Monitor Health

Som

eth

ing

to d

isp

atc

h a

ccess


Cloud Foundry components (part of it)

DEAHealth

Manager

Router

Cloud Controller

Understand role of Cloud Foundry components ２

component＝？

Cloud Controller

Router

DEAHealth

Manager

Component = Application

Cloud Controller(Ruby)

Router(Golang)

DEA(Ruby)

HealthManager(Golang)

All components can run on 1VM


Router(Golang)

DEA(Ruby)


VM

(Generally) In real situation, separate VM for each components


Router(Golang)

DEA(Ruby)


VM

VM

VM VM

Let’s take a look at each components

Router

Cloud Controller

Router

DEAHealth

Manager

Router

Based on URL, dispatches access to appropriate component. Similar to L7 load balancer.

This is different from “router” in network equipment.

It is implemented as an application (called Gorouter) which runs on Ubuntu. Written in Go.

Cloud Controller

Router

DEAHealth

Manager

router.register

Each component send “router.register” message to Router

Send

api.107.22.72.200.xip.io to

10.244.0.138:9022

Send dora.107.22.72.200.xip.io

to 10.244.0.26:61032

Cloud Controller

Router

DEAHealth

Manager

router.register

Router dispatches accesses based on collected information

dora.107.22.72.200.xip.io => 10.244.0.26:61032api.107.22.72.200.xip.io => 10.244.0.138:9022

api.107.22.72.200.xip.io

dora.107.22.72.200.xip.io

Cloud Controller

Router

DEAHealth

Manager

router.register

Multiple targets are handled for the same URL

dora.107.22.72.200.xip.io => 10.244.0.26:61032dora.107.22.72.200.xip.io => 10.244.0.27:58719api.107.22.72.200.xip.io => 10.244.0.138:9022api.107.22.72.200.xip.io => 10.244.0.139:9022

api.107.22.72.200.xip.io

dora.107.22.72.200.xip.io

Cloud Controller

DEA

Cloud Controller

Cloud Controller

Router

DEAHealth

Manager

Cloud Controller

A component to provide API.

Receive application from cf command, etc.

Control application start/stop against DEA.

Control creation of service such as database.

Etc, etc. Control overall Cloud Foundry.

POST /v2/apps?async=true HTTP/1.1{"name":"dora","space_guid":"dbf6ed0a-3108-45d8-9c78-a512d5071358"}HTTP/1.1 201 Created

GET /v2/shared_domains HTTP/1.1HTTP/1.1 200 OK

GET /v2/domains?inline-relations-depth=1&q=name%3A107.22.72.200.xip.io HTTP/1.1HTTP/1.1 200 OK

GET /v2/routes?inline-relations-depth=1&q=host%3Adora%3Bdomain_guid%3A524ba14d-4d3f-495f-a2c7-a5ed25a70e81 HTTP/1.1HTTP/1.1 200 OK

POST /v2/routes?async=true&inline-relations-depth=1 HTTP/1.1{"host":"dora","domain_guid":"524ba14d-4d3f-495f-a2c7-a5ed25a70e81","space_guid":"dbf6ed0a-3108-45d8-9c78-a512d5071358"}HTTP/1.1 201 Created

PUT /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/routes/ed912294-be29-4013-8b1c-735851acee24 HTTP/1.1HTTP/1.1 201 Created

PUT /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/bits?async=true HTTP/1.1Content-Type: multipart/form-data;HTTP/1.1 201 Created

GET /v2/jobs/6dd46fe7-40b0-41d1-94aa-dd2ee80fd3b2 HTTP/1.1HTTP/1.1 200 OK(queued)

GET /v2/jobs/6dd46fe7-40b0-41d1-94aa-dd2ee80fd3b2 HTTP/1.1HTTP/1.1 200 OK(finished)

CONNECTING TO WEBSOCKET: wss://loggregator.107.22.72.200.xip.io:443/tail/?app=0e948568-aa90-47e9-a128-9ff3cc680600

PUT /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600?async=true&inline-relations-depth=1 HTTP/1.1{"state":"STARTED"}







GET /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/summary HTTP/1.1HTTP/1.1 200 OK

GET /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/instances HTTP/1.1HTTP/1.1 200 OK

GET /v2/apps/0e948568-aa90-47e9-a128-9ff3cc680600/stats HTTP/1.1HTTP/1.1 200 OK

⇐ Create app

⇐ Obtain domain

⇐verify routes

⇐create routes

⇐ bind app and routes

⇐upload source files

⇐start app

DEA

Cloud Controller

Router

DEAHealth

Manager

DEA

A component to run user’s application.

Abbreviation of “Droplet Execution Agent"

In CF V2, it is responsible for user application staging

(e.g. Ruby bundle install).

DEA uses Linux container called Warden to run user

application. (Warden is not described this time)

Receive source files from client

Cloud Controller

Router

DEAHealth

Manager

cf push

Gemfilelib/bin/config.ruapp.rb

http://config.ru/

Staging request (staging.start)

Cloud Controller

Router

DEAHealth

Manager

cf push

Gemfilelib/bin/config.ruapp.rb

GemfileGemfile.lockvendor/lib/bin/config.ruapp.rb

http://config.ru/

http://config.ru/

Store Droplet

Cloud Controller

Router

DEAHealth

Manager

cf push


http://config.ru/

Start request (dea.start)

Cloud Controller

Router

DEAHealth

Manager

cf push



http://config.ru/

http://config.ru/

Start Request(dea.start)

Cloud Controller

Router

DEAHealth

Manager

cf push


http://config.ru/

Cloud Controller

Router

DEAHealth

Manager

cf scale -i 3


DEA

http://config.ru/

Start request(dea.start)

Cloud Controller

Router

DEA

Health

Manager

cf scale -i 3


DEA

http://config.ru/

Start request(dea.start)

Cloud Controller

Router

DEAHealth

Manager

cf scale -i 3


DEA

http://config.ru/

Health Manager

Cloud Controller

Router

DEAHealth

Manager

DEA

Health Manager

A component to monitor user application to see if the

status is “what it should be”.

The latest version is “hm9000” written in Go.

Suddenly instance dies

Cloud Controller

Router

DEAHealth

Manager

DEA

droplet.exited

Cloud Controller

Router

DEAHealth

Manager

DEA

hm9000.start

Cloud Controller

Router

DEAHealth

Manager

DEA

dea.start

Cloud Controller

Router

DEAHealth

Manager

DEA

When the number of instances are over…

Cloud Controller

Router

DEAHealth

Manager

DEA

dea.heartbeat

Cloud Controller

Router

DEAHealth

Manager

DEA

I got 4, it should be 3 ….

hm9000.stop

Cloud Controller

Router

DEAHealth

Manager

DEA

dea.stop

Cloud Controller

Router

DEAHealth

Manager

DEA

Did you understand the each relationship？

Cloud Controller

Router

DEAHealth

Manager

Internal Communication

External Access

Understand communications between Cloud Foundry components３

The last chart

Cloud Controller

Router

DEAHealth

Manager

Internal Communication

External Access

Cloud Controller

Router

DEAHealth

Manager

NATS

In reality

NATS

Publish-Subscribe model lightweight messaging

Developed by Derek Collison designed and architected

the industry's first Open PaaS, Cloud Foundry

Exists at the beginning of Cloud Foundry. Key component

in the architecture.

It was implemented in Ruby using EventMachine. It is re-

written using Go language (dnatsd)

Publish-Subscribe Model

Publisher Subscriber

NATS

Subject: foo.bar

Subject: foo.bar



NATS

Subject: foo.bar{“message”: “hogefuga”}

{“message”: “hogefuga”}

{“message”: “hogefuga”}



NATS

Cloud Controller

DEA

Router

Subject: router.register {“host":"10.244.0.138","port":9022,"uris":["api.107.22.72.200.xip.io"]}

Subject: router.register {“host":"10.244.0.111","port":37902,"uris":["dora.107.22.72.200.xip.io"]}

All subscriber receive the same message

NATS

Cloud Controller

DEA

Router

Subject: router.register {“host":"10.244.0.138","port":9022,"uris":["api.107.22.72.200.xip.io"]}

Subject: router.register {“host":"10.244.0.111","port":37902,"uris":["dora.107.22.72.200.xip.io"]}

Router

Router


When there is no NATS component…

NATS Direct Communication

1.Run Router

2.Router subscribe router.register

1.Register new Router in admin DB

2.Run Router

3.DEA or Cloud Controler obtain Router list from admin DB

4.Sead message to each Router

Cloud Controller

Router

DEA HealthManager

NATSRouter

Router

Cloud Controller

DEA

DEA

DEA

Cloud Controller

Cloud Controller

Cloud Foundry Architecture

• Centered around NATS, loosely coupled component communications. Scalable.

• Self-organized distributed system. Administrative information is minimal.

• Self healing. The point is that it heals by itself.

• Removes SPoF( Single Point of Failure). Except recent NATS.

Cloud Foundry V1What’s the difference ?

Difference between CF v1 and v2

Overall architecture has not been changed.

DEA became DEAng, and runs Linux container called Warden

Many components were re-written in Go.

Can use Heroku Buildpack

There is no API compatibility !

We will deep dive into Cloud Foundry next time.

End Note.

This material was translated by @ibmamnt

From original chart by @jacopen

http://www.slideshare.net/jacopen/cloud-foundry-33851040

cloud foundry | how it works

Technology

true http1

9ff3cc680600instances

adorain http1

spac http1

9ff3cc680600stats http1

3a5 http1

domains http1

aadd2ee80fd3b2 http1