lessions from building a high available cloud foudry on top of open stack

Lessions from Building a High Available Cloud

Foudry on top of OpenStack

1

Hello World!Jiang Yi Tao

IBM CDL Bluemix IaaS PaaS

2

Agenda• Open Stack

• Nova Neutron Cinder Swift

• Cloud Foundry

• LB Gorouter

• Database

• Backup

3

Releases

• CF 233

• OpenStack Kilo

4

OpenStack

5

OpenStack Intergration

6

• Image

• Router

• network, subnet

• Flavor

• Keypair

• Security Group

OpenStack Create VM

7

How to Deploy CF1.Target a bosh director using cli

2.Upload a stemcell

3.Upload a release

4.Gererate a manifest

5.bosh deploy

5.1.prepare deployment

5.2.complie the packages

5.3.create and bind vm

5.4.pull in job configurations

5.5.create needed job instances

8

OpenStack-CPI

• OpenStack CPI is an implementation of the BOSH CPI

• Leverages the Fog Ruby gem for OpenStack

http://fog.io

9

OpenStack Ready?

• Fog Version?

• bosh openstack cpi

• Validate OpenStack

• https://docs.cloudfoundry.org/deploying/openstack/validate_openstack.html

10

OpenStack Ready?(cont.)gem install XXX -v XXX

require 'fog'

EXCON_DEBUG=true

nova=Fog::Compute.new({:provider => 'openstack', :openstack_tenant => 'jiangytcn', :openstack_api_key => 'XXXX', :openstack_username => 'jiangytcn', :openstack_auth_url => 'https://XXX.cn.ibm.com:35357/v3/auth/tokens', :openstack_domain_name => 'default', :connection_options => { :ssl_verify_peer => false}})

nova.list_servers

nova.servers.create(name = 'trusty', image_ref = image, flavor_ref = flavor, key_name = my_key)

Similar for Volumes, Networks, Images, etc.

11

Nova• QEMU KVM

• Over Commit

• Flavor

• ephemeral >= 2* ram If has ephemeral

• Security Groups

• Only open required ports, remote CIDR/SG

• VM runs OK?

• ping 127.0.0.1

• (Compute Node) watch 'tail /sys/kernel/mm/ksm/*'

12

Resource Pool• Scheduler hints

• AZ Affinity-Group Anti-affinity-group

• No SPOF

• Need more physical resource

13

Neutron• Overlap

• Vlan gre vxlan

• Isolated

• IP translation, controlled

• Flat

• Flat-dhcp

• Shared

• Directly to switch✤MTU Selection

PMTU

ping -D -g <minsize> -G <maxsize> -h <sizeincr> <host>

✤ JUMBO FRAMES

✤meta-data/user-data

14

Cinder• Backend

• LVM CEPH ...

• Disk Readonly *

15

Swift - BlobStore• user-uploaded applications, buildpacks, droplets, and

application resources

• Tempurl

• X-Account-Meta-Temp-URL-Key

• runner download packages using tempurl

• Ratelimit(Optional. If no account or container limits are provided there will be no rate limiting)

• NTP

16

Keystone

• Tenant/Project with Large Quotas

• Day one & Day two

17

Deployment• keystone

• v2 or v3

• network type

• manual VS dynamic VS static

18

Cloud Foundry?

19

CF push workflow

20

Grouter & Load Balancer• Domains

• system, apps

• Gorouter only support single certs

• TLS Pass-Through

• Performant and a single TLS connection

• TLS Termination at Load Balancer

• Less performant option

• Multiple certificates to be used

• Internal

• Haproxy

• External LB

• Third party

• LBaaS

21

BlobStore• NFS

• Webdav

• Swift

22

Cloud Controller• resources clean up

app_events:

cutoff_age_in_days: 36

app_usage_events:

cutoff_age_in_days: 36

audit_events:

cutoff_age_in_days: 36

failed_jobs:

cutoff_age_in_days: 7

23

Database• Cluster

• Postgresql

• Mysql

• Backup

• Transaction Logs

• Full backup

24

DB Encrypt/Decrypt

25

CCDB.apps

DB Encrypt/Decrypt• Cloud Controller

• db_encryption_key

• row level encrypt, salt

26

def encrypt(input, salt)

return nil unless input

Base64.strict_encode64(run_cipher(make_cipher.encrypt, input, salt))

end

def decrypt(encrypted_input, salt)

return nil unless encrypted_input

run_cipher(make_cipher.decrypt, Base64.decode64(encrypted_input), salt)

end

def make_cipher

OpenSSL::Cipher::Cipher.new(ALGORITHM)

end

def run_cipher(cipher, input, salt)

cipher.pkcs5_keyivgen(db_encryption_key, salt)

cipher.update(input).tap { |result| result << cipher.final }

end

27

DB Encrypt/Decrypt

28

Buildpacks• application runtime

• installed buildpack

• system build pack will be override every update

29

Backing up Cloud Foundry• Platform

• BOSH DB • configuration files • CCDB UAADB • Blobstore • Credentials

• User • apps • service instance

30

Bosh DB• DNS

• for internal use

• VM and Volume mapping

31

Configuration• deployment manifest

bosh download manifest <DEPLOYMENT> cf.yml

kept it

• key pairs, certs

32

33

You can find me at: • @jiangytcn • jiangyt.cn@gmail.com

Validate OpenStack

core component vm metadata service

Cloud Foundry Network Type Data encrypt/decrypt

Backup DB services manifest