autoscaling distributed system with bosh (cloud foundry summit 2014)
DESCRIPTION
Technical Track presented by Yudai Iwasaki, Lead Engineer, Software Innovation Center, NTT. Yudai Iwasaki is a research engineer at NTT Software Innovation Center. He is a core member of the development team of Cloudn PaaS, which is a public PaaS solution provided by NTT Communications, and is leading the development of their deployment system for Cloud Foundry. He is also the leader of community relationships at NTT and is a member of the Japan Cloud Foundry Group, in which capacity he gives lectures on the structure of Cloud Foundry for Japanese Cloud Foundry developers. He is a developer of Nise BOSH, which is a light weight BOSH emulator, BOSH CloudStack CPI, and BOSH AutoScaler.TRANSCRIPT
![Page 1: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/1.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
AUTOSCALING CLOUD FOUNDRY WITH BOSH Yudai Iwasaki NTT Service Innovation Laboratory Group
CF Summit 2014
![Page 2: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/2.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
• Core developer of Cloudn PaaS • Working on Cloud Foundry since 2012
– Nise BOSH, BOSH CloudStack CPI and BOSH AutoScaler • Twitter: @i_yudai
![Page 3: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/3.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Agenda • Problem – System load is not fixed
• Solution – BOSH AutoScaler
• Getting Started
![Page 4: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/4.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Problem: System Load is
Not Fixed
![Page 5: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/5.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Daily, Weekly, Monthly, and Yearly Peaks
![Page 6: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/6.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Daily, Weekly, Monthly, and Yearly Peaks
![Page 7: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/7.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Manually Scaling? Always monitor load?
![Page 8: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/8.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Impossible No one wants to do such a boring job
![Page 9: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/9.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Solution: BOSH AutoScaler
![Page 10: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/10.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
BOSH AutoScaler • Extension for BOSH
• Flexible scaling policies in deployment manifest files
• Special support for Cloud Foundry
![Page 11: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/11.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Architecture Overview
![Page 12: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/12.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics(@buffers) run()
Agent Agent
Agent Agent
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF
CF Component CF Component
CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
![Page 13: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/13.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics(@buffers) run()
Agent Agent
Agent Agent
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF
CF Component CF Component
CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
![Page 14: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/14.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics(@buffers) run()
Agent Agent Agent Agent Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF CF Component
CF Component CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
![Page 15: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/15.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics(@buffers) run()
Agent Agent Agent Agent Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF CF Component
CF Component CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
![Page 16: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/16.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Policy Definitions
![Page 17: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/17.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Defining policies
• Adding “scale” section • Defining conditions for
each job • Various condition classes
to define flexible policies
-‐-‐-‐ name: cf .... scale: jobs: -‐ name: router cooldown: 300 out: limit: 10 unit: 2 conditions: -‐ class: CpuAverage larger_than: 80 duration: 300 -‐ class: MemoryAverage larger_than: 90 duration: 300 in: limit: 3 conditions: -‐ class: CpuAverage smaller_than: 10 duration: 300 -‐ class: MemoryAverage smaller_than: 20 duration: 300
![Page 18: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/18.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Conditions: BOSH Heartbeat • CpuAverage
– Average CPU percentage for duration • MemoryAverage
– Average memory percentage for duration • LoadAverage1
– Latest Load Average in 1 minute • LoadAverage5
– Latest Load Average in 5 minutes • LoadAverage15
– Latest Load Average in 15 minutes
![Page 19: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/19.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Conditions: CF Plugin • CFVarzAverage
– Average Varz value for duration
DEA available_memory_ratio available_disk_ratio
Router total_routes latency.1m
HM9000 NumberOfRunningInstances NumberOfCrashedInstances
etcd SendingRequestRate ReceivingRequestRate
Loggregator Server receivedMessageCount numberOfWebsocketSinks
Cloud Controller connection_count threadqueue.num_waiting
![Page 20: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/20.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Web UI
![Page 21: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/21.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
![Page 22: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/22.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
![Page 23: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/23.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
![Page 24: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/24.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Getting Started It’s easy to plug in
![Page 25: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/25.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
https://github.com/nttlabs/bosh-scaler
![Page 26: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/26.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
-‐-‐-‐ # BOSH NATS settings nats: &nats uri: mbus://192.168.50.4:21084 user: nats pass: nats # BOSH Director REST API settings rest: &rest endpoint_uri: https://192.168.50.4:25555 user: scaler password: scaler disable_verify_certification: true
collectors: -‐ class: BoshNatsCollector bosh_nats: *nats bosh_rest: *rest -‐ class: CfVarzTsdbCollector port: 4567 listeners: -‐ class: BoshScaler bosh_rest: *rest interval: 60 # seconds buffer_size: 1000 ui: enable: true port: 8888
Configuration
![Page 27: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/27.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
... properties: collector: use_tsdb: true deployment_name: cf opentsdb: # your AutoScaler address address: 192.168.15.139 port: 4567
Cloud Foundry Manifest
![Page 28: Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)](https://reader031.vdocuments.us/reader031/viewer/2022020207/54c6dfc24a7959e83e8b456f/html5/thumbnails/28.jpg)
© 2014 Nippon Telegraph and Telephone Corporation
Optimize Running Costs with BOSH AutoScaler