large-scale deployments with pogo at yahoo!
DESCRIPTION
Large-Scale Deployments With Pogo At Yahoo!. Mike Schilli YAPC::NA 06/14/2012. What’s Pogo?. Yahoo’s deployment system Highly scalable (thousands of targets) Highly parallelized rollouts In active use for several years Open Source version available (pre-alpha). Pogo in a CI environment. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/1.jpg)
Large-Scale Deployments With Pogo At Yahoo!
Mike SchilliYAPC::NA 06/14/2012
![Page 2: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/2.jpg)
What’s Pogo?
• Yahoo’s deployment system– Highly scalable (thousands of targets)– Highly parallelized rollouts– In active use for several years
• Open Source version available (pre-alpha)
![Page 3: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/3.jpg)
Pogo in a CI environment
![Page 4: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/4.jpg)
What’s “Deployment”?
• Run one command on many hosts• Each host “knows” what it’s supposed to look
like via a central database• One command brings host into “desired” state• Fetches package updates from edge-server
repos
![Page 5: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/5.jpg)
Run one command on many hosts
• Act on all hosts immediately
![Page 6: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/6.jpg)
Run one command on many hosts
• Act on all hosts immediately
![Page 7: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/7.jpg)
Example Server Farm
![Page 8: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/8.jpg)
Pogo Features
• Sequences• Parallelizes with constraints• Halts if errors violate constraints• Runs health checks (pre/post hooks)• Command agnostic (rpm, apt-get, custom,
etc.)
![Page 9: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/9.jpg)
Server Farm Release via Pogo
![Page 10: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/10.jpg)
Server Farm Release via Pogo
![Page 11: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/11.jpg)
Server Farm Release via Pogo
![Page 12: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/12.jpg)
Server Farm Release via Pogo
![Page 13: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/13.jpg)
Server Farm Release via Pogo
![Page 14: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/14.jpg)
Server Farm Release via Pogo
![Page 15: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/15.jpg)
Server Farm Release via Pogo
![Page 16: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/16.jpg)
Server Farm Release via Pogo
![Page 17: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/17.jpg)
Server Farm Release via Pogo
![Page 18: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/18.jpg)
Server Farm Release via Pogo
![Page 19: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/19.jpg)
Server Farm Release via Pogo
![Page 20: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/20.jpg)
Pogo Configuration
![Page 21: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/21.jpg)
Run a Pogo job (tags)
![Page 22: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/22.jpg)
Pogo UI – Single Job View
![Page 23: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/23.jpg)
Pogo UI Overview
![Page 24: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/24.jpg)
Pogo UI – Host View
![Page 25: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/25.jpg)
Pogo UI – Start it Up
$ perl -Ilib bin/pogo-apipogo-api-28> Listening on 0.0.0.0:7657
$ perl -Ilib bin/pogo-ui -vpogo-ui-22> Listening on port 5000
![Page 26: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/26.jpg)
Run a Pogo job (targets)
![Page 27: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/27.jpg)
Pogo Configuration
![Page 28: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/28.jpg)
State of Pogo
• Pre-alpha Open Source version on Github
• OSS Pogo will be completed in the coming months
![Page 29: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/29.jpg)
Pogo on Github
• Developed in the open:– https://github.com/ytoolshed/pogo
• Internally used with plugins to adapt to specific data sources
![Page 30: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/30.jpg)
CI Tests with travis-ci.org
![Page 31: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/31.jpg)
The Gory Details
![Page 32: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/32.jpg)
Pogo – User View
![Page 33: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/33.jpg)
Pogo Workflow
![Page 34: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/34.jpg)
Authentication/Authorization
• Web server authentication for submitting jobs• Target authentication via passwords or
priv/pub keys• Credentials encrypted with worker pubkey• Credentials never stored persistently (or
ZooKeeper)
![Page 35: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/35.jpg)
Pogo “Hooks”
• Scripts that run before or after Pogo commands
• Used to prepare host, perform check health, or other functions
• Installed locally on target hosts• rc.d style invocation: run in alphanumeric
order
![Page 36: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/36.jpg)
Pogo “Hooks” (cont.)
• Two types:– pre hook: run before the specified command– post hook: run after the specified command
• If any hook script fails:– No further scripts or commands run– Host is marked as failed
![Page 37: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/37.jpg)
Pre-hook Examples
• Take a host out of rotation• Drain database connections• Check that host is ready to be updated
![Page 38: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/38.jpg)
Post-hook Examples
• Check whether local webserver is serving expected content
• Ping required services• Put host back into rotation
![Page 39: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/39.jpg)
Host Failures and Concurrency
• If a host fails:– Command returns non-zero– Pre- or post-hook returns non-zero
• The failed host counts against the acceptable number/percent of unavailable hosts
• Example:– Four hosts in an app can run at a time– One host in the app fails– Only three hosts will be acted on at a time going forward
![Page 40: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/40.jpg)
Implementation
• AnyEvent Framework• Asynchronous, event-based• Added Object::Event for event pub/sub
![Page 41: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/41.jpg)
HTTP Client in AnyEvent
![Page 42: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/42.jpg)
HTTP Client in AnyEvent
![Page 43: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/43.jpg)
HTTP Client in AnyEvent
![Page 44: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/44.jpg)
AnyEvent and Object::Event
![Page 45: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/45.jpg)
AnyEvent and Object::Event
![Page 46: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/46.jpg)
AnyEvent and Object::Event
![Page 47: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/47.jpg)
Queue Processor with AnyEvent and Object::Event
![Page 48: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/48.jpg)
AnyEvent
bin/pogo-dispatcher
![Page 49: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/49.jpg)
Testing AnyEvent Components
![Page 50: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/50.jpg)
All Pogo Components
![Page 51: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/51.jpg)
All-In-One Pogo: pogo-one
![Page 52: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/52.jpg)
All-In-One Pogo: pogo-one
![Page 53: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/53.jpg)
All-In-One Pogo: pogo-one
![Page 54: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/54.jpg)
Questions?
![Page 55: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/55.jpg)
Thanks!
• Pogo on Github: – http://github.com/ytoolshed/pogo
![Page 56: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/56.jpg)
Lessons learned along the Way
![Page 57: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/57.jpg)
Set up Unit Tests with Travis-ci.org
![Page 58: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/58.jpg)
Runs immediately after Github checkins
![Page 59: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/59.jpg)
CI Tests with travis-ci.org
![Page 60: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/60.jpg)
Configure CI with travis-ci.org
![Page 61: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/61.jpg)
Select Github project
![Page 62: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/62.jpg)
Authorize travis-ci.org
![Page 63: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/63.jpg)
Travis-ci.org API Calls
![Page 64: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/64.jpg)
Filling in Passwords
![Page 65: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/65.jpg)
Filling in Passwords
• Test logging into machines without sshkeys• Convenient for testing• Answer to sudo prompts on targets• Answer to prompts for encrypted packages
![Page 66: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/66.jpg)
Filling in Passwords
• Good password prompt:
mschilli@localhost's password:
• Not password prompt!
… user ''@'localhost' (using password: NO)
![Page 67: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/67.jpg)
PasswordMonkey on CPAN
![Page 68: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/68.jpg)
Plack
![Page 69: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/69.jpg)
Plack
![Page 70: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/70.jpg)
Plack/PSGI
• Plack used for server-independent web application– API– Dispatcher Control Port
![Page 71: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/71.jpg)
Alternatives
• Capistrano• Rundeck• Aegir• Drush
![Page 72: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/72.jpg)
Pogo UI – Start it Up
$ perl -Ilib bin/pogo-apipogo-api-28> Listening on 0.0.0.0:7657
$ perl -Ilib bin/pogo-ui -vpogo-ui-22> Listening on port 5000
![Page 73: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/73.jpg)
Pogo UI
• All Cient Side-Javascript• Server just redirects to index.html– Except assets/* js/*
![Page 74: Large-Scale Deployments With Pogo At Yahoo!](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681688a550346895ddf099c/html5/thumbnails/74.jpg)
Thanks!
• Pogo on Github: – http://github.com/ytoolshed/pogo
• PasswordMonkey CPAN Module:– http://search.cpan.org/dist/PasswordMonkey/
• Travis CI Service: – http://travis-ci.org