building scalable applications while scaling your infrastructure by rhommel lamas
DESCRIPTION
"Building scalable applications while scaling your infrastructure" presented at Puppet Camp Munich 2013 by Rhommel Lamas 3scale NetworksTRANSCRIPT
![Page 1: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/1.jpg)
November 2013
Puppet at scale
Rhommel Lamas
Thursday, November 28, 13
![Page 2: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/2.jpg)
Who am I?
Thursday, November 28, 13
![Page 3: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/3.jpg)
Scale experience•Physical servers Xtratelecom.es.•AWS and other cloud providers at Wuaki.tv and 3scale.net.
Who am I?
Thursday, November 28, 13
![Page 4: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/4.jpg)
Scale experience•Physical servers Xtratelecom.es.•AWS and other cloud providers at Wuaki.tv and 3scale.net.
Who am I?
Config Management• Worked with CFEngine2 for 2 years.• Puppet user since 2009.
Thursday, November 28, 13
![Page 5: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/5.jpg)
Scale experience•Physical servers Xtratelecom.es.•AWS and other cloud providers at Wuaki.tv and 3scale.net.
Who am I?
Config Management• Worked with CFEngine2 for 2 years.• Puppet user since 2009.
Obsessed about config management and I use Emacs.
Thursday, November 28, 13
![Page 6: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/6.jpg)
What is 3scale?3scale provides tools such as traffic
management, API administration, traffic reports, and more, to help companies
publicize their APIs easily and power new types of connected applications.
Thursday, November 28, 13
![Page 7: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/7.jpg)
What does scaling stands for?
Thursday, November 28, 13
![Page 8: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/8.jpg)
Scalability is the ability of a system, network, or process to handle a growing amount of
work in a capable manner or its ability to be enlarged to accommodate that growth.
Wikipedia
What does scaling stands for?
Thursday, November 28, 13
![Page 9: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/9.jpg)
What are our scaling needs?
Thursday, November 28, 13
![Page 10: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/10.jpg)
• How many people do we need to scale our system infrastructure?
What are our scaling needs?
Thursday, November 28, 13
![Page 11: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/11.jpg)
• How many people do we need to scale our system infrastructure?
• How many environments and systems can our team maintain?
What are our scaling needs?
Thursday, November 28, 13
![Page 12: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/12.jpg)
• How many people do we need to scale our system infrastructure?
• How many environments and systems can our team maintain?
• Can we delegate configurations easily to our team members?
What are our scaling needs?
Thursday, November 28, 13
![Page 13: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/13.jpg)
Puppet dynamic environments
Thursday, November 28, 13
![Page 14: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/14.jpg)
• Purpose of Puppet environments.
Puppet dynamic environments
Thursday, November 28, 13
![Page 15: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/15.jpg)
• Purpose of Puppet environments.
• Puppet base environments at 3scale.
Puppet dynamic environments
Thursday, November 28, 13
![Page 16: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/16.jpg)
/etc/puppet
auth.conf
autosign.conf
config.ru
hiera.yaml
hipchat.yaml
fileserver.conf
unicorn.rb
environments
Puppet environments at 3scaleenvironments
production
manifests
production.pp
preview.pp
nodes
production
webserver.pp
preview
webserver.pp
modules
nginx
hieradata
common.yaml
production.yaml
preview.yaml
Capfile
preview
staging
Thursday, November 28, 13
![Page 17: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/17.jpg)
[main] server = puppet.example.com environment = production[master] environment = production manifest = $confdir/environments/$environment/manifests/site.pp modulepath = $confdir/environments/$environment/modules
Thursday, November 28, 13
![Page 18: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/18.jpg)
[main] server = puppet.example.com environment = production[master] environment = production manifest = $confdir/environments/$environment/manifests/$environment.pp modulepath = $confdir/environments/$environment/modules
# production.pp
import 'nodes/production/nodes.pp'import 'nodes/production/*.pp
Thursday, November 28, 13
![Page 19: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/19.jpg)
:hierarchy: - '%{::osfamily}' - '%{::environment}' - common:backends: - yaml - json:yaml: :datadir: "/etc/puppet/environments/%{environment}/hieradata"
Thursday, November 28, 13
![Page 20: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/20.jpg)
Thursday, November 28, 13
![Page 21: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/21.jpg)
• Purpose of Puppet environments.
• Puppet environments at 3scale.
• Puppet development workflow (Git).• Puppet-lint• Puppet-rspec• Code Review
Puppet dynamic environments
Thursday, November 28, 13
![Page 22: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/22.jpg)
Thursday, November 28, 13
![Page 23: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/23.jpg)
Thursday, November 28, 13
![Page 24: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/24.jpg)
Thursday, November 28, 13
![Page 25: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/25.jpg)
Puppet at 3scale.
Thursday, November 28, 13
![Page 26: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/26.jpg)
• Since July 2013 we are running Puppet 3.0+.
Puppet at 3scale.
Thursday, November 28, 13
![Page 27: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/27.jpg)
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.
Puppet at 3scale.
Thursday, November 28, 13
![Page 28: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/28.jpg)
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.• Nginx + Unicorn.
Puppet at 3scale.
Thursday, November 28, 13
![Page 29: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/29.jpg)
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.• Nginx + Unicorn.• 15 minutes splay with a splaylimit $runinterval/2.
Puppet at 3scale.
Thursday, November 28, 13
![Page 30: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/30.jpg)
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.• Nginx + Unicorn.• 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records.
Puppet at 3scale.
Thursday, November 28, 13
![Page 31: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/31.jpg)
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.• Nginx + Unicorn.• 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records.• The foreman as our ENC.
Puppet at 3scale.
Thursday, November 28, 13
![Page 32: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/32.jpg)
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.• Nginx + Unicorn.• 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records.• The foreman as our ENC.• Puppetdb and hiera.
Puppet at 3scale.
Thursday, November 28, 13
![Page 33: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/33.jpg)
• Since July 2013 we are running Puppet 3.0+.
• 2 Puppet master’s.• Nginx + Unicorn.• 15 minutes splay with a splaylimit $runinterval/2. • Using DNS SRV records.• The foreman as our ENC.• Puppetdb and hiera.
•MCollective with Rabbitmq connector.
Puppet at 3scale.
Thursday, November 28, 13
![Page 34: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/34.jpg)
ROUTE53
_x-puppet._tcp.3scale.net_x-puppet-ca._tcp.3scale.net
PuppetMaster
+RabbitMQ
Cluster+
MCollectiveClient
+PuppetDB
+The
Foreman
RabbitMQ Load Balancer
PostgreSQLRDS
MultiAZ
PuppetMaster
+RabbitMQ
Cluster+
MCollectiveClient
+PuppetDB
+The
Foreman
PuppetMaster
+RabbitMQ
Cluster+
MCollectiveClient
+PuppetDB
+The
Foreman
Puppet Agent
+ Mcollective
Server
Puppet Agent
+ Mcollective
Server
Puppet Agent
+ Mcollective
Server
Thursday, November 28, 13
![Page 35: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/35.jpg)
Thursday, November 28, 13
![Page 36: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/36.jpg)
What nobody will tell you about puppet.
Thursday, November 28, 13
![Page 37: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/37.jpg)
• Puppet-lint is your friend and it will help others to understand your code, so USE IT.
What nobody will tell you about puppet.
Thursday, November 28, 13
![Page 38: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/38.jpg)
• Puppet-lint is your friend and it will help others to understand your code, so USE IT.• Puppet-rspec will help you find errors on early stages so TEST your modules.
What nobody will tell you about puppet.
Thursday, November 28, 13
![Page 39: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/39.jpg)
• Puppet-lint is your friend and it will help others to understand your code, so USE IT.• Puppet-rspec will help you find errors on early stages so TEST your modules.• Don’t code remotely on your servers, use your local development environment and git.
What nobody will tell you about puppet.
Thursday, November 28, 13
![Page 40: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/40.jpg)
• Puppet-lint is your friend and it will help others to understand your code, so USE IT.• Puppet-rspec will help you find errors on early stages so TEST your modules.• Don’t code remotely on your servers, use your local development environment and git.• Read puppet documentation on every version.
What nobody will tell you about puppet.
Thursday, November 28, 13
![Page 41: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/41.jpg)
• Puppet-lint is your friend and it will help others to understand your code, so USE IT.• Puppet-rspec will help you find errors on early stages so TEST your modules.• Don’t code remotely on your servers, use your local development environment and git.• Read puppet documentation on every version.• Be careful about dependencies.
What nobody will tell you about puppet.
Thursday, November 28, 13
![Page 42: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/42.jpg)
• Puppet-lint is your friend and it will help others to understand your code, so USE IT.• Puppet-rspec will help you find errors on early stages so TEST your modules.• Don’t code remotely on your servers, use your local development environment and git.• Read puppet documentation on every version.• Be careful about dependencies.• Once you start to puppetize you become an Addict.
What nobody will tell you about puppet.
Thursday, November 28, 13
![Page 43: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/43.jpg)
Rhommel LamasTwitter: @rhoml
Questions?at 3scale
we’re hiringhttp://3scale.net/jobs
Thursday, November 28, 13
![Page 44: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/44.jpg)
Rhommel LamasTwitter: @rhoml
Thank you.at 3scale
we’re hiringhttp://3scale.net/jobs
Thursday, November 28, 13
![Page 45: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/45.jpg)
Thursday, November 28, 13
![Page 46: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/46.jpg)
Useful links
Thursday, November 28, 13
![Page 47: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/47.jpg)
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records
Useful links
Thursday, November 28, 13
![Page 48: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/48.jpg)
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html
Useful links
Thursday, November 28, 13
![Page 49: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/49.jpg)
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html• http://puppetlabs.com/blog/git-workflow-and-puppet-environments
Useful links
Thursday, November 28, 13
![Page 50: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/50.jpg)
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html• http://puppetlabs.com/blog/git-workflow-and-puppet-environments• http://projects.puppetlabs.com/projects/1/wiki/using_unicorn
Useful links
Thursday, November 28, 13
![Page 51: Building scalable applications while scaling your infrastructure by rhommel lamas](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f3dfab4c90572088b5113/html5/thumbnails/51.jpg)
• http://docs.puppetlabs.com/guides/scaling_multiple_masters.html#option-4-dns-srv-records• http://docs.puppetlabs.com/puppetdb/latest/install_via_module.html• http://puppetlabs.com/blog/git-workflow-and-puppet-environments• http://projects.puppetlabs.com/projects/1/wiki/using_unicorn• https://github.com/cernops/puppetdb_foreman
Useful links
Thursday, November 28, 13