infrastructure as code with chef / puppet
DESCRIPTION
TRANSCRIPT
Infrastructure as Code with Chef / Puppet
Edmund Haselwanter ([email protected])
Samstag, 16. November 13
cloudbau
Edmund Haselwanter
●Co-Founder and CSO cloudbau GmbH
●Dell Professional Services (Openstack/Crowbar/Automation)
●So!ware Development Background
●10 Years Infrastructure Automation
●@ehaselwanter on twitter/github/irc/...
Introduction
Samstag, 16. November 13
cloudbau
Business-Agility needs IT-Agility
„IT REVOLUTION MANIFESTO“, http://itrevolution.com
For most companies, IT functions as the nervous system and provides an increasing amount of the organizational muscle mass. Most critical business functions are entirely automated within IT, and 95% of all capital projects depend on IT to get done. Today, nearly every business decision will result in at least one IT change.
Gene Kim, Kevin Behr, George Spafford, 2013
E.M. Goldratt, 1984
Theory of Constraints → →
Samstag, 16. November 13
cloudbau
IT-Agility and the „Risk of Change“
The core, chronic con!ict that every IT leader faces is the need to simultaneously enable faster time to market (i.e., make as many changes as you can), while providing stable, secure and reliable IT services (i.e., make as few changes as you can).
„Lowering risk of change through tools and culture“John Allspaw, Paul Hammond, 2009
small changes, o!en
reproducible
process standards
Request Approve
Test Deploy
Develop
Water-
Scrum
- FallGene Kim
expect failure
cooperative culture
metrics
Samstag, 16. November 13
cloudbau
Tools and Culture
CPU-Virt.
Storage-Virt.
SDN
Con!g-DB
Programmable Infrastructure Description in Code
Process and Culture
Continuos Delivery
DevOpsCross-Functional Teams
Automatic Build
Anti-Fragile Organizations
Binary-Repo DescriptionOrchestration
Prod Env Stage Env Dev Env
Samstag, 16. November 13
cloudbau
Con!guration Management
Samstag, 16. November 13
cloudbau
●Just build it
●Keep notes in server.txt
●Move notes to the wiki
●Custom scripts (in scm?!)
●Snapshot & Clone
Evolving towards Con!guration Management
Samstag, 16. November 13
cloudbau
Applications
http://www."ickr.com/photos/steffenz/337700069/
http://www."ickr.com/photos/kky/704056791/
Samstag, 16. November 13
cloudbau
Infrastructure
http://www."ickr.com/photos/sbh/462754460/
Samstag, 16. November 13
cloudbau
Collection of Resources
http://www."ickr.com/photos/philliecasablanca/3354734116/
• Networking
• Files
• Directories
• Symlinks
• Mounts
• Routes
• Users
• Groups
• Tasks
• Packages
• Software
• Services
• Configuration
• Other Stuff
Samstag, 16. November 13
cloudbau
Acting in Concert
http://www."ickr.com/photos/glowjangles/4081048126/
Samstag, 16. November 13
cloudbau
To Provide a Service
http://www."ickr.com/photos/28309157@N08/3743455858/
Samstag, 16. November 13
cloudbau
And it Evolves
http://www."ickr.com/photos/16339684@N00/2681435235/
Samstag, 16. November 13
cloudbau
Application Server
See Node
Samstag, 16. November 13
cloudbau
Application Server
Application Database
See Nodes
Samstag, 16. November 13
cloudbau
Application Server
Application Databases
See Nodes Grow
Samstag, 16. November 13
cloudbau
Application Servers
Application Databases
See Nodes Grow
Samstag, 16. November 13
cloudbau
Application Servers
Application Databases
Load Balancer
See Nodes Grow
Samstag, 16. November 13
cloudbau
Application Servers
Application Databases
Load Balancers
See Nodes Grow
Samstag, 16. November 13
cloudbau
Application Servers
Application Database Cache
Load Balancers
Application Databases
See Nodes Grow
Samstag, 16. November 13
cloudbau
Application Servers
Application Database Cache
Load Balancers
Application Databases
Tied Together with Con!guration
Samstag, 16. November 13
cloudbau
So when this
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
Samstag, 16. November 13
cloudbau
Becomes This
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
Samstag, 16. November 13
cloudbau
This can happen automagically
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
Samstag, 16. November 13
cloudbau
Count the Resources
NagiosGraphite
Jboss App
Memcache
Postgres Slaves
• Load balancer config
• Nagios host ping
• Nagios host ssh
• Nagios host HTTP
• Nagios host app health
• Graphite CPU
• Graphite Memory
• Graphite Disk
• Graphite SNMP
• Memcache firewall
• Postgres firewall
• Postgres authZ config
• 12+ resource changes for 1 node addition
Samstag, 16. November 13
cloudbau
Focus Today: Chef and Puppet
Series A: 2,5 M$ (2009)Series B: 11 M$ (2010)
Series C: 19,5 M$ (2012)
Technologie
Series A: 2 M$ (2009)Series B: 5 M$ (2010)
Series C: 8,5 M$ (2011)Series D: 30 M$ (2013)
Technologie
*alle Angaben von www.crunchbase.com
Samstag, 16. November 13
cloudbau
Chef & Puppet
since 2005 since 2009
Declarative DSL Ruby DSL
Customers: Customers:
• CERN • AWS (OpsWorks)
• Twitter • Facebook
• Zynga • SAP
•VMware • IBM (Smartcloud)
• Microso#
Samstag, 16. November 13
cloudbau
What is Chef?
Recipes and Cookbooks that describe and deliver code.
Chef enables people to easily build & manage complex & dynamic applications at massive scale.
• New model for describing infrastructure that promotes reuse
• Programmatically provision and configure • Reconstruct business from code repository,
data backup, and bare metal resources
Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.
CHEF USES:
“!”!
What is Chef?
Recipes and Cookbooks that describe and deliver code.
Chef enables people to easily build & manage complex & dynamic applications at massive scale.
• New model for describing infrastructure that promotes reuse
• Programmatically provision and configure • Reconstruct business from code repository,
data backup, and bare metal resources
Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.
CHEF USES:
“!”!
What is Chef?
Recipes and Cookbooks that describe and deliver code.
Chef enables people to easily build & manage complex & dynamic applications at massive scale.
• New model for describing infrastructure that promotes reuse
• Programmatically provision and configure • Reconstruct business from code repository,
data backup, and bare metal resources
Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.
CHEF USES:
“!”!
What is Chef?
Recipes and Cookbooks that describe and deliver code.
Chef enables people to easily build & manage complex & dynamic applications at massive scale.
• New model for describing infrastructure that promotes reuse
• Programmatically provision and configure • Reconstruct business from code repository,
data backup, and bare metal resources
Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.
CHEF USES:
“!”!
Samstag, 16. November 13
cloudbau
What is Chef?
Recipes and Cookbooks that describe and deliver code.
Chef enables people to easily build & manage complex & dynamic applications at massive scale.
• New model for describing infrastructure that promotes reuse
• Programmatically provision and configure • Reconstruct business from code repository,
data backup, and bare metal resources
Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.
CHEF USES:
“!”!
What is Chef?
Recipes and Cookbooks that describe and deliver code.
Chef enables people to easily build & manage complex & dynamic applications at massive scale.
• New model for describing infrastructure that promotes reuse
• Programmatically provision and configure • Reconstruct business from code repository,
data backup, and bare metal resources
Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.
CHEF USES:
“!”!
What is Chef?
Recipes and Cookbooks that describe and deliver code.
Chef enables people to easily build & manage complex & dynamic applications at massive scale.
• New model for describing infrastructure that promotes reuse
• Programmatically provision and configure • Reconstruct business from code repository,
data backup, and bare metal resources
Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.
CHEF USES:
“!”!
Samstag, 16. November 13
cloudbau
Samstag, 16. November 13
cloudbau
Puppet is IT automation so"ware that helps system administrators manage infrastructure throughout its lifecycle, from provisioning and con$guration to orchestration and reporting.
What is Puppet?
How Puppet WorksPuppet uses a declarative, model-based approach to IT automation.
1. Define the desired state of the infrastructure's configuration using Puppet's declarative configuration language.
2. Simulate configuration changes before enforcing them.
3. Enforce the deployed desired state automatically, correcting any configuration drift.
4. Report on the differences between actual and desired states and any changes made enforcing the desired state.
Samstag, 16. November 13
cloudbau
What is Chef?
Recipes and Cookbooks that describe and deliver code.
Chef enables people to easily build & manage complex & dynamic applications at massive scale.
• New model for describing infrastructure that promotes reuse
• Programmatically provision and configure • Reconstruct business from code repository,
data backup, and bare metal resources
Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.
CHEF USES:
“!”!
What is Chef?
Recipes and Cookbooks that describe and deliver code.
Chef enables people to easily build & manage complex & dynamic applications at massive scale.
• New model for describing infrastructure that promotes reuse
• Programmatically provision and configure • Reconstruct business from code repository,
data backup, and bare metal resources
Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.
CHEF USES:
“!”!
What is Chef?
Recipes and Cookbooks that describe and deliver code.
Chef enables people to easily build & manage complex & dynamic applications at massive scale.
• New model for describing infrastructure that promotes reuse
• Programmatically provision and configure • Reconstruct business from code repository,
data backup, and bare metal resources
Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.
CHEF USES:
“!”!
Samstag, 16. November 13
cloudbau
Samstag, 16. November 13
cloudbau
Concepts
Samstag, 16. November 13
cloudbau
Declarative Language
File 1
Package A
Service IPackage B
Template 1
Samstag, 16. November 13
cloudbau
Convergence to desired State
File 1
Package A
Service IPackage B
Template 1
File 1
Package A
“run”
State X State Y
Samstag, 16. November 13
cloudbau
Idempotence
File 1
Package A
Service IPackage B
Template 1
“run”
State Y State Y
File 1
Package A
Service IPackage B
Template 1
Samstag, 16. November 13
cloudbau
The Implementations
Puppet and Chef
Samstag, 16. November 13
cloudbau
Terminology
Java Puppet Chef
*.java manifests *.pp recipes *.rb
package module cookbook
Samstag, 16. November 13
cloudbau
Terminology
Java Puppet Chef
singleton parametrized classes
recipes with attributes
class de#nes, classes de#nitions, libraries
Samstag, 16. November 13
cloudbau
Syntax
Puppet Chef
Limited DSL (by design) DSL + full ruby
Extended by De!nitionsDe!nitions, Resource/Providers, Light
Weight Resource
Custom Resource providersDe!nitions, Resource/Providers, Light
Weight Resource
ERB & Custom Libraries Providers, Libraries
Samstag, 16. November 13
cloudbau
Syntax Examples
Puppet Chef
class myapache { package “apache2” service “apache2”: ensure => “running”, require => Package[“apache2”]}
package “apache2”
service “apache2 do action [:enable , :start]end
Samstag, 16. November 13
cloudbau
Variables
Puppet Chef
#special syntax
$bla = “aa”
#string interpolation
$bla_string = “${bla}”
#native ruby
bla = “aa”
#string interpolation
bla_string = #{bla}
Samstag, 16. November 13
cloudbau
Loops
Puppet Chef
pass array of elements
erb template for loops
create_resource (:type, hash)
do while , repeat etc. available through native ruby
Samstag, 16. November 13
cloudbau
Conditionals
Puppet (since 2.7) Chef
if
case (execute block)
selectors (get value)
if, unless, case etc. available through native ruby
Samstag, 16. November 13
cloudbau
Project Structure
Puppet Modules Chef
files/
lib/
README.rdoc
manifests/
templates/
attributes/
definitions/
files/
libraries/
metadata.rb
providers/
README.rdoc
recipes/
resources/
templates/
Samstag, 16. November 13
cloudbau
Execution Model
Puppet Chef
Compile catalog (directed graph model) Then Execute Top Down execution
only code for state transition is visited every line of code is visited all the time
Samstag, 16. November 13
cloudbau
Execution Model
Puppet Chef
standalone standalone
client - serverclient - server
hosted
Samstag, 16. November 13
cloudbau
System Information
Puppet/Factor => Flat Key/Value
"hostname": "server-1", "fqdn": "server-1.example.com", "domain": "example.com", "network": { "interfaces": { "eth0": { "type": "eth", "number": "0", "encapsulation": "Ethernet", "addresses": { "00:0C:29:43:26:C5": { "family": "lladdr" }, "192.168.177.138": { "family": "inet", "broadcast": "192.168.177.255", "netmask": "255.255.255.0" },
Chef/Ohai => Nested Hash
Samstag, 16. November 13
cloudbau
●Selects Package Provider
●Selects Service Provider
●....
System Information
Samstag, 16. November 13
cloudbau
System Information (Chef Example)
execute "load sysctl" do command "/sbin/sysctl -p" action :nothingend
bytes = node['memory']['total'].split("kB")[0].to_i * 1024 / 3,pages = node['memory']['total'].split("kB")[0].to_i * 1024 / 3 / 2048
# adjust shared memory and semaphorestemplate "/etc/sysctl.conf" do source "sysctl.conf.erb" variables( :shmmax_in_bytes => bytes, :shmall_in_pages => pages ) notifies :run, "execute[load sysctl]", :immediatelyend
Samstag, 16. November 13
cloudbau
Templates
# Set up application listeners here.
listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>
Samstag, 16. November 13
cloudbau
Lint Tools
http://acrmp.github.io/foodcritic/
http://puppet-lint.com/
Samstag, 16. November 13
cloudbau
Dependency Managementhttps://github.com/applicationsonline/librarian-chef
http://librarian-puppet.com
http://berkshelf.com
Samstag, 16. November 13
cloudbau
Reports
Samstag, 16. November 13
cloudbau
Web Interface
Samstag, 16. November 13
cloudbau
●Razor for bare metal provisioning
●mcollectiv for remote execution/orchestration
●test-kitchen for infrastructure testing
●vagrant for rapid development
● ....
Tools
Samstag, 16. November 13
cloudbau
●Switch Con#g
●Driven by SDN Adoption
●Lots of Vendors (Arista ..)
Networking
Samstag, 16. November 13
cloudbau
Community
Samstag, 16. November 13
cloudbau
Chef Community
Community Overview!
25,000+ Community Members!!
1,000+ Community "Cookbooks!
!250,000+ Cookbook
Downloads!!
400+ Public Training Attendees in the last year!
!30+ Meetup Groups!
!
Over 200 Corporate Contributors
Samstag, 16. November 13
cloudbau
http://community.opscode.com/cookbooks
approx 1200 co0kbooks134 maintained by Opscode
Samstag, 16. November 13
cloudbau
http://forge.puppetlabs.com
approx 1700 modules72 maintained by Puppetlabs
Samstag, 16. November 13
cloudbau
The Price Tag
Samstag, 16. November 13
cloudbau
Puppet: How to BuyCumulative # of
NodesPer Node
Standard SupportPer Node Premium
Support*1-10 Download FREE NA
11-99 $ 99 Contact Sales
100-249 $ 93 Contact Sales
250-499 $ 88 $ 152
500-999 $ 83 $ 119
1000-2499 $ 79 $ 99
2500+ Contact Sales Contact Saleshttps://puppetlabs.com/puppet/how-to-buy
Samstag, 16. November 13
cloudbau
Puppet Enterprise vs. OpenSource
Samstag, 16. November 13
cloudbau
Chef: How to Buy
Free Launch Standard Premium
Price per Month Free $ 120 $ 300 $ 600
Nodes 5 20 50 100
Standard Support — ✔ ✔ ✔
http://www.opscode.com/enterprise-chef/#plans-pricing
Samstag, 16. November 13
cloudbau
Enterprise Chef vs. OpenSource Chef
Samstag, 16. November 13
Danke!
Edmund Haselwanter
@[email protected]+49 30 57701800www.cloudbau.de
cloudbau GmbHKörnerstr. 7-1010785 Berlin
cloudbau
Samstag, 16. November 13
Credits:• Patrick Debois (http://www.jedi.be/blog/) for some of his Puppet/Chef Slides• Opscode (http://www.opscode.com) for product info and some slides • Puppetlabs (http://puppetlabs.com) for product info
Samstag, 16. November 13