infrastructure as code with chef / puppet

Infrastructure as Code with Chef / Puppet

Edmund Haselwanter (e.haselwanter@cloudbau.de)

Samstag, 16. November 13

cloudbau

Edmund Haselwanter

●Co-Founder and CSO cloudbau GmbH

●Dell Professional Services (Openstack/Crowbar/Automation)

●So!ware Development Background

●10 Years Infrastructure Automation

●@ehaselwanter on twitter/github/irc/...

Introduction

cloudbau

Business-Agility needs IT-Agility

„IT REVOLUTION MANIFESTO“, http://itrevolution.com

For most companies, IT functions as the nervous system and provides an increasing amount of the organizational muscle mass. Most critical business functions are entirely automated within IT, and 95% of all capital projects depend on IT to get done. Today, nearly every business decision will result in at least one IT change.

Gene Kim, Kevin Behr, George Spafford, 2013

E.M. Goldratt, 1984

Theory of Constraints → →

cloudbau

IT-Agility and the „Risk of Change“

The core, chronic con!ict that every IT leader faces is the need to simultaneously enable faster time to market (i.e., make as many changes as you can), while providing stable, secure and reliable IT services (i.e., make as few changes as you can).

„Lowering risk of change through tools and culture“John Allspaw, Paul Hammond, 2009

small changes, o!en

reproducible

process standards

Request Approve

Test Deploy

Develop

Water-

- FallGene Kim

expect failure

cooperative culture

metrics

cloudbau

Tools and Culture

CPU-Virt.

Storage-Virt.

Con!g-DB

Programmable Infrastructure Description in Code

Process and Culture

Continuos Delivery

DevOpsCross-Functional Teams

Automatic Build

Anti-Fragile Organizations

Binary-Repo DescriptionOrchestration

Prod Env Stage Env Dev Env

cloudbau

Con!guration Management

cloudbau

●Just build it

●Keep notes in server.txt

●Move notes to the wiki

●Custom scripts (in scm?!)

●Snapshot & Clone

Evolving towards Con!guration Management

cloudbau

Applications

http://www."ickr.com/photos/steffenz/337700069/

http://www."ickr.com/photos/kky/704056791/

cloudbau

Infrastructure

http://www."ickr.com/photos/sbh/462754460/

cloudbau

Collection of Resources

http://www."ickr.com/photos/philliecasablanca/3354734116/

• Networking

• Files

• Directories

• Symlinks

• Mounts

• Routes

• Users

• Groups

• Tasks

• Packages

• Software

• Services

• Configuration

• Other Stuff

cloudbau

Acting in Concert

http://www."ickr.com/photos/glowjangles/4081048126/

cloudbau

To Provide a Service

http://www."ickr.com/photos/28309157@N08/3743455858/

cloudbau

And it Evolves

http://www."ickr.com/photos/16339684@N00/2681435235/

cloudbau

Application Server

See Node

cloudbau

Application Server

Application Database

See Nodes

cloudbau

Application Server

Application Databases

See Nodes Grow

cloudbau

Application Servers

See Nodes Grow

cloudbau

Application Servers

Load Balancer

See Nodes Grow

cloudbau

Application Servers

Load Balancers

See Nodes Grow

cloudbau

Application Servers

Application Database Cache

Load Balancers

See Nodes Grow

cloudbau

Application Servers

Application Database Cache

Load Balancers

Tied Together with Con!guration

cloudbau

So when this

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

cloudbau

Becomes This

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

cloudbau

This can happen automagically

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite

cloudbau

Count the Resources

NagiosGraphite

Jboss App

Memcache

Postgres Slaves

• Load balancer config

• Nagios host ping

• Nagios host ssh

• Nagios host HTTP

• Nagios host app health

• Graphite CPU

• Graphite Memory

• Graphite Disk

• Graphite SNMP

• Memcache firewall

• Postgres firewall

• Postgres authZ config

• 12+ resource changes for 1 node addition

cloudbau

Focus Today: Chef and Puppet

Series A: 2,5 M$ (2009)Series B: 11 M$ (2010)

Series C: 19,5 M$ (2012)

Technologie

Series A: 2 M$ (2009)Series B: 5 M$ (2010)

Series C: 8,5 M$ (2011)Series D: 30 M$ (2013)

Technologie

*alle Angaben von www.crunchbase.com

cloudbau

Chef & Puppet

since 2005 since 2009

Declarative DSL Ruby DSL

Customers: Customers:

• CERN • AWS (OpsWorks)

• Twitter • Facebook

• Zynga • SAP

•VMware • IBM (Smartcloud)

• Microso#

cloudbau

What is Chef?

Recipes and Cookbooks that describe and deliver code.

Chef enables people to easily build & manage complex & dynamic applications at massive scale.

• New model for describing infrastructure that promotes reuse

•  Programmatically provision and configure • Reconstruct business from code repository,

data backup, and bare metal resources

Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.

CHEF USES:

“!”!

What is Chef?

CHEF USES:

“!”!

What is Chef?

CHEF USES:

“!”!

What is Chef?

CHEF USES:

“!”!

cloudbau

What is Chef?

CHEF USES:

“!”!

What is Chef?

CHEF USES:

“!”!

What is Chef?

CHEF USES:

“!”!

cloudbau

Puppet is IT automation so"ware that helps system administrators manage infrastructure throughout its lifecycle, from provisioning and con$guration to orchestration and reporting.

What is Puppet?

How Puppet WorksPuppet uses a declarative, model-based approach to IT automation.

1. Define the desired state of the infrastructure's configuration using Puppet's declarative configuration language.

2. Simulate configuration changes before enforcing them.

3. Enforce the deployed desired state automatically, correcting any configuration drift.

4. Report on the differences between actual and desired states and any changes made enforcing the desired state.

cloudbau

What is Chef?

CHEF USES:

“!”!

What is Chef?

CHEF USES:

“!”!

What is Chef?

CHEF USES:

“!”!

cloudbau

Concepts

cloudbau

Declarative Language

File 1

Package A

Service IPackage B

Template 1

cloudbau

Convergence to desired State

File 1

Package A

Service IPackage B

Template 1

File 1

Package A

“run”

State X State Y

cloudbau

Idempotence

File 1

Package A

Service IPackage B

Template 1

“run”

State Y State Y

File 1

Package A

Service IPackage B

Template 1

cloudbau

The Implementations

Puppet and Chef

cloudbau

Terminology

Java Puppet Chef

*.java manifests *.pp recipes *.rb

package module cookbook

cloudbau

Terminology

Java Puppet Chef

singleton parametrized classes

recipes with attributes

class de#nes, classes de#nitions, libraries

cloudbau

Syntax

Puppet Chef

Limited DSL (by design) DSL + full ruby

Extended by De!nitionsDe!nitions, Resource/Providers, Light

Weight Resource

Custom Resource providersDe!nitions, Resource/Providers, Light

Weight Resource

ERB & Custom Libraries Providers, Libraries

cloudbau

Syntax Examples

Puppet Chef

class myapache { package “apache2” service “apache2”: ensure => “running”, require => Package[“apache2”]}

package “apache2”

service “apache2 do action [:enable , :start]end

cloudbau

Variables

Puppet Chef

#special syntax

$bla = “aa”

#string interpolation

$bla_string = “${bla}”

#native ruby

bla = “aa”

#string interpolation

bla_string = #{bla}

cloudbau

Puppet Chef

pass array of elements

erb template for loops

create_resource (:type, hash)

do while , repeat etc. available through native ruby

cloudbau

Conditionals

Puppet (since 2.7) Chef

case (execute block)

selectors (get value)

if, unless, case etc. available through native ruby

cloudbau

Project Structure

Puppet Modules Chef

files/

README.rdoc

manifests/

templates/

attributes/

definitions/

files/

libraries/

metadata.rb

providers/

README.rdoc

recipes/

resources/

templates/

cloudbau

Execution Model

Puppet Chef

Compile catalog (directed graph model) Then Execute Top Down execution

only code for state transition is visited every line of code is visited all the time

cloudbau

Execution Model

Puppet Chef

standalone standalone

client - serverclient - server

hosted

cloudbau

System Information

Puppet/Factor => Flat Key/Value

"hostname": "server-1", "fqdn": "server-1.example.com", "domain": "example.com", "network": { "interfaces": { "eth0": { "type": "eth", "number": "0", "encapsulation": "Ethernet", "addresses": { "00:0C:29:43:26:C5": { "family": "lladdr" }, "192.168.177.138": { "family": "inet", "broadcast": "192.168.177.255", "netmask": "255.255.255.0" },

Chef/Ohai => Nested Hash

cloudbau

●Selects Package Provider

●Selects Service Provider

●....

System Information

cloudbau

System Information (Chef Example)

execute "load sysctl" do command "/sbin/sysctl -p" action :nothingend

bytes = node['memory']['total'].split("kB")[0].to_i * 1024 / 3,pages = node['memory']['total'].split("kB")[0].to_i * 1024 / 3 / 2048

# adjust shared memory and semaphorestemplate "/etc/sysctl.conf" do source "sysctl.conf.erb" variables( :shmmax_in_bytes => bytes, :shmall_in_pages => pages ) notifies :run, "execute[load sysctl]", :immediatelyend

cloudbau

Templates

# Set up application listeners here.

listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>

cloudbau

Lint Tools

http://acrmp.github.io/foodcritic/

http://puppet-lint.com/

cloudbau

Dependency Managementhttps://github.com/applicationsonline/librarian-chef

http://librarian-puppet.com

http://berkshelf.com

cloudbau

Reports

cloudbau

Web Interface

cloudbau

●Razor for bare metal provisioning

●mcollectiv for remote execution/orchestration

●test-kitchen for infrastructure testing

●vagrant for rapid development

● ....

cloudbau

●Switch Con#g

●Driven by SDN Adoption

●Lots of Vendors (Arista ..)

Networking

cloudbau

Community

cloudbau

Chef Community

Community Overview!

25,000+ Community Members!!

1,000+ Community "Cookbooks!

!250,000+ Cookbook

Downloads!!

400+ Public Training Attendees in the last year!

!30+ Meetup Groups!

Over 200 Corporate Contributors

cloudbau

http://community.opscode.com/cookbooks

approx 1200 co0kbooks134 maintained by Opscode

cloudbau

http://forge.puppetlabs.com

approx 1700 modules72 maintained by Puppetlabs

cloudbau

The Price Tag

cloudbau

Puppet: How to BuyCumulative # of

NodesPer Node

Standard SupportPer Node Premium

Support*1-10 Download FREE NA

11-99 $ 99 Contact Sales

100-249 $ 93 Contact Sales

250-499 $ 88 $ 152

500-999 $ 83 $ 119

1000-2499 $ 79 $ 99

2500+ Contact Sales Contact Saleshttps://puppetlabs.com/puppet/how-to-buy

cloudbau

Puppet Enterprise vs. OpenSource

cloudbau

Chef: How to Buy

Free Launch Standard Premium

Price per Month Free $ 120 $ 300 $ 600

Nodes 5 20 50 100

Standard Support — ✔ ✔ ✔

http://www.opscode.com/enterprise-chef/#plans-pricing

cloudbau

Enterprise Chef vs. OpenSource Chef

Danke!

Edmund Haselwanter

@ehaselwantere.haselwanter@cloudbau.de+49 30 57701800www.cloudbau.de

cloudbau GmbHKörnerstr. 7-1010785 Berlin

cloudbau

Credits:• Patrick Debois (http://www.jedi.be/blog/) for some of his Puppet/Chef Slides• Opscode (http://www.opscode.com) for product info and some slides • Puppetlabs (http://puppetlabs.com) for product info

infrastructure as code with chef / puppet

culturecloudbau samstag

technologiecloudbau

concertcloudbau samstag

servicecloudbau samstag

evolvescloudbau samstag

infrastructurecloudbau

postgres master

series c

Technology

devops workshop (section 3) · infrastructure as code...

infrastructure modeling with chef

puppet on a string - nlvmug · puppet on a string. ......

puppet: infrastructure as code - it pro forum · 15-02-2011...

tame your infrastructure with puppet

plugging chocolatey into your puppet infrastructure...

capistrano, puppet, and chef

manageable puppet infrastructure - netways · manageable...

puppet camp berlin 2014: manageable puppet infrastructure

devops : automate your infrastructure with puppet

chef, puppet & ansible - comparing leading configuration...

cfengine vs puppet vs chef: a guide for stressed developers

automating db ops with ansible, chef, and puppet db ops...

ougn 2014 - provisioning oracle fusion middleware using...

devsecops: getting there from here - sans.org · devsecops:...

infrastructure as code on azure with puppet & chef · pdf...

if ( network == server ) magic happens...linux devopstools...

new to chef and puppet ? overview of chef and puppet and how...

the forrester wave™: configuration management ... -...

chef: smart infrastructure automation