Rapid scaling and management in the cloud with

puppet

Carl Caumcarl@puppetlabs.com

@ccaum

What is Puppet?

Express infrastructure as....• code

o Manage your infrastructure just like softwareo Use version controlo QA changeso Continuous Integrationo Prevent problems from re-occurring 

What is Puppet?

Express infrastructure as....• code• resources

o What, not howo Relationships not order

What is Puppet?

Express infrastructure as....• code• resources• state

o Idempotento What, not how

The Cycle

The Cycle

Resources

Resources are the building blocks of puppetAll resources have:• type• title• attributes

file { '/etc/motd':   ensure  => file,   owner   => 'root',   content => 'Managed by Puppet',   mode    => 0755,}

The Resource Abstraction Layer (RAL)

The Resource Abstraction Layer allows puppet to introspect the system about resource types.

Modules

Modules contain everything puppet needs to manage something. 

For example:• apache• bacula• mysql• subversion• etc

Modules

Layout:

module_name  |  -- manifests (puppet code)  |  -- files          (files to serve to clients)  |  -- templates (ERB templates)  |  -- lib             (puppet plugins)

Modules

Where do I get them?

The Forgehttp://forge.puppetlabs.com

Githubhttp://github.comPuppet Module Tool# puppet-module install puppetlabs/apache

QAQ:  How do I QA my puppet code before pushing to production?

QAQ:  How do I QA my puppet code before pushing to production?A: Environments!!

The GraphPuppet uses a graph to know the relationship between resources

package { 'ntp':   ensure => present, } 

file { '/etc/ntp.conf':   owner => 'root',  group => 'root',  mode => '0644',  require => Package['ntp'], } 

service { 'ntpd':   ensure => running,   enable => true,   subscribe => File['/etc/ntp.conf'], }

The GraphPuppet uses a graph to know the relationship between resources

Puppet is highly customizable

Using Ruby, you can add custom.....• puppet subcommands• types/providers• facts• report processors

Puppet is highly customizable

Custom Fact

Facter.add("role") do 

  setcode do     Facter::Util::Resolution.exec("cat /etc/role")   end 

end

Puppet is highly customizable

Custom Report ProcessorPuppet::Reports.register_report(:autoami) do

  def process    .. do stuff ..  end

end

Demo

 

Cloud Provisioner

Instant cloud management with puppet

Technologies

Puppet Faces

    A new API for creating Puppet subcommands and actions.    Introduced in 2.7.0  

    http://www.puppetlabs.com/faces/

Fog

    Ruby gem designed to control a variety of cloud services     through a unified API.

    https://github.com/geemus/fog

    

Simplicity and Scriptability

1. Create a new instance from an AMI2. Install Puppet (from community packages or Puppet Enterprise

installer)3. Generate and sign SSL cert for new agent on master node

# puppet node bootstrap   --image ami-d812efb1   --keyname my_keyname   --type m1.small   --login root   --keyfile /path/to/my_keypair.pem   --node-group webserver   --server puppet.domain.com   --enc-ssl --enc-auth-user console   --enc-auth-passwd console_pass   --enc-port 443

Simplicity and Scriptability

List nodes instances

# puppet node_aws listi-d22612b2:  created_at: Wed Oct 12 16:50:02 UTC 2011  dns_name: ec2-184-73-33-225.compute-1.amazonaws.com  id: i-d22612b2  state: runningi-f1b54b92:  created_at: Wed Oct 26 13:46:44 UTC 2011  dns_name: ec2-174-129-228-163.compute-1.amazonaws.com  id: i-f1b54b92  state: running

Simplicity and Scriptability

Destroy instances

# puppet node terminate ec2-75-101-181-145.compute-1.amazonaws.com

Simplicity and Scriptability

require 'puppet'require 'puppet/face'

opts = { :image      => 'ami-d812efb1',              :keyname => 'my_keypair',              :type        => 'm1.small',              :login       => 'root',              :keyfile     => '/path/to/my_keyfile.pem,              :server     => 'puppet.mydomain.com'}

Puppet::Face[:node_aws, '0.0.1'].bootstrap(opts)

AMI Management

This is a technique, not a tool!

AMI Management

• Not funo No, really. It sucks

AMI Management

• Not fun• Difficult to know when to update

o Usually requires a human to kick off a process

AMI Management

• Not fun• Difficult to know when to update• Needs to be registered with load balancer

o Whoever/whatever updates the AMI needs to register the new AMI with the load balancer and/or auto scaler

AMI Management

• Not fun• Difficult to know when to update• Needs to be registered with load balancer• Necessary?

o Many choose to just have puppet always configure a stock AMI

Initial Puppet Run

Can be slow if...• You haven't updated your AMIs in a while• You're running on a stock AMI to prevent image

management

Best of Both Worlds• Use Cloud Provisioner to spawn new instances of AMIs you

want to manage• Use Puppet custom report processors to detect if anything

changed• Use custom face to snapshot instances if anything changes.

Build new AMI off of snapshot• Register new AMI with load balancer and delete old one

EBS backed images (Elastic Block Store)

• Persistent• Allows for snapshots• Many public ones available

Autoami

Module containing custom face to manage images and custom report processor

http://github.com/ccaum/puppet-autoami

Autoami

Steps:1. Manually release production puppet code in VCS (git/svn)

Autoami

Steps:1. Manually release production puppet code in VCS (git/svn)– Launch an instance of every AMI you want to manage

Autoami

Steps:1. Manually release production puppet code in VCS (git/svn)– Launch an instance of every AMI you want to manage– Record the certificate name

Autoami

Steps:1. Manually release production puppet code in VCS (git/svn)– Launch an instance of every AMI you want to manage– Record the certificate name– Classify the instance with Console

Autoami

Steps:1. Manually release production puppet code in VCS (git/svn)– Launch an instance of every AMI you want to manage– Record the certificate name– Classify the instance with Console– Sign the certificate

Autoami

Steps:1. Manually release production puppet code in VCS (git/svn)– Launch an instance of every AMI you want to manage– Record the certificate name– Classify the instance with Console– Sign the certificate– Wait for instance to report

Autoami

Steps:1. Manually release production puppet code in VCS (git/svn)– Launch an instance of every AMI you want to manage– Record the certificate name– Classify the instance with Console– Sign the certificate– Wait for instance to report– If changes occurred (and were successful), generate new

AMI

Autoami

Steps:1. Manually release production puppet code in VCS (git/svn)– Launch an instance of every AMI you want to manage– Record the certificate name– Classify the instance with Console– Sign the certificate– Wait for instance to report– If changes occurred (and were successful), generate new

AMI– Record AMI for load balancers

Autoami

Custom Report Processor

Puppet::Reports.register_report(:autoami) do

  def process    .. do stuff ..  endend

Autoami

 

Demo