How I hack PuppetHow I hack Puppetand made my life soo much easier

by

Kris Buytaert

Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source

Consultant @inuits.euConsultant @inuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops● But mostly, trying to be good at my jobBut mostly, trying to be good at my job

TodayToday

• VagrantVagrant

• StyleStyle

• Testing PuppetTesting Puppet

• Modules vs ModulesModules vs Modules

• JenkinsJenkins

• Demo ?Demo ?

Typical EnvironmentsTypical EnvironmentsFor DevsFor Devs

● ScrumScrum

● Version ControlVersion Control

● Automated Build Automated Build

● Bugtracking Bugtracking

● Continous integrationContinous integration

● Integrated testingIntegrated testing

● Automated Automated deployment deployment

For OpsFor Ops

● KanbanKanban

● Version ControlVersion Control

● Automated Build Automated Build

● Bugtracking Bugtracking

● Continous integrationContinous integration

● Integrated testingIntegrated testing

● Automated Automated deployment deployment

Everybody is a developerEverybody is a developer

● Yes we write code alsoYes we write code also

● httpd.conf, squid.conf, my.cnfhttpd.conf, squid.conf, my.cnf

● Just crappy languages :)Just crappy languages :)

● shell, perl, ruby, python, puppetshell, perl, ruby, python, puppet

● So those rules apply for EveryoneSo those rules apply for Everyone

VagrantVagrant● Abstraction layer for Abstraction layer for

VirtualBoxVirtualBox

● Integrates well with Integrates well with Puppet/ChefPuppet/Chef

● Project = Project =

● VagrantfileVagrantfile

● Manifests / Manifests / CookbooksCookbooks

● Portable, Small , Portable, Small , VersionableVersionable

VeeweeVeewee● Vabgrantbox.esVabgrantbox.es

● Use veewee to build your boxenUse veewee to build your boxen

•gem install veeweegem install veewee

•veewee templatesveewee templates

•veewee init natty ubuntu-11.04-server-amd64veewee init natty ubuntu-11.04-server-amd64

•vagrant basebox build nattyvagrant basebox build natty

•vagrant box add 'natty' 'natty.box'vagrant box add 'natty' 'natty.box'

A Vagrant projectA Vagrant project

[sdog@mine vagrant-graphite]$ lsmanifests modules README TODO Vagrantfile [sdog@mine vagrant-graphite]$ tree -dL 2.├── manifests│   └── hosts└── modules ├── apache ├── collectd ├── graphite ├── jmxtrans ├── logster ├── statsd └── tattle

10 directories

A VagrantfileA VagrantfileVagrant::Config.run do |config| # All Vagrant configuration is done here. The most common configuration # options are documented and commented below. For a complete reference, # please see the online documentation at vagrantup.com.

config.vm.define :mongo1 do |mongo1_config| mongo1_config.ssh.max_tries = 100 mongo1_config.vm.box = "MyCentOS2" mongo1_config.vm.network("192.168.99.101") mongo1_config.vm.host_name = "mongo1" mongo1_config.vm.provision :puppet do |mongo1_puppet| mongo1_puppet.pp_path = "/tmp/vagrant-puppet" mongo1_puppet.manifests_path = "manifests" mongo1_puppet.module_path = "modules" mongo1_puppet.manifest_file = "site.pp" end end config.vm.define :mongo2 do |mongo2_config| mongo2_config.ssh.max_tries = 100 mongo2_config.vm.box = "MyCentOS2" mongo2_config.vm.network("192.168.99.102") mongo2_config.vm.host_name = "mongo2" mongo2_config.vm.provision :puppet do |mongo2_puppet| mongo2_puppet.pp_path = "/tmp/vagrant-puppet" mongo2_puppet.manifests_path = "manifests" mongo2_puppet.module_path = "modules" mongo2_puppet.manifest_file = "site.pp" end end

Vagrant RocksVagrant Rocks● Vagrant init Vagrant init

● Vagrant up Vagrant up

● Vagrant provision Vagrant provision

● Vagrant down Vagrant down

● Vagrant destroyVagrant destroy

Booting UpBooting Up[sdog@stillmine vagrant-graphite]$ vagrant up[default] VM already created. Booting if its not already running...[default] Preparing host only network...[default] Clearing any previously set forwarded ports...[default] Forwarding ports...[default] -- carbon: 2003 => 2021 (adapter 1)[default] -- http2: 8080 => 50051 (adapter 1)[default] -- http: 80 => 50050 (adapter 1)[default] -- ssh: 22 => 2222 (adapter 1)[default] Cleaning previously set shared folders...[default] Creating shared folders metadata...[default] Running any VM customizations...[default] Booting VM...[default] Waiting for VM to boot. This can take a few minutes.[default] VM booted and ready for use![default] Enabling host only network...[default] Setting host name...[default] Mounting shared folders...[default] -- v-root: /vagrant[default] -- manifests: /tmp/vagrant-puppet/manifests[default] -- v-pp-m0: /tmp/vagrant-puppet/modules-0[default] Running provisioner: Vagrant::Provisioners::Puppet...[default] Running Puppet with site.pp...[default] notice: /Stage[main]/Default-repo/Yumrepo[epel]/descr: descr changed 'Extra Packages for Enterprise Linux 6.x' to 'Extra Packages for Enterprise Linux 6.x '[default] [default] notice: /Stage[main]/Up-graph::Document-throughput/File[/usr/local/up-graph/document-throughput/graph.sh]/content: content changed '{md5}5085413bb8465edc1815024e1e06c0e6' to '{md5}c7d73b7f91a751632742f569030cb3d7'[default] [default] notice: /Stage[main]/Collectd/Service[collectd]/ensure: ensure changed 'stopped' to 'running'[default] [default] notice: Finished catalog run in 12.45 seconds[default]

Failed ProvisioningFailed Provisioning[sdog@stillmine vagrant-graphite]$ vagrant provision [default] Running provisioner: Vagrant::Provisioners::Puppet...[default] Running Puppet with site.pp...[default] Could not parse for environment production: Syntax error at 'import'; expected '}' at /tmp/vagrant-puppet/manifests/site.pp:5 on node graphite.lan[default] The following SSH command responded with a non-zero exit status.Vagrant assumes that this means the command failed!

cd /tmp/vagrant-puppet/manifestspuppet apply --modulepath '/tmp/vagrant-puppet/modules-0' site.pp

The output of the command prior to failing is outputted below:

[no output]

Vagrant TipsVagrant Tips

Set up a puppetmaster node Set up a puppetmaster node

Stored configsStored configs

Hiera Hiera

Mount Symlink /etc/puppet dirs Mount Symlink /etc/puppet dirs

/vagrant is your host folder/vagrant is your host folder

Handover to devsHandover to devs● Gitrepo Gitrepo

•VagrantfileVagrantfile

•A Box definitionA Box definition

•Puppet Manifests Puppet Manifests

•Puppet modules Puppet modules

•Their codeTheir code

=> A reproducable environment easy to => A reproducable environment easy to migrate to test/uat/prod.migrate to test/uat/prod.

StyleStyle● I never caredI never cared

● JanJan Coworkers complained Coworkers complained

● http://docs.puppetlabs.com/guides/style_guidehttp://docs.puppetlabs.com/guides/style_guide

BeforeBeforenode 'nova' {

yumrepo {

'epel': baseurl => $operatingsystemrelease ? { '6.0' => "http://mirror.eurid.eu/epel/6/$hardwaremodel/", '*' => "http://mirror.eurid.eu/epel/5/$hardwaremodel/", }, descr => $operatingsystemrelease ? { '6.0' => 'Extra Packages for Enterprise Linux 6.x ', '*' => 'Extra Packages for Enterprise Linux 5.x', }, gpgcheck => 0, enabled => 1;

'openstack': baseurl => "http://yum.griddynamics.net/yum/diablo-centos", descr => "openstack diablo at griddynamics", gpgcheck => 0, enabled => 1;

}

service { "iptables": ensure => "stopped", enable => "false"; }

package { "centos-release-cr": ensure => "present"; "openstack-nova": ensure => "present"; }

AfterAfternode 'nova' {

yumrepo { 'epel': baseurl => $operatingsystemrelease ? { '6.0' => "http://mirror.eurid.eu/epel/6/$hardwaremodel/", '*' => "http://mirror.eurid.eu/epel/5/$hardwaremodel/", }, descr => $operatingsystemrelease ? { '6.0' => 'Extra Packages for Enterprise Linux 6.x', '*' => 'Extra Packages for Enterprise Linux 5.x', }, gpgcheck => 0, enabled => 1;

'openstack': baseurl => "http://yum.griddynamics.net/yum/diablo-centos", descr => "openstack diablo at griddynamics", gpgcheck => 0, enabled => 1; }

service { "iptables": ensure => "stopped", enable => "false"; }

package { "centos-release-cr": ensure => "present"; "openstack-nova": ensure => "present"; }

Puppet LintPuppet Lintpuppet-lint nova.pp WARNING: double quoted string containing no variables on line 18WARNING: double quoted string containing no variables on line 19WARNING: double quoted string containing no variables on line 30WARNING: double quoted string containing no variables on line 31WARNING: double quoted string containing no variables on line 32WARNING: double quoted string containing no variables on line 36WARNING: double quoted string containing no variables on line 37WARNING: double quoted string containing no variables on line 38WARNING: double quoted string containing no variables on line 39WARNING: => on line 47 isn't aligned with the previous lineWARNING: => on line 48 isn't aligned with the previous lineWARNING: => on line 49 isn't aligned with the previous lineWARNING: => on line 52 isn't aligned with the previous lineWARNING: => on line 53 isn't aligned with the previous lineWARNING: => on line 54 isn't aligned with the previous lineWARNING: => on line 55 isn't aligned with the previous line

Note: Buy Tim Sharpe more beer ...

Testing PuppetTesting Puppet● puppet --parseonly test.pppuppet --parseonly test.pp

● puppet parser validate test.pppuppet parser validate test.pp

● erb -x -T '-' foreman-vhost.conf.erb | ruby -c erb -x -T '-' foreman-vhost.conf.erb | ruby -c

● Cucumber-puppetCucumber-puppet

● rspec-puppetrspec-puppet

Editor pluginsEditor plugins● Vim : https://github.com/rodjek/vim-puppetVim : https://github.com/rodjek/vim-puppet

● Emacs etc .. Emacs etc ..

Puppet-modulePuppet-module● Generates a clean structureGenerates a clean structure

● and unused code and unused code

● Does stuff with forge (Does stuff with forge (Does anyone actually use forge ? )Does anyone actually use forge ? )

● + shell script also create class stubs + shell script also create class stubs

Classes vs ModulesClasses vs Modules● Module : Module :

● Abstract definition on configuring a Abstract definition on configuring a serviceservice

● Class : Class :

● Specific implementation of your use case Specific implementation of your use case of such a module of such a module

•e.g usernames / passwords / hosts do not e.g usernames / passwords / hosts do not belong in modulesbelong in modules

Parametrized ClassesParametrized Classes

Services and ModulesServices and ModulesNot in my cluster Not in my cluster please ! please !

service {service {

'blah':'blah':

Ensure => running,Ensure => running,

}}

class autofs::disable inherits autofs {class autofs::disable inherits autofs {

Service['autofs'] {Service['autofs'] {

ensure => 'stopped' ,ensure => 'stopped' ,

enable => false,enable => false,

}}

}}

GitHubGitHub

The Nr 1 place for your modulesThe Nr 1 place for your modules(Till forge.puppetlabs.org is revamped)(Till forge.puppetlabs.org is revamped)

A github projectA github project

[sdog@mine vagrant-graphite]$ lsmanifests modules README TODO Vagrantfile [sdog@mine vagrant-graphite]$ tree -dL 2.├── manifests│   └── hosts└── modules ├── apache ├── collectd ├── graphite ├── jmxtrans ├── logster ├── statsd └── tattle

10 directories

SubmodulesSubmodules git submodule git submodule

-d132d2ba4b8c4848acee3ee8e6144b68e86bedc3 modules/apache-d132d2ba4b8c4848acee3ee8e6144b68e86bedc3 modules/apache

6a44fb1179e23ff42f476233d9c65b545be76470 modules/collectd 6a44fb1179e23ff42f476233d9c65b545be76470 modules/collectd (heads/master)(heads/master)

+24560116dd498033d81a444f5e326ba9b8b7aefc modules/graphite +24560116dd498033d81a444f5e326ba9b8b7aefc modules/graphite (heads/master)(heads/master)

9b5c07afcf4e0b18f48346093053a7d46222a19a modules/jmxtrans 9b5c07afcf4e0b18f48346093053a7d46222a19a modules/jmxtrans (heads/master)(heads/master)

1a200f9dfae4f346b7f10f7b8f33076d99670ef8 modules/logster (heads/master)1a200f9dfae4f346b7f10f7b8f33076d99670ef8 modules/logster (heads/master)

-7f801701898dec12b578aafa35f7df8fa64f2c05 modules/statsd-7f801701898dec12b578aafa35f7df8fa64f2c05 modules/statsd

2d506cac88a731d142b434a99646926c015af6a3 modules/tattle (heads/master)2d506cac88a731d142b434a99646926c015af6a3 modules/tattle (heads/master)

Git FlowGit Flow● git flow feature start blahgit flow feature start blah

● git flow feature finish blah git flow feature finish blah

● git flow release start blah git flow release start blah

● git flow release finish blah git flow release finish blah

JenkinsJenkins● Pull git Pull git

● ““Build” Build”

•puppet-syntax-puppet-syntax-check.shcheck.sh

•puppet-lint-check.shpuppet-lint-check.sh

•Generate-puppet-Generate-puppet-doc.shdoc.sh

● Package & deploy Package & deploy

DemoDemo$git clon$git clone e git@github.comgit@github.com:KrisBuytaert/vagrant-:KrisBuytaert/vagrant-graphite.gitgraphite.git

$cd vagrant-graphite$cd vagrant-graphite

$git submodule init$git submodule init

$git submodule update $git submodule update There's a submodule in a submodule !There's a submodule in a submodule !

$vagrant up$vagrant up

$vagrant provision$vagrant provision

$vagrant provision (or fix your ordering) $vagrant provision (or fix your ordering)

My code sucked, but I try to make it suck less My code sucked, but I try to make it suck less step by step, step by step,

patches welcomepatches welcome

Homework: Homework: ● puppet-rspecpuppet-rspec

● cucumber-puppetcucumber-puppet

● guardguard

● jenkins vagrant pluginjenkins vagrant plugin

ContactContactKris Buytaert Kris Buytaert Kris.Buytaert@inuits.euKris.Buytaert@inuits.eu

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://krisbuytaert.be/blog/http://krisbuytaert.be/blog/http://www.inuits.eu/http://www.inuits.eu/

InuitsInuitsDuboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231

+32 475 961221+32 475 961221