continuous deployment of puppet modules

CONTINUOUSDEPLOYMENT OF PUPPET

MODULESHOW WE DO IT AT MAILCHIMP

BILL O'NEILL@WONEILL

EMAIL SERVICE PROVIDERDeliver ~500 million emails daily723 million emails delivered on Cyber Monday

of 97Sender Scorehttp://delivery.mailchimp.com/

HISTORY

Image source: https://blog.engineyard.com/2014/con�gure-before-you-boot

CONFIGURATION MANAGEMENT IS HARD"With Chef, Puppet, and CFEngine we found anot-insigni�cant learning curve on setting upthe different server daemons and learning theDSL. This was particularly challenging whenwe were con�guring unique software not yet

given recipes by the existing community.Given our cluster sizes, we also didn't really

need any of the advanced features thosesystems provided."

- README from internally built tool

MOVE TO COLOCATIONBuy vs. Lease analogyGrow our Operations teamNeeded a tool with dry-run mode

PEOPLE MAKE MISTAKES

HOW DO WE CATCH THESE MISTAKES AS EARLYAS POSSIBLE?

AUTONOMATION"automation with a human touch"

1. Detect the abnormality2. Stop3. Fix or correct the immediate condition

HOW DO WE CATCH THESE MISTAKES AS EARLYAS POSSIBLE?

DSL toolsEditor SupportSource Code ManagementContinuous Integration

DSL TOOLSPuppetERBYAMLPuppet Style Guide

PUPPETpuppet parser validate mymanifest.pp

package { 'openssh-server': ensure => installed, }

file { '/etc/ssh/sshd_config': source => 'puppet:///modules/sshd/sshd_config', owner => 'root', group => 'root', mode => '640', notify => Service['sshd'] /* sshd will restart whenever you edit this file. */ require => Package['openssh-server'], }

service { 'sshd': ensure => running, enable => 'true', hasstatus => 'true', hasrestart => 'true', }

$ puppet parser validate validate_1.pp Error: Could not parse for environment production: Syntax error at 'require'; expected '}' at validate_1.pp:12

ERBerb -P -x -T '-' mytemplate.erb | ruby -c

restrict default kod nomodify notrap nopeer<% unless @service %> noquery restrict 127.0.0.1 restrict -6 ::1

driftfile /var/lib/ntp/drift

<% @serverlist.sort.each do |server| -%>server <%= server %> iburst maxpoll 6 restrict <%= server %> mask 255.255.255.255 nomodify notrap noquery <% end -%>

$ erb -P -x -T '-' broken-ntp.conf.erb | ruby -c -:11: syntax error, unexpected $end, expecting kEND

YAMLnpm install -g js-yaml; js-yaml hiera.yaml

ruby -e "require 'yaml'; YAML.load_file('hiera.yaml')"

--- ntp::servers: - 0.us.pool.ntp.org - 1.us.pool.ntp.org - 2.us.pool.ntp.org - 3.us.pool.ntp.org hp::ilo::settings: ssh_status type: global value: true ssh_port type: global value: '22' http_port type: global value: '80' https_port type: global value: '443'

$ js-yaml hiera.yaml JS-YAML: bad indentation of a mapping entry at line 9, column 13: type: global ̂ $ ruby -e "require 'yaml'; YAML.load_file('hiera.yaml')" yaml.rb:133:in ̀load': syntax error on line 9, col 14: ̀ value: true' (ArgumentError) from yaml.rb:133:in ̀load' from yaml.rb:144:in ̀load_file' from yaml.rb:143:in ̀open' from yaml.rb:143:in ̀load_file' from -e:1

PUPPET STYLE GUIDEhttps://docs.puppetlabs.com/guides/style_guide.htmlgem install puppet-lint

puppet-lint --fix /my/puppet/code

package { 'openssh-server': ensure => installed, }

file { '/etc/ssh/sshd_config': source => 'puppet:///modules/sshd/sshd_config', owner => 'root', group => 'root', mode => '640', notify => Service['sshd'], /* sshd will restart whenever you edit this file. */ require => Package['openssh-server'], }

service { 'sshd': ensure => running, enable => 'true', hasstatus => 'true', hasrestart => 'true', }

$ puppet-lint validate_2.pp WARNING: quoted boolean value found on line 16 WARNING: quoted boolean value found on line 17 WARNING: quoted boolean value found on line 18 WARNING: indentation of => is not properly aligned on line 6 WARNING: indentation of => is not properly aligned on line 7 WARNING: indentation of => is not properly aligned on line 8 WARNING: indentation of => is not properly aligned on line 9 WARNING: indentation of => is not properly aligned on line 10 WARNING: mode should be represented as a 4 digit octal value or symbolic mode on line 9 WARNING: /* */ comment found on line 10

$ puppet-lint --fix validate_2.pp FIXED: quoted boolean value found on line 16 FIXED: quoted boolean value found on line 17 FIXED: quoted boolean value found on line 18 FIXED: indentation of => is not properly aligned on line 6 FIXED: indentation of => is not properly aligned on line 7 FIXED: indentation of => is not properly aligned on line 8 FIXED: indentation of => is not properly aligned on line 9 FIXED: indentation of => is not properly aligned on line 10 FIXED: mode should be represented as a 4 digit octal value or symbolic mode on line 9 FIXED: /* */ comment found on line 10

package { 'openssh-server': ensure => installed, }

file { '/etc/ssh/sshd_config': source => 'puppet:///modules/sshd/sshd_config', owner => 'root', group => 'root', mode => '0640', notify => Service['sshd'], # sshd will restart whenever you # edit this file. require => Package['openssh-server'], }

service { 'sshd': ensure => running, enable => true, hasstatus => true, hasrestart => true, }

--- validate_2.pp 2014-12-08 09:43:38.000000000 -0500 +++ validate_2.pp-fixed 2014-12-08 09:50:51.000000000 -0500 @@ -3,18 +3,18 @@ } file { '/etc/ssh/sshd_config': - source => 'puppet:///modules/sshd/sshd_config', - owner => 'root', - group => 'root', - mode => '640', - notify => Service['sshd'], /* sshd will restart whenever you - edit this file. */ + source => 'puppet:///modules/sshd/sshd_config', + owner => 'root', + group => 'root', + mode => '0640', + notify => Service['sshd'], # sshd will restart whenever you + # edit this file. require => Package['openssh-server'], } service { 'sshd': ensure => running, - enable => 'true',- hasstatus => 'true',- hasrestart => 'true',+ enable => true, + hasstatus => true, + hasrestart => true, }

EDITOR SUPPORT

VIMPLUGINS FTW

with

Syntasticvim-puppetUltiSnips vim-snippets

EMACShttp://www.emacswiki.org/emacs/PuppetProgramming

GEPPETTOhttp://puppetlabs.github.io/geppetto/index.html

SOURCE CODEMANAGEMENT

COMMIT HOOKSSCRIPT RUNNING THE DSL TOOLS AGAINST NEW FILES

PEER REVIEW

TRUNK BASEDDEPLOYMENT

CONTINUOUSINTEGRATION

JENKINSHTTPS://GITHUB.COM/VSTONE/JENKINS-

PUPPET-SCRIPTS

WHY NOT RSPEC ORBEAKER?

CONTINUOUSDEPLOYMENT

REMEMBER TRUNK BASEDDEPLOYMENT?

# Keep environment up-to-date vcsrepo { '/etc/puppet/environments/production': ensure => latest, provider => hg, source => 'https://localhost/mercurial/puppet-modules', }

REVIEW TIME!Catch mistakes earlyAutomation with a human touchTrunk Based Deployments

QUESTIONS?

Slide sources at

THANKS!BILL O'NEILL

WONEILL@POBOX.COM@WONEILL

http://github.com/woneill/puppetcamp_atlanta_2014