Streamlining your puppet development workflow

Tomas  Doran  @bobtfish  2014-­‐07-­‐07

Why invest in your workflow?

2

More time

3

To concentrate on

4

The important things in life

5

Like dwarf fortress!

6

Why invest in your workflow?• Productivity!

• Work smarter, not harder

• You spend a lot of time writing/testing/debugging code • Optimizing that is worthwhile

• Faster development cycle is more productive • 5% faster cycle • 5% more time for testing • Less bugs

7

Revision control

• You must have your code in revision control

• git is preferred • fast + cheap branches • everyone else uses it • github • gitolite

!• svn is also ‘workable’

8

How do you run puppet?

• I like cron (daemon also fine) • Two possible approaches • —noop mode automatically + manual apply • Automatic apply

9

How do you run puppet?

• I like cron (daemon also fine) • Two possible approaches • —noop mode automatically + manual apply • Automatic apply

10

How do you run puppet?

• I like cron (daemon also fine) • Two possible approaches • —noop mode automatically + manual apply • Automatic apply

• I recommend automatic apply • Scary (don’t push to master unless you’re

confident!) • puppet agent —disable (monitor this!) • Testing workflow • Eventual consistency

11

Move just fast enough to not break everything

• Test so that you’re confident • Branch for every significant change • Reduce batch size • Small scary change easier to test • Easier to roll back

• Otherwise - applying months of changes at once • Really scary! • Don’t even know desired effects!

• Communicate!

12

Standalone modepuppet apply --modulepath modules/:vendor/modules/ --show_diff --hiera_config=./standalone-hiera.yaml $@ manifests/site-standalone.pp !

13

—noop

• Use —noop mode for testing!

ssh -A "$HOST" -- "sh -c 'cd $DESTDIR/$PUPPET_DIRNAME; ./tools/puppet-standalone --verbose --show_diff —noop'"

• tools/what-would-happen-on

14

Dynamic environmentsgit branch => puppet environment

puppet agent -t —environment my_test_branch !

• puppet >= 2.7 has environment support

• Use puppetupdate or r10k to push branches (Links at the end!)

15

16

Reporting

• Need to know what puppet did • Puppet has logs + reporting functionality • Push reports to: • irc • email (eww!) • elasticsearch • mysql • puppetdb

• Saves compiled catalogs to disk tools/what-just-happened-on

17

Foreman

18

Norman

19

Puppet enterprise

20

puppet-syntax

• Ruby gem • Trivial to add to your project • Checks .pp, .erb, .yaml • Fast enough to run pre-commit

echo ‘bundle exec rake syntax’ \ >.git/hooks/pre-commit \ chmod 755 .git/hooks/pre-commit

21

r10k/librarian - Puppetfile• Awesome module deployment - with robots! • Easy vendor/modules directory for modules from the forge • Makes module = git repository pattern easier • Not every module from the forge is useable immediately

• Fork on github (and make your changes open source)? • Pull request and get them back upstream! • Fork into internal git and modify.

• gitolite mirrors • Improve performance • No external dependencies

• Private forge (puppet-library)22

Module template

• ‘puppet module generate’ uses a template • Start from the GDS example one:

github.com/gds-operations/puppet-module-skeleton • Modify to your taste!

23

‘Real’ testing

• No hard rules. • Invest to the level that’s right for you!

• Dev heavy teams • Know about unit testing!

• Sysadmin heavy teams • Less enthusiastic

• Do what provides value!24

Feedback!

• Tighten your OODA loop! • Don’t care how! • N.B. Automated tests don’t work unless they’re

automated. • I.E. MUST run on commit

• Whatever’s effective for your org25

rspec-puppet

• Unit testing • At least write a compile test for your code! • Put it in your module template. • Explicit dependencies FTW

!• Use puppetlabs-spec-helper • Inject mocks into spec/fixtures/manifests/site.pp

echo ‘define my::complex::dependency ($foo, $bar) {}’ >> spec/fixtures/manifests/site.pp

26

serverspec

• Spin up Vagrant VM and apply your code • Check properties of: • files • ports • services

• Acceptance testing • Slower and heavier weight than unit tests • Can be highly valuable!

27

Jenkins - simple• Put your tests together so that they can be run as one job

!rake test

task :test => [:syntax, :spec, :integration]

• Get Jenkins to run it on commit to master • git polling • + add a post-receive hook to curl Jenkins

• Shout in email + irc!28

Jenkins - less simple

• If branches are cheap (i.e. git!) • Encourages people to push branches • Code review++ • Adhoc • Or pick your poison

• Run syntax checks and unit and/or integration tests on every branch.

• Report back to committer • irc notification • write in code review

29

Jenkins integration branches• Jenkins can merge branches! • Push a branch • Jenkins picks it up • Merges with master • Runs tests • If they pass, pushes results

30

Jenkins integration branches• Jenkins can merge branches! • Push a branch • Jenkins picks it up • Merges with master • Runs tests • If they pass, pushes results

31

Do everything in irc

32

True believers

33

34

Policy vs Automation

• Testing is awesome • Code review is awesome • ‘Process is the scar tissue from previous problems’ • Empower people to change the process!

• If you make the tools simple to use… • People will use them! • Make doing the right thing a no-brainer

35

We’re hiring! NY, SF, Dublin, London! !

• https://slideshare.net/bobtfish/ • http://puppetlabs.com/blog/git-workflow-and-puppet-environments • http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-1/ • http://garylarizza.com/blog/2014/02/18/puppet-workflow-part-3/ • https://github.com/youdevise/puppetupdate • https://github.com/adrienthebo/r10k • http://gitolite.com/gitolite/index.html • http://puppetlabs.com/puppet/puppet-enterprise/ • http://docs.puppetlabs.com/guides/reporting.html • http://theforeman.org/ • https://github.com/youdevise/norman/ • https://github.com/youdevise/puppet-logstash-reporter/ • https://github.com/gds-operations/puppet-module-skeleton • https://github.com/drrb/puppet-library 36