Transcript
Page 1: Sock Puppets: Growing Your Puppet Codebase

Sock Puppets: Growing your puppet codebase

Fabio Lessa & Andrew Myers

Page 2: Sock Puppets: Growing Your Puppet Codebase
Page 3: Sock Puppets: Growing Your Puppet Codebase

Part 1

Page 4: Sock Puppets: Growing Your Puppet Codebase

Empty server room

Page 5: Sock Puppets: Growing Your Puppet Codebase

Project teams

teams

Page 6: Sock Puppets: Growing Your Puppet Codebase
Page 7: Sock Puppets: Growing Your Puppet Codebase

OPS team for app support

Page 8: Sock Puppets: Growing Your Puppet Codebase

Something that represents a mess

The codebase we've started with

Page 9: Sock Puppets: Growing Your Puppet Codebase

Standard Puppet Master setup

Page 10: Sock Puppets: Growing Your Puppet Codebase

Code was split by environments

Page 11: Sock Puppets: Growing Your Puppet Codebase
Page 12: Sock Puppets: Growing Your Puppet Codebase

Workflow: 1. Push change 2. Wait for it to propagate to that environment 3. Check it worked If you need that applied to another environment, go to 1

Page 13: Sock Puppets: Growing Your Puppet Codebase

The real Workflow: 1. Push change 1.1. Stop the puppet daemon in all the critical nodes for that environment 1.2. Run puppetd manually in one host to ensure it works 1.3. Start the daemon again (on all the hosts you remember) 2. Wait for it to propagate to that environment 3. Check it worked (meh) If you need that applied to another environment, go to 1 (if you remember, that is)

Page 14: Sock Puppets: Growing Your Puppet Codebase
Page 15: Sock Puppets: Growing Your Puppet Codebase

UAT

Production

System Testing

CI

Page 16: Sock Puppets: Growing Your Puppet Codebase

Making changes was painful

Page 17: Sock Puppets: Growing Your Puppet Codebase

Part 2

Page 18: Sock Puppets: Growing Your Puppet Codebase
Page 19: Sock Puppets: Growing Your Puppet Codebase

"don't duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Don't test if puppet is doing it's job,

test that your logic it's doing it's job.

- Nikolay Sturm

Page 20: Sock Puppets: Growing Your Puppet Codebase

Before After

The goal...

Page 21: Sock Puppets: Growing Your Puppet Codebase

So how do we test/refactor

Page 22: Sock Puppets: Growing Your Puppet Codebase

Compile Puppet Catalogues

Page 23: Sock Puppets: Growing Your Puppet Codebase

Compare Versions

Page 24: Sock Puppets: Growing Your Puppet Codebase
Page 25: Sock Puppets: Growing Your Puppet Codebase

Application specific tests

Page 26: Sock Puppets: Growing Your Puppet Codebase

Tying it all together

Page 27: Sock Puppets: Growing Your Puppet Codebase

Part 3

Page 28: Sock Puppets: Growing Your Puppet Codebase

Continuous Delivery

Page 29: Sock Puppets: Growing Your Puppet Codebase

We were moving application deploys, why not do the same with infrastructure?

Page 30: Sock Puppets: Growing Your Puppet Codebase

MCollective

Page 31: Sock Puppets: Growing Your Puppet Codebase

The new setup

Page 32: Sock Puppets: Growing Your Puppet Codebase

Picture of the pipelines

Page 33: Sock Puppets: Growing Your Puppet Codebase

info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/hostinfo/lib/facter/hostinfo.rb info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/concat/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/puppetmaster/manifests/repos.pp:36

Page 34: Sock Puppets: Growing Your Puppet Codebase

info: Retrieving plugin info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: You cannot specify more than one of content, source, target at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/sbs/manifests/init.pp:73

Page 35: Sock Puppets: Growing Your Puppet Codebase

notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message: current_value absent, should be Setting java the following java as default: 1.6.0.31 sun 1.6.0 (noop) notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh' from 1 events notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered 'refresh' from 1 events notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value absent, should be directory (noop) err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not evaluate: Could not retrieve information from environment noop source(s) puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/manifests/init.pp:30 notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value directory, should be absent (noop) notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure: current_value file, should be absent (noop)

Page 36: Sock Puppets: Growing Your Puppet Codebase

We were moving application deploys, why not do the same with infrastructure?

Page 37: Sock Puppets: Growing Your Puppet Codebase
Page 38: Sock Puppets: Growing Your Puppet Codebase

Our MCollective agent

Page 39: Sock Puppets: Growing Your Puppet Codebase

The new action

Page 40: Sock Puppets: Growing Your Puppet Codebase

At every push, all nodes connect to the noop environment and run in noop mode

Page 41: Sock Puppets: Growing Your Puppet Codebase
Page 42: Sock Puppets: Growing Your Puppet Codebase

Picture of a report

Page 43: Sock Puppets: Growing Your Puppet Codebase

Change management process ?

Page 44: Sock Puppets: Growing Your Puppet Codebase

What changed?

Page 45: Sock Puppets: Growing Your Puppet Codebase

What did you test?

Page 46: Sock Puppets: Growing Your Puppet Codebase

Who will do it?

Page 47: Sock Puppets: Growing Your Puppet Codebase

Part 4

Page 48: Sock Puppets: Growing Your Puppet Codebase

New data centre is ready!

Page 49: Sock Puppets: Growing Your Puppet Codebase

Should be easy to migrate....

Page 50: Sock Puppets: Growing Your Puppet Codebase

Okay so our manifests are incomplete, but we need to be able to rebuild machines...

Virtualisation to the rescue...

Page 51: Sock Puppets: Growing Your Puppet Codebase

Magnets?

Page 52: Sock Puppets: Growing Your Puppet Codebase

What happens

Page 53: Sock Puppets: Growing Your Puppet Codebase

Wrap Up

Page 54: Sock Puppets: Growing Your Puppet Codebase

120 pushes to production in the last 8 months: 2 failures

Page 55: Sock Puppets: Growing Your Puppet Codebase
Page 56: Sock Puppets: Growing Your Puppet Codebase

5 minutes!

MTTR FTW!

Page 57: Sock Puppets: Growing Your Puppet Codebase

Thank you!


Top Related