sock puppets

57
Sock Puppets: Growing your puppet codebase Fabio Lessa & Andrew Myers

Upload: fabio-lessa

Post on 25-Dec-2014

1.101 views

Category:

Technology


0 download

DESCRIPTION

 

TRANSCRIPT

Page 1: Sock puppets

Sock Puppets: Growing your puppet codebase

Fabio Lessa & Andrew Myers

Page 2: Sock puppets
Page 3: Sock puppets

Part 1

Page 4: Sock puppets

Empty server room

Page 5: Sock puppets

Project teams

teams

Page 6: Sock puppets
Page 7: Sock puppets

OPS team for app support

Page 8: Sock puppets

Something that represents a mess

The codebase we've started with

Page 9: Sock puppets

Standard Puppet Master setup

Page 10: Sock puppets

Code was split by environments

Page 11: Sock puppets
Page 12: Sock puppets

Workflow: 1. Push change 2. Wait for it to propagate to that environment 3. Check it worked If you need that applied to another environment, go to 1

Page 13: Sock puppets

The real Workflow: 1. Push change 1.1. Stop the puppet daemon in all the critical nodes for that environment 1.2. Run puppetd manually in one host to ensure it works 1.3. Start the daemon again (on all the hosts you remember) 2. Wait for it to propagate to that environment 3. Check it worked (meh) If you need that applied to another environment, go to 1 (if you remember, that is)

Page 14: Sock puppets
Page 15: Sock puppets

UAT

Production

System Testing

CI

Page 16: Sock puppets

Making changes was painful

Page 17: Sock puppets

Part 2

Page 18: Sock puppets
Page 19: Sock puppets

"don't duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Don't test if puppet is doing it's job,

test that your logic it's doing it's job.

- Nikolay Sturm

Page 20: Sock puppets

Before After

The goal...

Page 21: Sock puppets

So how do we test/refactor

Page 22: Sock puppets

Compile Puppet Catalogues

Page 23: Sock puppets

Compare Versions

Page 24: Sock puppets
Page 25: Sock puppets

Application specific tests

Page 26: Sock puppets

Tying it all together

Page 27: Sock puppets

Part 3

Page 28: Sock puppets

Continuous Delivery

Page 29: Sock puppets

We were moving application deploys, why not do the same with infrastructure?

Page 30: Sock puppets

MCollective

Page 31: Sock puppets

The new setup

Page 32: Sock puppets

Picture of the pipelines

Page 33: Sock puppets

info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/hostinfo/lib/facter/hostinfo.rb info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/concat/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/puppetmaster/manifests/repos.pp:36

Page 34: Sock puppets

info: Retrieving plugin info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: You cannot specify more than one of content, source, target at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/sbs/manifests/init.pp:73

Page 35: Sock puppets

notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message: current_value absent, should be Setting java the following java as default: 1.6.0.31 sun 1.6.0 (noop) notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh' from 1 events notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered 'refresh' from 1 events notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value absent, should be directory (noop) err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not evaluate: Could not retrieve information from environment noop source(s) puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/manifests/init.pp:30 notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value directory, should be absent (noop) notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure: current_value file, should be absent (noop)

Page 36: Sock puppets

We were moving application deploys, why not do the same with infrastructure?

Page 37: Sock puppets
Page 38: Sock puppets

Our MCollective agent

Page 39: Sock puppets

The new action

Page 40: Sock puppets

At every push, all nodes connect to the noop environment and run in noop mode

Page 41: Sock puppets
Page 42: Sock puppets

Picture of a report

Page 43: Sock puppets

Change management process ?

Page 44: Sock puppets

What changed?

Page 45: Sock puppets

What did you test?

Page 46: Sock puppets

Who will do it?

Page 47: Sock puppets

Part 4

Page 48: Sock puppets

New data centre is ready!

Page 49: Sock puppets

Should be easy to migrate....

Page 50: Sock puppets

Okay so our manifests are incomplete, but we need to be able to rebuild machines...

Virtualisation to the rescue...

Page 51: Sock puppets

Magnets?

Page 52: Sock puppets

What happens

Page 53: Sock puppets

Wrap Up

Page 54: Sock puppets

120 pushes to production in the last 8 months: 2 failures

Page 55: Sock puppets
Page 56: Sock puppets

5 minutes!

MTTR FTW!

Page 57: Sock puppets

Thank you!