Puppet @ Citygrid

PuppetCamp LA 2012

Who am I ?

My name is Julien RottenbergRecovering System AdministratorPuppet user since 2009 (at GN)I like to write code and answer questions

Puppet promises

No need to ssh to a server to understand how it works

Look at the puppet code

"Those who cannot remember the past are condemned to repeat it."

George Santayana

literally... go to a server, run `history` as root.

Puppet promises

Guarantee that a server is setup exactly the same way on all environments

Puppet promises

Puppet promises

All the building steps are auditable - SDLCDrift management Open knowledge, no blackbox

Puppet promises

and finally...

Toolset

Open source tools :

● No need to reinvent the wheel● You are not alone with 'that problem',

get help

Toolset

Toolset

puppet doc ● validate code (puppet parser validate)● produce html doc

Running on commit through Jenkins ~ 10 sec

Toolset

Toolset

puppet-lint == coding standard

Takes ~ 2 minPart of the 'Ci-Game'

Toolsettrunk = work in progress - could break but fix ASAPproduction = tested code - do not break

Toolset Trunk --> Production (CherryPicking)We use svnmerge.py

Code in progress = technical debtIt needs to be merged and deployed or reverted Fail the build if pending Cherry Picks > 30 Can't CherryPick yourself ~ code review

ToolsetSvn even with svnmerge has limitations

Git ?

ToolsetVagrant

Sits on top of VirtualboxAllows to 'go crazy'Test, experiment, share configured box$ vagrant provision == puppet apply

ToolsetnVentory

As an External Node ClassifierClasses:

- default_install- gcd

ToolsetnVentory

Helper to build configuration files, etc.ie replacement for Stored Config

ToolsetForeman Api for puppet status, visibility of puppetrun

(Failed attempt on ENC - probably pebkac)

Toolset

Man ! that's a lot of tools !1!!

And it's fine, design by contract. Each application does one thing and was designed to do that thing only (so does it well)Pipe them

Dennis Ritchie

Toolset Boto for

● ec2● route53● elb● sqs● sns

River dolphin native to the amazon WP

Culture cloudEC2 glitches, expect them It used to `just work` :

● gmond (split view dns side effect, using gmetric)

● jmx (need to specify public name at runtime)● ELB pseudo load balancer● Bad neighbors

Culture cloud

One AMI to rule them all (map set per region)

# Build 10 nodes $ create_ec2_instance.py -e dev -s m1.medium -c thd -n 10 # assign them to the hadoop::datanode class (specific install on top of our vanilla)$ nv --name aws1devthd --addtonodegroup hadoop::datanode # send traffic & enable monitoring (configs to be updated on the next puppet run)$ nv --name aws1devthd --set status=inservice

Culture cloud

Culture Collaborate

Culture Collaborate

"If the company's doing well and people don't hate each other, you're probably doing ok."

John Allspaw, speaking at Devopsdays 2011 MV.

You guys ! Make it better : - Integrated function for 'secrets' (no get_var or hiera + gpg)Something like an encrypted data luggage

- puppet agent stability, puppetd vs cron ?

Guys !

Yesterday...

Two last things...

On call rotation side effect

Don't be a human keyboard

Shameless plug

We aggregate advertising & content from local businesses and distribute across our network of web & mobile sites

● Ingest : Hadoop & Cassandra● Serve : Solr & MongoDb + Tomcat● Code : Java mostly

Q/A

puppet doc : http://docs.puppetlabs.com/guides/style_guide.html#puppet-docpuppet-lint : http://puppet-lint.com/Jenkins : http://jenkins-ci.org/svn strategy : http://weblogs.java.net/blog/johnsmart/archive/2009/05/ci_branching_an.htmlsvnmerge.py : http://www.orcaware.com/svn/wiki/Svnmerge.pyvagrant : http://vagrantup.com/nventory : http://sourceforge.net/apps/trac/nventory/wikiforeman : http://theforeman.org/boto : https://github.com/boto/boto Don't be an Ahole : http://www.hollenback.net/DevOpsMeansDontBeAnAholeDevelopers on call - Mitchell Hashimoto : http://vimeo.com/31367609Don't be a human keyboard : http://sysadvent.blogspot.com/2010/12/day-13-dont-be-human-keyboard.html Citygrid is hiring : http://careers.citygrid.com/ Puppet Master LA User Group : http://www.meetup.com/LAPMUG/

This page : goo.gl/soxEs