puppet at twitter - puppet camp silicon valley

36
Puppet @Twitter Andrew Hamilton @ahamilton55

Upload: puppet-labs

Post on 23-Dec-2014

846 views

Category:

Documents


0 download

DESCRIPTION

Andrew Hamilton gives a talk about "How Twitter uses Puppet" at Puppet Camp Silicon Valley 2014

TRANSCRIPT

Page 1: Puppet At Twitter - Puppet Camp Silicon Valley

Puppet @Twitter

Andrew Hamilton

@ahamilton55

Page 2: Puppet At Twitter - Puppet Camp Silicon Valley

Who am I?

Page 3: Puppet At Twitter - Puppet Camp Silicon Valley

What we’ll cover

Page 4: Puppet At Twitter - Puppet Camp Silicon Valley

The Dev in DevOps

Page 5: Puppet At Twitter - Puppet Camp Silicon Valley

Version Control

Page 6: Puppet At Twitter - Puppet Camp Silicon Valley

Peer review of code

Page 7: Puppet At Twitter - Puppet Camp Silicon Valley

Review Boardhttp://www.reviewboard.org/

Page 8: Puppet At Twitter - Puppet Camp Silicon Valley

Define your style

Page 9: Puppet At Twitter - Puppet Camp Silicon Valley

OWNERS

Page 10: Puppet At Twitter - Puppet Camp Silicon Valley

OWNERS example

Moving from Puppet 2.7 to Puppet 3.x

Need to go from:

to:source => ‘puppet:///<module>/<filename>’

source => ‘puppet:///modules/<module>/<filename>’

Page 11: Puppet At Twitter - Puppet Camp Silicon Valley

But of course…

Page 12: Puppet At Twitter - Puppet Camp Silicon Valley

You can automate all of this!

Page 13: Puppet At Twitter - Puppet Camp Silicon Valley

Review Board – git review tools

Style Guide – puppet-lint

OWNERS – Script a hook and git review tools

Version control hooks

Page 14: Puppet At Twitter - Puppet Camp Silicon Valley

And always iterate on this

Page 15: Puppet At Twitter - Puppet Camp Silicon Valley

Branching

Page 16: Puppet At Twitter - Puppet Camp Silicon Valley

Head

Our true testing branch

Where development happens

Systems don’t live on this branch

Page 17: Puppet At Twitter - Puppet Camp Silicon Valley

Testing

Works along with our canary system for deployments

Major changes can sit here for a couple days to weeks

Page 18: Puppet At Twitter - Puppet Camp Silicon Valley

Production

99% of our systems are on the production branch

Testing on puppet and head should prevent any error from propagating to production

Page 19: Puppet At Twitter - Puppet Camp Silicon Valley

Workflow

Head

Develop

Review Board

Commit

Test

Testing Production

Cherry pick

Review Board

Test

Commit

Cherry pick

Review Board

Commit

Page 20: Puppet At Twitter - Puppet Camp Silicon Valley

But it could be better

Page 21: Puppet At Twitter - Puppet Camp Silicon Valley

puppetmasterd

Page 22: Puppet At Twitter - Puppet Camp Silicon Valley

Each host runs 3 puppetmasterd processes

Each can serve between ~1k – 2k hosts

We use round-robin DNS

Page 23: Puppet At Twitter - Puppet Camp Silicon Valley

ENC

Page 24: Puppet At Twitter - Puppet Camp Silicon Valley

Audubon

Page 25: Puppet At Twitter - Puppet Camp Silicon Valley

Facts

Mostly deprecated, legacy

Pulled info from Facter

Page 26: Puppet At Twitter - Puppet Camp Silicon Valley

Attributes

Basic host information

Two levels - server, role

Allows for inheritance

Page 27: Puppet At Twitter - Puppet Camp Silicon Valley

Groups

Allows for a logical grouping of hosts

Tells you things like role, rack, cluster, etc

Page 28: Puppet At Twitter - Puppet Camp Silicon Valley

Pulling in info from the ENC is easy

We have functions that pull in each type of change in a manifest

attribute(‘<attr_name>’, ‘<default_value>’)

group(‘<group_name>’, ‘<default_value>’)

Page 29: Puppet At Twitter - Puppet Camp Silicon Valley

Our ENC data store is also what controls the puppet branch our hosts use

We set a group called puppet_branch which is used by our client cronjob

Page 30: Puppet At Twitter - Puppet Camp Silicon Valley

Agents

Page 31: Puppet At Twitter - Puppet Camp Silicon Valley

1. Find branch the host is using

2. Find the port that branch uses on masters

3. Get a master from DNS

1. Health check the master

2. If health continue, else repeat until timeout

4. Run puppet apply against the master

and port

Page 32: Puppet At Twitter - Puppet Camp Silicon Valley

Modules

Page 33: Puppet At Twitter - Puppet Camp Silicon Valley

Team Modules

Base Modules

Service Modules

Page 34: Puppet At Twitter - Puppet Camp Silicon Valley

Create a module for each system service

Chain these together in your service modules

If you do something often, create a module that every group can use

Page 35: Puppet At Twitter - Puppet Camp Silicon Valley

Build for self-service

Page 36: Puppet At Twitter - Puppet Camp Silicon Valley

Looking forward