puppet atbazaarvoice
TRANSCRIPT
Puppet at Bazaarvoice
Hi, my name is Dave.
• DevOps Engineer at Bazaarvoice.• Started working with puppet in 2008 while working at
Bioware. First version was 0.24.• At Bioware, puppet configured over 14k nodes that
comprised of web servers, databases and game servers.• All 5 datacenters (in California, Virginia, Ireland,
Australia and Texas) housed puppet managed nodes that all reported back to a centralized puppet dashboard.
• My contact info:– http://www.linkedin.com/in/jamesbarcelo
Bazaarvoice Plug
• We do embedded DevOps!• Application teams are responsible for their
application’s operation success.• 2.0 stack is 100% in Amazon!• Conferences!• Work on awesome projects with spiffy tech like
Cassandra or ElasticSearch.• Send me your resume!
Preview presentation
• Puppet in the legacy stack.• Puppet in the Data Infrastructure Team.– Focus on client/server.
• Puppet in the Data Services Team.– Focus on masterless puppet.
Puppet in the Legacy Infrastructure
• Traditional puppet use with client/server.• Multiple levels of inheritance.• Puppet managed instances were configured
according to DNS naming convention.
node /my-hostname/ { … …}
• Some issues encountered:– Very hard to work with. Very complex.– Large codebase. Adds to complexity. – No confidence in making changes. Side effects
feared after code change. A jinga tower of puppet code.
– Too many pivot points. Many places to configure. Adds to complexity.
– Lots of code rot. Had not been refactored.
Puppet master/client in Data Infrastructure teams
Architecture
• Each server type we care about will be referenced by its role. We only care about roles, not hostnames.
• Centered around an uber IT tools server that runs everything ops (including puppet) to do work in an environment. The Mothership.
• Hiera and parameterized classes will be used to create generic puppet modules that can be reused for different roles.
• Development will be centered on using puppet environments on the Mothership to protect devs from stepping on each other.
Mothership
• Contains a cocktail of different application tools for doing work in the environment. Tools included:– Mcollective/ActiveMQ– Puppet server– Puppet managed operation scripts.
• Motherships configured to be highly available in regular AWS fashion(Autoscaling, cluster multiple activeMQ, etc).
• Advertises multiple puppet environments that clients can switch between via –environments.
What is a Universe?
Anatomy of a Mothership
Methods of passing in data
Getting environment data into puppet configuration. • Hiera datastore.• Puppet stdlib/tags.txt• Cloud formation parameters – Universe, VPC
Puppet Stdlib/facts.d
• Bootstrap process (EC2 user data) populates /etc/facter/facts.d/tags.txt with mappings. These mappings become facters.
• Example of data in tags.txt:– Universe value.– Ec2 metadata.
/etc/facter/facts.d/tags.txt:universe=devTag_region=us-east-1
Hiera datastore
• Hiera is used extensively where different data needs to be passed into puppet according to context. Different contexts would include:– Amazon region.– Instance role.– Universe.
• Example usage:$app_version = hiera(‘app_version’, nil)
/etc/hiera.yaml--:logger: console
:hierarchy:
- %{fqdn} - 10-team/%{team}/10-region/%{tag_region}/10-universe/%{universe}/10-roles/%{role} - 10-team/%{team}/10-region/%{tag_region}/10-universe/%{universe}/20-common - 10-team/%{team}/10-region/%{tag_region}/20-roles/%{role} - 10-team/%{team}/10-region/%{tag_region}/30-common - 10-team/%{team}/20-universe/%{universe}/10-roles/%{role} - 10-team/%{team}/20-universe/%{universe}/20-common - 10-team/%{team}/30-roles/%{role} - 10-team/%{team}/40-common
- 40-common
- environments
:backends: - yaml - json
:yaml: :datadir: /etc/puppet_env/%{environment}/manifests/hieradata
:json: :datadir: /etc/puppet/env/global_hieradata
Puppet Environments/Universe
Mothership Execution Flow
Puppet code on the Mothership• The files that do the things:– /etc/puppet/puppet.conf– /etc/puppet/env/global_hieradata/
environments.json– /etc/puppet/puppetmaster.conf– /etc/puppet_env/{puppet_env}/…• manifests/…• modules/…
/etc/puppet/puppet.conf# File managed by Puppet.
[main] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl
usecacheonfailure = true pluginsync = true factpath = $vardir/lib/facter preferred_serialization_format = yaml
[user] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet
ssldir = $vardir/ssl
[agent] runinterval = 1800 ca_server = <%= ca_srv %> server = <%= logical_srv %> certificate_revocation = False environment = <%= environment %> report = true
/etc/puppet/env/global_hieradata/environments.json
{ "environments": [ { "cert": [ { "modulepath": "/etc/puppet_env/cert/modules" }, { "manifestdir": "/etc/puppet_env/cert/manifests" }, { "manifest": "/etc/puppet_env/cert/manifests/site.pp" } ] }}
/etc/puppet/puppetmaster.conf[main] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl
usecacheonfailure = true pluginsync = true factpath = $vardir/lib/facter preferred_serialization_format = yaml syslogfacility = local1
[master] certname=<%= certname %> ca = True certificate_revocation=False dns_alt_names=<%= logical_srv %> ssl_client_header = SSL_CLIENT_S_DN ssl_client_verify_header = SSL_CLIENT_VERIFY autosign = true
# For puppet dashboard reporting. reports = store, datadog_reports
<% if store_configs == true %> # Puppetdb.
storeconfigs = true storeconfigs_backend = puppetdb <% end %>
[user] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl
# Environments<% environments.each do |env_val| -%><% env_val.keys.each do |env_key| -%>[<%= env_key -%>]<% env_val[env_key].each do |env_data| -%><% env_data.each_pair do |k, v| -%><%= k %> = <%= v -%><% end %><% end %><% end %><% end %>
/etc/puppet_env/{env}/manifests/site.pp
import 'roles/*.pp'
node default { class { 'basenode_role': }
class { "$tag_role": }}
/etc/puppet_env/{env}/manifests/role/00_basenode.pp
class basenode_role { class { ‘security’: } class { ‘monitoring’: } …..}
/etc/puppet_env/{env}/manifests/role/mothership_role.pp
class mothership_role {
class { 'puppet': master => true, ca_srv => $tag_caserver, logical_srv => $tag_puppet_server, }
}
Mothership Dev Workflow
Masterless Puppet in Data Services Teams
Architecture
Still keeping bits of the Mothership project:• Applications/Services scoped in zookeeper by
Universe.• Emphasis is put on making things simpler.
Puppet code will not be monolithic. Individual application teams will only need to maintain there own modules/manifests.
• Changes to modules/manifests will not impact other teams.
Methods of passing in data
• The usual suspects:– Puppet stdlib/tags.txt.– Hiera.– Cloud formation parameters – Universe, VPC
• Some new ones:– EC2 data/metadata -> facter.– Zookeeper.– Cloud formation parameters - DeployTag
getEC2data_cache.rb
• Script runs out of /etc/facts.d that converts EC2 data/metadata into facts.
Zookeeper/Ostrich
• Custom functions to pull data from zookeeper the same way applications do discovery.
Masterless Execution Flow
Puppet code in Masterless
• No more Mothership. All work is done via puppet apply.– /etc/hiera.yaml– /etc/puppet/manifests/{role}.pp– /etc/puppet/manifests/00_common.pp– /etc/puppet/manifests/01_users.pp– /etc/puppet/modules/…
/etc/hiera.yaml--:logger: console
:hierarchy: - %{fqdn}
- 10-universe/%{universe}/10-roles/%{role} - 10-universe/%{universe}/20-common - 20-roles/%{role} - 30-common
:backends: - yaml
:yaml: :datadir: /etc/puppet/manifests/hieradata
/etc/puppet/manifests/{role}.ppimport '00_common'
node default {
# This class contains common modules that should be used by all roles. class { 'common': }
class { 'activemq’: } -> class { 'mcollective': server => true, client => true, }
}
/etc/puppet/manifests/00_common.ppimport '01_users'###################################### Common#####################################
class common {
class { 'stdlib': }
file { '/opt/bazaarvoice': ensure => directory, }
# Authorized keys for project developers. class { 'user_setup': stage => setup, }
host { 'internal_ip': ensure => 'present', name => $fqdn, ip => $ipaddress, }
class { 'prompt': }}
/etc/puppet/manifests/01_users.ppclass user_setup {
include users
users::user { 'dbarcelo': groups => 'wheel', sshKey => 'ssh-dss AAAAB3NzaC1kc3MAAACBANL1zoZdYJp/6vQ4G5iNQXjdJ7NGmK0J2eqHbztvuD0CBPyqMuEtuYKRg14tFd4iwp5EpnT4UWpv8kXF/dkEN3b5xgN/R+1hYq7/3mnRLchMFTl0tyryLuARC9zTI003mQrXd/W9jzXaNlCTpxh8Ihj2Ov3lvAAX65tN9nijxhCTAAAAFQCgMU0obmTLo5CRYtRwDCkj1mb2hQAAAIAiZF9axkCvMa9vwigDiAf3rNMbut1gtqtwdzux8c9T1inApKV5sccjg5POKm+4WmWTBOtQfYR8cNot2Mn/mO+MRiKH8sYapYnU2es+KRBmhdARE+N7EqdD0WqoP7NrsNVbObHwDQBNkODuc3ZPyTQuqv/w4poTXaS5u5M1XZbgZwAAAIEAjt4r7SN1I/m0V/TvedTVxJvKln4wZkFxyI5CAgpsAr435kwSLM08R9Hd0/5Vy9LfhYpH1aZTBaoTqmTCtnv3mp1coXoscEp5nE0llfm+4DX3YvWnR80S/OeMUe71Ucm1ORwFpST/K4WKQoZ30TAVVsc8nYy2hyD7hyozjzsS09o= dave.barcelo@dbarcelo-mbpro' } users::user { 'lwadhwani': groups => 'wheel', sshKey => 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAysFCPpffw9LIOqAEFZxOOb52m2FbHhumBFc07o8sm3c4cmdLq/bBtr5TyuQp89zVNEaTGRbw1nMpQCDno4i5ipTvCLoKkOE1PRdtyJw6PGu6VV/0U1ghK+1xmveM2jDX/otj5hjnQiRm1+Fx/orYwNBkywDlDHZQCGxalWaFgXVyReCRUqq0jBwj3EKJfsQgoxuSrh7F6GjsQ6DUOsA3wBfewS25hPmhulEqvga4/P58BMHemL9d4Ugu98Vg7fgaur/b1adX+LzbmE6C2T4Gn1kzAOEct6bFgLPRj3n5/EaspdOsZ/Nnik0LUvIwZNHgDCLgkS0D8aMIsiUrB4OqSw== luveen@Pantalaimon' }}
/etc/puppet/modules/…
• Do stuff!• Code is still generic but it does not have to be.
Masterless Dev Workflow