performance tuning your puppet infrastructure - puppetconf 2014
DESCRIPTION
Performance Tuning Your Puppet Infrastructure - Nic Benders, New RelicTRANSCRIPT
![Page 1: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/1.jpg)
Performance TuningYour (Puppet) Infrastructure
Nic Benders
OWNING
http://nicbenders.com/presentations/puppetconf-2014
![Page 2: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/2.jpg)
I work at New Relic. !
This is a story about treating your infrastructure the same
way you treat your applications. !
This is also a story about how we use New Relic,
at New Relic.
http://nicbenders.com/presentations/puppetconf-2014
![Page 3: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/3.jpg)
Managing Our Applications
• Source Control
• Continuous Deployment
• Performance Monitoring
• Operational Analytics
• Log Collection
GitHub with Pull Requests
Jenkins + Capistrano
New Relic APM
New Relic Insights
Heka + ElasticSearch + Kibana
![Page 4: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/4.jpg)
Managing Our Infrastructure
• Source Control
• Continuous Deployment
• Performance Monitoring
• Operational Analytics
• Log Collection
GitHub with Pull Requests
Jenkins + Capistrano
New Relic APM
New Relic Insights
Heka + ElasticSearch + Kibana
![Page 5: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/5.jpg)
Source Control
![Page 6: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/6.jpg)
![Page 7: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/7.jpg)
Continuous Deployment
![Page 8: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/8.jpg)
![Page 9: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/9.jpg)
Performance Monitoring
![Page 10: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/10.jpg)
DON’T PANIC• These example are for Puppet Enterprise 3.1, but this will work with Puppet
Open Source, or with newer versions. You’ll just need to make a few changes here and there.
• If you don’t have a New Relic account, you can sign-up for a free “Lite” account.
• All of the examples are up in a more useful format at the URL below. (The URL will also be shown at the end of the presentation.)
http://nicbenders.com/presentations/puppetconf-2014
![Page 11: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/11.jpg)
Instrumenting the Puppet Master
• Install newrelic_rpm using the Puppet’s Ruby interpreter
• Modify the config.ru in the Puppet Master root dir
• Create a newrelic.yml in the same directory
# /var/opt/lib/pe-‐puppetmaster/config.ru !# Setup the New Relic Ruby Agent before anything else require 'rubygems' require 'newrelic_rpm' # END New Relic Ruby Agent setup
# /var/opt/lib/pe-‐puppetmaster/newrelic.yml !common: &default_settings license_key: '000-‐your-‐license-‐key-‐here-‐000' app_name: "Puppet Master" log_level: info log_file_path: '/var/log/pe-‐puppet/' ssl: false capture_params: true transaction_tracer: enabled: true error_collector: enabled: true ignore_errors: "ActionController::RoutingError" production: <<: *default_settings monitor_mode: true
![Page 12: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/12.jpg)
![Page 13: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/13.jpg)
Getting good transaction names
• Use set_transaction_name in a Rack Middleware to tell the Ruby Agent how we are going to name the “web transactions” for the Puppet Master
• Use add_custom_parameters to pass the environment and other useful data along
# # We don't want every transaction to just say "call", so define a # Transaction "namer" for the Ruby Agent based on the URL structure # class CoolNamer def initialize(app, opts = {}) @app = app end def call(env) req = Rack::Request.new(env) _, environment, kind_of_thing, resource = req.path.split('/',4) NewRelic::Agent.set_transaction_name(kind_of_thing, :category => :rack) NewRelic::Agent.add_custom_parameters(:puppet_env => environment) NewRelic::Agent.add_custom_parameters(:puppet_resource => resource) @app.call(env) end end use CoolNamer # END Transaction "namer"
![Page 14: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/14.jpg)
![Page 15: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/15.jpg)
Getting more detail• The Ruby Agent’s MethodTracer
allows us to get details of what is going on inside a Transaction.
• By hooking into Puppet’s built-in Profiler calls, we can quickly instrument all the most important things.
• This example is for Puppet 3.3.1. In newer versions it is much easier.
require "puppet/util/profiler" module Puppet::Util::Profiler include ::NewRelic::Agent::MethodTracer def self.profile(message, &block) if /^Processed request / === message # Top level trace, skip for now yield else metric_name = self.message_to_metric(message) trace_execution_scoped([metric_name]) do yield end end end def self.message_to_metric(message) message.chomp! message.gsub!(/^(Compiled catalog) for .*/, '\1') message.gsub!(/^(Filtered result for catalog) .*/, '\1') message.gsub!(/\(\d+\) /, '') message.gsub!(/^(Evaluated resource) /, '\1/') message.gsub!(/\[.*\]/,'') message.gsub!(/\:\s/,'/') message.gsub!(/^(Called) /, '\1/') "Custom/Puppet/#{message}" end end
![Page 16: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/16.jpg)
![Page 17: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/17.jpg)
Instrumenting the Puppet DB
• Download the Java Agent from your “Account Settings” page.
• Create an /opt/puppet/newrelic dir and place the newrelic.jar and newrelic.yaml file into it.
• Modify the newrelic.yaml
• Modify JAVA_ARGS
# /opt/puppet/newrelic/newrelic.yml !common: &default_settings license_key: '000-‐your-‐license-‐key-‐here-‐000' ! app_name: 'Puppet DB' log_level: info log_file_path: /var/log/pe-‐puppetdb/ ssl: false transaction_tracer: enabled: true transaction_threshold: apdex_f record_sql: obfuscated stack_trace_threshold: 0.2 explain_enabled: true explain_threshold: 0.2 browser_monitoring: auto_instrument: false production: <<: *default_settings
# /etc/syslog/pe-‐puppetdb !# Enable the New Relic Java Agent by adding this # to the bottom of the file JAVA_ARGS="-‐javaagent:/opt/puppet/newrelic/newrelic.jar ${JAVA_ARGS}"
![Page 18: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/18.jpg)
![Page 19: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/19.jpg)
![Page 20: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/20.jpg)
![Page 21: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/21.jpg)
Operational Analytics
![Page 22: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/22.jpg)
Log Collection
![Page 23: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/23.jpg)
Puppet Agent already goes into Syslog
• Agent syslogs have a ton of useful information. We just need to get it to a useful place.
• Our application and syslog data is collected by a Heka process on each server. It then makes its way to ElasticSearch, where it can be queried by Kibana.
![Page 24: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/24.jpg)
Send run reports to the same place
• Add your log system as a Report processor!
• We use https://github.com/logstash/puppet-logstash-reporter
• To get the data into Heka, we had to write some Lua, but it was pretty straightforward.
![Page 25: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/25.jpg)
# Logstash-‐style report processer through Heka class { 'logstash_reporter': logstash_port => '22253', logstash_host => hiera('logstash_reporter::logstash_host'), }
pe_puppet::master::config_hash: reports: 'http,puppetdb,logstash' !logstash_reporter::logstash_host: 'our-‐heka-‐server'
![Page 26: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/26.jpg)
Managing Our Infrastructure
• Source Control
• Continuous Deployment
• Performance Monitoring
• Operational Analytics
• Log Collection
GitHub with Pull Requests
Jenkins + Capistrano
New Relic APM
New Relic Insights
Heka + ElasticSearch + Kibana
![Page 27: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/27.jpg)
Site Reliability Engineers MySQL Experts Engineering Managers Data Pipeline Engineers newrelic.com/about/careers
We’re Hiring
![Page 28: Performance Tuning Your Puppet Infrastructure - PuppetConf 2014](https://reader034.vdocuments.us/reader034/viewer/2022052619/5560cac1d8b42a19088b4af2/html5/thumbnails/28.jpg)
http://nicbenders.com/presentations/puppetconf-2014