puppet camp düsseldorf 2014: monitoring with puppet (beginner)
Post on 25-Dec-2014
458 Views
Preview:
DESCRIPTION
TRANSCRIPT
Monitoring (with) PuppetMonitoring (with) Puppet
PuppetCamp Dusseldorf
@KrisBuytaert
KrisKris BuytaertBuytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Offcer and Open Source Consultant Chief Trolling Offcer and Open Source Consultant
@inuits.eu@inuits.eu● Everything is an effng DNS ProblemEverything is an effng DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Organising too many confs , #devopsdays, Organising too many confs , #devopsdays,
#loadays, ... #loadays, ... ● Evangelizing devopsEvangelizing devops
Do you know what your children do at 5 am in the Do you know what your children do at 5 am in the morning ?morning ?
● Sleeping ?Sleeping ?
● Crashing a Party ?Crashing a Party ?
● Why are the cops at your front door ?Why are the cops at your front door ?
● What happened ?What happened ?
● How long have they been gone ?How long have they been gone ?
devops = clamsdevops = clams● CultureCulture
● (Lean)(Lean)
● Automate all the things ... Automate all the things ...
• Build Automation Build Automation
• Test Automation Test Automation
• IACIAC
● Monitoring , Metrics ... Monitoring , Metrics ...
● SharingSharing
Monitoring is usually an aftertoughtMonitoring is usually an aftertoughtENOBUDGET, ENOTIMEENOBUDGET, ENOTIME
#monitoringsucks#monitoringsucks● John Vincent (@lusis)John Vincent (@lusis)
● A sub movement A sub movement
● https://github.com/monitoringsucks/https://github.com/monitoringsucks/
#monitoringlove#monitoringlove• #monitoringlove hacksessions#monitoringlove hacksessions
• #monitorama#monitorama
For years we've tolerated humans to to For years we've tolerated humans to to make structural manual changes to the make structural manual changes to the infrastructure our critical applications are infrastructure our critical applications are running on.running on.
Whilst at the same time demanding those Whilst at the same time demanding those critical applications to go trough rigid test critical applications to go trough rigid test
scenarios.scenarios.
Who let this happen ?Who let this happen ?
Infrastructure as CodeInfrastructure as Code● Model our infrastructure Model our infrastructure
● A fast reproducable platformA fast reproducable platform
● Disaster discovery for “free”Disaster discovery for “free”
Infrastructure as CodeInfrastructure as Code● Code = Code Code = Code
● Version Control Version Control
● Quality ChecksQuality Checks
● TestingTesting
● Continuous IntegrationContinuous Integration
● Continous DeliveryContinous Delivery
Infrastructure as CodeInfrastructure as Code● Core Infrastructure Core Infrastructure
● Middleware deployment and integrationMiddleware deployment and integration
● Automated continuous application Automated continuous application deploymentdeployment
● Integrated Security enforcementIntegrated Security enforcement
● Host, Service and Application Monitoring Host, Service and Application Monitoring confguredconfgured
Why #monitoringsucksWhy #monitoringsucks● Manual confg (gui)Manual confg (gui)
● Not in sync with realityNot in sync with reality
● Hosts onlyHosts only
● Services sometimesServices sometimes
● Appliccation neverAppliccation never
● ChaosChaos
• Really ?Really ?
• Service,Service,
• FunctionalitiesFunctionalities
• eg. vhosts etceg. vhosts etc
• Single Source of TruthSingle Source of Truth
But tools do AutodetectionBut tools do Autodetection
Monitoring 101Monitoring 101● Deploy a host,Deploy a host,
● Add it to the monitoringAdd it to the monitoring
● Add collection toolsAdd collection tools
● Add check defnitionsAdd check defnitions
● Update the monitoring tool confgUpdate the monitoring tool confg
● RestartRestart
Collect Metrics 101Collect Metrics 101
CollectdCollectd● CollectsCollects
● Zillion PluginsZillion Plugins
• Nginx,apache, mysql, diskNginx,apache, mysql, disk
● Graphite Carbon PluginGraphite Carbon Plugin
● Send metrics to graphiteSend metrics to graphite
● https://github.com/KrisBuytaert/puppet-https://github.com/KrisBuytaert/puppet-collectdcollectd
Puppet and GraphitePuppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/https://github.com/KrisBuytaert/vagrant-graphite/
● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules ! Tattle and more modules as submodules !
● git clonegit clone
● git submodule init git submodule init
● git submodule update git submodule update
● vagrant up vagrant up
2014 style dashboards2014 style dashboards
Alerting 101Alerting 101
Alert all the thingsAlert all the thingsAnd get alertfatigueAnd get alertfatigue
● We are in Germany, EuropeWe are in Germany, Europe
● It needs to be confgurable It needs to be confgurable
=> Icinga=> Icinga
But Monitoring confg is Data ?But Monitoring confg is Data ?● If it can be generated it's not user If it can be generated it's not user
generated content anymoregenerated content anymore
● Your computer can generate your confg !Your computer can generate your confg !
Stored ConfgsStored Confgs
Collection and ExportCollection and Export
Export :Export :
@@resource { @@resource {
... }... }
Collect:Collect:
Resource <<| query Resource <<| query |>>|>>
Clean out nodes that dissapearClean out nodes that dissapear
puppet node clean puppet node clean
Exporting and Collecting Exporting and Collecting
Default Puppet TypesDefault Puppet Types
Puppet-icinga modulePuppet-icinga module● https://github.com/inuits/puppet-icingahttps://github.com/inuits/puppet-icinga
Monitoring a VhostMonitoring a Vhost
Alternative ApproachesAlternative Approaches
● https://gist.github.com/jfryman/5808537https://gist.github.com/jfryman/5808537
● https://github.com/favoretti/puppetdb-https://github.com/favoretti/puppetdb-external-naginatorexternal-naginator
● Deploy a new appDeploy a new app
● Add monitoringAdd monitoring
● Add Real application monitoringAdd Real application monitoring
● Both on infra and on app levelBoth on infra and on app level
Monitoring Puppet & FriendsMonitoring Puppet & Friends
Puppet RunsPuppet Runs
PuppetMasterPuppetMaster
PuppetMasterPuppetMaster @@nagios_service{"check_socket_8140_puppet_${::fqdn}":@@nagios_service{"check_socket_8140_puppet_${::fqdn}":
check_command => 'check_tcp!8140',check_command => 'check_tcp!8140',
service_description => 'TCP puppet on port 8140',service_description => 'TCP puppet on port 8140',
host_name => $::fqdn,host_name => $::fqdn,
use => 'generic-service',use => 'generic-service',
contact_groups => $::environment,contact_groups => $::environment,
notifcation_period => $::icinga::notifcation_period,notifcation_period => $::icinga::notifcation_period,
notifcations_enabled => $::icinga::notifcations_enabled,notifcations_enabled => $::icinga::notifcations_enabled,
target => "/etc/icinga/objects/services/${::fqdn}.cfg",target => "/etc/icinga/objects/services/${::fqdn}.cfg",
}}
PuppetDBPuppetDB
PuppetDB(2)PuppetDB(2)● check_puppetdb_memorycheck_puppetdb_memory
• Java heap memoryJava heap memory
● check_puppetdb_processedcheck_puppetdb_processed
• Nr of reports processedNr of reports processed
● check_puppetdb_populationscheck_puppetdb_populations
• Resources, nodes, resources per nodeResources, nodes, resources per node
● check_puppetdb_queuecheck_puppetdb_queue
• Is pgsql down ? :)Is pgsql down ? :)
Puppet DashboardPuppet Dashboard @@nagios_service{"check_http_puppet_dashboard_${::fqdn}":@@nagios_service{"check_http_puppet_dashboard_${::fqdn}":
check_command => "check_http!-H ${::fqdn} -p 3000 -e 200",check_command => "check_http!-H ${::fqdn} -p 3000 -e 200",
service_description => 'HTTP PuppetDashboard on port 3000',service_description => 'HTTP PuppetDashboard on port 3000',
host_name => $::fqdn,host_name => $::fqdn,
use => 'generic-service',use => 'generic-service',
contact_groups => $::environment,contact_groups => $::environment,
notifcation_period => $::icinga::notifcation_period,notifcation_period => $::icinga::notifcation_period,
notifcations_enabled => $::icinga::notifcations_enabled,notifcations_enabled => $::icinga::notifcations_enabled,
target => "/etc/icinga/objects/services/${::fqdn}.cfg",target => "/etc/icinga/objects/services/${::fqdn}.cfg",
}}
Puppet Dashboard(2)Puppet Dashboard(2)
/usr/share/puppet-dashboard/spool/usr/share/puppet-dashboard/spool
#MonitoringSucks#MonitoringSucks● Puppetruns break our Icinga boxenPuppetruns break our Icinga boxen
● BadlyBadly
● FrequentlyFrequently
It ain't borkenIt ain't borken● Successful puppet runSuccessful puppet run
● Successful Icinga reconfgureSuccessful Icinga reconfgure
● Disk usage growsDisk usage grows
● FastFast
Be aware of bucketsBe aware of buckets
A Puppet BugA Puppet Bug
Triggers on GraphsTriggers on Graphs● Export Java MetricsExport Java Metrics
● JMXTransJMXTrans
● Export JMXConfgsExport JMXConfgs
● Confgure NRPE CheckConfgure NRPE Check
● Export NagiosCheckExport NagiosCheck
● Collect JMX Exports on Collect JMX Exports on JMXTransNodeJMXTransNode
● Graph EmGraph Em
Collect Nagios Confgs on Collect Nagios Confgs on Nagios ServerNagios Server
Triggers on GraphsTriggers on Graphs
Triggers on GraphsTriggers on Graphs
SummarySummary● Honour your parentsHonour your parents
● Don't manually do what machines can do Don't manually do what machines can do for youfor you
● Monitor your puppet infrastructure too !Monitor your puppet infrastructure too !
● Send Pull RequestsSend Pull Requests
● Icinga2 module in the worksIcinga2 module in the works
ContactContactKris.Buytaert@inuits.euKris.Buytaert@inuits.eu
Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.eu/http://www.inuits.eu/
InuitsInuits
Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231
+32 475 961221+32 475 961221
top related