monitoring in the cloud with puppet
DESCRIPTION
My PuppetCamp Ghent 2013 talk on building monitoring solutions with PuppetTRANSCRIPT
Monitoring in the CloudMonitoring in the Cloudwith Puppet
PuppetCamp Ghent 2013
Kris Buytaert
Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source
Consultant @inuits.euConsultant @inuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops
Do you know what your children do at 5 am Do you know what your children do at 5 am in the morning ?in the morning ?
● Sleeping ?Sleeping ?
● Crashing a Party ?Crashing a Party ?
● Why are the cops at your front door ?Why are the cops at your front door ?
● What happened ?What happened ?
● How long have they been gone ?How long have they been gone ?
devopsdevops● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim
Monitoring is usually an Monitoring is usually an aftertoughtaftertought
ENOBUDGET, ENOTIMEENOBUDGET, ENOTIME
A sub movementA sub movement● #monitoringsucks trending#monitoringsucks trending
● https://github.com/monitoringsucks/https://github.com/monitoringsucks/
● 2008 Study :Nagios + Friends2008 Study :Nagios + Friends
● 2011 Conclusion : Nagios/Icinga are 2011 Conclusion : Nagios/Icinga are the only automatable alternatives the only automatable alternatives
#monitoringsucks#monitoringsucksMonitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love: Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love:
● Having my hands tied with the model of host and service bindings. Having my hands tied with the model of host and service bindings.
● Having to set up "fake" hosts just to group arbitrary metrics together Having to set up "fake" hosts just to group arbitrary metrics together
● Having to either collect metrics twice - once for alerting and another for trending Having to either collect metrics twice - once for alerting and another for trending
● Only being able to see my metrics in 5 minute intervals Only being able to see my metrics in 5 minute intervals
● Having to chose between shitty interface but great monitoring or shitty monitoring Having to chose between shitty interface but great monitoring or shitty monitoring but great interface but great interface
● Dealing with a monitoring system that thinks IT is the system of truth for my Dealing with a monitoring system that thinks IT is the system of truth for my environment environment
● Not actually having any real choicesNot actually having any real choices
John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-monitoring-sucks.htmlmonitoring-sucks.html
Let's forget aboutLet's forget about● GroundWorkGroundWork
● ZenossZenoss
● ZabbixZabbix
● HypericHyperic
● CactiCacti
● MuninMunin
● BigBrotherBigBrother
What's different in the cloud ?What's different in the cloud ?
● ScaleScale
● VelocityVelocity
● ChangeChange
ChallengesChallenges● ReproducabilityReproducability
● SpeedSpeed
● AuditingAuditing
● Keeping stuff in sync Keeping stuff in sync
Thank Thank godgod Luke for Puppet Luke for Puppet
And Mark for CfengineAnd Mark for Cfengine
}}#monitoringsucks #monitoringsucks 20132013
What we wantWhat we want
● Small , wel suited componentsSmall , wel suited components
• CollectCollect
• Transport / MangleTransport / Mangle
• Analyse / ActAnalyse / Act
• VisualizeVisualize
CollectdCollectd● CollectsCollects
● Zillion PluginsZillion Plugins
• Nginx,apache, mysql, diskNginx,apache, mysql, disk
● Graphite Carbon PluginGraphite Carbon Plugin
● Send metrics to graphiteSend metrics to graphite
● https://github.com/KrisBuytaert/pupphttps://github.com/KrisBuytaert/puppet-collectdet-collectd
● Collect from Collect from anywhereanywhere
● FilterFilter
● Send anywhereSend anywhere
● Queuing supportQueuing support
Puppet and LogstashPuppet and Logstash● https://github.com/KrisBuytaert/vagrant-puppet-logstashhttps://github.com/KrisBuytaert/vagrant-puppet-logstash
● Includes Logstash / Kibana / Grok / ES modules as Includes Logstash / Kibana / Grok / ES modules as submodules ! submodules !
git clonegit clone
● git submodule init git submodule init
● git submodule update git submodule update
● vagrant up vagrant up
Graphite Graphite ● Graphing at ScaleGraphing at Scale
● Graphing at EaseGraphing at Ease
● Any metric is a graphAny metric is a graph
● echo "somestring $somevalue $timestamp" | echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003nc <%= graphitehost %> 2003
Graphite ComposerGraphite Composer
x
Graphite APIGraphite API
Gdash In actionGdash In action
Puppet and GraphitePuppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/https://github.com/KrisBuytaert/vagrant-graphite/
● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules ! Statsd / Tattle and more modules as submodules !
● git clonegit clone
● git submodule init git submodule init
● git submodule update git submodule update
● vagrant up vagrant up
Collectd & GraphiteCollectd & Graphite
Configuring a ServiceConfiguring a Service● PackagePackage
● ConfigConfig
● Service Service
<- we pray status is correct<- we pray status is correct
Apache Example:Apache Example:
Definition of Done:Definition of Done:
Done means : being monitoredDone means : being monitored
Icinga ?Icinga ?• Isn't nagios dead ?Isn't nagios dead ?
• Vibrant CommunityVibrant Community
• Throw great parties in NurnbergThrow great parties in Nurnberg
• Nobody can pronounce it anyhowNobody can pronounce it anyhow
• https://github.com/Inuits/puppet-icinga/https://github.com/Inuits/puppet-icinga/
Monitoring BaselineMonitoring Baseline● Deploy a host,Deploy a host,
● Add it to the monitoringAdd it to the monitoring
● Add collection toolsAdd collection tools
● Add check definitionsAdd check definitions
● Update the monitoring tool configUpdate the monitoring tool config
Stored ConfigsStored Configs
Collection and ExportCollection and Export
Export :Export :
@@resource { @@resource {
... }... }
Collect:Collect:
Resource <<| Resource <<| query |>>query |>>
Clean out nodes that dissapearClean out nodes that dissapear
puppet node clean puppet node clean
Exporting and Collecting Exporting and Collecting
Monitoring a VhostMonitoring a Vhost
It is not in production if it is not It is not in production if it is not monitoredmonitored
Triggers on GraphsTriggers on Graphs● Export Java MetricsExport Java Metrics
● JMXTransJMXTrans
● Export JMXConfigsExport JMXConfigs
● Configure NRPE CheckConfigure NRPE Check
● Export NagiosCheckExport NagiosCheck
● Collect JMX Exports on Collect JMX Exports on JMXTransNodeJMXTransNode
● Graph EmGraph Em
Collect Nagios Configs Collect Nagios Configs on Nagios Serveron Nagios Server
Triggers on GraphsTriggers on Graphs
Triggers on GraphsTriggers on Graphs
Conclusion:Conclusion:● Reproducable monitoring setupReproducable monitoring setup
● Dynamically generated monitoring Dynamically generated monitoring configconfig
● Code is available at Code is available at github.com/KrisBuytaert/snippetsgithub.com/KrisBuytaert/snippets
But,But,(pronounced in Ozzy dialect)(pronounced in Ozzy dialect)
htt
p:/
/ww
w.k
art
ar.
net/
201
3/0
1/m
onit
ori
ng
-suck
s---
a-r
ant/
#monitoringlove#monitoringlove
HomeworkHomeworkSensuSensu
ExtrememonExtrememon
......
RiemannRiemann
EsperEsper
[email protected]@inuits.eu
Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.eu/http://www.inuits.eu/
InuitsInuits
Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231
+32 475 961221+32 475 961221
http://loadays.orghttp://loadays.org