![Page 1: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/1.jpg)
Leveling Up Monitoring:
A Decade of Automating and Scaling Nagios
Katherine Daniels and Laurie Denness
@beerops - @lozzd Velocity 2016
![Page 2: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/2.jpg)
@beerops - @lozzd Velocity 2016
Katherine Daniels@beerops
Senior Operations Engineer, Etsy Co-Author of Effective DevOps
Laurie Denness @lozzd
Staff Operations Engineer, Etsy Official Graph Enthusiast
![Page 3: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/3.jpg)
3
![Page 4: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/4.jpg)
Agenda
@beerops - @lozzd Velocity 2016
Au to mat i o n
2
D e p loy i nato r
3
S c a l i ng + To o l i ng
4
I n T h e B e g i n n i ng . . .
1
![Page 5: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/5.jpg)
![Page 6: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/6.jpg)
25MActive Buyers
About Etsy
1.6MActive Sellers
$2.39B2015 Annual GMS
(As of March 31, 2016)
![Page 7: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/7.jpg)
Monitoring!
![Page 8: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/8.jpg)
@beerops - @lozzd Velocity 2016
![Page 9: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/9.jpg)
@beerops - @lozzd Velocity 2016
![Page 11: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/11.jpg)
https://kartar.net/2015/08/monitoring-survey-2015---tools/
![Page 13: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/13.jpg)
@beerops - @lozzd Velocity 2016
In The Beginning
![Page 14: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/14.jpg)
@beerops - @lozzd Velocity 2016
![Page 15: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/15.jpg)
@beerops - @lozzd Velocity 2016
![Page 16: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/16.jpg)
@beerops - @lozzd Velocity 2016
Sometimes your statement needs emphasis with a black background.
![Page 17: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/17.jpg)
@beerops - @lozzd Velocity 2016
![Page 18: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/18.jpg)
@beerops - @lozzd Velocity 2016
L E S S O N S L E A R N E D :
Templates are awesome.
![Page 19: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/19.jpg)
@beerops - @lozzd Velocity 2016
![Page 20: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/20.jpg)
@beerops - @lozzd Velocity 2016
![Page 21: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/21.jpg)
@beerops - @lozzd Velocity 2016
![Page 22: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/22.jpg)
@beerops - @lozzd Velocity 2016
![Page 23: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/23.jpg)
@beerops - @lozzd Velocity 2016
define service { use generic-service hostgroups Linux_hosts,!email-only-servers service_description SSH check_command check_ssh }
![Page 24: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/24.jpg)
@beerops - @lozzd Velocity 2016
define service { use disk-space-service hostgroup_name email-only-servers contact_groups ops_nonurgent }
![Page 25: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/25.jpg)
@beerops - @lozzd Velocity 2016
![Page 26: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/26.jpg)
@beerops - @lozzd Velocity 2016
L E S S O N S L E A R N E D :
Start small.
![Page 27: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/27.jpg)
@beerops - @lozzd Velocity 2016
Nagios and Chef
![Page 28: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/28.jpg)
@beerops - @lozzd Velocity 2016
![Page 29: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/29.jpg)
@beerops - @lozzd Velocity 2016
![Page 30: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/30.jpg)
24
![Page 31: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/31.jpg)
![Page 32: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/32.jpg)
![Page 33: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/33.jpg)
@beerops - @lozzd Velocity 2016
L E S S O N S L E A R N E D :
Automation is awesome!
![Page 34: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/34.jpg)
@beerops - @lozzd Velocity 2016
L E S S O N S L E A R N E D :
Automation is awesome!
HA HA JUST KIDDING
![Page 35: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/35.jpg)
@beerops - @lozzd Velocity 2016
![Page 36: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/36.jpg)
@beerops - @lozzd Velocity 2016
![Page 37: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/37.jpg)
@beerops - @lozzd Velocity 2016
L E S S O N S L E A R N E D :
Trust but verify.
![Page 38: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/38.jpg)
@beerops - @lozzd Velocity 2016
How Many Repos?
![Page 39: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/39.jpg)
@beerops - @lozzd Velocity 2016
![Page 40: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/40.jpg)
![Page 41: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/41.jpg)
![Page 42: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/42.jpg)
![Page 43: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/43.jpg)
@beerops - @lozzd Velocity 2016
![Page 44: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/44.jpg)
@beerops - @lozzd Velocity 2016
L E S S O N S L E A R N E D :
?!?!?!?!??!?!
![Page 45: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/45.jpg)
@beerops - @lozzd Velocity 2016
L E S S O N S L E A R N E D :
Try, fail, learn, and try again.
![Page 46: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/46.jpg)
Problems
![Page 47: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/47.jpg)
Problems
• Four git repos, inconsistent mess, duplication
![Page 48: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/48.jpg)
Problems
• Four git repos, inconsistent mess, duplication
• Broken semi-useful automation - need to regain trust
![Page 49: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/49.jpg)
Problems
• Four git repos, inconsistent mess, duplication
• Broken semi-useful automation - need to regain trust
• Some shared config, some unique
![Page 50: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/50.jpg)
Problems
• Four git repos, inconsistent mess, duplication
• Broken semi-useful automation - need to regain trust
• Some shared config, some unique
• Gain confidence in changes
![Page 51: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/51.jpg)
Problems
• Four git repos, inconsistent mess, duplication
• Broken semi-useful automation - need to regain trust
• Some shared config, some unique
• Gain confidence in changes
• Stop editing on the production box
![Page 52: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/52.jpg)
@beerops - @lozzd Velocity 2016
Nagios and Chef
![Page 53: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/53.jpg)
@beerops - @lozzd Velocity 2016
Nagios and Chefand Deployinator!
![Page 54: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/54.jpg)
@beerops - @lozzd Velocity 2016
Solution 1: Merge everything: find and remove duplication,
shared configs
![Page 55: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/55.jpg)
@beerops - @lozzd Velocity 2016
Thanks Murphy!
![Page 56: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/56.jpg)
![Page 57: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/57.jpg)
@beerops - @lozzd Velocity 2016
Super Secret Option!!!
![Page 58: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/58.jpg)
@beerops - @lozzd Velocity 2016
![Page 59: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/59.jpg)
@beerops - @lozzd Velocity 2016
![Page 60: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/60.jpg)
@beerops - @lozzd Velocity 2016
![Page 61: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/61.jpg)
![Page 62: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/62.jpg)
![Page 63: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/63.jpg)
@beerops - @lozzd Velocity 2016
Solution 2:
Using Jenkins CI to test changes before production
![Page 64: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/64.jpg)
![Page 65: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/65.jpg)
![Page 66: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/66.jpg)
![Page 67: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/67.jpg)
![Page 68: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/68.jpg)
![Page 69: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/69.jpg)
@beerops - @lozzd Velocity 2016
Solution 3:
Use Deployinator to run Chef recipe to generate automated configs
![Page 70: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/70.jpg)
![Page 71: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/71.jpg)
![Page 72: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/72.jpg)
Chart Tit le
![Page 73: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/73.jpg)
Chart Tit le
![Page 74: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/74.jpg)
@beerops - @lozzd Velocity 2016
Solution 4:
Use Deployinator to rsync config to all boxes
![Page 75: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/75.jpg)
![Page 76: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/76.jpg)
• git pull repo on deploy host
![Page 77: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/77.jpg)
• git pull repo on deploy host
• Run Chef recipe to add automated pieces
![Page 78: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/78.jpg)
• git pull repo on deploy host
• Run Chef recipe to add automated pieces
• Re-run the try-nagios script against that
![Page 79: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/79.jpg)
• git pull repo on deploy host
• Run Chef recipe to add automated pieces
• Re-run the try-nagios script against that
• rsync copy from deploy box to Nagios hosts
![Page 80: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/80.jpg)
• git pull repo on deploy host
• Run Chef recipe to add automated pieces
• Re-run the try-nagios script against that
• rsync copy from deploy box to Nagios hosts
• Create symlink for nagios.cfg
![Page 81: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/81.jpg)
• git pull repo on deploy host
• Run Chef recipe to add automated pieces
• Re-run the try-nagios script against that
• rsync copy from deploy box to Nagios hosts
• Create symlink for nagios.cfg
• Restart Nagios
![Page 82: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/82.jpg)
![Page 83: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/83.jpg)
@beerops - @lozzd Velocity 2016
L E S S O N S L E A R N E D :
Use the tools you have.
![Page 84: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/84.jpg)
@beerops - @lozzd Velocity 2016
Scaling things up!
![Page 85: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/85.jpg)
![Page 86: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/86.jpg)
![Page 87: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/87.jpg)
@beerops - @lozzd Velocity 2016
![Page 88: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/88.jpg)
@beerops - @lozzd Velocity 2016
![Page 89: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/89.jpg)
![Page 90: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/90.jpg)
![Page 91: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/91.jpg)
@beerops - @lozzd Velocity 2016
![Page 92: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/92.jpg)
@beerops - @lozzd Velocity 2016
![Page 93: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/93.jpg)
@beerops - @lozzd Velocity 2016
Core Workers
![Page 94: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/94.jpg)
@beerops - @lozzd Velocity 2016
Core Workers
![Page 95: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/95.jpg)
@beerops - @lozzd Velocity 2016
![Page 96: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/96.jpg)
![Page 97: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/97.jpg)
![Page 98: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/98.jpg)
@beerops - @lozzd Velocity 2016
L E S S O N S L E A R N E D :
If at first you don’t succeed, rub some webscale on it.
![Page 99: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/99.jpg)
@beerops - @lozzd Velocity 2016
Iterating and Iterating
![Page 100: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/100.jpg)
![Page 101: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/101.jpg)
![Page 102: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/102.jpg)
![Page 103: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/103.jpg)
![Page 104: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/104.jpg)
![Page 105: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/105.jpg)
![Page 106: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/106.jpg)
![Page 107: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/107.jpg)
![Page 108: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/108.jpg)
![Page 109: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/109.jpg)
![Page 110: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/110.jpg)
![Page 111: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/111.jpg)
@beerops - @lozzd Velocity 2016
L E S S O N S L E A R N E D :
Iterate
Iterate
Iterate
![Page 112: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/112.jpg)
@beerops - @lozzd Velocity 2016
To Infinity and Beyond
![Page 113: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/113.jpg)
![Page 114: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/114.jpg)
@beerops - @lozzd Velocity 2016
![Page 115: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/115.jpg)
http://github.com/etsy/opsweekly
![Page 116: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/116.jpg)
http://github.com/etsy/opsweekly
![Page 117: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/117.jpg)
Chart Tit le
![Page 118: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/118.jpg)
Chart Tit le
![Page 119: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/119.jpg)
![Page 120: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/120.jpg)
![Page 121: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/121.jpg)
Final Lessons Learned
![Page 122: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/122.jpg)
• Templates are awesome
• Start small
• Automation is awesome
• Trust but verify
• Learn from (y)our mistakes
• Iterate on the tools you have
![Page 123: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/123.jpg)
Open Source Summary
![Page 124: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/124.jpg)
Open Source Summary
• http://github.com/etsy/deployinator
• http://github.com/etsy/pushbot
• http://github.com/etsy/trylib
• http://github.com/etsy/opsweekly
• http://github.com/etsy/nagios-herald
• http://github.com/RJ/irccat
![Page 125: Leveling up monitoring: A decade of automating and scaling Nagios](https://reader034.vdocuments.us/reader034/viewer/2022052116/58802b401a28ab9f0f8b56ef/html5/thumbnails/125.jpg)
THANK YOU!
@beerops - @lozzd Velocity 2016