monitor your atlassian stack like the nsa
DESCRIPTION
Monitor your Atlassian stack like the NSA by Tom Moors, Atlassian Expert at ACA IT-Solutions.TRANSCRIPT
aca-it.be - 2014
Monitor your Atlassian stack like the NSA
http://www.flickr.com/photos/zigazou76/7670889680
Tom Moors January 16th 2014 Atlassian User Group Belgium
@tommoors
aca-it.be - 2014
Who is Tom Moors?✓ Atlassian Expert @ ACA IT-Solutions
✓ Poor drawer
✓ @tommoors
aca-it.be - 2014
@tommoors
Why monitoring?
aca-it.be - 2014
@tommoors
Why monitoring?Costs money
Needs extra time
aca-it.be - 2014
@tommoors
Why monitoring?Reduce costs
Save timeOffer insights
Opportunities
Improve service
Detect fraud
…
aca-it.be - 2014
–Lord Kelvin
“If you can not measure it, you can not improve it.”
aca-it.be - 2014
@tommoors
…but #monitoringsucks!
Great movement!
@tommoors
aca-it.be - 2014
Agenda
• Introduction
• Building blocks
• Monitoring levels
• Conclusion
@tommoors
aca-it.be - 2014
What is monitoring?
Observe and check the progress or quality of (something) over a period of time; keep under systematic review
http://www.oxforddictionaries.com/definition/english/monitor
@tommoors
aca-it.be - 2014
What is monitoring?
Super Monitoring
@tommoors
aca-it.be - 2014
Approach to monitoring?• Ride on the DevOps/Infrastructure as Code wave
• Build a toolchain
@tommoors
aca-it.be - 2014
GoalAlerts
...when things go wrong
...before disaster strikes
...no spamming / false metrics
User Interface
• Context for post-mortem
• Trends
• Current state
@tommoors
aca-it.be - 2014
Monitoring basics• Gather data
• Transport data
• Process and store data
• Present data
• Alerts
• Trends
• History / Future
@tommoors
aca-it.be - 2014
Monitoring basics
http://www.slideshare.net/roidelapluie/devops-andmonitoringowf13
Metric✓ Time ✓ Name ✓ Value
Event✓ Time ✓ Fields
can be converted to
@tommoors
aca-it.be - 2014
Monitoring levels
• Availability
• Infrastructure
• Process
• Application
• Business
BONUS: Logs
@tommoors
aca-it.be - 2014
Availability• Built-in checks: ping
• Service Delivery?
@tommoors
aca-it.be - 2014
Monitoring levels
• Availability
• Infrastructure
• Process
• Application
• Business
BONUS: Logs
@tommoors
aca-it.be - 2014
InfrastructureSystem
• CPU usage/min/max/average
• Memory usage/min/max/average
• Disk space
• IO Wait
• Process load
Services
• Network
• Database
• Query timing
• LDAP / Active Directory
Other
• SSL/TLS Certificates
@tommoors
aca-it.be - 2014
Infrastructure• Do-It-Yourself (Not Invented Here Syndrome)
• CollectD
• StatsD
• Cacti
• Ganglia
• Hyperic
• Nagios, Icinga
• Zabbix
@tommoors
aca-it.be - 2014
Infrastructuregoogle.com results
Cacti 1,740,000
Nagios 1,690,000
StatsD 1,470,000
Ganglia 1,150,000
Zabbix 922,000
Icinga 730,000
Hyperic 524,000
CollectD 225,000
@tommoors
aca-it.be - 2014
Infrastructure: Icinga
@tommoors
aca-it.be - 2014
Infrastructure: Icinga
@tommoors
aca-it.be - 2014
Icinga: AlertsExtensible with plugins
• JIRA issue
• Jabber/XMPP
• HipChat
• SMS
• PagerDuty
• …
@tommoors
aca-it.be - 2014
StatsD• UDP Network daemon
• Aggregates data
• Buckets with values
• Type
• Counter
• Timer
• Gauge
• Backend: eg Graphite
@tommoors
aca-it.be - 2014
Graphite• Stores numeric time-series
• Render graph of this data on demand
!
Architecture
• carbon: daemon
• whisper: data storage
• graphite webapp: frontend
@tommoors
aca-it.be - 2014
Graphite
@tommoors
aca-it.be - 2014
Monitoring levels
• Availability
• Infrastructure
• Process
• Application
• Business
BONUS: Logs
@tommoors
aca-it.be - 2014
Process
• Is the service/process still running?
• Watchdog to (repair and) restart
eg upstart, Monit, jsvc
!
Potential problem ➞ send event
@tommoors
aca-it.be - 2014
Monitoring levels
• Availability
• Infrastructure
• Process
• Application
• Business
BONUS: Logs
@tommoors
aca-it.be - 2014
JavaMelody• Plugin for JIRA, Confluence, Bamboo
• Lightweight
• Statistics and charts
@tommoors
aca-it.be - 2014
Java Applications: JMX
Java Management eXtensions
• MBeans representing device, application, resource
• Set of readable (writable) attributes
• Set of operations
@tommoors
aca-it.be - 2014
Java Applications: JMX
@tommoors
aca-it.be - 2014
Java Applications: JMX
@tommoors
aca-it.be - 2014
Working with JMX
• Well-supported in tools (plugins)
• Jolokia
• jmxtrans
@tommoors
aca-it.be - 2014
Commercial solutions• New Relic
• Keynote
@tommoors
aca-it.be - 2014
Monitoring levels
• Availability
• Infrastructure
• Process
• Application
• Business
BONUS: Logs
@tommoors
aca-it.be - 2014
Business ValueHow many…
• new spaces?
• issues created?
• users active?
• popular pages?
• …
Very specific metrics
@tommoors
aca-it.be - 2014
Business Value
• Google Analytics
• Piwik
• (Hummingbird)
• JIRA: reporting
• Custom development
@tommoors
aca-it.be - 2014
Monitoring levels
• Availability
• Infrastructure
• Process
• Application
• Business
BONUS: Logs
@tommoors
aca-it.be - 2014
Logs
• Most up-to-date information
• Best insights, eg why a HTTP 500?
• Unstructured data
• Massive amount from multiple sources
@tommoors
aca-it.be - 2014
Working with Logs
Logstash
Scribe
free and open sourcecommercial
aca-it.be - 2014
@tommoors
Logstashhttp://logstash.net
@tommoors
aca-it.be - 2014
LogStash: design
Input Filter Output
@tommoors
aca-it.be - 2014
LogStash: design
Input Filter Output
2013-10-29 12:48:43,663 Thread-67 ERROR
[jira.issue.managers.DefaultCustomFieldManager] Could not load custom field type
plugin with key 'com.pyxis.greenhopper.jira:gh-epic-label'. Is the plugin present and
enabled?
@tommoors
aca-it.be - 2014
LogStash: design
Input Filter Output
2013-10-29 12:48:43,663 Thread-67 ERROR
[jira.issue.managers.DefaultCustomFieldManager] Could not load custom field type
plugin with key 'com.pyxis.greenhopper.jira:gh-epic-label'. Is the plugin present and
enabled?
@tommoors
aca-it.be - 2014
LogStash: design
Input
• file
• imap
• stdin
• log4j
• syslog
• tcp
• udp
• rabbitmq
• collectd
• redis
• …
Input Filter Output
@tommoors
aca-it.be - 2014
LogStash: design
Filter
• grep
• grok
• geo ip
• anonymize
• drop
• split
• dns
• extractnumbers
• mutate
• …
Input Filter Output
@tommoors
aca-it.be - 2014
LogStash: design
Output
• elasticsearch
• graphite
• HipChat
• JIRA
• http
• StatsD
• Nagios
• PagerDuty
• Cloudwatch
• …
Input Filter Output
@tommoors
aca-it.be - 2014
LogStash: architecture
Shipper
Broker IndexerShipper
Shipper
Search & Storage
Web Interface
@tommoors
aca-it.be - 2014
LogStash: Kibana
aca-it.be - 2014
@tommoors
One more thing…
@tommoors
aca-it.be - 2014
Dashboards• Buy a big TV for the office
• Combine the most important metrics
• SLA
• Builds
• Work done
• Deployments
• …
@tommoors
aca-it.be - 2014
Dashboards
JIRA Wallboard Plugin
@tommoors
aca-it.be - 2014
Dashboards
AtlasBoard
@tommoors
aca-it.be - 2014
Dashboards
GDash
@tommoors
aca-it.be - 2014
Dashboards
Graphene
@tommoors
aca-it.be - 2014
Dashboards
Giraffe
@tommoors
aca-it.be - 2014
Conclusion
LogStash StatsD Graphite
Icinga
System
Possible toolchain
@tommoors
aca-it.be - 2014
Conclusion
• Monitoring is a must
• Great tools support great responsibility
• There is no one-fits-all
• Iterate over your configuration
• Get close(r) with logs
• Dashboards are a great reason to buy a big TV
@tommoors
aca-it.be - 2014
Further Reading• https://github.com/monitoringsucks
• http://jedi.be/blog/2012/01/03/monitoring-wonderland-survey-introduction
• http://jedi.be/blog/2012/01/03/monitoring-wonderland-metrics-api-gateways
• http://jedi.be/blog/2012/01/04/monitoring-wonderland-moving-up-the-stack-application-user-metrics
• http://jedi.be/blog/2012/01/03/monitoring-wonderland-nagios-the-mighty-beast
• http://www.paperplanes.de/2011/1/5/the_virtues_of_monitoring.html
• http://assets.en.oreilly.com/1/event/65/Metrics-driven%20Engineering%20at%20Etsy%20Presentation.pdf
• http://www.devco.net/archives/2011/03/19/thinking_about_monitoring_frameworks.php
• http://www.slideshare.net/roidelapluie/devops-andmonitoringowf13
• http://serialized.net/2011/02/getting-more-signal-from-your-noise
• http://obfuscurity.com/static/TrendingWithPurpose_20110318.pdf