j-day kraków: listen to the sounds of your application
DESCRIPTION
TRANSCRIPT
Listen to the sounds of your application
!2
Krzysztof Ciesielski
@kpciesielski
softwaremill.com
Lublin Software Craftsmen
!4
• Monitoring, huh?
• Introducing Graphite
• Log analysis – the whys
• Logstash architecture & use cases
• Exploring logs with Kibana
Agenda
!5
Monitoring, huh?
!7
Types of measurements
!7
Types of measurements
Network
!7
Types of measurements
Network
Machine
!7
Types of measurements
Network
Machine
Application
!9
measurement > prediction
!10
!11
measurement >> prediction
!12
Our stack
!12
Our stack
Server
App
Server
App
…
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
!12
Our stack
Server
App
Server
App
…
Graphite Logstash
Yammer Metrics
!13
Introducing Graphite
!14
Graphite
!15
PORT=2003 SERVER=graphite.your.org echo "local.random.diceroll 4 `date +%s`" | nc ${SERVER} ${PORT};
!16
successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297
!16
successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297
=> successful.login.attempt = 1
!17
Yammer Metrics
!17
Yammer Metrics
private final Meter successfulLogins = metrics.meter(name(LoginHandler.class, "successful")); !public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }
!18
Metrics
!18
MetricsTypes:
!18
MetricsTypes:
•Gauges
!18
MetricsTypes:
•Gauges
•Counters
!18
MetricsTypes:
•Gauges
•Counters
•Meters
!18
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
!18
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
•Timers
!18
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
•Timers
•Health Checks
!18
MetricsTypes:
•Gauges
•Counters
•Meters
•Histograms
•Timers
•Health Checks
Reporters:
•STDOUT
•CSV
•SLF4J
•JMX
•Graphite
•Ganglia
!19
Counters, meters… vs.
Gauges
!20
Aggregation
!22
You can also check out• collectd https://collectd.org
• StatsD https://github.com/etsy/statsd/
• Riemman http://riemann.io/
• Twitter’s Ostrich https://github.com/twitter/ostrich
• Ganglia http://ganglia.sourceforge.net/
• Dashboards: http://shopify.github.io/dashing/ https://github.com/obfuscurity/descarteshttps://github.com/obfuscurity/dusk
!23
Event Sourcing
Tracing and storing all the events
Current app state: replaying the stream
Great for data mining and analysis
!24
Log aggregation
Less invasive way to build an “event stream”
Logs are data with plenty of value
!25
Log aggregation
Log source
Log source
Log source
Central storage
Query engine
Web view
!26
Logstash
Open Source (written in JRuby)
!26
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
!26
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Parses and extracts metadata
!26
Logstash
Open Source (written in JRuby)
Gathers logs from various inputs
Writes to various outputs
Parses and extracts metadata
!26
Logstash
Similar tools: !Splunk, Graylog, Fluentd, log.io
Open Source (written in JRuby)
Gathers logs from various inputs
Writes to various outputs
Parses and extracts metadata
!27
Logstash centralized architecture
!28
Logstash grok
2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext !- Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
!28
Logstash grok
2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext !- Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
time 2013-11-10 19:41:25.321
thread main
loglevel INFO
source o.a.camel.impl.DefaultCamelContext
message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
!29
Logstash simple use cases
Output: e-mail alert
ExceptionsSecurity issuesRare and business-meaningful events
Grok:
!30
Kibana
!31
Kibana checking general app activity
!32
Kibana Spotting anomalies
!33
Kibana Checking regular events
!34
Kibana How often is this feature used?
!35
Kibana Displaying multiple query results
Query: exception Type: applog
Query: index Type: mongolog
!36
References• Graphite http://graphite.readthedocs.org/en/latest/
• Yammer Metrics http://metrics.codahale.com/
• Logstash http://logstash.net/
• Surfing the event stream by Sam Newman at Geecon http://www.slideshare.net/spnewman/surfing-the-event-stream
• Lessons from Building and Scaling LinkedIn by Jay Krepshttp://www.infoq.com/presentations/linkedin-architecture-stack
• Code as Craft http://codeascraft.com/
!37
