application logging with the elk stack
TRANSCRIPT
Application Logging With The ELK Stack
@bwaine - #DPC15
Monday, 29 June 15
2
Ben Andersen-Waine
Software Engineer Contractor
Deployed ELK To Prod Numerous Times
Monday, 29 June 15
Logging?
Monday, 29 June 15
System Logs
Monday, 29 June 15
5
Monday, 29 June 15
Application Log
Monday, 29 June 15
Debug Information - Errors (connections, uncaught exceptions, resource exhaustion)
Narrative Information - Methods Calls, Event Triggers
Business Events - Purchases, Logins, Registrations, Unsubscribes
7
Application Log
Monday, 29 June 15
ssh [email protected] -f /var/log/nginx/my-site.access.logtail -f /var/log/my.application.log
ssh [email protected] -f /var/log/mysql/mysql.log
ssh [email protected] -f /var/log/rabbitmq/nodename.log
8
Keeping Track Of All This....
Monday, 29 June 15
9
The Elk Stack
Monday, 29 June 15
Monday, 29 June 15
1) Monolog2) Everything else....
11
PHP Logging Tools
Monday, 29 June 15
1) Monolog: Loggers And Handlers2) Monolog: Tags & Formatters3) Logging business events
12
Basic Logging Examples
Monday, 29 June 15
use Monolog\Logger;use Monolog\Handler\FingersCrossedHandler;use Monolog\Handler\StreamHandler;
$logEnv = getenv('LOG_LEVEL');$level = empty($logLevel) ? $logEnv : Logger::WARNING;
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG); $fcHandler = new FingersCrossedHandler($strHandler, $level);
$appLog−>pushHandler($fcHandler);$appLog−>debug('LOGGING!');
EG1: Loggers And Handlers
13
Monday, 29 June 15
// Set A Log Level$logEnv = getenv('LOG_LEVEL');$level = empty($logLevel) ? $logEnv : Logger::WARNING;
// Create A Logger$appLog = new Logger('AppLog');
14
Monday, 29 June 15
$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG);
$fcHandler= new FingersCrossedHandler($strHandler, $level);
// Create Handlers
$appLog−>pushHandler($fcHandler);
$appLog−>debug('Start Logging!');$appLog−>emergency('Something Terrible Happened');
// Push The Handler And Start Logging
15
Monday, 29 June 15
EG 2: Tagging Formatting
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);$formatter = new LogstashFormatter("helloapp", "application");
$strHandler−>setFormatter($formatter); $appLog−>pushHandler($strHandler));
$id = $_SERVER('X_VARNISH');$tag = new TagProcessor(['request−id' => $id])
$appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!");
16
Monday, 29 June 15
// Create A Logger$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/lg.lg', $level);$formatter = new LogstashFormatter("helloapp", "app");
// Create A Handler & Formatter
// Set Formatter Onto Handler$strHandler−>setFormatter($formatter);
$appLog−>pushHandler($strHandler));
//Push Handler Onto Logger
17
Monday, 29 June 15
$id = $_SERVER('X_VARNISH');$tag = new TagProcessor(['request−id' => $id])$appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!");
// Capture A Unique Id, Create A Tag Processor, Push
18
Monday, 29 June 15
2009 - RFC 5424 - Syslog Protocol
Code / Severity
0 Emergency: system is unusable1 Alert: action must be taken immediately2 Critical: critical conditions3 Error: error conditions4 Warning: warning conditions5 Notice: normal but significant condition6 Informational: informational messages7 Debug: debug-level messages
https://tools.ietf.org/html/rfc542419
Log Levels
Monday, 29 June 15
2013 - PSR03 - PHP Logging Interface Standard
http://www.php-fig.org/psr/psr-3/
20
PSR3
Monday, 29 June 15
EG 3: Event Logginguse Monolog\Logger;use Symfony\Component\EventDispatcher\EventDispatcher;
$dispatcher = new EventDispatcher();
$dispatcher−>addListener( "business.registration.post", function () use ($busLog) { $busLog−>info("Customer registered"); });
$dispatcher−>dispatch("business.registration.post");
Monday, 29 June 15
Logstash Architecture
1. Logstash Shipper ships logs to logstash
2. Logstash processes them
3. Logstash Inserts Into Elastic Search
4. Kibana exposes a web interface to Elastic Search data
Monday, 29 June 15
Logstash Architecture
Monday, 29 June 15
Why not rate the talk now BEFORE the demo?
24
https://joind.in/talk/view/14235
Monday, 29 June 15
ELK Demo
25
1) Discover Data (search / diagnose)2) Visualize Data 3) Produce A Dashboard 4) Demonstrate ‘the new hotness’ of Kibana 4
Monday, 29 June 15
26
https://github.com/LoveSoftware/getting-started-with-the-elk-stack
Monday, 29 June 15
Monday, 29 June 15
Monday, 29 June 15
Monday, 29 June 15
Monday, 29 June 15
Logstash Config
31
Monday, 29 June 15
Logstash Collecting{ "network": { "servers": [ "logs.logstashdemo.com:5000" ], "timeout": 15, "ssl ca": "/etc/pki/tls/certs/logstash−forwarder.crt" }, "files": [ { "paths": [ "/var/log/nginx/helloapp.access.log" ], "fields": { "type": "nginx−access" } } ] }
32
Monday, 29 June 15
Logstash Processing
input { lumberjack { port => 5000 ssl_certificate => "/etc/pki/tls/certs/logstash−forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash−forwarder.key"}
}
Input
33
Monday, 29 June 15
Logstash ProcessingFilteringfilter { if [type] == "nginx−access" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } date { match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ] } } }
34
Monday, 29 June 15
Logstash ProcessingOutput
output { elasticsearch { host => localhost }}
35
Monday, 29 June 15
Groking grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns
http://grokdebug.herokuapp.com/
55.3.244.1 GET /index.html 15824 0.043
%{IP:client}%{WORD:method}%{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
Monday, 29 June 15
37
Hey Ben.... Have you got time for that
gratuitously flashy geo data demo?
Monday, 29 June 15
Monday, 29 June 15
Logging IdeasRelease MarkerError rates of various applications over timeLatency in various percentiles of each application tierHTTP Responses: 400 series responsesHTTP Responses: 500 series responsesAuto git blame production errorsAuth and Syslogs
39
Monday, 29 June 15
Go Forth And Log....BUT
Remember log rotation
Beware running out of space
Beware file logging on NFS
40
Monday, 29 June 15
Questions?
41
Monday, 29 June 15
https://joind.in/talk/view/14235
42
Monday, 29 June 15