turbo charge your logs

Turbo Charge your Logs

Turbo charge your logs

Who?

Ex-pat Englishman, now living in Southern Ontario.

Web developer for 5 years, mostly PHP.

(Almost) senior software engineer at TribeHR.

Co-organiser of Guelph PHP User Group.

Ex-professional musician.

Why logging?

Of all the things you would come to a conference like this to hear about...

Your app is trying to talk to you!

Logs are the way your app speaks to you.

Ignore log messages at your peril...

Logging is the L in LUCID development.

http://crisscott.com/2012/09/11/lucid-development/

Crisscott.com seems to be Scott Mattocks.LoggingUnit TestingConfigurationIsolates featuresDocumentedYou can't optimise what you can't measure...

It's confessional time...

How many people monitor log files regularly?How many only look at them during a major crisis?

Using log data is hard

Not in a human friendly format.

A lot of data.

Many log files.

Potentially many servers.

Many log files generated by many applications/pieces of software.Last time want to be digging through this is in a crisis.

Use your logs pro-actively

How can we stop using log data reactively and start using it pro-actively?

Mention that I can't tell you how to do this. This talk will introduce some tools that can get you to this point.Combination of tools will get you to a pro-active log monitoring solution.Also mention that for each tool I'm talking about there are many alternatives...Mention closed source alternatives.Mention that this is being used in production at MRX.

The 'ideal' logging setup

Centralised.

Accepts logs from application code, software and the OS.

Performant.

Scalable.

Easily searchable.

Alarms and alerting.

Of course this will be different for everyone!

RFC 5424 logging levels

Debug

Info

Notice

Warning

Error

Critical

Alert

Emergency

Logging from your code

What's wrong with error_log?

Nothing at all but...

It's limited:Have to format the message yourself.

Limited number of destinations.

Doesn't support logging levels defined in RFC 5424.

Also mention that it's specifically for logging errors, not informational or debug messages.Difficult to format messages.Destinations: file or email.Define log levels in RFC 5425

Introducing Monolog

PHP 5.3+ logging library by Jordi Boggiano based on Python's Log Book library.

PSR-3 compliant.

Supports RFC 5424.

Mention that there are many logging libraries but Monolog has seemed to have gained the most traction.Describe what PSR-3 is.

Installing Monolog

Symfony2, Laravel4, Silex and PPI all come with Monolog.

CakePHP and Slim have have plug-ins to use it.

Most easily installed with Composer:

PPI takes pieces of Zend 2, Sf2 and Doctrine2 and mashes them!Silex allows you to register a Monolog provider.

Monolog concepts

Channels.

Handlers.

Formatters.

Processors.

Channels

A channel is a name or category for a logger.

Each logger instance is given a channel when instantiated.

Allows for multiple loggers, each with a different channel.

Channel equates to facility in Syslog.Makes it easy to use different loggers for different parts/functionality in an app.

Handlers

Handlers write log messages to a storage medium.

Multiple handlers can be attached to each logger.

Set lowest level handler logs at and if it 'bubbles'.

Many handlers available or you can write your own.

The handlers constructor accepts the minimum log level that the handler should accept. Defaults differently depending on handler.

Handlers can be shared between multiple loggers.

Needs care when not bubbling! Add more specific handlers later.

Example handlers

Files/SyslogStream Handler

Rotating File Handler

Syslog Handler

NotificationsMail handlers

Pushover Handler

HipChat Handler

DebuggingFirePHP Handler

ChromePHP Handler

Networked LoggingSocket Handler

AMQP Handler

Gelf Handler

Zend Monitor Handler

Rotating File Handler: Creates one file per day but meant as a quick + dirty solution.Mail handlers include native mail and Swiftmail handlers.Pushover handler sends mobile notifications through the Pushover API.HipChatHandler send notification to a HipChat chat room (Rafael Dohms wrote it)FirePHP and ChromePHP write to FireBug or Chrome consoles. DEV ONLY!!

Formatters

Processes a log message into the appropriate format for a handler.

Each handler has a default formatter to use but this can be overridden.

Use Handler::setFormatter() method to set the formatter for a handler.

Simple example

Mention that logging a message accepts up to two arguments:The message (string) and an array of context.

Using multiple handlers

Leveraging bubbling

Mention that handlers added last are called first.

Processors

Used to amend or add to the log message.

PHP callable, called when a message is logged.

Built in processors available:IntrospectionProcessor

WebProcessor

MemoryUsageProcessor

MemoryPeakUsageProcessor

ProcessIdProcessor

UidProcessor

Mention that this takes away some of the repetition of adding context to each log message.IntrospectionProcessor: Adds the line/file/class/method from which the log call originated.WebProcessor: Adds the current request URI, request method and client IP to a log record.MemoryUsageProcessor: Adds the current memory usage to a log record.MemoryPeakUsageProcessor: Adds the peak memory usage to a log record.ProcessIdProcessor: Adds the process id to a log record.UidProcessor: Adds a unique identifier to a log record.

Processor example

Where does this get us
to?

Centralised. Maybe...

Accepts messages from application code, software and the OS.

Performant. Maybe...

Scalable. Maybe...

Easily searchable.

Alarms and alerting. Yes but crude.

We can do better!

Leveraging Syslog

Why Syslog?

Loggable events don't only happen in code!

To get a full picture of what's going on we need to monitor what's going on in other services too.

Problems often caused by the intersection of different pieces of software.

Syslog basics

OS daemon to process log messages.

Messages are assigned a facility, such as auth, authpriv, daemon or cron or a custom one.

Messages are also assigned a severity, defined in RFC 5424.

Messages can be sent to files, console or a remote location.

Which Syslog daemon
to use?

In part will depend on your OS.

Things to consider:Syslog is the oldest with not as many features.

Syslog-ng is produced under a dual license.

Rsyslog fully featured and open source.

Mention that you can often replace the default syslog daemon in an OS.

Introduction to Rsyslog

Fork of syslog by Rainer Gerhards.

Drop in replacement for syslog.

Many, many features including plugin system for extending.

Default syslogger in Debian, can be installed on other distros too.

Remote logging with Rsyslog

Rsyslog can be configured to work in a client-server setup.One or more machines are setup as clients to forward log messages.

One machine is setup to receive and store them.

Probably want to filter sender on the receiving machine...

Mention that not going into all features of Rsyslog, just focusing on remote logging.Suggest 'man rsyslog' or 'man rsyslog.conf'.Also mention that can use something like Rsyslog or IPTables to filter remote loggers.

Rsyslog client setup

Note this should be added to main rsyslog config file or a file that's included in it.This is for UDP forwarding. TCP would use @@.

Rsyslog server setup

Mention that normally you would need just one of these. Also that the corresponding port needs to be opened in the server config.This would only load the handler for the remote logs. Still needs to be processed with other directives.

Leveling up with Rsyslog

Apache can send all error logs to syslog directly.

Rsyslog can also monitor other log files using the Text File Input module.Example of monitoring Apache access log at https://gist.github.com/joseph12631/2580615

Where does this get us?

Centralised. Yes.

Accepts messages from application code, software and the OS. Possibly.

Performant. Depends.

Scalable. Depends.

Easily searchable.

Alarms and alerting. Yes but crude.

Note that if all you want is to centralise all of your logs this could be the solution...

Taking it further with Logstash

What is Logstash?

Tool to collect, filter and output log messages.

Built in web interface or richer web interface project called Kibana available.

Full information at http://logstash.net/ and Kibana demo at http://demo.logstash.net/

Mention that Logstash is written in Java.34 inputs, has 28 filters and 47 different outputs.

Installing Logstash

Current release is 1.3.3 and can be downloaded from here.

Run from cli, use supervisord or an init.d/upstart script (cookbook entry on how to do this at http://cookbook.logstash.net/).

Inputs, filters and outputs

InputsAMQP/RabbitMQ

Syslog

Varnishlog

FiltersAnonymize

Grok

Geoip

Mutate

OutputsStatsd

XMPP

AMQP/RabbitMQ

Nagios

Graphite

Varnishlog input from Varnishes memory log.Anonymize anonymise fields using a consistent hash.Grok regex library for parsing log messages and processing matches.Geoip add geo data to ip addresses in log messages.Mutate General mutations (rename, remove, replace, modify) to fields.

Logstash config

When starting specify the path to a config file for Logstash to use.

Three main sections: input, filter and output.

Each section may have multiple instances of each type.

Sample configuration file

input { file { path => "/var/log/apache2/*access.log" type => "apache" }} filter { if [type] == "apache" { grok { pattern => "%{COMBINEDAPACHELOG}" } }} output { redis { host => "10.0.0.5" data_type => "list" key => "logstash" }}See http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/

Where does this get us?

Centralised. Yes.

Accepts messages from application code, software and the OS. Yes.

Performant. Yes.

Scalable. Yes.

Easily searchable. Possibly.

Alarms and alerting. Yes.


Introducing Graylog2

What is Graylog2?

Log storage and search application.

Can accept thousands of messages per second and store terabytes of data.

Web interface for searching and analytics.

Built in alerting and metrics.

Discuss advantages and disadvantages to using Graylog or Logstash.

Installing Graylog2

Components:Elasticsearch

MongoDb

Graylog2 server

Graylog2 web interface

Full info on installing at http://support.torch.sh/help/kb

Live demo at http://public-graylog2.taulia.com/login

Mention that graylog server and elasticsearch are written in Java, web interface is a Rails app.Mention login details for the demo username admin or user, password graylog2.

Getting log messages into Graylog2

Can accept log messages in 3 ways:Graylog Extended Log Format (GELF) via UDP .

Syslog via UDP or TCP.

AMQP.

Multiple Graylog2 server instances can be run in parallel.

Benefits of UDP 'Fire and forget'.Drawbacks of UDP Lack of acknowledgement of receiving messages.TCP can mitigate packet loss but slower.AMQP guarantees delivery, but more complex to setup and run.GELF is basically JSON. Ideal for sending messages from app code. Libraries in many languages, including a Monolog handler.

Graylog2 web interface

Main view shows recent log messages and graphs of recent message numbers.

Single message can be clicked on to view all details for it.

Dashboard views.

Full search functionality.

Analytics dashboard and metrics.

Web interface view

Details of an individual message

Dashboard view

Searches and streams

Web interface allows fine grained searching by different fields.

Frequently used searches can be saved as streams.

Streams can be marked as favourites by users and can be viewed as dashboards.

Stream alarms

Alarms can be sent for a stream with user defined sensitivity.

Plugins for sending alarms include:Email

PagerDuty

HipChat

Twilio SMS

Jabber/XMPP

You can also write your own plugins.

Where does this get to?

Centralised. Yes.

Accepts messages from application code, software and the OS. Yes.

Performant. Yes.

Scalable. Yes.

Easily searchable. Yes.

Alarms and alerting. Yes.


Thanks for listening

Contact me:[email protected]

@JCook21

Questions?

Putting it all together

A few possible implementations.

turbo charge your logs

Technology