logging application behavior to mongodb

Logging Application Behavior to MongoDB

Robert Stewart @wombatnation

1/1/2012

TOC

• Why Log to MongoDB?

•  Logging Library Basics

•  Logging by Language

• Other Ways to Distribute Your Logging

Why Log to MongoDB?

•  Centralized application logging

•  Easy remote access, compared to files

•  Log events good fit for document model

•  Flexible schema

•  Indexes for frequent queries

•  Capped collections are very efficient

•  Analyze data in-place with JavaScript MapReduce

Event Storage

•  Create a database for your app

•  Create a capped collection to store log events

•  Time-based eviction more natural, but less efficient

•  Capped collection reduces chance of filling up disk

Log Analysis

•  Compared to files, easier to analyze app behavior across servers and days

•  Log values to custom keys where possible

•  JavaScript MapReduce

Miscellaneous Free Advice

•  Take advantage of separating storage from presentation

•  Be aware of which classes your Mongo driver knows how to BSONify

•  If logging from multiple languages, sticking to Log4Lang family can help

Logging Library Basics

•  Some provide API with pluggable implementation

•  Configuration

•  Programmatic

•  Declarative

•  Dynamic

Logging Concepts

• Named, Hierarchical Logger Objects

•  Identify a class or an area of an app

•  Log Level, e.g., error, warn, info, debug

•  Thresholds determine if event is logged

•  Logging Event

•  The message

•  Metadata, e.g., time, level, class, method, line

Logging Concepts

•  Handler - determines event destination

•  File, console, socket, email, syslog, MongoDB, etc.

•  Handler level threshold

•  Async vs. sync

•  Filter - for logger and/or handler

•  Formatter - event and message

•  Pattern with placeholder tokens

App calls logger.info(“freak out”)

Common Flow for Logging

Is logger info

enabled?

Passes filter, if any?

For each handler:

No

No

Yes

Yes

Is handler info

enabled?

Passes filter, if any?

No

Yes

Yes

Format and store event

No

Polyglotism

•  Good news - lots of client language bindings and logging libraries for MongoDB

• Mostly good news - Log4J very influential

•  Sort of bad news - default log event formats can be very different

Logging by Language

•  Java

•  Python

•  Ruby

•  PHP

•  C#

Java

•  APIs

•  Apache Commons Logging

•  SLF4J

•  java.util.logging, a.k.a., JUL

•  Limited feature set

•  Destinations - console, file, socket, memory

•  JVM-level config

Log4J •  Apache Log4J is dominant logging library

•  Logback is potential successor

•  Logger - usually, represents class logging the event

•  LoggingEvent - message and metadata

•  Appender - sends LoggingEvent to destination

•  Layout - formats the event

•  Converter - convert placeholder to value

Log4mongo-java

•  Set of Log4J Appenders for MongoDB

•  log4mongo.org

• Open source project on GitHub

•  I’m a committer with Peter Monks and Jozef Ševčík

•  I used it at Voxify with speech reco applications and infrastructure services

•  Peter has used it at Alfresco

MongoDbAppender

•  Stores a Log4J LoggingEvent object as a document in a collection

•  Format is very similar to structure of LoggingEvent Java class

• Where possible, data is stored as appropriate BSON objects, rather than just strings

MongoDbPatternLayoutAppender

•  Stores log event based on user-specified pattern

•  Standard Log4J pattern layout, parser and converter functionality

•  Very flexible formatting at cost of additional config

•  Values stored as strings or arrays

•  Custom values determined at time event is logged

BsonAppender

•  Custom log format by extending BsonAppender

•  Can use it to log BSON objects to other data stores

•  Includes code for BSONifying Java exceptions

Enabling log4mongo-java • Associate a name with an appender implementation log4j.appender.MongoDB = org.log4mongo.MongoDbAppender!

• Specify host (optional port, username and password). For replica set, specify all active hosts.

log4j.appender.MongoDB.hostname = localhost

• Specify database and collection log4j.appender.MongoDB.databaseName = app_name

log4j.appender.MongoDB.collectionName = log

• Add named appender to rootLogger list log4j.rootLogger = INFO, file, MongoDB

Using Pattern Layout Appender • Specify pattern layout appender

log4j.appender.MongoDB = org.log4mongo.MongoDbPatternLayoutAppender

• Specify  your  custom  Pa2ernLayout  class  log4j.appender.MongoDB.layout = com.voxify.log.log4j.MongoDBPatternLayout

• Specify a conversion pattern

•  Must be a valid JSON document

•  Can have sub-documents

•  Values can be strings or arrays

log4j.appender.MongoDB.layout.ConversionPattern = {"ts":"%d{yyyy-MM-dd HH:mm:ss,SSS}", "level":"%p","class":"%c{1}","message":"%m"}

Shell and Tail

In the Shell

> use app_name switched to db app_name > db.log.findOne() {

"_id" : ObjectId("4c200c4d28cf037460e82363"), "ts" : "2010-10-21 18:05:17,237", "level" : "INFO", "class" : "NodeFlagGenerator", "message" : "Generating me some node flags”

}

Recent Log Events If using mongo 1.9.1+, put useful functions like this in .mongorc.js

> function last(count) {

if (count==null) count = 1;

return db.log.find({}, {_id:0}).sort({ts:-1}).limit(count);

}

> last(2)

{ "ts" : "2010-10-28 23:49:14,567", "level" : "INFO", "session" : "122", "class" : "Site", "message" : "uninformative message" }

{ "ts" : "2010-10-28 23:49:14,566", "level" : "INFO", "session" : "122", "class" : "Site", "message" : "some info kind of message" }

Tailing Logs

•  You’ll really miss ability to tail logfiles

• Or, ... will you?

• MongoDB offers tailable cursors

•  Specify cursor, e.g., from find, as tailable

• While cursor alive, print new docs then sleep

Python Tailing Example from pymongo import Connection import time db = Connection().my_db coll = db.my_collection cursor = coll.find(tailable=True) while cursor.alive: try: doc = cursor.next() print doc except StopIteration: time.sleep(1)

Python

•  Feature-rich logging module in Python 2.3

•  Influenced by Log4J

Components •  Logger - represents area of library or project

•  LogRecord

•  Handler

•  Filter

•  Formatter

•  LoggerAdapter (2.6) - add contextual info

mongodb-log

•  Python logging handler to log to MongoDB

•  Stores dictionary directly as BSON doc

•  https://github.com/andreisavu/mongodb-log

log4mongo-python

• Similar design to log4mongo PHP and .NET appenders

• Example programmatic handler config and usage: import logging from log4mongo.handlers import MongoHandler

logger = logging.getLogger('test') logger.addHandler(MongoHandler(host='localhost')) logger.warning('test')

• http://log4mongo.org/display/PUB/Log4mongo+for+Python

• https://github.com/log4mongo/log4mongo-python

Ruby

•  Built-in Logger library

•  Limited feature set

•  Console and file outputs

• Message format is fixed

•  Datetime format is configurable

Log4r •  Influenced by Log4J

•  Logger

•  Outputter

•  Formatter

•  Configurator

•  http://log4r.rubyforge.org

•  Code for using Log4r with MongoMapper

•  http://gist.github.com/207347

Logging

•  Similar to Log4r with a few enhancements

•  Supports JSON as output format

•  http://logging.rubyforge.org/

mongo_db_logger •  Rails plugin which uses Rails Logger, which can use

Ruby Logger, Log4r, etc.

•  Logs a single document per request

•  IP address, time, etc.

•  All messages logged during request

•  Custom request data

•  Configure via database.yml

•  http://github.com/peburrows/mongo_db_logger

Central Logger

•  Forked from mongo_db_logger

•  Added Rails 3 support and gemified

• Web UI for searching, filtering and analyzing logs

•  Configure via database.yml (or mongoid.yml or central_logger.yml)

•  http://www.slideshare.net/leopard_me/logging-rails-application-behavior-to-mongodb

•  http://files.meetup.com/1742411/Logging%20With%20MongoDB.pdf

•  https://github.com/customink/central_logger

MongodbLogger

•  Based on central_logger

•  Different web UI for searching, filtering and analyzing logs

•  Configure via database.yml (or mongoid.yml or mongodb_logger.yml)

•  http://mongodb-logger.catware.org/

PHP

•  error_log() logs to file, email address or system logger (e.g., syslog or NT EventLog)

•  Log4PHP - heavily influenced by Log4J

•  PEAR Log - PHP-specific logging library

•  Zend_Log - Logging for Zend Framework

Log4mongo-php

•  Log4PHP appender

•  Programmatic & declarative config

•  Timestamp stored as Mongo Date

•  Exception stored as array - msg, code, stacktrace

•  Inner exception stored if available

•  https://github.com/log4mongo/log4mongo-php

PEAR Log

•  PEAR Log class - lots of handlers (Firebug!), but none yet for MongoDB

•  Programmatic config

•  Specify handler when creating logger object

•  Composite handler for multiple destinations

•  http://pear.php.net/package/Log/

Log Writer for Zend���

•  Project -> Recordshelf_Log_Writer_MongoDb

•  Described as a “prototypesque implementation” •  Standard Zend tools to config and init

•  Zend log event is associative array

•  Project has code for accessing & filtering events •  http://raphaelstolt.blogspot.com/2009/09/logging-to-mongodb-and-accessing-log.html

C#/.NET

•  Log4net

•  Heavily influenced by Log4J

•  http://logging.apache.org/log4net/

Log4mongo-net

•  Uses community developed C# driver, but may move to official driver

•  Tested with .NET 3.5+ and Mono 2.8

•  Standard App.config/Web.config or programmatic

•  http://log4mongo.org/display/PUB/Log4mongo+for+.NET

Other Options •  Destinations include MongoDB

•  Flume, logstash, Fluentd

•  Primary Destination is HDFS

•  Chukwa and Scribe

•  Commercial Hosted

•  Loggly

•  Commercial On-Premise

•  Splunk

Flume •  If you can’t change the source of logs ...

•  Can aggregate files, syslog, etc.

•  Agent --> Collector --> Storage

•  Storage plugins for HDFS, MongoDB, etc.

•  Decorators transform data & extract metadata

•  https://github.com/cloudera/flume

•  https://github.com/mongodb/mongo-hadoop

•  Chukwa, Honu and Scribe more HDFS-centric

logstash

•  Log event storage, search and graphing

•  Read from file, syslog, AMQP, tcp, etc.

•  Store to ElasticSearch, AMQP queue, websockets, MongoDB, etc.

•  http://logstash.net/

•  http://code.google.com/p/logstash

Fluentd •  Ruby-based open source log collector

•  Input, output and buffer plug-ins

•  For example, plug-ins for tailing Apache log files and storing log events to MongoDB

•  http://blog.treasure-data.com/post/13766262632/real-time-log-collection-with-fluentd-and-mongodb

•  http://fluentd.org/doc/

•  https://github.com/fluent/fluentd

Questions?

•  Photo/Image Credits •  Log analysis - ww.flickr.com/photos/linuxtuxguy/2579295702

•  Don’t Stop Believing - www.flickr.com/photos/loozrboy/3908830690/

•  Homeland Security Advisory System - www.theonion.com/articles/iraq-adopts-terror-alert-system,1258/

•  Advice - www.flickr.com/photos/wurzle/659315/

•  Shell - http://www.flickr.com/photos/geishaboy500/305555652/

•  Tail - http://www.flickr.com/photos/autumnsonata/2632194442/

•  Log flume - www.flickr.com/photos/halloweenjack/1330806722/

•  Cufflinks - www.flickr.com/photos/oberazzi/318947873/