state of the art logging

Copyright 2013 BalaBit IT Security Ltd.

State of the art loggingSyslog-ng, journal, CEE/Lumberjack and ELSA

Péter Czanikcommunity manager


Topics

• No, it is not about cutting trees :-)• What is syslog? And syslog-ng?• Free-form messages against name-value pairs• The new buzzword: journal• Standardization efforts: CEE/Lumberjack• Name-value pairs at work: ELSA


What is syslog?

• Logging: recording events

• Syslog:- Application: collecting events- Protocol: forwarding events


What is syslog-ng?

• “Next Generation” syslog server• “Swiss army knife” of logging

• More input sources (files, sockets, and so on)• Better filtering (not only priority, facility)• Processing (rewrite, normalize, correlate, and so

on)• More destinations (databases, encrypted network,

and so on)


What is new since 2.0

• 2.0 is best known, but EOL• Most important new features since 2.0:

- PatternDB and CSV message parsing- Correlation- SQL and MongoDB destinations- JSON formatting- Modularization- Multi-threading

• Next: 3.4- JSON parsing- More flexible configuration


Free form log messages

• Most logs are in /var/log• Most are from syslog (but also wtmp, apache, and

so on)• Most are: date + hostname + text

Mar 11 13:37:56 linux-6965 sshd[4547]: Accepted keyboard-interactive/pam for root from 127.0.0.1 port 46048 ssh2

• Text = English sentence with some variable parts• Easy to read


Why it does not scale?

• Few logs (workstation) → easy to find information• Many logs (server) → difficult to find information• Relevant information is presented differently by

each application• Difficult to process them with scripts

• Answer: structured logging- Events represented as name value pairs


Solution from syslog-ng: PatternDB

• Most messages are static texts with some variable parts embedded

• PatternDB parser:- Can extract useful information into name-value pairs- Add status fields based on message text

• Example:- user=root- action=login- status=failure

• It requires patterns• syslog-ng: name-value pairs inside


Journal

• The logging component of systemd• Name-value pairs inside:

- Message- Trusted properties- Any additional name-value pairs

• Native support for name-value pair storage


Journal: the enemy?

• FAQ: Q: is journal the enemy? A: No!• Journal is limited to Linux/systemd (syslog-ng: all

Linux/BSD/UNIX)• Journal is local only (syslog-ng: client – server)• Journal does not filter or process log messages

• Journal + syslog-ng complement each other• Logs forwarded to syslog-ng through:

/run/systemd/journal/syslog

• syslog-ng can filter, process and forward logs to many different destinations (one day also to journal)


CEE

• Journal, syslog-ng, Windows eventlog, rsyslog, auditd, and so on are based on name-value pairs

• All use different field names• Standardization is a must: CEE → Common Event

Expression• Events: name-value pairs instead of free-form text

- Taxonomy: name-value pairs to describe events (example: status)- Dictionary: name-value pairs for event parameters (example: user)

• PatternDB can turn free-form messages into CEE


Lumberjack

• Make CEE happen → implementation• Coordinated by RedHat

- CEE (Mitre), syslog-ng, rsyslog, and so on- Open, with high traffic mailing list- https://fedorahosted.org/lumberjack/

• API(s) to make structured logging easier• Work on dictionary, taxonomy, transport issues

https://fedorahosted.org/lumberjack/




Name-value pairs in action: ELSA

• ELSA: Enterprise Log Search and Archive• Based on syslog-ng, PatternDB and MySQL• Simple and powerful web GUI• Extreme scalability• Patterns focused on network security (Cisco,

Snort, HTTP, Bro, and so on)


Some logs


Diagram


A few extras


Questions?

• Questions?


Thank You!Péter Czanik

community [email protected]

state of the art logging

Technology

events syslog

linuxsystemd syslogng

patterns syslogng

art logging syslogng

ceelumberjacknamevalue

value pairs insidecopyright

network security cisco

value pairscopyright