osdc 2014: jordan sissel & lennart koopmann - intro to log management
DESCRIPTION
Log management can be overwhelming! This talk provide an overview of log management. You will learn about different kinds of logs, structured/unstructured logs, retention, archival, and more. We will also briefly introduce several common tools in the log management space.TRANSCRIPT
![Page 1: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/1.jpg)
Log ManagementAn Introduction
Lennart Koopmann Jordan Sissel
![Page 2: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/2.jpg)
What is a Log?
![Page 3: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/3.jpg)
time + data
What is a Log?
![Page 4: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/4.jpg)
37.5.55.31 - - [08/Apr/2014:15:31:30 -0400] "GET /images/web/2009/banner.png HTTP/1.1" 304 - "http://semicomplete.com/style2.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0"
What is a Log? time + data
![Page 5: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/5.jpg)
120707 0:40:34 4 Connect root@localhost on 4 Query select @@version_comment limit 1 120707 0:40:45 4 Query select * from mysql.user
What is a Log? time + data
?????? ???????
![Page 6: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/6.jpg)
Kinds of Logs
![Page 7: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/7.jpg)
Trace and Debug
Kinds of Logs
![Page 8: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/8.jpg)
Accounting
Kinds of Logs
![Page 9: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/9.jpg)
Transaction
Kinds of Logs
![Page 10: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/10.jpg)
Problems
![Page 11: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/11.jpg)
Difficult to Access
Problems
![Page 12: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/12.jpg)
Too Many Logs
Problems Difficult to Access
![Page 13: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/13.jpg)
Too Many Servers
Problems Difficult to Access
![Page 14: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/14.jpg)
No Permissions :(
Problems Difficult to Access
![Page 15: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/15.jpg)
Difficult to Consume
Problems
![Page 16: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/16.jpg)
Unstructured
Problems Difficult to Consume
![Page 17: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/17.jpg)
Requires Expertise
Problems Difficult to Consume
![Page 18: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/18.jpg)
Requires Maintenance
Problems
![Page 19: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/19.jpg)
Configuration
Problems Requires Maintenance
![Page 20: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/20.jpg)
Log Retention
Problems Requires Maintenance
![Page 21: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/21.jpg)
Bad Tooling
Problems
![Page 22: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/22.jpg)
grep, ssh, awk
Problems Bad Tooling
![Page 23: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/23.jpg)
Life of a Log
![Page 24: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/24.jpg)
Life of a Log
Record
Transport
Search & Analyze
Archive
Delete
![Page 25: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/25.jpg)
Sources of Logs
![Page 26: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/26.jpg)
Vendor Hardware
Sources of Logs
Routers, VPNs, Printers, Phones, AWS CloudTrail, etc
![Page 27: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/27.jpg)
Vendor Software
Sources of Logs
Nginx, Wordpress, Jira
![Page 28: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/28.jpg)
In-house Software
Sources of Logs
Your company controls it
![Page 29: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/29.jpg)
Solutions(Open Source!)
![Page 30: OSDC 2014: Jordan Sissel & Lennart Koopmann - Intro to log management](https://reader033.vdocuments.us/reader033/viewer/2022060108/554f6d95b4c9058a148b506b/html5/thumbnails/30.jpg)
Solutions
Logstashfrom Elasticsearch
Graylog2from Torch