night owl by boyd meyer of pros
TRANSCRIPT
![Page 1: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/1.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Night OwlLog Monitoring using Elasticsearch and Hadoop
Boyd Meier ([email protected])Hadoop Meetup – October 16, 2013
![Page 2: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/2.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Problem
![Page 3: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/3.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Application Performance Monitoring
● Many servers● Many applications● Many log formats● Many places to go look for information
● What if we could just look in one place and see everything?
![Page 4: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/4.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Advanced Analysis
● The logs are too low-level● The servers need the existing capacity● The amount of data to be analyzed is huge● Some analysis needs to be across multiple servers● What if we want to change the analysis algorithms?
● How we can do analysis in the most flexible way possible?
![Page 5: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/5.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Proactive Support
● See problems coming before they become crises● Watch for errors and exceptions● Track performance of the application● Track usage of the application● Enable checks we haven’t thought of yet
![Page 6: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/6.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Some Analysis Questions
● What errors happen, and how often?● Who did what, when?● How long did it take to do a task?● What else was happening on the server?
![Page 7: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/7.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Constraints
● Very little budget – as much free stuff as possible● Can’t use client machines● Communications need to be secure● Large amounts of data (Gb/day/client)● Minimize support’s dependence on client IT
![Page 8: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/8.jpg)
Approach
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
![Page 9: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/9.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Hadoop
● We have a lot of data (~2 GB day with 3 clients)● We need to process it in reasonable time● We can’t afford a big machine for this● We have lots of old machines lying around
● Sounds like a job for the elephant! But what about query?
![Page 10: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/10.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Elasticsearch
● Query performance on base Hadoop is painful● Ad-hoc queries are required● Hadoop integration● Cluster deployment
● Looks promising! How do we get the data into the server?
![Page 11: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/11.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Logstash
● Handle many sources, not just logs● Fan-in architecture to server● Compressed, SSL encrypted data● Can offload some logic on the client if desired● Massively configurable● Output to Elasticsearch
● Great! Now how about visualization?
![Page 12: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/12.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Kibana
● Backed by Elasticsearch● Supports dynamic queries● View information over time● Built-in support for Logstash ● Configurable, shareable dashboards
![Page 13: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/13.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
High Level Software Architecture
KibanaElasticsearch
LogstashClient
LogstashServer
![Page 14: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/14.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Hadoop Processing
● Pig scripts process the data● Wonderdog from InfoChimps to integrate Pig and
Elasticsearch– There are issues:• Cluster stability using Wonderdog• Wonderdog Pig interface has not been updated in a while• Currently evaluating elasticsearch-hadoop project from Elasticsearch.org
● Analysis results are stored in Elasticsearch for ease of access
![Page 15: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/15.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Demo
![Page 16: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/16.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Configuration Details
![Page 17: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/17.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
![Page 18: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/18.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Software
● Ubuntu 12.04.2 LTS (Precise)● Cloudera CDH 4.3.1–Hadoop 2.0.0–Hbase 0.94–Hive 0.10– Pig 0.11
● Elasticsearch 0.90.3● Logstash 1.1.12● Kibana 3 M3
![Page 19: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/19.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Hardware Architecture
● 27 node cluster of commodity machines ● 42 TB of disk space● Connected via 10 gigabit switch● Each machine has:– 8 GB RAM– 2 TB SATA HDD–Gigabit Ethernet
![Page 20: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/20.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Performance
● Over the month of September:– 188 million events ingested from 3 clients– 57.5 GB storage used (1.92 GB / day)
● At that rate, 42 TB is enough space for:– 142 billion events– 60 years of data from these clients– 1 year of data from 180 clients at the same volume per client
![Page 21: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/21.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
Resources
● Elasticsearch - http://www.elasticsearch.org/overview/• http://github.com/elasticsearch/elasticsearch
● Logstash - http://www.elasticsearch.org/overview/logstash/• https://github.com/logstash/logstash
● Kibana - http://www.elasticsearch.org/overview/kibana/• https://github.com/elasticsearch/kibana
● ES – Hadoop - http://www.elasticsearch.org/overview/hadoop/• http://github.com/elasticsearch/elasticsearch-hadoop
● Cloudera - http://www.cloudera.com/
![Page 22: Night owl by Boyd Meyer of PROS](https://reader036.vdocuments.us/reader036/viewer/2022081519/554f850fb4c905435d8b4b64/html5/thumbnails/22.jpg)
© COPYRIGHT PROS, INC. 2013 | CONFIDENTIAL AND PROPRIETARY
PROS Germany GmbHFeringastrasse 685774 UnterfoehringMunichTel.: +49 89 99216 270 Fax: +49 89 99216 200
Regional Office - Austin, TX3600 Parmer Lane, Suite 205Austin, Texas 78727Regional Office - Cary, North Carolina1000 Centre Green Way, #200Cary, NC 27513Phone:+1 919-228-6334
World Headquarters3100 Main Street, Suite #900Houston, TX 77002 Phone: +1 713-335-5151Sales: +1 855-846-0641Fax: +1 713-335-8144
European Headquarters - United KingdomLakeside House1 Furzeground WayStockley ParkHeathrowUB11 1BDPhone: +44 (0) 208 622 3555Fax: +44 208 622 3230