the elk stack - get to know logs
TRANSCRIPT
©2015 GlobalLogic Inc. CONFIDENTIAL
The ELK stack - get to know logs
Igor RudykDevOps / System Integrator
2 CONFIDENTIAL
Agenda• Introduction. What is ELK, and why do we need it?• The ELK stack
LogstashElasticSearchKibana
•Architecture•Demo
3 CONFIDENTIAL
Can you check the errors from yesterday between 9:09 and 9:27
4 CONFIDENTIAL
So what’s a log
5 CONFIDENTIAL
Log = timestamp + data
6 CONFIDENTIAL
Lifecycle of a log
Rec
ord
Transmit Analyze
Store
Delete
Real Real
7 CONFIDENTIAL
Default problem
• Multiple log time formats•Apr 28 20:21:59
•[27/Apr/2015:07:05:28 +0000]
•071012 09:27:32
•Mon, 27-Apr-15 06:27:02 UTC
•2015-04-28 20:07:51 +0000
• Starts not with timestamp or without timestamp•Error messages with really unhelpful info•No rotation•No scaling
Tools?
• grep
• awk / sed / cut
•less / tail
•vi / vim
•regular expression
•...
8 CONFIDENTIAL
Logging SolutionsSolutions
Collections Transport Parsing Storage Analysis Alerting
Visualizer
Commercial
Logstash
Logstash shipper or logstash-forwarder (Lumberjack)
RPM installation
Logstash shipper or logstash-forwarder (Lumberjack, encrypted transport is the default)
RPM installation
Output plugins
Central server-master with a hot-standby in case of failure
Codecs plugins
Grok debugger
ElasticSearch,
MongoDB,
AWS S3
and much more
Kibana,
graylog2
Riemann Kibana,
graylog2
NO
fluentd Input plugins
Install from source or via gem
Output plugins
Load-balance between multiple hosts or have a master with a hot-standby in case of failure
Plugins Doesn’t provide any storage tier itself but allows you to easily configure where your logs should be collected
Kibana,
graylog2
Riemann Kibana,
graylog2
NO
splunk Splunk Universal Forwarder
RPM installation
Splunk Universal Forwarder
SSL security
Splunk Splunk Splunk Splunk Splunk YES
Graylog2
Graylog2
Logstash
Graylog2
Logstash
Graylog2 ElasticSearch Kibana,
graylog2
graylog2 graylog2 NO
loggly loggly loggly loggly Hosted loggly loggly loggly YES
9 CONFIDENTIAL
What is ELK, and why do we need it?
• ELK is a stack of programs that help dealing with logs.
• Includes:
– Aggregation of logs
– Search capabilities
– Aggregation of statistics
– Visualizations
10 CONFIDENTIAL
Logstash
Unstructured
Filters Outputs
Documents
11 CONFIDENTIAL
Inputs➔ Logs:
● Lumberjack - resilient, compressed, secure (logstash-forwarder)● Remote syslog● Files
➔ Devices:● Event log, Collectd● Netflow, WMI
➔ Event Queue:● Redis, RabbitMQ● Kafka, ZeroMQ
➔ Streaming APIs:● Twitter
➔ Email (IMAP)➔ Amazon S3, ganglia, sqs, varnishlog, etc➔ ....
http://logstash.net/docs/1.4.2/ - Full list
12 CONFIDENTIAL
Filters➔ grep➔ date➔ json➔ grok➔ ....
http://logstash.net/docs/1.4.2/ - Full list
Why Do I like Logstash?It uses Grok filter for parsing standard and non standard logs:
Log Line:27/10/14 07:39:28 [localhost-startStop-1] [] INFO com.vidmind.config.LoggingPropertyPlaceholderConfigurer - streams.limit.general = 0
Pattern:
%{DATESTAMP} %{SYSLOG5424SD} ?? %{WORD:ErrorLevel} %{JAVACLASS}
13 CONFIDENTIAL
Outputs
➔ Storage:● ElasticSearch● MongoDB● S3● Graphite● File● ...
➔ Notification: ● Zabbix ● Nagios● Riemann● PagerDuty● Email
➔ Event Queue:● Redis, RabbitMQ● Kafka, ZeroMQ● tcp/udp
➔ SaaS:● AWS CloudWatch● Hipchat● Jira
➔ ....http://logstash.net/docs/1.4.2/ - Full list
14 CONFIDENTIAL
Logstash - Forwarder (Shipper)Configuration file
{ "network": { "servers": [[logstash_indexers]] "timeout": 15, "ssl ca": "logstash-forwarder.crt" }, "files": [ { "paths": [ "/usr/share/tomcat7/logs/*.json.log" ], "fields": { "type": "tomcat", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" } },
{ "paths": [ "/usr/share/tomcat7/logs/*.activities.log" ], "fields": { "type": "activities", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]"
} }
]}
15 CONFIDENTIAL
Logstash-IndexerConfiguration file
input { lumberjack { codec => json{} port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" }}
output {elasticsearch { host => "127.0.0.1" protocol => "http" cluster => "[[elasticsearch_cluster_name]]" manage_template => false index => "logstash-%{system}-%{type}-%{+YYYY.MM.dd}" }}
16 CONFIDENTIAL
ElasticSearchConfiguration file (yaml-based configuration)
cluster.name: [[elasticsearch_cluster_name]]node.name: "[[node_name]]"
node.master: false / truenode.data: false / true
index.number_of_replicas: 1
#Securitydiscovery.zen.ping.multicast.enabled: falsediscovery.zen.ping.unicast.hosts: [[elasticsearch_servers]]
action.disable_close_all_indices: trueaction.disable_delete_all_indices: trueaction.disable_shutdown: truescript.disable_dynamic: true
17 CONFIDENTIAL
The ELK stackGeneral Architecture
Logstash ElasticSearch Kibana
18 CONFIDENTIAL
KibanaElasticSearch
Logstash-Forwarder
Logstash-Indexer
Logstash-Indexer
Logstash-Indexer
The ELK stackOur Scaled Architecture
ElasticSearch
ElasticSearch
Kibana
Kibana
©2015 GlobalLogic Inc. CONFIDENTIAL
DEMO