data discovery and systems diagnostics with elasticsearch, logstash and kibana
TRANSCRIPT
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
Data Discovery and Systems Diagnostics with the ELK stack
Rittman Mead - BI Forum 2015, BrightonRobin Moffatt, Principal Consultant Rittman Mead
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
•Principal Consultant with Rittman Mead ‣OBIEE & ODI ‣SysAdmin ‣Performance
•Previously … •OBIEE/DW developer at large UK retailer •SQL Server DBA, Business Objects, DB2, COBOL….
•Oracle ACE
•Frequent blogger for Rittman Mead : http://ritt.md/rmoff •Twitter: @rmoff • IRC: rmoff / #obihackers / freenode
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
About Me
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
About Rittman Mead
•Oracle BI and DW Gold partner •Winner of five UKOUG Partner of the Year awards in 2013 and 2014 - including BI •World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI
•Approximately 80 consultants worldwide •All expert in Oracle BI and DW •Offices in US (Atlanta), Europe, Australia and India •Skills in broad range of supporting Oracle tools: ‣OBIEE, OBIA ‣ODIEE ‣Essbase, Oracle OLAP ‣GoldenGate ‣Endeca
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
ELK
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
•Elasticsearch - schema-free, document-orientated, distributed data store
•Logstash - centralised data processing
•Kibana - analytics and visualisation
ELK
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Getting started is easy!
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.comT : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Kibana
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.comT : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Kibana
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.comT : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Kibana
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Data Discovery
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Information Management and Big Data - A Reference Architecture
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Logstash
Elasticsearch
Kibana
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
DEMO!
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
logstashElasticsearch
Kibana
csv
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
DEMO!
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Elasticsearch
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Elasticsearch
•The core component of the ELK stack
•Based on Apache Lucene (same as Cloudera’s Solr)
•Distributed for scalability & resilience
•Near-realtime document indexing
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Elasticsearch Uses - Search and Analytics
•Search ‣Soundcloud ‣GitHub
•Analytics ‣The Guardian’s Ophan application
https://www.elastic.co/assets/bltd061cc55096a5780/case-study-the-guardian.pdf
A quarter of a billion events per day … typically the lag before something shows up on the dashboard is somewhere between three to five seconds…http://tnw.to/s3NV5
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Elasticsearch
•Stores data as JSON documents within an index
•An index is made up of shards
•Shards are distributed around a cluster automatically ‣Resilience and scale-out are simple
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Elasticsearch Administration
https://github.com/lmenezes/elasticsearch-kopf
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Elasticsearch REST API
$ curl -XPOST 'http://es:9200/viz/characters/' -d '{"name":"finbarr saunders"}'
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
$ curl -XPOST 'http://es:9200/viz/characters/' -d '{"name":"roger mellie”, "notes":"the man on the tele"}'
Elasticsearch REST API
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
$ curl -XGET 'http://localhost:9200/viz/_search?q=roger' […] "hits" : { "total" : 1, "max_score" : 0.11506981, "hits" : [ { "_index" : "viz", "_type" : "characters", "_id" : "AUyyNUrTI0Rm5Pb-t8_l", "_score" : 0.11506981, "_source":{"name":"roger mellie" ,"notes":"the man on the tele"}
Elasticsearch REST API
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
$ curl -XDELETE 'http://localhost:9200/viz'
Elasticsearch REST API
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Logstash
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.comT : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Logstash
inpu
tfil
ter
outp
ut elasticsearch email
kafka
nagios
pagerduty stdout file
grok geoip mutate drop
kafkalog csv
tsv
json
syslog tcp log4j
stdin
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.comT : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Logstash
•Does Logstash support <foo> …. yes, probably! ‣Vast number of supported input (and output) formats
couchdb_changes drupal_dblog elasticsearch exec eventlog file ganglia gelf generator graphite github heartbeat heroku irc imap
jmx kafka log4j lumberjack meetup pipe puppet_facter relp rss rackspace rabbitmq redis snmptrap stdin sqlite
s3 sqs stomp syslog tcp twitter unix udp varnishlog wmi websocket xmpp zenoss zeromq
Outputsboundary circonus csv cloudwatch datadog datadog_metrics email elasticsearch exec file google_bigquery google_cloud_storage ganglia gelf graphtastic
graphite hipchat http irc influxdb juggernaut jira kafka lumberjack librato loggly mongodb metriccatcher nagios null
nagios_nsca opentsdb pagerduty pipe riemann redmine rackspace rabbitmq redis riak s3 sqs stomp statsd solr_http
sns syslog stdout tcp udp websocket xmpp zabbix zeromq
Inputs
http://www.elastic.co/guide/en/logstash/master/input-plugins.html http://www.elastic.co/guide/en/logstash/master/output-plugins.html
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.comT : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Logstash Filters
•Powerful data processing ‣Extract fields from input (grok) ‣Enrich data (geoip, dns) ‣Reformat (split, multiline, json, xml)
alter anonymize collate csv cidr clone cipher checksum date dns drop elasticsearch extractnumbers environment elapsed
fingerprint geoip grok i18n json json_encode kv mutate metrics multiline metaevent prune punct ruby range
syslog_pri sleep split throttle translate uuid urldecode useragent xml zeromq
Filters
http://www.elastic.co/guide/en/logstash/master/filter-plugins.html
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Grok — Time to get your RegEx on!
Input data
Grok pattern
Key/Value output
http://grokdebug.herokuapp.com/
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Logstash in Action
filter { grok { match => [ "message", "\[%{TIMESTAMP_ISO8601:timestamp}\] \[%{DATA:Component}\] \[%{WORD:Severity} (:%{NUMBER:LogLevelNum})?\]
input {file { path => ["nqserver.log" ] }}
output { elasticsearch { host => "localhost" }}
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Logstash -> Elasticsearch
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
•Two-way connector between Hadoop and Elasticsearch •Read/Write with Elasticsearch from Hive, Pig, Spark, etc
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Elasticsearch-Hadoop
https://www.elastic.co/products/hadoop
Hive
MongoDB HDFS
Elasticsearch
Tweets Website logs Blog post metadata
Datasift Flume CSV
mongo-hadoop
elasticsearch-hadoop
Kibana
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
Other Elasticsearch Input Methods
•JDBC ‣“River” or “Feeder” method ‣Pull data from any Oracle, mysql, etc with schema intact
•Native libraries for common languages: ‣Perl / Python / Ruby / PHP / Groovy / Scala / .NET / R / etc etc
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Systems Diagnostics
with ELK
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
System Diagnostics
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
System Monitoring
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
System Monitoring
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Performance Diagnostics
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Summary
Summary
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Summary
•Data Data Discovery With ELK
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Data Discovery
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
Summary System Diagnostics
With ELK
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
web
http://ritt.md/rmoff
@rmoff
irc
rmoff @ #obihackers
Interested? Data Discovery
http://ritt.md/go-elk-1
System Diagnostics & Monitoring http://ritt.md/go-elk-2 http://ritt.md/go-elk-3