logstash + elasticsearch + kibana presentation on startit tech meetup
DESCRIPTION
TRANSCRIPT
Logstash + Elasticsearch + Kibana Centralized Log server
(as Splunk replacement)
Marko Ojleski DevOps Engineer
Grok filter
(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-
9]{1,2}))(?![0-9])
Grok filter
(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-
9]{1,2}))(?![0-9])
IP
`$=`;$_=\%!;($_)=/(.)/;$==++$|;($.,$/,$,,$\,$",$;,$^,$#,$~,$*,$:,@%)=( $!=~/(.)(.).(.)(.)(.)(.)..(.)(.)(.)..(.)......(.)/,$"),$=++;$.++;$.++;
$_++;$_++;($_,$\,$,)=($~.$"."$;$/$%[$?]$_$\$,$:$%[$?]",$"&$~,$#,);$,++ ;$,++;$^|=$";`$_$\$,$/$:$;$~$*$%[$?]$.$~$*${#}$%[$?]$;$\$"$^$~$*.>&$=`
`$=`;$_=\%!;($_)=/(.)/;$==++$|;($.,$/,$,,$\,$",$;,$^,$#,$~,$*,$:,@%)=( $!=~/(.)(.).(.)(.)(.)(.)..(.)(.)(.)..(.)......(.)/,$"),$=++;$.++;$.++;
$_++;$_++;($_,$\,$,)=($~.$"."$;$/$%[$?]$_$\$,$:$%[$?]",$"&$~,$#,);$,++ ;$,++;$^|=$";`$_$\$,$/$:$;$~$*$%[$?]$.$~$*${#}$%[$?]$;$\$"$^$~$*.>&$=`
Just another Perl hacker.
Grok filter
2.10.146.54 - 2013-12-01T13:37:57Z - some really boring message
%{IP:client} - %{TIMESTAMP_ISO8601:time} - %{GREEDYDATA:message}
Grok filter
client => 2.10.146.54 time => 2013-12-01T13:37:57Z message = > some really boring message
Grok filter
input { tcp { type => “server1" host => "192.168.1.1" port => "5555" }
filter { if [type] == “server1" { grok { match => { "message" => "%{IP:client} - %{TIMESTAMP_ISO8601:time} - %{GREEDYDATA:message} "} } }
1. RESTful api 2. JSON-oriented 3. Horizontal scale 4. HA 5. Full Text search 6. Based on Lucene
Elasticsearch Distributed RESTful
search server
Logstash => elasticsearch
input { tcp { type => “server1" host => "192.168.1.1" port => "5555" } filter { if [type] == “server1" { grok { match => { "message" => "%{IP:client} - %{TIMESTAMP_ISO8601:time} - %{GREEDYDATA:message} "} } } output { elasticsearch {} }
1. Clean and simple UI 2. Fully customizable 3. Bootstrap based 4. Old version running on Ruby 5. Milestone 3 fully rewritten in HTML/Angular.js
Kibana Awesome Elasticsearch
Web Frontend to search/graph
Scenario 1
L2 switch
Cisco ASA
L3 switch
Syslog broker (lightweight shipper)
Logstash (main log server)
Elasticsearch
Kibana
UDP
UDP
UDP