javantura v3 - elk – big data for devops – maarten mulders

29
ELK BigData for DevOps Javantura v3 // February 20, 2016 Maarten Mulders // @mthmulders

Category:

Technology


3 download

TRANSCRIPT

ELKBigData for DevOps

Javantura v3 // February 20, 2016 Maarten Mulders // @mthmulders

AgendaE, L, KReal-world use caseQ & A

ELK?

elastic (search)"search, analyze in real-time. sweet"

logstash"scrub, parse and enrich. like soap for your data"

kibana"line graphs, pie charts... yeah we got that"

all together nowlogstash → collect log files

elastic → storage and analysis

kibana → visualisation

input { file { path => "/path/to/file.log" } output { path => "/path/to/copied.log" } }

logstashvery modular: various inputs, filters and outputs

 

input: various application log files, but also syslog, stdin, xmpp, log4jsocket, irc, ...

filter: extract semantics (geo info, grok), add information, removeinformation, match fields (cidr, dates, numbers, dns, user agent), ...

output: send events to another system such as graphite,elasticsearch, email, file, stdout, irc, jira, nagios, s3, redis, xmpp, ...

elasticsearch and analytics engine

very scalable

stores collected log events in an uniform way

events can be filtered and queried by clients (e.g. kibana)

kibanaanalytics and search dashboard for elastic

 

just html and javascript (dashboards can be saved to elastic, too)

filtering determines what data is used to populate the dashboard,queries categorise data inside the dashboard

Real-world use case

logstash setupprocessess technical logging and audit logging

adds information (hostname, environment, application name)

removes information (sensitive details about customers,transactions)

transforms information to a more usable form

 

ship events to redis

elastic setuplarge cluster that contains data

one month of history

also hosts kibana files and stores its dashboards

kibana configurationfilters based on environment and timestamp (last 24h)

automatically refreshed

queries for 'error', 'orange cell', specific error codes

rows and panels for optimal screen usage

logstash inputinput { file { path => "/path/to/application.log" codec => multiline { pattern => "̂%{TIMESTAMP_ISO8601} " negate => true what => previous } type => "application" } file { path => "/path/to/audit.log" type => "audit" } }

logstash filtersregular application log file

filter { if [type] == "application" { grok { match => { "message" => "(?m)%{TIMESTAMP_ISO8601:timestamp} \[%DATA\] %{LOGLEVEL:level} %{JAVACLASS} %{GREEDYDATA:line}" } remove_field => "message" } } }

logstash filters (ctd)audit log file

2015-01-28 01:32:15,098 [thread-1] INFO nl.ing.application.Class eventId=1401751935098~|~inChannel=MINGZ~|~odBeneficiaryAccount=NL28INGB0000000001

filter { if [type] == "audit" { grok { match => { "message" => "(?m)%{TIMESTAMP_ISO8601:timestamp} \[%DATA\] %{LOGLEVEL} %{JAVACLASS} %{GREEDYDATA:audit_message}" } remove_field => "message" } mutate { gsub => ["audit_message", "\~\|\~", "̀"] } kv { source => "audit_message" field_split => "̀" remove_field => "audit_message" } prune { blacklist_names => "̂od.+$" } } }

{ timestamp: "2015-01-28 01:32:15,098", eventId: "1401751935098", inChannel: "MINGZ" }

logstash filters (ctd)just in case...

filter { if "_grokparsefailures" in [tags] { prune { blacklist_names => [ "message", "audit_message" ] } } }

logstash outputoutput { redis { host => "redis-host" data_type => "list" key => "logstash" } }

Questions?