ELK - StackA perfect match for your Log Management

Steve Behrendt

@derStevedersteve.com

2

The problem Distributed systems Service-oriented Architectures Microservices Multi-language systems Multi-technology stack Multiple Datastores (SQL, noSQL, File stores)

3

Traditional Architecture

4

Browser

IIS

Store App

SQL Server

Product Schema

Invoice Schema

Billing Schema

Customer Schema

Product Module

Customer Module

Azure DocumentDB

MySQL

Azure SQL Server

Microservices

5

Browser

IIS

Apache / Tomcat

Azure

Product UI

Service

Customer UI

Service

Invoicing

Service

CMS Service

Customer Schema

SQL Server

Product Schema

Content Schema

Billing Schema

Azure DocumentDB

MySQL

Azure SQL Server

Be the logs with you

6

Browser

IIS

Apache / Tomcat

Azure

Product UI

Service

Customer UI

Service

Invoicing

Service

CMS Service

Customer Schema

SQL Server

Product Schema

Content Schema

Billing Schema

The challenges Different log formats Each log has its expert Different log locations (machines/servers) Different date formats Internet of things – decentralised log creation

and storing Searching files by keyword is hard Combination of different messages Setting the log context

7

One solution:

ELK

8

ELK E - Elasticsearch L - Logstash K – Kibana … Lucene Shield Marvel

9

The ELK architecture

10

LogsLogstas

h Elasticsearch Kibana

IIS

Syslog

EntLib

Broker

Indexer SearchStorage

Visualize

Visualize

Visualize

Visualize

Logstash Collecting, Filtering, Normalizing, Sending logs to a central location

Understands the logs

11

Logstash Pipeline

12

• Log FileInput• grok• date• geoip• useragent

Filter

• Elasticsearch• Console

Output

Grok debugger

http://grokdebug.herokuapp.com/

13

Forwarder - lumberjack Separate service to forward messages to a

remote endpoint, e.g. logstash instance or elasticsearch

14

Logstash DEMO

15

Elasticsearch Based on lucene for indexing and searching - but

lucene is just a library and very complex Provides (simple) Restful - API abstraction on top

of lucene Stores documents in json format

16

Elasticsearch - Scaling Supports vertical (bigger hardware) and

horizontal scaling (more hardware)

Horizontal scaling is hard, but Elasticsearch is distributed by nature

17

The empty cluster

18

Node: Is a running instance of elasticsearch

Cluster: A cluster consists of one or more nodes with the same cluster name that are working together to share their data and workload

Index and shards

19

Shard: low-level worker holding a slice of data. A single instance of lucene.

Index: logical namespace that points to one or more physical shards

Replicas / Failover

20

Primary and replica shards: Primary and associated replica shard store the same documents.

Newly indexed document first stored on a primary shard, then copied in parallel to the associated replica shard(s).

Horizontal scaling

21

3 shards spread across 3 from 2 nodes.

Each shard is full fledged search engine. Scaling by increasing number of replica shards.

Cluster Discovery Discovering nodes inside a cluster and electing a

master node

Zen discovery

22

Types, Documents, Fields

23

Relational Database

Elasticsearch

Databases

Indices

Tables

Types

Rows

Documents

Columns

Fields

Storing documents

PUT /customer/employee/1 { "first_name" : "John", "last_name" : "Smith", "age" : 25, "about" : "I love to go rock climbing", "interests": [ "sports", "music" ] }

24

Retrieving document

GET /customer/employee/1

Search lite

GET /customer/employee/search

25

Query DSL

GET /customer/employee/_search{ "query" : { "match" : { "last_name" : "Smith" } }}

26

Elasticsearch

DEMO

27

Kibana Data Visualization + Data Discovery

28

Kibana DEMO

29

What’s missing? Security Alerting

30

Alternatives?

31

32

Go and grok some logs