the devops toolbox: open source log analytics

logz.io | @logzio | @tomerlevy | @asafyigal

The DevOps Toolbox: Open Source Log Analytics

Tomer Levy & Asaf YigalCofounders, Logz.io

logz.io | @logzio | @tomerlevy | @asafyigal

Is anyone using ELK to process logs?

logz.io | @logzio | @tomerlevy | @asafyigal

Is anyone using the public cloud? AWS?

logz.io | @logzio | @tomerlevy | @asafyigal

Is anyone doing kite-surfing?

Windsurfing?

Paragliding?

Sailing?

logz.io | @logzio | @tomerlevy | @asafyigal

ELK – What is it good for?

logz.io | @logzio | @tomerlevy | @asafyigal

This is our office

logz.io | @logzio | @tomerlevy | @asafyigal

And this is the view from the office

logz.io | @logzio | @tomerlevy | @asafyigal

When should we leave everything and go

kitesurfing?

This is our challenge

logz.io | @logzio | @tomerlevy | @asafyigal

Wind analytics

logz.io | @logzio | @tomerlevy | @asafyigal

Wind across locations

logz.io | @logzio | @tomerlevy | @asafyigal

Next steps

logz.io | @logzio | @tomerlevy | @asafyigal

Wind analytics — next items

• Alerts

• Wind forecast combined with real wind

• Wind predictions!

logz.io | @logzio | @tomerlevy | @asafyigal

Who are we?

• Logz.io Insights: Behavioural intelligence to pinpoint

what actually matters in logs

• ELK++ as a Service – Infinitely scalable

– Secured

– Highly Available

– Additional Features (alerts, role-based access)

logz.io | @logzio | @tomerlevy | @asafyigal

ELK implementation —creating the right architecture

logz.io | @logzio | @tomerlevy | @asafyigal

ELK implementation —creating the right architecture

Curator

Curator

3x Master Nodes + 1 data

logz.io | @logzio | @tomerlevy | @asafyigal

ELK implementation —creating the right architecture

Curator

3x Master Nodes + 1 data

Index Failures Handler

logz.io | @logzio | @tomerlevy | @asafyigal

ELK basic implementation —find the weak spots

AZ-1

AZ-2

ELBLB

logz.io | @logzio | @tomerlevy | @asafyigal

• Grok – parse logs to extract the relevant fields…

• Try our blog for some help on grok/plugins etc’

• blog.logz.io

ELK basic implementation — configuration

logz.io | @logzio | @tomerlevy | @asafyigal

1. Use Elasticsearch AWS Plugin

2. EBS are challenging for big environment Use PIOPS if you can afford

3. Don’t run AWS cluster on the same AZ (but don’t run them on different zones!) - use Shard allocation awareness

4. S3 Snapshots are cool! Things tend to break…

Elasticsearch basic implementation —configuration

logz.io | @logzio | @tomerlevy | @asafyigal

Let’s see a high-level view of how we process logs

Demo

logz.io | @logzio | @tomerlevy | @asafyigal

Want to try our product?

• Email us with questions:

tomer@logz.io

asaf@logz.io

• Visit our website for more information!

logz.io | @logzio | @tomerlevy | @asafyigal

We’re hiring — big time!

• Elasticsearch experts

• Java developers

• Machine-learning experts

• See our job listings page