the devops toolbox: open source log analytics
TRANSCRIPT
logz.io | @logzio | @tomerlevy | @asafyigal
The DevOps Toolbox: Open Source Log Analytics
Tomer Levy & Asaf YigalCofounders, Logz.io
logz.io | @logzio | @tomerlevy | @asafyigal
Is anyone using ELK to process logs?
logz.io | @logzio | @tomerlevy | @asafyigal
Is anyone using the public cloud? AWS?
logz.io | @logzio | @tomerlevy | @asafyigal
Is anyone doing kite-surfing?
Windsurfing?
Paragliding?
Sailing?
logz.io | @logzio | @tomerlevy | @asafyigal
And this is the view from the office
logz.io | @logzio | @tomerlevy | @asafyigal
When should we leave everything and go
kitesurfing?
This is our challenge
logz.io | @logzio | @tomerlevy | @asafyigal
Wind analytics — next items
• Alerts
• Wind forecast combined with real wind
• Wind predictions!
logz.io | @logzio | @tomerlevy | @asafyigal
Who are we?
• Logz.io Insights: Behavioural intelligence to pinpoint
what actually matters in logs
• ELK++ as a Service – Infinitely scalable
– Secured
– Highly Available
– Additional Features (alerts, role-based access)
logz.io | @logzio | @tomerlevy | @asafyigal
ELK implementation —creating the right architecture
logz.io | @logzio | @tomerlevy | @asafyigal
ELK implementation —creating the right architecture
Curator
Curator
3x Master Nodes + 1 data
logz.io | @logzio | @tomerlevy | @asafyigal
ELK implementation —creating the right architecture
Curator
3x Master Nodes + 1 data
Index Failures Handler
logz.io | @logzio | @tomerlevy | @asafyigal
ELK basic implementation —find the weak spots
AZ-1
AZ-2
ELBLB
logz.io | @logzio | @tomerlevy | @asafyigal
• Grok – parse logs to extract the relevant fields…
• Try our blog for some help on grok/plugins etc’
• blog.logz.io
ELK basic implementation — configuration
logz.io | @logzio | @tomerlevy | @asafyigal
1. Use Elasticsearch AWS Plugin
2. EBS are challenging for big environment Use PIOPS if you can afford
3. Don’t run AWS cluster on the same AZ (but don’t run them on different zones!) - use Shard allocation awareness
4. S3 Snapshots are cool! Things tend to break…
Elasticsearch basic implementation —configuration
logz.io | @logzio | @tomerlevy | @asafyigal
Let’s see a high-level view of how we process logs
Demo
logz.io | @logzio | @tomerlevy | @asafyigal
Want to try our product?
• Email us with questions:
• Visit our website for more information!
logz.io | @logzio | @tomerlevy | @asafyigal
We’re hiring — big time!
• Elasticsearch experts
• Java developers
• Machine-learning experts
• See our job listings page