big data expo - machine learning in the elastic stack

Dr Stephen Dodson, Machine Learning, Elastic

Machine Learning in the Elastic Stack

2

Elastic Stack

Beats

Logstash

Kibana

Elasticsearch

X-PackX-Pack X-Pack

Security

Alerting

Monitoring

Reporting

Graph

Machine Learning

3

Extracting useful, valuable information is hard

4

Search

Aggregations

Visualization

Machine Learning

Extracting useful, valuable information is hard

Machine Learning1Algorithms and methods for data driven prediction, decision making, and modelling

Supervised Learning

Prediction based on examples of correct behavior

1Machine Learning Overview, Tommi Jaakkola, MIT

Unsupervised Learning

No explicit target, only data, goal to model/discover

Semi-supervised Learning

Supplement limited annotations with unsupervised learning

Active Learning

Learn to query the examples actually needed for learning

Transfer Learning

How to apply what you have learned from A to B

Reinforcement Learning

Learning to act, not just predict; goal to optimize the consequences of

actions

Other! …

Machine Learning1Algorithms and methods for data driven prediction, decision making, and modelling

Supervised Learning

Prediction based on examples of correct behavior

1Machine Learning Overview, Tommi Jaakkola, MIT

Unsupervised Learning

No explicit target, only data, goal to model/discover

Semi-supervised Learning

Supplement limited annotations with unsupervised learning

Active Learning

Learn to query the examples actually needed for learning

Transfer Learning

How to apply what you have learned from A to B

Reinforcement Learning

Learning to act, not just predict; goal to optimize the consequences of

actions

Other! …

Time Series Anomaly Detection

Machine Learning - Technical Debt

7

• Machine Learning Systems generally involved a large amount of plumbing and complexity that incur large ongoing maintenance costs

From Google Paper: Sculley, D., et al. "Machine learning - The high-interest credit card of technical debt." (2014).

© Elasticsearch BV

Has my order rate dropped significantly?

Has my order rate dropped significantly?

• Learn models from past behaviour (training, modelling)

• Use models to predict future behaviour (prediction)

• Use predictions to make decisions

Expected value @ 15:05 = 1859

Actual value @ 15:05 = 280

Probability = 0.0000174025

Demo: Simple Time Series

Has my system changed behaviour?

i-5cfd3dcb

...

…

i-f1e94994

i-ece626ff i-ebc323df

Has my system changed behaviour?

i-5cfd3dcb i-f1e94994

i-ece626ff i-ebc323df

...

…

Demo: Multiple Time Series

Do my application logs contain unusual messages?

Do my application logs contain unusual messages?Classify unstructured log messages by clustering similar messages

Nor

mal

Log

Mes

sage

sU

nusu

al lo

g M

essa

ges

Demo: Log Messages

17

Ingest, Enrich, Visualize, Analyse, Alert

Elasticsearch

X-pack

Master Nodes (3)

Ingest Nodes (X)

Data Nodes - Hot (X)

Data Nodes - Warm (X)

Beats

Log Files Metrics

Wire Data your(beat)

Filebeat Module

NGINX

Kibana

X-pack

Instances (X)

18

Coming soon - forecasting…and more…

Exceptwhereotherwisenoted,thisworkislicensedunderhttp://creativecommons.org/licenses/by-nd/4.0/

CreativeCommonsandthedoubleCinacircleareregisteredtrademarksofCreativeCommonsintheUnitedStatesandothercountries.

Thirdpartymarksandbrandsarethepropertyoftheirrespectiveholders.

Please attribute Elastic with a link to elastic.co