presentation meetup elasticsearch paris #10

19
OUR STORY WITH ELASTICSEARCH November 2014

Upload: renaud-boutet

Post on 17-Jul-2015

134 views

Category:

Software


0 download

TRANSCRIPT

OUR STORY WITH ELASTICSEARCH

November 2014

WHERE DO WE COME FROM?

• ActivePivot by QuartetFS:

– InMemory ultra fast business

intelligence tool

– Mainly for traders and market risk

analysts

• One of the biggest french

success (& secret) story

• We are big data crunchers for a

long time now

HOW DID EVERYTHING START?

• Created in may 2012

• We wanted to build the perfect tool

to understand the social WEB

• We started with a very famous

NoSQL engine

But we quickly had problems

(performance, clustering, query/txns

overlaps, etc…)

ELASTICSEARCH 2 YEARS AGO

BigData ready

Easy to use/manage

Performance/Scalability

Analytical capabilities

Primary document storage?

IT DIDN’T MATCH OUR ANALYTICS REQUIREMENTS

• FieldData cache

– High memory consumption

– Memory is expensive on the cloud

• No multi-field & multi-metric aggregations

– We could not build the product of our dreams

• But ElasticSearch is modular

– We decided to implement our own analytics plugin

A SEARCH ENGINE WITH OLAP SKILLS

• Support lazy loading of fields

• Multi-Fields & Multi-Metric aggregations

• Ultra-Fast & Efficient

– Usage of columnar storage with primitive types

– Sub-seconds queries over tens of millions of elements

CUSTOM AGGREGATION EXAMPLE

The query

Our facet

2 dimensions

2 metrics

The result

WE BUILT FOCUSMATIC

WE BUILT FOCUSMATIC

WE BUILT FOCUSMATIC

ELASTICSEARCH GETS EVEN BETTER

• Release 1.0.0 – February 2014– First version of the Aggregation Engine

– Introduction of doc_values

• Release 1.2.0 – May 2014– global_ordinals / Faster Aggregations

• Release 1.4.0 – November 2014– Improv. Circuit Breakers / Safer Aggregations

– Improv. doc_values

• Every releases since 1.0.0– More stability

– More aggregation capabilies

• We had more time to develop other things!

INTRODUCING LOGMATIC

LOG GOODNESS POWERED BY ELASTICSEARCH

• Introduced Logmatic.io in private beta this year

– Beginning of 2014: A lot of logs projects around us

– Our logs experiment: It was an eye opener! (30 VMs / ~6 apps)

– 2 friendly startups tried: they went live

– Market Study: 12 projects launched

– We faced new challenges and had to build a new product!!

OUR TYPICAL USER

?

OUR APPROACH

Customer’s applications Our cloud based

infrastructureCTO, devops,

developers

Tomorrow, we’ll even have

business people. We’ll tell

you more…

alerts, reports

queries

• Centralises & enriches all

data

• Fully hosted (SaaS)

• Advanced analytics

• Real-Time

• Beautiful dataviz

• Rapidfire answers

securedconnection

And more…

UDPTCP (SSL)

HTTP(S)

DEMO

INCOMING FEATURES

• Integrated Grok parsing:

– Logs shipping should be

the only concern

– Logs structuration is done

totally in the cloud

– We extended Grok to

simplify issues like date

parsing

INCOMING FEATURES

• Integrated Grok parsing:

– Logs shipping should be

the only concern

– Logs structuration is done

totally in the cloud

– We extended Grok to

simplify issues like date

parsing

• And much more…

– Security and limited views

– Dimension contexts

– Complex metrics /

formulas

WE CONTINUE WITH ELASTICSEARCH

• Scalability

• Heterogeneity

• Query performance

• Great analytics

• Reactivity of the team

logmatic.io

@logmatic_

[email protected]

We’d love to hear from you and answer the

questions you might have