Monitoring the unknown, 1000*100 series a day

Quentin ADAM - @waxzce

Who am I ?Quentin ADAM from the Clever Cloud

@waxzce on twitter – github- soundcloud – instagram ….

My day to day work : , the IT automation company

Keep your apps online. made with node.js,scala, java, ruby, php, python, go…

Cloud & on premise

And learn a lot of things about your code, apps, and good/bad design…

Give back to the community

NEVER GONNA LET YOU DOWN

clever-cloud.com

Automate infrastructure

What are we talking about?

Immutable infrastructureFR : https://www.youtube.com/watch?v=WrZCbgQsPVU

EN : https://www.infoq.com/presentations/server-cloud

Ephemeral instances

Pet!=Cattle

Monitoring service!=Monitoring host

Monitoring the unknowPart 1

Which tech stackpeoples will use?

Business metrics

Application metrics

middleware app

system metrics

• logins

• leads

• sales

• clicks

• ...

• SQL sessions

• User session opened

• Connections pool

• …

• GC

• JVM management

• request per seconds

• Threads

• Jobs scheduling

• …

• cpu

• load

• storage

• RAM

• ...

Monitoring the unknowPart 2

There is no ready solution…

Let’s build something

Data needs

Store measure in time

Analyze itMost of the time using a micro batching approach

Traditional needs not required

• No ACID

• No Transaction

• Just a few full text

• Eventual consistency if enough

SCALE

Time Series DB

Series and Labels

• Instance.xxxxxx.ram.free = 2078

• {• Application: myAppId

• Platform: Java

• Os: Linux

• …

• }

Watch out cardinality

Time series DB is an hot noSQL topic

Influx DB

Prometeus

Timescale DB

Based on

Open TSDB

Cassandra + Akka persistence/stream

Warp10

Standalone Java fat jar

Distributed

• Consensus using Zookeepers

• Store and analysis on Hadoop

• Event bus with Kafka

• Several roles in a single JAR, all stateless

Warp10 roles

• Ingress

• Directory

• Store

• Egress

• Webcall

• Plasma

• Runner

• Fetcher

Ingress🔑

Kafka

Hadoop Hbasecluster

Store

Dictionary

So, We can now store data

How to send some

Agent: Telegraf

Agent goals

Collect data

• System

• Many connector to specific softs

• Open protocols…

• Push or Pull?

JMX

Statsd Simple UDP protocol

Prometeus

Buffer to send batched data

~100 metric point / minutes

Token based security

More tokens: JWT and Macaroon

EN : https://www.youtube.com/watch?v=SO5G0Gsm5-w

FR : https://www.youtube.com/watch?v=mvKeCsxGZhE

✅ Send data from all VMs

✅ Send data from others infrastructures parts

✅ Store data

❔ Analyze data

❔ Trigger data

❔ Display data

WarpScript, distributed on hadoop

µBatching WarpScriptµBatchingWarpScriptµBatchingWarpScript

Build a new series

Listen on Kafka or Plasma for triggers

Immutable DataBase

✅ Send data from all VMs

✅ Send data from others infrastructures parts

✅ Store data

✅ Analyze data

✅ Trigger data

❔ Display data

Quantum

Tool

+

PolymerJS lib

Quantum

Tool

+

PolymerJS lib

Which data expose?

Templating the data

Templating the data

Templating the data

Give access to the API

Enhance developer work, happiness and productivity

Give access to metrics and logs

Thank you

find me on twitter

@waxzce

Gift coupon for clever-cloud.com:

bigDataConf2017