Monitoring the unknown, 1000*100 series a day
Quentin ADAM - @waxzce
Who am I ?Quentin ADAM from the Clever Cloud
@waxzce on twitter – github- soundcloud – instagram ….
My day to day work : , the IT automation company
Keep your apps online. made with node.js,scala, java, ruby, php, python, go…
Cloud & on premise
And learn a lot of things about your code, apps, and good/bad design…
Give back to the community
NEVER GONNA LET YOU DOWN
clever-cloud.com
Automate infrastructure
What are we talking about?
Immutable infrastructureFR : https://www.youtube.com/watch?v=WrZCbgQsPVU
EN : https://www.infoq.com/presentations/server-cloud
Ephemeral instances
Pet!=Cattle
Monitoring service!=Monitoring host
Monitoring the unknowPart 1
Business metrics
Application metrics
middleware app
system metrics
• logins
• leads
• sales
• clicks
• ...
• SQL sessions
• User session opened
• Connections pool
• …
• GC
• JVM management
• request per seconds
• Threads
• Jobs scheduling
• …
• cpu
• load
• storage
• RAM
• ...
Monitoring the unknowPart 2
There is no ready solution…
Let’s build something
Data needs
Store measure in time
Analyze itMost of the time using a micro batching approach
Traditional needs not required
• No ACID
• No Transaction
• Just a few full text
• Eventual consistency if enough
SCALE
Time Series DB
Series and Labels
• Instance.xxxxxx.ram.free = 2078
• {• Application: myAppId
• Platform: Java
• Os: Linux
• …
• }
Watch out cardinality
Time series DB is an hot noSQL topic
Influx DB
Prometeus
Timescale DB
Based on
Open TSDB
Cassandra + Akka persistence/stream
Warp10
Standalone Java fat jar
Distributed
• Consensus using Zookeepers
• Store and analysis on Hadoop
• Event bus with Kafka
• Several roles in a single JAR, all stateless
Warp10 roles
• Ingress
• Directory
• Store
• Egress
• Webcall
• Plasma
• Runner
• Fetcher
Ingress🔑
Kafka
Hadoop Hbasecluster
Store
Dictionary
So, We can now store data
How to send some
Agent: Telegraf
Agent goals
Collect data
• System
• Many connector to specific softs
• Open protocols…
• Push or Pull?
JMX
Statsd Simple UDP protocol
Prometeus
Buffer to send batched data
~100 metric point / minutes
Token based security
More tokens: JWT and Macaroon
EN : https://www.youtube.com/watch?v=SO5G0Gsm5-w
FR : https://www.youtube.com/watch?v=mvKeCsxGZhE
✅ Send data from all VMs
✅ Send data from others infrastructures parts
✅ Store data
❔ Analyze data
❔ Trigger data
❔ Display data
WarpScript, distributed on hadoop
µBatching WarpScriptµBatchingWarpScriptµBatchingWarpScript
Build a new series
Listen on Kafka or Plasma for triggers
Immutable DataBase
✅ Send data from all VMs
✅ Send data from others infrastructures parts
✅ Store data
✅ Analyze data
✅ Trigger data
❔ Display data
Quantum
Tool
+
PolymerJS lib
Quantum
Tool
+
PolymerJS lib
Which data expose?
Templating the data
Templating the data
Templating the data
Give access to the API
Enhance developer work, happiness and productivity
Give access to metrics and logs
Thank you
find me on twitter
@waxzce
Gift coupon for clever-cloud.com:
bigDataConf2017