time series database and tick stack
TRANSCRIPT
Time Series DataTick Stack
Hello!
I am Gianluca ArbezzanoSoftware Engineer at CurrencyFair
@gianarb on Twitter and GitHub
2
3
Open Source Maintainer and Contributors
penny.gianarb.it vim-php.org
4
scaledocker.comThe book that drives you in production with
docker
“
“Try again. Fail again.Fail better.”cit. Samuel Beckett
5
Trust your system
To be familiar with your applications you need to know what they are doing.
6
1
Widespread Monitoring Tools
tail -f /var/log/app.log
7
8
9
2016/04/15 15:42:46 [warn] 2330#0: *167 using uninitialized variable, client: 10.0.1.1, server: localhost.dev, request: "POST /auth HTTP/1.1", host: "localhost"
2016/04/15 15:44:44 [error] 2330#0: *171 FastCGI sent in stderr: " PHP message: PHP Fatal error: Uncaught exception 'RuntimeException' with message 'All broken)[500]' in /var/www/my/project.php:237
Stack trace:#0 /var/www/index.php:45 ObjectService->flush()#1 [internal function] ->save()
2
Expensive to store
10
2
Difficult to index
11
Difficult not impossible
They do an amazing work. And there are other tools!!
12
They are awesome for some use cases
▪ Extract informations
▪ They can be “human readable”
▪ … and others, and others
13
Keep your life amazing
Reduce your time series to a timestamp
and a value (int or float)
14
This is cheap and useful
We are here to speak about Time Series
[ { "name": "log_lines", "columns": ["time", "line"], "point": [1400425947368, "here's some useful log info"] }]
15
EASY! EASY! EASY!
{ "name": "cpu_percent_use", "columns": ["value"], "point": 40}
16
““Any intelligent fool can make things
bigger, more complex, and more violent. It takes a touch of genius — and a lot of
courage to move in the opposite direction.”
Cit. Ernest F. Shumacher
17
Time is a perfect sharding key
It means that Time Series scale really well
18
We have a set of tools to use
19
20
InfluxDB
● Optimized to store time series data
● Open Source and easy to install (go
binary)
● Big community and ecosystem to
manage alert and collect metrics
21
Easy
Install and start a go binary
wget https://dl.influxdata.com/influxdb/releases/influxdb_1.0.0_amd64.debsudo dpkg -i influxdb_1.0.0_amd64.debInfluxd -config /usr/local/etc/influxdb.conf
22
Easy
HTTP API on port 8086
Support for UDP connections
Admin Panel on port 8083
23
Easy
SELECT value
FROM cpu_load_short
WHERE region='us-west'
24
T-Shirts time!
25
26
27
[key] [fields] [timestamp]temperature,machine=unit internal=3,external=10 1434055562000000035
Inline Protocol thinked to be smart and slim
28
Corley\Benchmarks\InfluxDB\AdapterEvent Method Name Iterations Average Time Ops/second ------------------------ ------------ -------------- ------------- sendDataUsingHttpAdapter: [1,000 ] [0.0026700308323] [374.52751] sendDataUsingUdpAdapter : [1,000 ] [0.0000436344147] [22,917.69026]
UDP vs TCP protocol
29
CREATE CONTINUOUS QUERY minnie ON world BEGIN SELECT min(mouse) INTO min_mouse FROM zoo GROUP BY time(30m) END
Continuous Query
Telegrafhttps://github.com/influxdata/telegraf
Collector to grab and send data from different sources to InfluxDB and other databases
30
Based on Input and out Plugin System
31
Telegraf Plugins
Kapacitorhttps://github.com/influxdata/kapacitor
Trigger notifications and make action in case of specific
behaviors
32
framework for processing, monitoring, and alerting on time series data
33
Kapacitor high CPU alert
stream
|from()
.measurement('cpu_usage_idle')
.groupBy('host')
|window()
.period(1m)
.every(1m)
|mean('value')
|eval(lambda: 100.0 - "mean").as('used')
|alert()
.message('{{ .Level}}: {{ .Name }}/{{ index .Tags "host" }} has high cpu: {{ index .Fields "used" }}')
.warn(lambda: "used" > 70.0)
.crit(lambda: "used" > 85.0)
// Send alert to hander of choice.
// Slack
.slack()
.channel('#alerts')
// PagerDuty
.pagerDuty()
34
Demohttps://github.com/gianarb/tick-php
35
When you start to work with "micro"services understand the topology of your connections is really important Time series can help you
36
Use a dashboard to put together different metrics and create not obvious relations
37
Why InfluxDB and not something else?https://www.influxdata.com/influxdb-is-27x-faster-vs-mongodb-for-time-series-workloads/
27x greater write throughput
84x less disk space
38
That’s it!
A series of great tools to monitor your applications and your infrastructure
39
A monitoring system isn’t for all
Thanks!
See you around
twitter.com/gianarb
40