opentsdbopentsdb.net/misc/opentsdb-strata.pdf · opentsdb a distributed, scalable, time series...
TRANSCRIPT
![Page 1: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/1.jpg)
OpenTSDBA Distributed, Scalable, Time Series Database
“Monitoring at an unprecedented level of granularity”
Benoît “tsuna” [email protected]
![Page 2: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/2.jpg)
Old Plow
Where’s my Paradigm
Shift?
![Page 3: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/3.jpg)
Get Real-Time Data from your Infrastructure
Backbone #2 by AndiH
![Page 4: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/4.jpg)
Working at Scale
Finding needle in haystack by Bindaas Madhavi
![Page 5: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/5.jpg)
No SPoF
![Page 6: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/6.jpg)
HBase
Bright Ideas by purplemattfish
Distributed Scalable
Reliable Efficient
![Page 7: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/7.jpg)
Design Goals
• Distributed storage of monitoring data
• No Single Point of Failure
• Pulling custom graphs must be trivial & fast
• Scale to:
• Thousands of machines
• Many billions of data points
![Page 8: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/8.jpg)
Key concepts
• Data Points(time, value)
• Metricsproc.loadavg.1m
• Tagshost=web42 pool=static
• Metric + Tags = Time Series
put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
![Page 9: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/9.jpg)
Key concepts
• Data Points(time, value)
• Metricsproc.loadavg.1m
• Tagshost=web42 pool=static
• Metric + Tags = Time Series
put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
![Page 10: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/10.jpg)
Key concepts
• Data Points(time, value)
• Metricsproc.loadavg.1m
• Tagshost=web42 pool=static
• Metric + Tags = Time Series
put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
![Page 11: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/11.jpg)
Key concepts
• Data Points(time, value)
• Metricsproc.loadavg.1m
• Tagshost=web42 pool=static
• Metric + Tags = Time Series
put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
![Page 12: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/12.jpg)
The Big Picture™
Applications
Metrics
tcollectorPeriodic
polling
TSD TSD TSD
HBase
Push data points(telnet-style proto, RPC soon)
Put / Scan(Hadoop RPC)
Browser
Graph request(HTTP)
![Page 13: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/13.jpg)
12 Bytes Per Datapoint
4TB per year for 1000 machines
![Page 14: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/14.jpg)
OpenTSDB @
150 Million Datapoints/Dayin a typical datacenter
![Page 15: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/15.jpg)
Demo Time!
![Page 16: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/16.jpg)
Set it up in 15 minutes
• JDK + Gnuplot 1 minute (1 command)
• Single-node HBase 4 minutes (3 commands)
• OpenTSDB 5 minutes (5 commands)
• Deploy tcollector 5 minutes
With zero prior experience
![Page 17: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/17.jpg)
Under the Hood
Nettysu async
hbase asyncTSD
core
Local Disk(cache)
![Page 18: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/18.jpg)
Under the Hood
Nettysu async
hbase asyncTSD
core
Local Disk(cache)
Write Path
put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
HBase1s delay
max.
>2000 data points / sec / core
![Page 19: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/19.jpg)
Under the Hood
Nettysu async
hbase asyncTSD
core
Local Disk(cache)
Write Path
put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
HBase
>2000 data points / sec / core
![Page 20: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/20.jpg)
Under the Hood
Nettysu async
hbase asyncTSD
core
Local Disk(cache)
HBase
Read Path
scan
GET /q?...
![Page 21: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/21.jpg)
Under the Hood
Nettysu async
hbase asyncTSD
core
Local Disk(cache)
HBase
Read Path
scan
write to cache
GET /q?...
Gnuplot
![Page 22: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/22.jpg)
100% Natural, Organic Free & Open-Source
Danger in the Corn by Roger Smith
![Page 24: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/24.jpg)
Inside HBase
put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
} } }0 5 2 0 0 1 0 2 8} }0 4 7 0 0 1
Row KeyColumn Family: nameColumn Family: nameColumn Family: name Column Family: idColumn Family: idColumn Family: id
Row Keymetrics tagk tagv metrics tagk tagv
host static
proc.loadavg.1m
host
proc.loadavg.1m 0 5 2
0 5 2
0 0 1
0 0 1
Table:tsdb-uid
![Page 25: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/25.jpg)
Inside HBase
put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
Row KeyColumn Family: tColumn Family: tColumn Family: tColumn Family: tColumn Family: tColumn Family: t
Row Key+0 +15 +20 ... +90 +600
0.69 0.51 0.42
0.99 0.72
} } } }0 5 2 0 0 1 0 2 8} }0 4 7 0 0 1=1234567800+90
}73 -106 2 120
Table:tsdb
![Page 26: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/26.jpg)
Inside HBase
put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
Row KeyColumn Family: tColumn Family: tColumn Family: tColumn Family: tColumn Family: tColumn Family: t
Row Key+0 +15 +20 ... +90 +600
0.69 0.51 0.42
0.99 0.72
} } } }0 5 2 0 0 1 0 2 8} }0 4 7 0 0 1=1234567800+90
}73 -106 2 120
Table:tsdb
![Page 27: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/27.jpg)
Inside HBase
put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
Row KeyColumn Family: tColumn Family: tColumn Family: tColumn Family: tColumn Family: tColumn Family: t
Row Key+0 +15 +20 ... +90 +600
0.69 0.51 0.42
0.99 0.72
} } } }0 5 2 0 0 1 0 2 8} }0 4 7 0 0 1=1234567800+90
}73 -106 2 120
Table:tsdb
![Page 28: OpenTSDBopentsdb.net/misc/opentsdb-strata.pdf · OpenTSDB A Distributed, Scalable, Time Series Database “Monitoring at an unprecedented level of granularity” Benoît “tsuna”](https://reader031.vdocuments.us/reader031/viewer/2022030414/5a9f97717f8b9a7f178d031f/html5/thumbnails/28.jpg)
Inside HBase
put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
Row KeyColumn Family: tColumn Family: tColumn Family: tColumn Family: tColumn Family: tColumn Family: t
Row Key+0 +15 +20 ... +90 +600
0.69 0.51 0.42
0.99 0.72
} } } }0 5 2 0 0 1 0 2 8} }0 4 7 0 0 1=1234567800+90
}73 -106 2 120
Table:tsdb