stream processing inside librato [monitorama pdx 2015]

hi.

hi.dave@librato.com@davejosephsen

github: djosephsen

Graphing Nagios

hi.dave@librato.com@davejosephsen

github: djosephsen

github: djosephsen

Who put Stream Processing in my Monitoring Conference?

github: djosephsen

Graphing NagiosLet’s talk about signal processing

github: djosephsen

Graphing NagiosLets talk about signal processing

RealTime Audio Processing

github: djosephsen

Graphing NagiosLets talk about signal processing

Graphing NagiosLets talk about signal processing

github: djosephsen

Graphing NagiosLets talk about signal processing

github: djosephsen

Graphing NagiosLets talk about signal processing

github: djosephsen

Graphing NagiosLets talk about signal processing

github: djosephsen

Graphing NagiosLets talk about signal processing

github: djosephsen

Lets talk about signal processing

WhisperDB

WhisperDB

WhisperDB

WhisperDB

github: djosephsen

Lets talk about signal processing

github: djosephsen

Graphing NagiosLets talk about signal processing

github: djosephsen

Graphing NagiosLets talk about signal processing

github: djosephsen

github: djosephsen

Nagios

Zabbix

Collectd

DAC (transform and normalize the signal)

(e.g., translate input to a standard json blob)

github: djosephsen

Zabbix

Filters (de-duplication, noise reduction)

(e.g., compute percentiles to filter outliers)

github: djosephsen

Multiplex (aggregation)

(e.g., aggregate http 200’s from multiple hosts)

github: djosephsen

Effects (ad-hoc external signal processing)

??

github: djosephsen

Effects (ad hoc external signal processing)

??

Say “Exponential Smoothing”

again

Disk Latency

Queue Latency

Model A

Model B

ing

MultiVariate Data Science Menu

Specials Gaussian

Fuzzy

Discrete

Hybridized

SmoothingExponentialSmoothingHolt-Winters

Chicken Fried Exponential SmoothingNaieve Baysian

linear regression

Markov Chains

extreme studentized

Kung-Pow distance vector groupingstream decompisiton

Organic, free-range Ansecomb Quartet

really really smooth Exponential Smoothingxtra seasonality available on request

Downsampling

Statistical Convergence

Lacunary Sequence

Hashsum Distance

Lambda Convergence

Lambda ANYTHINGSeasonal Hybridized Whatever

HALF OFF!

Anything Named after a turn-of-the-century Member of the Russian Intelligencia

Spacial Forecasting

deviant TestQuantile isolation

Probablistic merging

Hybridized/Fuzzy Combo platter!

Collaborative n-gram cosine matching

Robust Sequential Map-Reduce

Micro-lot N-variate chi-square

Ask about our locally-sourced quantile robustness

Roasted Daily!

Scientist gonna science

Scientist gonna scienceEngineer gonna.. yeah.

Scientist gonna scienceData Engineer gonna ??

Making a model is just the beginning

Nagios

GraphiosNagios

Graphios StatsdNagios

Graphios Statsd GraphiteNagios

Graphios Statsd Graphite Grafana

Graphios Statsd Graphite Grafana

Math thingy ???

github: djosephsen

Effects (ad hoc external signal processing)

??

Stuff Statsd Graphite Grafana

Stuff Graphite GrafanaHeka

Stuff Graphite GrafanaHeka

Model A

Model B

Stuff Graphite GrafanaHeka

Model A

Model Bcarbon relay

carbon relay

carbon relay

carbon relay

carbon relay

carbon relay

carbon relay

carbon relay

Stuff Graphite GrafanaHeka

Model A

Model Bcarbon relay

carbon relay

carbon relay

carbon relay

carbon relay

carbon relay

carbon relay

carbon relay

Elastisearch

Elastisearch

Data in Truth out

X

XXX

XXX X X

XX

Kafka

SSA

Kafka

SSA

Kafka

SSAr60

Kafka

SSAr60

r900

Kafka

SSA

SSA

r60

Raw Writer

SSA

r60

Raw Writer

r900 r3600

SSA

r60

Raw Writer

Alerts

SSA

r60

Raw Writer

Alerts

SSA

MySQL

SSA

Alerts

MySQL

SSA

Alerts

MySQL

SSA

Alerts

MySQL

{ {

SSA

Alerts

MySQL

Storm is “pretty OK”

Lacks visibility into flow, performance, and failures

Netty? JVM Hops? Ser/Der?

Samza? Spark?

NIH? ( kafka + zk + mezos + disruptor+? )

Questions?