Is this normal?Finding anomalies in real-time data.

Who am I?

I’m Theo (@postwait on Twitter)I write a lot of code

50+ open source projectsseveral commercial code bases

I wrote “Scalable Internet Architectures”I sit on the ACM Queue and Professions boards.I spend all day looking at telemetry data at Circonus

What is real-time?Hard real-time systems are those where the outputs of a system based on specific inputs are considered incorrect if the latency of their delivery is above a specified amount.

Soft real-time systems are similar,but “less useful” instead of “incorrect.”

I don’t design life support systems, avionicsor other systems where lives are at stake,so it’s a soft real-time life for me.

A survey of big data sytems.

Traditional:

Oracle, Postgres, MySQL, Teradata,Vertica, Netezza, Greenplum, Tableau, K

The shiny:

Hadoop, Hive, HBase, Pig, Cassandra

The real-time:

SQLstream, S4, Flumebase, Truviso, Esper, Storm

Big data the old way

Relational databases, both column store and not.

Just work.

Likely store more data than your “big data.”

Big data the distributed way

distributed systems allow much larger data sets, but

markedly change the data analytics methods

hard for existing quants to roll up their sleeves

highly scalable and accommodate growth

Big data the real-time way

what we do needs a different approach

the old (and even the distributed)

do not design for soft real-time complex observation of data.

Notable exceptions are S4 and Storm.

So, what’s your problem?

We have telemetry...

over 10 trillion data points on near-line storage

growing super-linearly

Data, what kind?

Most data is numeric:

counts, averages, derivatives, stddevs, etc.

Some data is:

text changes (ssh fingerprints, production launches)

histograms

highly dimensional event streams.

Data rates.

Quantity of data isn’t such a big deal

okay, yes it is, but we’ll get to that later.

The rate of new data arrival makes the problem hard.

low end: 15k datum / second

high end: 300k datum / second

growing rapidly

What we use.We use Esper

Esper is very powerful,elegantly coded and performance focused

Like any good toolthat allows users towrite queries...

http://www.flickr.com/photos/mcertou/

What we do with Esper

Detect absence in streams:select b from pattern[every a=Event -> (timer:interval(30 sec) and not b=Event(id=a.id, metric=a.metric)]

Detect ad-hoc threshold violation:select * from Event(id=”host1”, metric=”disk1”)where value > 95

etc. etc. etc. [1]

Making the problem harder.

So, it just wasn’t enough.

We want to do long term trendingand apply that information to anomaly detection

Think: Holt-Winters (or multivariate regressions)

Look at historic data

Use that to predict the immediate futurewith some quantifiable confidence.

How we do it.

We implemented the Snowth for storage of data. [2]

We implemented a C/lua distributed system to analyze4 weeks of data (~8k statistical aggregates)yielding a prediction with confidences(triple exponential smoothing) [3]

To keep the system real-time,we need to ensure that queries return inless than 2ms (our goal is 100µs).

Cheating is winning.

Our predictions work on 5 minute windows.

4 weeks of data is 8064 windows.

Given Pred(T-8063 .. T0) -> (P1, C1)

Given Pred(T-8062 .. T0, P1) -> ~(P2, C2)

Tolerably inaccurate.

When V arrives,we determine the prediction window WN we need.

If WN isn’t in cache, we assume V is within tolerances.

If WN+1 isn’t in cache,we query the Snowth for WN, WN+1placing in cache

Cache accesses are local and always < 100µs.

I see challenges

How do I

take offline data analytics techniques andapply them online to high-volume, low-latencyevent streams

quickly?

without deep expertise?

Thank you.Circonus is hiring:

software engineers,quants, andvisualization engineers.

[1] http://esper.codehaus.org/tutorials/solution_patterns/solution_patterns.html

[2] http://omniti.com/surge/2011/speakers/theo-schlossnagle

[3] http://labs.omniti.com/people/jesus/papers/holtwinters.pdf