devops - yow! conference...event processing in clojure andy marks @andee_marks...

FUNCTIONALDEVOPSEVENT PROCESSING IN CLOJURE

Andy Marks @andee_marks

amarks@thoughtworks.com

Background

we all code!

(prn “Hello World”)

(ns hello.core)

(defn -main [] (prn “Hello World”))

Jetty

(ns hello.core)

(defn -main [] (prn “Hello World”))

✓access requests ✓queues ✓sessions

AWS EC2

Jetty

(ns hello.core)

(defn -main [] (prn “Hello World”))

✓disk ✓network interfaces ✓memory ✓cores/cpus

AWS EC2

Jetty

AWS EC2

Jetty

AWS EC2

Jetty

DNS (Master)

DNS (Slave)

✓records ✓replication ✓resolution

AWS EC2

Jetty

AWS EC2

Jetty

AWS EC2

Jetty

DNS (Master)

DNS (Slave)

DB

✓performance ✓indexing ✓locking ✓connections

AWS EC2

Jetty

AWS EC2

Jetty

AWS EC2

Jetty

DNS (Master)

DNS (Slave)

DB

Load Balancer

AWS EC2

Jetty

AWS EC2

Jetty

✓connections ✓DoS ✓Clients

AWS EC2

Jetty

AWS EC2

Jetty

AWS EC2

Jetty

DNS (Master)

DNS (Slave)

DB

Load Balancer

AWS EC2

Jetty

AWS EC2

Jetty

Message Q

ueue

✓queues ✓timeouts

that is a lot of information…

from a lot of different places

this is a problem

this is the problem Riemann helps

solve

Riemann is written in

Clojure

52 source files7256 lines

377 functions30 macros

Riemann is written and configured

in Clojure

And so… to Riemann

kyl

e ki

ngsb

ury (

@ap

hyr)

OVERVIEW

Correlate events from disparate sources

What is the latency of the web servers when the DB servers are generating timeouts?

Filter events based on key criteria Ignore all network events with a normal status

Aggregate events to perform statistical analysis (e.g., percentiles) What is the average response rate for 95% of requests?

Monitor status of sources by availability of events

Which services haven’t sent any events for the last 5 minutes?

Route event streams to multiple consumers

Send all error events from DB servers to the DBA team as well as the main operational dashboards

SAMPLE RIEMANN USE CASES

i = f(e)e: thundering herd of events i: information ready for downstream processing

Hypothesis

Event processing is function application at large scale… using

Clojure to build/configure Riemann is a logical choice.

(defrecord Event [ service state description metric tags time ttl

KEY CONCEPTS: EVENTS

“reg1.foo.bar” “reg-service http” “ok” “apache-log” 201 [“reg1” “http”] 104534892 60

riem

ann

sour

ce

KEY CONCEPTS: STREAMS

(logging/init {:file "riemann.log" :console true})

(instrumentation {:interval 5}) ;; self monitor every 5 seconds (periodically-expire 1) ;; check for expired events every second

(tcp-server) (repl-server)

(streams ;; insert magic here!!! prn)

riem

ann

confi

g

KEY CONCEPTS: INDEX

flow of events

index

riemann process

General FP goodness

HOMOICONICITY

“[It] is where a program's source code is written as a basic data structure that the

programming language knows how to access.” Wikipedia

(defn validate-config [file]

(try

(read-strings (slurp file)) (catch clojure.lang.LispReader$ReaderException e (throw (logging/nice-syntax-error e file))))) ri

eman

n so

urce

HIGHER ORDER FUNCTIONS

“A stream is just a function that takes a variable number of child streams and returns a function

that takes an event and responds to the event it is passed when it is invoked.”

http://riemann.io/quickstart.html

“a higher-order function… does at least one of the following:

[1] takes one or more functions as an input,

[2] outputs a function” Wikipedia

HIGHER ORDER FUNCTIONS

“A stream is just a function that takes a variable number of child streams and returns a function

that takes an event and responds to the event it is passed when it is invoked.”

http://riemann.io/quickstart.html

(streams

(rate 5 prn) )

riem

ann

confi

g

IMMUTABILITY

“Streams ‘change’ events by sending altered, shared-structure *copies* of events

downstream.” http://riemann.io/howto.html#split-streams

(streams

(where (host "db04") (with :service "foo" prn) (with :service "bar" prn)) )

riem

ann

confi

g

Specific Clojure goodness

“clojure's remarkably fast for what it is, sexprs make the tree structure of the streams visually apparent, it makes writing the concurrent algos much simpler, and macros allow us to express things like 'by and 'pipe in ways that would be awkward in other languages without building our own AST & interpreter or transpiler, etc.”

“clojure's remarkably fast for what it is, sexprs make the tree structure of the streams visually apparent, it makes writing the concurrent algos much simpler, and macros allow us to express things like 'by and 'pipe in ways that would be awkward in other languages without building our own AST & interpreter or transpiler, etc.”

@APHYR SAID…

S-EXPRESSIONS

“a notation for nested list (tree-structured) data… popularised by the programming

language Lisp” Wikipedia

S-EXPRESSIONS

(streams

(where (and (service #"^riak") (state “critical”)) (email “delacroix@vonbraun.com")) )

1 Filter certain events…

2 Let people know…

1

2

riem

ann

confi

g

“make the tree structure of the streams visually apparent”

3

1

2

S-EXPRESSIONS

1 Split on event fields…

2 Detect state changes…

3 Collate and email(streams

(by [:host :service] (changed :state (rollup 5 3600 (email “delacroix@vonbraun.com")))) ) ri

eman

n co

nfig

“make the tree structure of the streams visually apparent”

MACROS

(defmacro pipe [marker & stages] `(let [~@(->> stages reverse (interpose marker) (cons marker))] ~marker))

(streams

(pipe - (splitp = service "http req rate" - "0mq req rate" (scale 2 -)) (rate 5

(graphite {:host “127.0.0.1” :port 2003}))) )

riem

ann

confi

g

riem

ann

sour

ce

MACROS: PIPE

(streams

(let [aggregate (rate 5 (graphite {:host “127.0.0.1” :port 2003}))] (splitp = service "http req rate" aggregate "0mq req rate" (scale 2 aggregate))) )

or

riem

ann

confi

gri

eman

n co

nfig

(streams

(pipe - (splitp = service "http req rate" - "0mq req rate" (scale 2 -)) (rate 5

(graphite {:host “127.0.0.1” :port 2003}))) )

Conclusion

Hypothesis

Event processing is function application at large scale: using Clojure to build/configure Riemann is a logical

choice.

homoiconicityhigher order functions

s-expressionsmacros

immutability

CLOJURE AD ABSURDUM

(defn do-you-trust-me? [event] (prn (load-string (:description event))))

(streams

(where (service "event-gen") do-you-trust-me?) )

(ns event-gen.core (:require [riemann.client :refer :all]) (:gen-class :main true))

(defn -main [& args] (let [c (tcp-client {:host "127.0.0.1"})]

(send-event c {:service "event-gen" :description "(+ 1 1)"}) (close-client c)))

riem

ann

clie

ntri

eman

n co

nfig

CLOJURE AD ABSURDUM

(defrecord Event [service description ])

“event-gen” “(+ 1 1)”

Riemann (Clojure)

riemann.config (Clojure)

Client (Clojure)

Clojure

=> “2”

THANKS

Andy Marks @andee_marks

amarks@thoughtworks.com