ali asad lotia (devops at beamly) - riemann stream processing at #doxlon
TRANSCRIPT
Metrics Insight with Riemann!Ali Asad Lotia | [email protected] | @aalotia !!
Clarification!
• Riemann usage at Beamly is WIP
Platform visibility• Dashboards – Plots
– Summary values (min, max, mean)
• Alerts
Dashboard Limitations• Dashboard overload
• Hard to be proactive
• Mostly reactive
Alerting challenges• Service interdependencies
• Multiple services + multiple metrics
• Alerting before outage
If only…• A tool to gain insight from the constant
stream of metrics
• Analyse incoming metrics in near real time
Well, there is…• http://riemann.io/
• Started by Aphyr (Kyle Kingsbury)
• Named for Bernhard Riemann – http://en.wikipedia.org/wiki/Bernhard_Riemann
Requirements• Java 7 or greater
• Sufficient CPU and RAM
• Curiosity
• Patience
Concepts• Events
• Streams
• Index
• Event manipulation (via functions)
EventStruct/Dictionary/Map
{:host "foo"! :service "My test service"! :state "good"! :time 1414687710! :description "A test that I am running"! :tags ["foo", "bar"]! :metric 1.0! :ttl 900}
Configuration• Written in clojure – Pro: Power of a programming language
– Con: Power of a programming language
• Updatable on the fly
Beamly metrics flow
Sensu
Server
Server
Server
Server
Graphite
Riemann
Accepting graphite metrics!(let [host "0.0.0.0"! port 2003]! (graphite-server {:host host! :port port})! (tcp-server {:host host})! (udp-server {:host host! :max-size 65000})! (repl-server {:host "127.0.0.1"})! (ws-server {:host host}))!
Transforming graphite metrics!!(defn process-graphite-metric-name! "Split the graphite metric name and set the event map values! accordingly. Currently a somewhat naive function since it assumes! the format of the graphite metric key."! [{:keys [service metric time]}]! (let [[component source environment region aws-host metric-details]! (cs/split service #"\." 6)! host-ip! (extract-host-ip-from-aws-hostname aws-host)! new-service! (cs/join "." (vector component source metric-details))]! {:service new-service! :environment environment! :region region! :host host-ip! :component component! :metric (or metric 0)! :time time! :state "ok"! :description service! :tags ["from-sensu" component]}))!
Indexing select events(let [index (default :ttl 900 (index))]! (streams! (where (= (:component event) "comp-a")! #(info (format "received event from component %s" (:component %)) %)! index)! (where (= (:component event) "comp-b")! (with :ttl 300 index! #(info (format "received event from component %s" (:component %)) %)))! (where (= (:component event) "comp-c")! (with :ttl 300 index! #(info (format "received event from component %s" (:component %)) %)))! (where (and (not (nil? service)) (service #"\.sys\.swapused$") (> metric 0))! (with :state "swapping"! #(info (format "swap used on component %s" (:component %)) %)! index))))!
Output• Email
• Pagerduty
• Graphite, InfluxDB
• custom plugin
Riemann Dashboardhttp://riemann.io/dashboard.html !Can query riemann index
Gotchas• Stateful
• Dashboard docs sparse
• Scaling possible by cascading to other Riemann instances
Resources• Website: http://riemann.io
• IRC: #riemann on freenode
• Riemann Users Google Group https://groups.google.com/forum/#!forum/riemann-users
• Guardian riemann-config repo on Github https://github.com/guardian/riemann-config
LONDON !Drury House
34-43 Russell Street London
WC2B 5HA
NEW YORK CITY !84 Wooster Street
Suite 703 New York NY 10012
SYDNEY !22-36 Mountain Street
Suite 1.10 Ultimo
NSW 2007
© Beamly 2014. All Rights Reserved.