confidential and not for reproduction without prior written consent. © of the sporting exchange...

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

SHOW ME THE METRICS

James Brooks

Born in 1999UK, Romania, Portugal, Ireland, Malta, Gibraltar, USAEngineering Blog: www.betsandbits.com700+ Engineers and Growing

ExchangeSportsbookGames

~1M active users~100M daily transactions~2.5Bn daily API calls~100k/s monitoring pointsE2E P95 transaction times <4ms

One of UK’s most successful .coms with ~£2Bn Market Cap2

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

OpenTSDB & OpenTSP at Betfair

3

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

4

What is OpenTSDB?

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

5

What is OpenTSDB?

A database optimised for storing and querying Time Series Metrics

os.filesystem.size

1413978032

912234340host=fe-108 cluster=bf_c mount=/

type=Free

Name Timestamp

Value

metadata

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

6

OpenTSDB Architecture

Hadoop

•TSDB Read•TSDB Write

Frontend Server

•TSDB Read•TSDB Write

Frontend Server

•TSDB Read•TSDB Write

Frontend Server

Load B

ala

nce

r

Metrics

Queries

TSDB

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

7

OpenTSDB Architecture

Metrics

Queries

Original

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

8

OpenTSDB Architecture (ctd)

TSDB

Source :http://mo.nitor.me/hadoop-hbase-opentsdb/

Original

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

10

OpenTSDB Architecture (ctd)

MetrilyxTSDB

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

11

Original

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

12

OpenTSDB Architecture (ctd)

MetrilyxTSDB Visualiser

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

13

tcollector

collect-os

collect-jmx

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

14

OpenTSDB Architecture (ctd)

TSDBtcollector

collect-os

collect-jmx

tcollector

collect-os

collect-jmx

• TSDB Use Cases

• Troubleshooting / Root Cause analysis

• Performance Analysis

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

15

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

16

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

17

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

18

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

19

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

20

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

21

• TSDB Use Cases

• Troubleshooting / Root Cause analysis

• Performance Analysis

• Monitoring and Alerting

• Limitations

• Load

• Recency

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

22

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

23

• TSDB Use Cases

• Troubleshooting / Root Cause analysis

• Performance Analysis

• Monitoring and Alerting

• Limitations

• Load

• Recency

• TSP

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

24

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

25

OpenTSP

tcollector

collect-os

collect-jmx

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

26

OpenTSP

TSDBtcollector

collect-os

collect-jmx

forwarder

collect-os

collect-jmx

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

27

OpenTSP Architecture: Forwarder

TAG

FILTERcollect-statse

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

28

OpenTSP Architecture: Statse

application

collect-statse

countmeanmedia

np95p99

forwarder

collect-statse

countmeanmedia

np95p99

forwarder

application

collect-statse

countmeanmedia

np95p99

forwarder

application

poller

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

29

OpenTSP Architecture: Poller

collect-snmp

collect-f5collect-

netscaler

switchesroutersfirewalls

load balancersstorage

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

30

OpenTSP Architecture: Aggregator

forwarder

forwarder

poller

aggregator

collect-site

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

31

OpenTSP Architecture: Controller

controller

collectors

aggregator

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

32

[james@ohlookahostname~]$ CONTROL_HOST=tspconfig.inf.betfair tsp-forwarder -t{ "Filter": [ { "Match": [ "", "host", "^$" ], "Set": [ "", "host", “ohlookahostname.inf.betfair" ] } ], "LogPath": "/var/log/tsp/forwarder.log", "Relay": { "aggregator": { "DropRepeats": false, "Host": “tspaggregator.inf.betfair", "MaxConnsPerHost": 1, "OnQueueFull": "Drop" }, "tsd": { "DropRepeats": true, "Host": "tsdb.inf.betfair", "MaxConnsPerHost": 1, "OnQueueFull": "Drop" } }, "CollectPath": "/etc/tsp/collect.d"}

Visualiser

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

33

TSP at Betfaircon

troller collectors TSDB

Visualiser

tsdb2mysql

mysqld

feedhealth

Riemann

future?

aggregator

aggregator

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

34

Open Source• tsp-forwarder

• collect-statse• tsp-poller

• collect-f5• collect-netscaler• collect-snmp

• tsp-aggregator• tsp-controller

Apache Licence v2

http://github.com/betfair/opentsp

james.brooks@betfair.com

CONFIDENTIAL and not for reproduction without prior written consent. © of The Sporting Exchange Limited.

35