building a scalable monitoring system · inconsistent alerts reported data required no action 27....

@molly_struve

Building A Scalable Monitoring System

�1

@molly_struve

Monitoring Mistakes Overhauling the System The Payoff

�2

@molly_struve

Monitoring Mistakes

�3

Overhauling the System The Payoff

@molly_struve

�4

@molly_struve

Monitoring Mistakes

�5


@molly_struve

Monitoring Mistakes Overhauling the System

�6


@molly_struve


�7

@molly_struve


�8

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�9


�10


�11


�12


�13


�14


�15


�16


�17


�18


�19


�20


�21


�22


�23


�24


�25


Incredibly Inconsistent

�26


Inconsistent Alerts

Required no actionReported data

�27


Inconsistent Alerts

Required no actionReported data

Immediate action required

�28

@molly_struve

make on-call devs miserable


�29


�30


�31


�32

@molly_struve


�33

@molly_struve


�34

@molly_struve

Monitoring Must Haves

1

2

3

4


Consolidate Monitoring To a Single Place

�35


!" #$

%

�36


�37


�38


�39


�40


�41


�42


�43

@molly_struve

Make ALL Alerts Actionable


1

2

3

4



�44


�45

Action


�46

Action


Action Required No action Needed

�47

@molly_struve

#ops_alerts


#dev_alerts


�48

@molly_struve

#ops_alerts


#ops_reporting#dev_alerts


#dev_reporting

�49

@molly_struve

Make Alerts MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place


1

2

3

4


�50

@molly_struve

Make Alerts MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place


1

2

3

4


�51


30 60 90minutes

�52


�53


�54

Miss new alerts


�55


�56


�57


�58


�59

@molly_struve

Track Alert HistoryMake Sure Alerts Are MutableMake ALL Alerts Actionable


1

2

3

4



�60

@molly_struve

Tracking Alert History


�61

@molly_struve



�62

@molly_struve



�63

@molly_struve

Track Alert HistoryMake Sure Alerts Are MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place


1

2

3

4


�64

@molly_struve


�65

@molly_struve


�66

@molly_struve

On boarding is a breeze


�67

@molly_struve

3 On-boarding steps:

1

2

3


�68

@molly_struve

Show them the monitoring setup1

2

3



�69

@molly_struve


2

3


If an alert goes off you have to address it


�70

@molly_struve


2

3


How to mute a triggered alert


�71

If an alert goes off you have to address it

@molly_struve

On boarding is a breeze


�72


Happier on-call developers

�73


All alerts must be actionable

�74


�75


No more noise

�76


�77

@molly_struve

Developers began helping to improve our monitoring system


�78

@molly_struve

30-40 Alerts


�79

@molly_struve

>90 Alerts


�80


�81

@molly_struve

If you want…

1

2

3


�82

@molly_struve

On boarding to be a breeze.1

2

3


If you want…

�83

@molly_struve

On-call developers to be a lot happier

1

2

3


If you want…

On boarding to be a breeze.

�84

@molly_struve

Developers to help improve your monitoring systems

1

2

3


If you want…

On boarding to be a breeze.

�85

On-call developers to be a lot happier

@molly_struve

Track Alert HistoryMake Sure Alerts Are MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place


1

2

3

4

�86

@molly_struve

❤❤❤

�87

@molly_struve

Questions?

�88

building a scalable monitoring system · inconsistent alerts reported data required no action 27....

Documents