building a scalable monitoring system · inconsistent alerts reported data required no action 27....

@molly_struve

Building A Scalable Monitoring System

�1

@molly_struve

Monitoring Mistakes Overhauling the System The Payoff

�2

@molly_struve

Monitoring Mistakes

�3

Overhauling the System The Payoff

@molly_struve

�4

@molly_struve

Monitoring Mistakes

�5

Overhauling the System The Payoff

@molly_struve

Monitoring Mistakes Overhauling the System

�6

Overhauling the System The Payoff

@molly_struve

Monitoring Mistakes Overhauling the System The Payoff

�7

@molly_struve

Monitoring Mistakes Overhauling the System The Payoff

�8

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�9

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�10

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�11

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�12

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�13

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�14

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�15

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�16

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�17

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�18

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�19

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�20

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�21

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�22

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�23

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�24

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�25

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

Incredibly Inconsistent

�26

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

Inconsistent Alerts

Required no actionReported data

�27

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

Inconsistent Alerts

Required no actionReported data

Immediate action required

�28

@molly_struve

make on-call devs miserable

Monitoring Mistakes Overhauling the System The Payoff

�29

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�30

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�31

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�32

@molly_struve

Monitoring Mistakes Overhauling the System The Payoff

�33

@molly_struve

Monitoring Mistakes Overhauling the System The Payoff

�34

@molly_struve

Monitoring Must Haves

1

2

3

4

Monitoring Mistakes Overhauling the System The Payoff

Consolidate Monitoring To a Single Place

�35

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

!" #$

%

�36

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�37

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�38

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�39

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�40

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�41

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�42

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�43

@molly_struve

Make ALL Alerts Actionable

Monitoring Must Haves

1

2

3

4

Monitoring Mistakes Overhauling the System The Payoff

Consolidate Monitoring To a Single Place

�44

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�45

Action

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�46

Action

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

Action Required No action Needed

�47

@molly_struve

#ops_alerts

Monitoring Mistakes Overhauling the System The Payoff

#dev_alerts

Action Required No action Needed

�48

@molly_struve

#ops_alerts

Monitoring Mistakes Overhauling the System The Payoff

#ops_reporting#dev_alerts

Action Required No action Needed

#dev_reporting

�49

@molly_struve

Make Alerts MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place

Monitoring Must Haves

1

2

3

4

Monitoring Mistakes Overhauling the System The Payoff

�50

@molly_struve

Make Alerts MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place

Monitoring Must Haves

1

2

3

4

Monitoring Mistakes Overhauling the System The Payoff

�51

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

30 60 90minutes

�52

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�53

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�54

Miss new alerts

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�55

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�56

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�57

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�58

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�59

@molly_struve

Track Alert HistoryMake Sure Alerts Are MutableMake ALL Alerts Actionable

Monitoring Must Haves

1

2

3

4

Monitoring Mistakes Overhauling the System The Payoff

Consolidate Monitoring To a Single Place

�60

@molly_struve

Tracking Alert History

Monitoring Mistakes Overhauling the System The Payoff

�61

@molly_struve

Tracking Alert History

Monitoring Mistakes Overhauling the System The Payoff

�62

@molly_struve

Tracking Alert History

Monitoring Mistakes Overhauling the System The Payoff

�63

@molly_struve

Track Alert HistoryMake Sure Alerts Are MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place

Monitoring Must Haves

1

2

3

4

Monitoring Mistakes Overhauling the System The Payoff

�64

@molly_struve

Monitoring Mistakes Overhauling the System The Payoff

�65

@molly_struve

Monitoring Mistakes Overhauling the System The Payoff

�66

@molly_struve

On boarding is a breeze

Monitoring Mistakes Overhauling the System The Payoff

�67

@molly_struve

3 On-boarding steps:

1

2

3

Monitoring Mistakes Overhauling the System The Payoff

�68

@molly_struve

Show them the monitoring setup1

2

3

Monitoring Mistakes Overhauling the System The Payoff

3 On-boarding steps:

�69

@molly_struve

Show them the monitoring setup1

2

3

Monitoring Mistakes Overhauling the System The Payoff

If an alert goes off you have to address it

3 On-boarding steps:

�70

@molly_struve

Show them the monitoring setup1

2

3

Monitoring Mistakes Overhauling the System The Payoff

How to mute a triggered alert

3 On-boarding steps:

�71

If an alert goes off you have to address it

@molly_struve

On boarding is a breeze

Monitoring Mistakes Overhauling the System The Payoff

�72

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

Happier on-call developers

�73

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

All alerts must be actionable

�74

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�75

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

No more noise

�76

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�77

@molly_struve

Developers began helping to improve our monitoring system

Monitoring Mistakes Overhauling the System The Payoff

�78

@molly_struve

30-40 Alerts

Monitoring Mistakes Overhauling the System The Payoff

�79

@molly_struve

>90 Alerts

Monitoring Mistakes Overhauling the System The Payoff

�80

@molly_struveMonitoring Mistakes Overhauling the System The Payoff

�81

@molly_struve

If you want…

1

2

3

Monitoring Mistakes Overhauling the System The Payoff

�82

@molly_struve

On boarding to be a breeze.1

2

3

Monitoring Mistakes Overhauling the System The Payoff

If you want…

�83

@molly_struve

On-call developers to be a lot happier

1

2

3

Monitoring Mistakes Overhauling the System The Payoff

If you want…

On boarding to be a breeze.

�84

@molly_struve

Developers to help improve your monitoring systems

1

2

3

Monitoring Mistakes Overhauling the System The Payoff

If you want…

On boarding to be a breeze.

�85

On-call developers to be a lot happier

@molly_struve

Track Alert HistoryMake Sure Alerts Are MutableMake ALL Alerts ActionableConsolidate Monitoring To a Single Place

Monitoring Must Haves

1

2

3

4

�86

@molly_struve

❤❤❤

�87

@molly_struve

Questions?

�88