How to make your monitoring shut up

Observations on what to monitor and when to alert

Hello!

@moegyver@EttLejon(tho I hardly use twitter…)(easier to find me as lejonet @Freenode/OFTC IRC)

What is your name?Ops/DevOps

What is your quest?A healthy and functioning server

environment

What is… your favourite time to be woken up by an alert?Erh.. Never.. Erh office hou

aaaaaaaaaah!

Example Checklist for monitoring vs alerting

1. What am I actually monitoring, service or details of part of a service (hardware, connectivity, etc)?

If the answer isn’t a service, then it might not be fit for alerting

2. Will I get fired for not responding directly to this? (aka is it business critical)

If the answer to that is yes, maybe its fit for alerting

3. Is what I want to check possibly covered by another alert or monitoring check? E.g.

connectivity is also answered by a check for if / in the webroot can be requested, in combination with answering if the webserver is properly configured

And for the love of all that is compiling, keep all the monitoring and alerting in _ONE_ system!

(if you need/want to use more than one, atleast aggregate the end result to _ONE_ system)

Arguing about how to monitor/alert or with what takes away from the fact that you’re

not that interested in that the monitoring is done but what information it gives you.

AKA

Even someone pressing F5 in their browser to verify that the site is up is more

informative than: Server has 1337 MB RAM free, / is 42% free and there is 1 (management ;) ) zombie process

Alerting

alert(“a service wants attention”)

TMI!

DISK CRITICAL - free space: / 7002 MB (9% inode 60%): /data 16273093 MB (26% inode 99%)

???

DISK CRITICAL - free space: / down to 9%

✓

DISK CRITICAL - root will be out of space in 2 h

Define sane SLOs

Always fix your alerts!

A bad fix is better than no fix

Once upon a time...

License:CC BY-ND 2.0, Wisconsin Department of Natural Resources

How did we get there?

Let’s do this!

Let’s do this!

Let’s do this!

License: CC-SA, Marco Mayer

2 years and counting...

You are never done done

Qs?