call me never (building apps that don't wake you up)
TRANSCRIPT
Call Me Never
Building apps that don't wake you up
David Dossot
Great in movies ...
... Tough in real life.
5 million visits a day...
... And counting.
Aiming for five nines...
... And 100x traffic
ReadinessDegradation
RecoveryStrengthening
99.9962( °□°╯ ）╯︵ ┻━┻
Load test and profiling
Readiness
Readiness
Replay access logs
Readiness
Welcome to the Internet
GET /admin.php
GET /catalog/compare/?action=DELETE_FROM_COMPARE_RESULT&IBLOCK_ID=3
Host: -- DROP DATABASE users;
Degradation
Decouple with queues and topics
Degradation
Delay, don't fall down
Degradation
Short circuit blocking-IOs
cache.get(key)
Degradation
Beware of benign code
Recovery
Self heal from transient failures
Recovery
Self heal from transient failures
Recovery
Maintain stable state
Recovery
Replay stable state
Strengthening
Healthy health checks
Strengthening
Healthy health checks
Strengthening
Code defensively and log abundantly
try {
// risky stuff
} catch {
// useful log
}
Strengthening
Failure is opportunity for more tests