normal accidents and outpatient surgeries
DESCRIPTION
Presentation to Box TechOps team in 2011TRANSCRIPT
Normal Accidents andOutpatient Surgeries
Resilience Engineering Done Right
Safety in a Complex and Changing Environment"...so safety isn't about the absence of something...that you need to count errors or monitor violations. But the presence of something. But the presence of what?
When we need to find that things go right under difficult circumstances, it's mostly because of people's adaptive capability; their ability to recognize, adapt to, and absorb changes and disruptions, some of which might fall outside of what the system is designed or trained to handle"-Sidney Dekker
Safety in a Complex and Changing Environment
"...so safety isn't about the absence of something...that you need to count errors or monitor violations. But the presence of something. But the presence of what?
When we need to find that things go right under difficult circumstances, it's mostly because of people's adaptive capability; their ability to recognize, adapt to, and absorb changes and disruptions, some of which might fall outside of what the system is designed or trained to handle"-Sidney Dekker
RESILIENCE
Vocabulary Lesson
Continuous Integration: The ability to quickly make sure the system is ready for production.
Vocabulary Lesson
Continuous Integration: The ability to quickly make sure the system is ready for production.
Resilience: The intrinsic ability of a system to adjust its functioning prior to, during, or following changes and disturbances in order to sustain required operations.
Vocabulary Lesson
Continuous Integration: The ability to quickly make sure the system is ready for production.
Resilience: The intrinsic ability of a system to adjust its functioning prior to, during, or following changes and disturbances in order to sustain required operations.
Maintainability: Characteristic of design and installation which determines the probability that a failed equipment, machine, or system can be restored to its normal state within a given timeframe.
Vocabulary Lesson
Continuous Integration: The ability to quickly make sure the system is ready for production.
Resilience: The intrinsic ability of a system to adjust its functioning prior to, during, or following changes and disturbances in order to sustain required operations.
Maintainability: Characteristic of design and installation which determines the probability that a failed equipment, machine, or system can be restored to its normal state within a given timeframe.
The SYSTEM includes all the hardware and software, but also all of the PEOPLE involved.
Maintainability = Uptime Goodness
MTTR vs. MTBF
Maintainability = Uptime Goodness
MTTR vs. MTBF
Low MTTR > Low MTBF
Maintainability = Uptime Goodness
MTTR vs. MTBF
Low MTTR > Low MTBF
Low MTTR = Better Uptime for most types of F
Maintainability = Uptime Goodness
MTTR vs. MTBF
Low MTTR > Low MTBF
Low MTTR = Better Uptime for most types of F
Low MTTR Requires: • more useful metrics• intelligent data analysis• pre-planned, purposeful resilience• cooperation between application and infrastructure
Your Average Operations Engineer
Your Average Operations Engineer
Automation as a Default:"One of the best ways to eliminate human problems is to take the human out of the problem. Machines are very good at doing things repeatedly and doing them the same way every single time. Humans are not good at this. Let the machines do it.”
Rapid Recovery:"Do we spend an unpredictable amount of time trying to solve some obscure issue, or do we simply recreate the instance providing the service from configuration management"
blog.lusis.org/blog/2011/10/18/deploy-all-the-things/
Automation as a Default:"One of the best ways to eliminate human problems is to take the human out of the problem. Machines are very good at doing things repeatedly and doing them the same way every single time. Humans are not good at this. Let the machines do it."
Rapid Recovery:"Do we spend an unpredictable amount of time trying to solve some obscure issue, or do we simply recreate the instance providing the service from configuration management"
blog.lusis.org/blog/2011/10/18/deploy-all-the-things/
PUPPET + KICKSTART+ Network Automation
Automation as a Default:"One of the best ways to eliminate human problems is to take the human out of the problem. Machines are very good at doing things repeatedly and doing them the same way every single time. Humans are not good at this. Let the machines do it."
Rapid Recovery:"Do we spend an unpredictable amount of time trying to solve some obscure issue, or do we simply recreate the instance providing the service from configuration management"
blog.lusis.org/blog/2011/10/18/deploy-all-the-things/
PUPPET + KICKSTART + Network Automation
ESPER + HEALTHCHECK + NAGIOS + SPLUNK+ OHSHIT
Comfortable Changes1) Are Small• Many Small Changes = Fewer Incidents with lower
MTTR
Comfortable Changes1) Are Small• Many Small Changes = Fewer Incidents with lower
MTTR
2) Are ReproducibleRPM:• Really Peaceful Mornings• Reduce Paging Monitors• Reusable Provisioning Methods
Comfortable Changes1) Are Small• Many Small Changes = Fewer Incidents with lower
MTTR
2) Are ReproducibleRPM:• Really Peaceful Mornings• Reduce Paging Monitors• Reusable Provisioning Methods
Rule # 81: If you are logging into servers, you are doing it wrong.
Comfortable Changes3) Are easily understood by your most junior team members
Comfortable Changes3) Are easily understood by your most junior team members
Rule # 4: Keep it Simple, because you are smart. Do not make it overly complex because you can.
Comfortable Changes3) Are easily understood by your most junior team members
Rule # 4: Keep it Simple, because you are smart. Do not make it overly complex because you can.
4) Can be deployed to a subset of production systems
Comfortable Changes5) Follow Process
Comfortable Changes5) Follow Process
Change control, deployment processes, peer review, all of these things matter for a world-class OPS organization.
Comfortable Changes6) Have been approved by a GO / NO-GO process with all relevant parties checking in.
Comfortable Changes6) Have been approved by a GO / NO-GO process with all relevant parties checking in.
Ensure that all teams involved in a change have signed off, including ON-CALL and CUSTOMER SERVICE
Tracking Changes
Small ChangesJohn Allspaw presented these graphs of data gathered at Etsy.
More Smaller Deploymentsmeans
Faster MTTRmeans
Fewer Minutes of Disruption
Operations Meta-MetricsWhen in doubt, COLLECT DATA, Build a Timeline!
Things to Monitor:
Changes (who/what/when/type)
Incidents(Type/Severity/Duration)
Responses to Incidents(TTD/TTR)
Things to Collect:IRC/Jabber LogsJira Logs
Search your Data: Use HBASE+PIG/HIVE, ESPER, SOLR and SPLUNK
Store everything, even stuff you don't yet know how to use.
Tracking Incidents - MTTD
1.Frequency2.Severity3.Root Cause: Five Whys Mentality
o why was the website down? The CPU utilization on all our front-end servers went to 100%
o why did the CPU usage spike? A new bit of code contained an infinite loop!
o why did that code get written? So-and-so made a mistakeo why did his mistake get checked in? He didn't write a unit test for the
featureo why didn't he write a unit test? He's a new employee, and he was not
properly trained
4.Time-to-Detect5.Time-to-Resolve
Tracking Incidents - MTTD
Rule # 18: Monitor EVERYTHING, alert on actionable items only, record other for trend information.
Rule # 20: Do not make the monitoring system so noisy it is useless.
Tracking Incidents - MTTD
Data Points to source these metrics from:Output from Application, CLOG, Puppet, Jabber, Jira, healthcheck, hardware, Eluna, Nagios....all collectible data
Handling Incident Response - MTTRDetect a ProblemCommunicate to Support/Community/Executives
Begin to take ActionCommunicate to Support/Community/Executives
Coordinate Troubleshooting/DiagnosisCommunicate to Support/Community/Executives
Confirm Stability, Resolving StepsCommunicate to Support/Community/Executives
Handling Incident Response - MTTRRule # 24: Assign people to be point people for every bit of technology
Rule # 25: Assign Backup People to those People
Rule #12: Know your bottlenecks, and how to spot them.
Rule # 42: Create gigantic poster size drawings of the physical layouts of your data center
Rule #43: Create gigantic poster size drawings of the logical flows of each part of your product.
XKCD #974:
I find that when someone is taking time to do something right in the present, they're a perfectionist with no ability to prioritize, whereas when someone took time to do something right in the past, they're a master artisan of great foresight.