staying alive: patterns for failure management from the bottom of the ocean - ronnie chen -...

STAYING ALIVEPATTERNS FOR FAILURE MANAGEMENT FROM THE BOTTOM OF THE OCEAN

RONNIE CHEN SLACK

1 — Ronnie Chen @rondoftw

WHY DID I BECOME A DIVER?


A WHOLE NEW WORLD


TECHNICAL DIVING

▸ longer dive times▸ deeper dives▸ overhead ceiling

▸ decompression obligations▸ more gear. a lot more.▸ higher pressure▸ more risks


RISKS MAY INCLUDE...

1. hypoxia2. hyperoxia

3. nitrogen narcosis4. carbon dioxide buildup5. oxygen sensor failure

6. deep tissure isobaric counterdiffusion (ICD)7. high pressure nervous syndrome (HPNS)

8. software failure9. exhausting your carbon dioxide scrubber

10. carbon dioxide channeling from a poorly packed scrubber11. carbon buildup causing an spark leading to an oxygen fire. underwater.

12. flooding of breathing loop or circuitry13. water mixing with the scrubbing agent to produce a toxic caustic soda that will give you chemical burns on your mouth, airway, and lungs

14. plain old decompression sickness


If you own a rebreather for five years, two percent of you

are going to die on it.

— Jill Heinerth, underwater explorer


HOLD UP!15 — Ronnie Chen @rondoftw

THIS IS A TALK ABOUTCOMMUNICATION AND PROCESS

? ? ?16 — Ronnie Chen @rondoftw

(it was a trap)


YOU CAME TO HEAR COOL STORIES...


BUT YOU'RE GETTING A MEANDERING MEDITATION ONBEST PRACTICES* WHEN DEALING WITH COMPLEX SYSTEMS INSTEAD

* These guidelines have only been shown to work for life or death situations under the ocean.They have not been proven to work for tech.


How failures really happen


Complex systems are designedto protect against simple failures.


But accidents still happen.


CATASTROPHES ARE CAUSED BY A FAILURE CASCADE▸ you have a rebreather malfunction

▸ which you would have caught it if you were testing your equipment on a regular basis▸ your backup tank had a leak and is running low and that wasn't caught either

▸ and your buddy is too far away and isn't checking in with you▸ and your dive light that you use to communicate at a distance is out of power

▸ and in the excitement you kick up silt and the visibility drops▸ and in your panic your air consumption goes up and then you breathe through the last of the air in your tank

▸ so you swim for the surface even though you have a decompression obligation


A post-mortem that blames this incident on a simple mechanical malfunction would only cover 12.5% of the issues that led up to this

accident.


Complex system failures don't happen because a single part of the system fails. They happen because all the safety procedures that are supposed to protect them from the simple system failure didn't work.


CORE RULES OF SAFETY SYSTEMS

1. An unused safety system doesn't exist.


NORMALIZATION OF DEVIANCEThat natural human tendency,

particularly in pressure circumstances,to take a safety shortcut.

To accept a lower standard of performance.

— Colonel Mike Mullane, astronaut


FALSE FEEDBACKthe absence of something bad happening means that it was safe

ADAPTATIONexperience is no longer a suitable gauge of risk

SOCIAL PRESSUREthis is just how we do things



2. An untested safety system doesn't exist either!



3.Unused or untested safety systems are more dangerous than not having one at all. Therefore, safety systems must be tested at regular

intervals.

The length of this interval should be determined not only by how likely it is for this system to fail but also how great the impact will be if it does.


A QUICK SIDENOTE ON ASSESSING RISK

▸ Make assessments based on likelihood of occurrence.▸ Make assessments based on magnitude of regret.

If you are only evaluating risk based on the chance of it happening, you must be prepared to experience the corresponding level of regret if it

does.


failures will happen


WHAT IS SAFETY?


FAILURE MANAGEMENT

▸ A framework for redundancy▸ The training and judgment to use it


FAILURE MANAGEMENT FOR SYSTEMS

▸ Have redundancy for systems that you cannot survive without.▸ Have a redundant pathway to success: a procedure for graceful

degradation for systems that are important but not critical.▸ Have a process for changing over from primary to redundant

systems.


FAILURE MANAGEMENT FOR SYSTEMS (CONT)

▸ Keep failures contained so that they don't bring down other systems▸ Make it easy to do the right thing and hard to do the dangerous

things


FAILURE MANAGEMENT FOR HUMAN SYSTEMS


TRAINING FOR PRESSURE


TRAINING: INEXPERIENCED PEOPLE TO THE FRONT

▸ Most inexperienced person leads▸ Experienced person advises and intervenes only when necessary▸ Team is invested in personal success to ensure mission success


TRAINING: INEXPERIENCED PEOPLE TO THE FRONT (CONT)

▸ Frees up more experienced people from micromanaging▸ Opportunity to revise and improve problematic systems▸ One of the best ways to equalize a gap in experience


GOOD JUDGMENT

Good judgment enables the reshaping of rules and frameworksto adapt to a changing environment.


REFINING JUDGMENT

▸ Post-Mortems▸ Pre-Mortems▸ Fire Drills

▸ Revisit Past Decisions


POST-MORTEMS▸ Look at the safety procedures that failed to stop the cascade▸ Look for opportunities to create new safety systems at critical

points


PRE-MORTEMS▸ Don't wait for failures to build safety frameworks

▸ Identify potential avenues of of failure and make plans for them▸ Include both likely failures and high regret failures


FIRE DRILLS▸ Vet your plans and safety systems▸ Perform targeted training

▸ Evaluate effectiveness of tools and documentation


REVISIT PAST DECISIONS▸ Examine successful operations to see what key insights were helpful

▸ Identify any dependency on luck in previous projects▸ Share rationale for decisions


RECOGNIZING SUCCESS


I WANT TO LEARN MORE!1. Diane Vaughn - The Challenger Launch Decision2. Richard I. Cook - How Complex Systems Fail3. Mike Mullane - https://www.youtube.com/watch?v=Ljzj9Msli5o4. Steve Lewis aka decodoppler - Staying Alive5. Sidney Dekker - Drift into Failure


Any Questions?


staying alive: patterns for failure management from the bottom of the ocean - ronnie chen -...

Technology