Learn how to Fail from World’s Top Experts

Big Disasters

Gwen Shapira, Senior Consultant

© 2012 – Pythian

Lessons from Really Big Disasters.

© 2012 – Pythian

Swiss Cheese Model“For a catastrophic error to occur, the holes need to align for each step in the process.”

Sometimes this is the right models.Other times the causes are fairly simple.Sometimes there is a “Swiss Cheese Illusion” – Many theories about cause make it seem like there are many causes.

© 2012 – Pythian

1986 was a Bad Year

© 2012 – Pythian

Chernobyl

© 2012 – Pythian

Sequence of Events1. Critical test planned for day shift

2. Gradual shutdown initiated before day shift

3. Test delayed and re-starts at 11PM

4. Night shift took over, with no time to prepare

5. Due to mistake, power dropped too low for the test

6. Attempt to restore power

7. Unstable core temperature and coolant flow

8. Lots of alarms and emergency signals

9. No control rods, coolant close to boiling.

10.… and the test began!

© 2012 – Pythian

More events…1. Turbines shut down and Diesel engines started

2. Decreased water flow, increased vapors

3. Which causes a positive feedback loop in this reactor

4. More steam -> more power -> more heat -> more steam

5. Automatic system inserting control rods

6. Emergency shutdown initiated

7. All rods inserted. Displacing some fluid

8. Massive power spike and first explosion

© 2012 – Pythian

And there is more!1. Some rods broke and blocked.

2. Rise in power, increased temperature, steam buildup

3. Last reading on control panel – 30GW output

4. Probably steam explosion

5. Destroying reactor casing and 2000 ton upper plate

6. Total water loss caused even higher power output

7. Another explosion

8. Dispersing radioactive material.

9. Graphite fire burning by now

10.Inaccurate dosimeters indicate reactor is still working

© 2012 – Pythian

Causes• Bypass of many procedures• Operator errors• Operator lack of training • Operator lack of experience• Non-intuitive reactor design• Dangerous reactor design• Non-compliance with standards• Total belief in in-accurate monitors• Disabled safety features

© 2012 – Pythian

Challenger

© 2012 – Pythian

Sequence of Events• Destroyed on minute two of tenth mission• Flame leaked from SRB to external fuel tank• Damage to tank caused released of hydrogen• Pushing hydrogen tank into liquid oxygen tank• Resulting in massive explosion• Caused by O-Ring Failure• Due to unusually low temperatures during lift-off

© 2012 – Pythian

Causes• NASA organizational culture and decision making are key cause• Problem with O-Ring was known• Disregarded warnings from engineers• O-Ring not certified for low temperatures• No test data for these conditions• Customer intimidation• Lack of clarity in information presentation

© 2012 – Pythian

K219

© 2012 – Pythian

Sequence of Events• K219 was patrolling near Bermuda• Seal in missile hatch failed and water went in• Causing poison gas, explosion, fire and war-head ejection• One missile hatch was already disabled• Vessel surfaced. Nuclear reactors shut down.• One seaman died while securing reactor• Towing attempts unsuccessful• Poison gas leaks • Captain evacuates ship against orders• Submarine sunk. Maybe on purpose.

© 2012 – Pythian

Admiral Nakhimov

© 2012 – Pythian

Sequence of Events• Passenger ship •Minutes into voyage, pilot noticed collision course with bulk carrier• Radioed warning. • Answer: “Don’t worry. We will take care of everything”.• Carrier didn’t take care of anything• Kept radioing the carrier• Eventually both carrier and Admiral Nakhimov changed course. Hard.• Too late.• Unofficial root cause: Both captains were drunk.

© 2012 – Pythian

Mikhail Lermontov

© 2012 – Pythian

Sequence of Events• Left Picton, Australia toward Marlborough Sounds• Experienced Australian Captain…•…Who believed Cape Jackson was twice its real width• And that there are no dangerous rocks• And that he doesn’t need a chart• So he made last minute decision to go through the passage• Despite advice from officers• Hit rocks, water poured in.• Ship was beached and eventually sank from damage• One crew member died. Passengers rescued.

© 2012 – Pythian

Ufa

© 2012 – Pythian

Sequence of Events• Engineers noticed drop of pressure in gas pipeline• To solve the problem, pressure was increased• No additional checks or analysis was done• Leaked gas formed a flammable cloud• Ignited by two passenger trains passing through• Estimated explosion of 200 to 10,000 tons of TNT• 575 dead, 800 injured•Monitoring by “Robot Pigs” was added after the disaster to detect leaks.

© 2012 – Pythian

Bhopal

© 2012 – Pythian

Sequence of Events• History of leaks in plant since 1979. Many events 1982-1984.•Warning by engineers never reached management• Safety systems not functioning• Tank contained more MIC than regulation allowed• During night, water entered the tank• Exothermic reaction. • Pressure was vented• Releasing poison gas• No consensus on how water entered the tank

© 2012 – Pythian

Top Tips to Avoid Disasters1. Avoid being the USSR

2. Communicate. Over-communicate.

3. If your engineers say there is a problem – There is a problem.

4. Fix all issues ASAP

5. Never ignore “almost accidents”

6. Never ignore monitors

7. Always troubleshoot

8. Follow processes and procedures

9. Escalate to the most qualified employees ASAP

10.Have a DR plan. Many of them.

© 2012 – Pythian

To contact us…

Thank you and Q&A

http://www.pythian.com/news/

http://www.facebook.com/pages/The-Pythian-Group/

http://twitter.com/pythian

http://www.linkedin.com/company/pythian

1-866-PYTHIAN

sales@pythian.com

To follow us…