accident investigation techniques and methodologies chuck dejohn, d.o., m.p.h federal aviation...
TRANSCRIPT
Accident Investigation Techniques and Methodologies
Chuck DeJohn, D.O., M.P.HFederal Aviation AdministrationCivil Aerospace Medical Institute
http://www.iprr.org/Papers/Defectslides/sld004.htm
Frei R, Kingston J, Koornneef F, & Schallier P. Investigation Tools in Context. JRC European Commission Institute for Energy Seminar. Investigation of Accidents. May 2003. Petten Netherlands.
Methods vs. MethodologyMethodology: A system of principles, practices and
body of procedures (methods) applied to a specific branch of knowledge. An overall approach to a field such as accident investigation. Examples: Adversarial, Commission, Events
Reconstruction, Modeling, Simulation
Method: A technique or tool. A regular, disciplined, systematic set of procedures used according to an underlying, detailed, logically ordered plan. Examples: Multi-linear Events Sequencing (MES), Fault
Tree Analysis (FTA), Management Oversight and Risk Tree (MORT)
Benner, L. Methodology biases which undermine accident investigation. Proceedings of ISASI 18th Annual Seminar. Washington, DC. 1981.
Methods vs. MethodologyProblems
Defined differently: By different authors By the same author in different articles By the same author in the same article!
Methods and Methodologies are often used interchangeably Examples include Fault Tree Analysis (FTA) and
Management Oversight and Risk Analysis Tree (MORT)
Benner, L. Rating accident models and investigation methodologies. J Safe Res. 1985. Vol 12, No.3; 105-26.
Methodology Classification Schemes
Unstructured Adversarial Events
Reconstruction Modeling Simulation
SurveyArchivalHistoricalExperimentalCase Study
vs.
Common SenseAdversarialEngineeringStatisticalSymbolic Modeling
vs.
Benner, L. Accident investigations – a case for new perceptions and methodologies. Archives of personal papers.
Benner L. Methodological biases which undermine accident investigations. 1981. Proceedings of the 18th Annual ISASI Seminar. Washington, D.C.
Unstructured Methodology
“Common Sense” or “Hunt-and-Peck”“Who, what, when, where, how and why?” Sequential ordering of eventsExplanation of the accident is acceptable if it
“makes sense”Truth is determined by the investigator
Benner, L. Methodology biases which undermine accident investigation. Proceedings of ISASI 18th Annual Seminar. Washington, DC. 1981
Adversarial MethodologyRules of evidence and judicial procedures Opposing interests will bring out the truth
Facts are gathered by the parties and informally tested by discussion against hypothesis for logic and consistency
Reasoned conclusions logically drawn from technical evidence
ExamplesU.S. Party System used by the NTSBCommission inquires
Benner, L. Methodology biases which undermine accident investigation. Proceedings of ISASI 18th Annual Seminar. Washington, DC. 1981.
Benner, L. Accident investigations – a case for new perceptions and methodologies. Archives of personal papers ex libris.
Events Reconstruction Methodology
Reconstruction of sequence of events (SOE): Physical evidence Witness interviews Speculation by investigator
Methodology is not rigorous “Events” are undefined and highly variable Logic trees often culminate in event(s) selected by
investigator without showing time relationshipsProbable cause (PC) often selected from one or more of the
events
Benner, L. Accident investigations – a case for new perceptions and methodologies. Archives of personal papers ex libris.
Symbolic Modeling Methodologies
Pictorial representations of the SOEFault Trees
Failure selected and all possible factors that can contribute to the event are diagramed in the form of a tree
Not always considered an overall methodologyExamples: Logic Tree Analysis, Fault Tree Analysis
(FTA), Management Oversight and Risk Tree (MORT), Multilinear Events Sequencing (MES)
Ferry TS. Modern accident investigation and analysis. Pp 134-44. 1981. John Wiley & Sons. New York.
Harvey MD. Models for accident investigation. 1985. Alberta Occupational Health and Safey Division. Occupational Health and Safey Division.
Benner, L. Accident investigations – a case for new perceptions and methodologies. Archives of personal papers ex libris.
EM 1110-2-6050. 30 Jun 99. Appendix F. Use of Logic Trees in Probabilistic Seismic Hazard Analysis.
Simulation Methodologies
Reenactments that allow investigators to vary assumed events and asses effects of changesFormulate hypothesesDevelop data where there are gapsExamples:
Computerized modelingScale modelingUse of actual aircraft/systems
Benner, L. Accident investigations – a case for new perceptions and methodologies. Archives of personal papers ex libris
Methodology Rankings
Compare simultaneous investigations of the same accident using different methodologiesVery resource intensive
1985 Benner Study:17 U.S. Federal Government Agencies10 evaluation criteria
Benner L. Investigating investigation methodologies. Starline Software Ltd. Oakton, VA. 2003. http://members.cox.net/lbjr99/papersa/IRIA03bennerf.pdf on 5/20/04.
Benner, L. Rating accident models and investigation methodologies. J Safe Res. 1985. Vol 12, No.3; 105-26.
Methodology RankingsAgencies Studied
Consumer Product Safety Commission
Department of Agriculture
Department of the Air Force
Department of the Army
Department of Energy
Department of Labor Mine Safety and Health Administration
Department of Labor Occupational Safety and Health Administration
US Coast Guard
Federal Highway Administration
National Highway Traffic Safety Administration
General Services Administration
Library of Congress
National Aeronautics and Space Administration
National Institute of Occupational Safety and Health
National Transportation Safety Board
Navy Department
Nuclear Regulatory CommissionBenner, L. Rating accident models and investigation methodologies. J Safe Res. 1985. Vol 12, No.3; 105-26.
Methodology RankingsRating Criteria
Encouragement: Encourages harmonious participation.
Independence: Produces unimpeachable results.
Initiatives: Supports personal initiative.
Discovery: Supports timely discovery of facts.
Competence: Provides/improves employee competence.
Standards: Provides for review of safety and health standards.
Enforcement: Supports the enforcement program.
States: Encourages states to take responsibility.
Accuracy: Outputs can be tested for completeness, validity, logic and relevance.
Closed Loop: Compatible with pre-investigation outputs.
Benner, L. Rating accident models and investigation methodologies. J Safe Res. 1985. Vol 12, No.3; 105-26.
Methodology RankingsTop Three
Event reconstructionModeling
MORTFault Tree
Adversarial
Benner, L. Rating accident models and investigation methodologies. J Safe Res. 1985. Vol 12, No.3; 105-26.
Accident Investigation MethodsMethods are Tools used by the investigator, not
an overall system or branch of knowledge Most methods are sequencing tools – Reduce
accidents to a collection of events using cause and effect relationships
Fault Tree Analysis (FTA)Management Oversight and Risk Tree Analysis (MORT)Multilinear Events Sequencing (MES)Sequentially Timed Events Plotting (STEP) Events and Causal Factors Analysis (ECFA)Root Cause Analysis (RCA)
Benner, L. Methodology biases which undermine accident investigation. Proceedings of ISASI 18th Annual Seminar. Washington, DC. 1981.
Accident Investigation Methods
To select the best method you should know:The name of the method you use nowWhich methods are availableWhich methods are better than othersThe outputs of the method you chose
FTA Created at Bell Laboratories, refined by Boeing to
analyze Minuteman missile problems and later adopted by DOD.
Selected failure and all possible factors that can contribute are diagrammed into a tree. The accident is the “top event.”
Top-down approach to determine how “top events” can be caused by individual or combined lower level failures.
Events – Failures that lead to accidents. Gates – Ways failures combine to cause accidents.
Useful for large accident investigations.
Ferry TS. Modern accident investigation and analysis. Pp. 134-44. 1981. John Wiley & Sons. New York.
Schiodtz K. Fault tree analysis in the application of accident analysis. 2003.
FTA
AdvantageConveniently represents main causes/factors
of an accident
DisadvantagesNo temporal relationships between events No ordering of events“Actors” not shown
Ferry TS. Modern accident investigation and analysis. Pp. 134-44. 1981. John Wiley & Sons. New York.
FTA of Aircraft Runway Overrun Accident
Erickson, C.A. Accident Investigation Using EEFTA. Proceedings of the 18th International System Safety Conference. Seattle, Washington. 2000.
MORT
Developed in the 1960s in response to the lack of accident investigation techniques that existed to support rigorous analysis
Pre-designed, systematized logic tree in a generic graphical checklist format of approximately 1500 items
Best suited to large complex accident Requires extensive training
Ferry TS. Modern accident investigation and analysis. Pp. 134-44. 1981. John Wiley & Sons. New York.
American Society of Safety Engineers. Northern Illinois University. TECH 438. MORT, Mini-MORT & PET. 2003.
Mort Event Symbols
American Society of Safety Engineers. Northern Illinois University. TECH 438. MORT, Mini-MORT & PET. 2003.
Mort Logic Gates
American Society of Safety Engineers. Northern Illinois University. TECH 438. MORT, Mini-MORT & PET. 2003.
MORT Advantages
Systematically examines all possible causal factors Ideal when there is a shortage of expertise to ask the right questions Evaluates multiple causes
Works well for complex accidents involving multiple systemsAddresses root causes and contributory causes
Looks beyond immediate causes including management/program factors
Disadvantages Time consuming and tedious to use
Requires extensive training Inappropriate for relatively simple accidents
Can focus more on management than the accident event May lead to recommendations that are too broad (i.e. more training,
more supervision) No temporal relationships between events
Department of Energy. Accident Investigation Program. Section 7 – Analyzing Data. Oct 1999.
Department of Energy Accident Investigation Program. Root cause analysis. January 19, 2001.
Abbreviated MORT Diagram
LTA implies Less Than Adequate performance
PG Bishop, et al. Learning from incidents involving E/E/PE systems. Part 1. 2003. HSE Books. Norwich.
MESTime line chart of the accident process:Time
line is displayed at the bottom of the chart and conditions and events are shown in logical order.
Event = Actor + ActionEvent: Something of significance caused by an action.
Actor: One who causes an event to occur. Does not have to be a person.
Action: Acts performed by the actor.
Ferry TS. Modern accident investigation and analysis. Pp. 134-44. 1981. John Wiley & Sons. New York.
Harvey MD. Models for Accident Investigation. Workers Health, Safety and Compensation, Occupational Health and Safety Diovision. Alberta, Canada. April, 1985.
Keong TH. Accident analysis techniques. Multilinear Events Sequencing.
MES
Accident sequence begins at to
Stable situation is disturbedBeginning of the act which had to be detected,
adapted, corrected, or otherwise changed for the course of events to have had a different outcome
Accident sequence ends at tn
Last consecutive harmful event connected directly with the accident
Ferry TS. Modern accident investigation and analysis. Pp. 134-44. 1981. John Wiley & Sons. New York.
MES
Adapted from: Benner L. Accident investigations: Multilinear events sequencing methods. J Safe Res. June 1975. Vol. 7. No. 2.
to = 11:01 tn = 11:02
MES
AdvantagesIncludes temporal relationship of eventsLimits focus to the accident rather than focusing
on managementHas been called the best model available by some
investigators
DisadvantageFocuses almost exclusively on the accident and
ignores management
Harvey MD. Models for Accident Investigation. Workers Health, Safety and Compensation, Occupational Health and Safety Division. Alberta, Canada. April, 1985.
STEP Developed by Hendrick and Benner in 1987 Refinement of the MES technique Each actor’s actions are traced from the start of an
accident to the finish Actor + Action: Who (person or object) must do what to
produce the next event
Events are positioned along a timeline Causal links are represented by arrows connecting
events Includes quality control with sufficient logic testing to
assure consistency and validityLivingston AS, Jackson G, & Priestly K. Root causes analysis: Literature review. Health and Safety Executive. Birchwood, Warrington. 2001.
NASA. QS/Safety and Risk Management Division. Procedures and guidelines for mishap reporting, investigating, and recordkeeping. NPG:8621.1. June 2, 2000.
STEP
Livingston AS, Jackson G, & Priestly K. Root causes analysis: Literature review. Health and Safety Executive. Birchwood, Warrington.
ECFA
Identifies causal factors for each significant event in an accident sequence
Designed as a stand-alone technique but most effective when used with other methods (i.e. MORT, RCA)
No “timeline” but temporal relationships are accounted for
Buys JR, Clark JL, Kingston-Howlett J, and Nelson HK. Events and causal factors analysis. Scientech, Inc. Idaho Falls, ID. August 1995.
ECFAEvaluate events to determine significant events:
The accident would not have occurred if the significant event had not occurred
The event deviated from what was planned or intended
The event had unwanted consequences
Determine the causal factors that allowed each significant event to occur:Who, why, what and how?
Department of Energy Accident Investigation Program. Events and causal factors analysis. January 19, 2001.
ECFAExample of Accident Chronology
Department of Energy Accident Investigation Program. 1/19/01.
Inspection of rudder PCU deleted from annual inspection
Rudder PCU failure mode not identified
Rudder PCU hydraulics contaminated
Rudder hard-over in-flight19:02:47
Crash 19:03:00
1994 September September 8 September 9
ECFAConditions for 1st Event
Event
4
Event
3
Event
2
Rudder hard-over in-flight
Crash
Crew fails to respond to unusual attitude
Crew fails to analyze unusual attitude
Crew fails to recognize rudder problem
Department of Energy Accident Investigation Program. 1/19/01.
ECFAConditions for 2nd Event
Event
4
Event
3Rudder PCU hydraulics contaminated
Event
1Crash
New maintenance personnel do not detect
Hydraulic fluid becomes contaminated
Change in maintenance services contract
Department of Energy Accident Investigation Program. 1/19/01.
ECFACausal Factors for 1st Event
Event
4
Event
3
Event
2
Rudder hard-over in-flight
Crash
Conditions
Need for UA training unrecognized
Potential need for recognizing rudder problems unrecognized
Department of Energy Accident Investigation Program. 1/19/01.
ECFACausal Factors for 2nd Event
Event
4
Event
3Rudder PCU contaminated
Event
1Crash
Conditions
Need to screen service contract provider unrecognized
Potential for hydraulic fluid contamination unrecognized
Department of Energy Accident Investigation Program. 1/19/01.
ECFA Advantages
Temporal relationships of significant events preserved Ideal for multi-faceted problems with long or complex
causal chain Causal factors for each significant event determined Recommendations easily arrived at from causal factors Helps to identify where deviations from acceptable
procedures occurred Disadvantages
Requires a broad perspective of the event to identify unrelated problems
Time consuming Requires training/and or familiarity with the process
US Department of Energy, Office of Nuclear Energy, Office of Safety Policy and Standards. Root cause analysis guidance document. DOE-NE-STD-1004-92. Washington, D.C. February 1992.
RCARoot Causes are causal factors that, if
corrected, would prevent the recurrence of the same or similar accident.Local Root Causes are specific deficiencies
that, if corrected, would prevent the recurrence of the same accident.
Systemic Root Causes are deficiencies in a management system that, if corrected, would prevent the occurrence of a class of similar accidents.
Department of Energy Accident Investigation Program. Root cause analysis. January 19, 2001.
RCA
Root Cause Analysis (RCA) is a structured procedure to identify and evaluate the underlying causes of an accident to prevent a recurrence.
Goal of RCA is not merely to determine the cause of an accident but to prevent it from occurring again.
NASA, Office of Safety and Mission Assurance, Chief Engineers Office. Root cause analysis overview. July 2003.
Rimson IJ. Investigating “causes” and assigning “blame.” The Investigation Process Research Library. August 2003.
Decision Systems, Inc. What is root cause analysis? Longview, TX. 1999.
RCAProcedure
Phase I: Clearly define the undesired outcome.Phase II: Data Collection. Phase III: Assessment.
Identify the problem and significance of the problem
Identify the causes working back to the fundamental cause, which if corrected, would have prevented the accident (root cause)FTAMORTECFA
NASA, Office of Safety and Mission Assurance, Chief Engineers Office. Root cause analysis overview. July 2003.
Department of Energy Accident Investigation Program. Root cause analysis. January 19, 2001.
RCAProcedure
Phase IV: Corrective actions for each identified cause to prevent recurrence.
Phase V: Follow-up by determining if corrective action effectively prevents recurrence.
Department of Energy Accident Investigation Program. Root cause analysis. January 19, 2001.
RCA
http://www.hq.nasa.gov/office/codeq/rca/rootcauseppt.pdfon
Conclusions
Methodologies largely determined by organization
Methods may be selected Not all methods suitable
for each accident Simplest method that
yields the required results
Frei R, Kingston J, Koornneef F, & Schallier P. Investigation Tools in Context. Noordwijk Risk Initiative Foundation.