Line OperationsSafety Audit (LOSA)

Robert L. Helmreich, PhD, Robert L. Helmreich, PhD, FRAeSFRAeS

James Klinect, PhD Cand.James Klinect, PhD Cand.

North American Aviation Safety North American Aviation Safety ConferenceConference

Atlanta, February 2003Atlanta, February 2003

Human Factors Research ProjectHuman Factors Research ProjectThe University of Texas at AustinThe University of Texas at Austin

Data to Identify Safety IssuesData to Identify Safety Issues Accident investigationAccident investigation

Limited, non-representative sampleLimited, non-representative sample

Incident reportsIncident reports (ASRS and ASAP) (ASRS and ASAP) Data slanted to events resulting from system and flight crew failuresData slanted to events resulting from system and flight crew failures UT HF developing data category system with AA & COUT HF developing data category system with AA & CO

Formal checkrides (Line and Proficiency)Formal checkrides (Line and Proficiency) Data show crew capability and procedural knowledgeData show crew capability and procedural knowledge

Flight Data Recorders – FOQAFlight Data Recorders – FOQA Data show “what happened” in terms of flight parametersData show “what happened” in terms of flight parameters

Non-jeopardy observation of normal flightsNon-jeopardy observation of normal flights Give data on why things happen and how they are managedGive data on why things happen and how they are managed Provides realistic baseline of safety dataProvides realistic baseline of safety data Is proactiveIs proactive

LOSALOSA Jump seat observations of flight crew Jump seat observations of flight crew

performance during regular scheduled flightsperformance during regular scheduled flights Observers unobtrusive – collecting Observers unobtrusive – collecting

data not participating in flightdata not participating in flight Team of observers from different Team of observers from different

backgrounds backgrounds Line pilots / Union representativesLine pilots / Union representatives Check airmenCheck airmen Safety and Training pilotsSafety and Training pilots UT observersUT observers

All data are DE-IDENTIFIED and All data are DE-IDENTIFIED and CONFIDENTIALCONFIDENTIAL

Scientific BackgroundScientific Background

LOSA uses systematic observation of LOSA uses systematic observation of behaviorbehavior

Systematic observation is a validated Systematic observation is a validated methodology using observers trained and methodology using observers trained and calibrated to high reliabilitycalibrated to high reliability

Systematic observation has been employed Systematic observation has been employed in scientific studies of crew performance in in scientific studies of crew performance in demanding environmentsdemanding environments For example, Aquanauts in undersea habitats For example, Aquanauts in undersea habitats

during Project SEALAB and Project Tektiteduring Project SEALAB and Project Tektite

Purpose of LOSAPurpose of LOSA Collect valid empirical data on:Collect valid empirical data on:

1.1. Crew performanceCrew performance – strengths and – strengths and weaknessesweaknesses ProficiencyProficiency Decision-MakingDecision-Making CRM skillsCRM skills Procedural complianceProcedural compliance Threat and error managementThreat and error management

2.2. System performanceSystem performance – strengths and – strengths and weaknessesweaknesses CultureCulture Airspace System – airports and navigational AidsAirspace System – airports and navigational Aids Aircraft design / automationAircraft design / automation Standards / Training / Safety / MaintenanceStandards / Training / Safety / Maintenance Crew support – ATC, Cabin, Ground, and DispatchCrew support – ATC, Cabin, Ground, and Dispatch

History of LOSAHistory of LOSA

Formal LOSA developed at request of Delta Formal LOSA developed at request of Delta Airlines to validate operational impact of Airlines to validate operational impact of Human Factors (CRM) trainingHuman Factors (CRM) training

The focus of initial LOSA was systematic The focus of initial LOSA was systematic assessment of CRM-related crew behaviorsassessment of CRM-related crew behaviors

In 1997, collaborating with Continental In 1997, collaborating with Continental Airlines, LOSA was expanded to record Airlines, LOSA was expanded to record threats and errors and their management threats and errors and their management during flight operationsduring flight operations

• Delta domesticDelta domestic• Delta internationalDelta international• TWATWA• AmericanAmerican• Continental Continental • USAirwaysUSAirways

CRM LOSA 1992 - 1996

Threat and Error Management Threat and Error Management LOSA (1997-2002)LOSA (1997-2002)

• Continental Latin Continental Latin AmericaAmerica

• Continental Continental ExpressExpress

• Gulfstream Gulfstream ExpressExpress

• Air New ZealandAir New Zealand• Air MicronesiaAir Micronesia• ContinentalContinental• DeltaDelta

• USAirwaysUSAirways• Cathay PacificCathay Pacific• EVA AirEVA Air• Uni Air (Taiwan)Uni Air (Taiwan)• FrontierFrontier• QANTASQANTAS• BraathensBraathens• Singapore Singapore

(forthcoming)(forthcoming)

1900+ flight segments. All foreign carriers are U.S. codeshares

LOSA ComponentsLOSA Components Part 1. Flight crew surveyPart 1. Flight crew survey Part 2. Flight crew interviewPart 2. Flight crew interview Part 3. Descent and approachPart 3. Descent and approach Part 4. CRM CountermeasuresPart 4. CRM Countermeasures

Behavioral markersBehavioral markers Part 5. Threat and error mgtPart 5. Threat and error mgt

LOSA Data: Flight CrewLOSA Data: Flight Crew

General Flight Crew General Flight Crew InformationInformation

DemographicsDemographics Attitudes / perceptions / Attitudes / perceptions /

safety culture (FMAQ)safety culture (FMAQ) Safety interview Safety interview

commentscomments

Flight DescriptionFlight Description Observer narrativeObserver narrative Overt threatsOvert threats Operational complexityOperational complexity

Flight Crew PerformanceFlight Crew Performance Behavioral markersBehavioral markers Crew errors and violationsCrew errors and violations Undesired aircraft statesUndesired aircraft states Technical data for Technical data for

approachesapproaches Type and stabilityType and stability

Threat managementThreat management Error managementError management Undesired state Undesired state

managementmanagement

For each flight segment, observers collect data on:

ThreatThreat

Threat ManagementThreat Management

Adverse weatherAdverse weather TerrainTerrain TrafficTraffic Airport conditionsAirport conditions A/C malfunctionsA/C malfunctions Automation eventsAutomation events Communication eventsCommunication events

Operational time pressuresOperational time pressures Non-normal operationsNon-normal operations ATC command events / errorsATC command events / errors Cabin events / errorsCabin events / errors MX events / errorsMX events / errors Dispatch events / errorsDispatch events / errors Ground crew events / errorsGround crew events / errors

• Threats - Originate outside the flight crew’s influence but require active management to prevent them from becoming consequential to safety

Representative Threat FindingsRepresentative Threat Findings

90% of flight segments had one or 90% of flight segments had one or more threatsmore threats

Organizational range – 84% to 97%Organizational range – 84% to 97% Most frequent threatsMost frequent threats

Adverse weather – 19%Adverse weather – 19% ATC clearances/late changes 16%ATC clearances/late changes 16%

ErrorError

Error AvoidanceError Avoidance

Complete error avoidance is Complete error avoidance is impossible – errors are inevitableimpossible – errors are inevitable

63% of flights observed had one or 63% of flights observed had one or more errorsmore errors Must look for sources of error to Must look for sources of error to

strengthen system defensesstrengthen system defenses

Human Limitations Lead to Human Limitations Lead to ErrorError

Limited memory capacityLimited memory capacity Limited processing capacityLimited processing capacity

multi-tasking capabilitymulti-tasking capability Limits imposed by stressors Limits imposed by stressors

tunnel visiontunnel vision Limits imposed by fatigue and other Limits imposed by fatigue and other

physiological factorsphysiological factors Poor group dynamicsPoor group dynamics Cultural influencesCultural influences

Flight Crew Error TypesFlight Crew Error TypesIntentional NoncomplianceIntentional Noncompliance – violations – violations

ex) Performing a checklist from memoryex) Performing a checklist from memory

ProceduralProcedural – Followed procedures but wrong – Followed procedures but wrong executionexecution

ex) Wrong altitude setting dialed into the MCP ex) Wrong altitude setting dialed into the MCP

CommunicationCommunication – Missing information or – Missing information or misinterpretation within cockpitmisinterpretation within cockpit

ex) Miscommunication by crew with ATCex) Miscommunication by crew with ATC

DecisionDecision – Discretionary decision that – Discretionary decision that unnecessarily increases riskunnecessarily increases risk

ex) Unnecessary navigation through adverse weatherex) Unnecessary navigation through adverse weather

Decision ErrorDecision Error

Choice increasing risk in a situation withChoice increasing risk in a situation with multiple courses of action possiblemultiple courses of action possible time available to evaluate alternatives time available to evaluate alternatives no discussion of consequences of no discussion of consequences of

alternate courses of actionalternate courses of action no formal procedures to followno formal procedures to follow

Error ResponseError Response

45% Undetected45% Undetected 29% Detected and action taken29% Detected and action taken 5% Detected and ignored5% Detected and ignored

Error OutcomesError Outcomes

Inconsequential – 71%Inconsequential – 71% Consequential – 29%Consequential – 29%

Additional Errors – 8% (Error Chains)Additional Errors – 8% (Error Chains) Undesired Aircraft States – 21%Undesired Aircraft States – 21%

Undesired Aircraft StateUndesired Aircraft State

• Lateral deviation

• Vertical deviation

• Speed too high

• Speed too low

• Incorrect aircraft configuration– Flight controls– Systems– Fuel– Automation

• Unstable approach

• Abrupt aircraft control

• Long landing – no go

around

• Firm landing

• Forced landing

• Runway incursion

• Wrong taxiway, ramp,

runway, country

A compromised situation placing the flight at increased risk

Phase of Flight EffectsPhase of Flight Effects

Phase of FlightPhase of Flight Threats by Threats by PhasePhase

Errors by Errors by PhasePhase

Pre-Departure / TaxiPre-Departure / Taxi 42%42% 2626%%

Takeoff / ClimbTakeoff / Climb 14%14% 23%23%CruiseCruise 8%8% 4%4%

Descent / Approach / Descent / Approach / LandLand 32%32% 42%42%

Taxi / ParkTaxi / Park 4%4% 5%5%

Descent / approach / land phase contains the most Descent / approach / land phase contains the most variability in crew performance and consequential errorsvariability in crew performance and consequential errors

Organizational DifferencesOrganizational DifferencesAirline AAirline A Airline BAirline B Airline CAirline C

Mean Mean ThreatsThreats

2.252.25 4.644.64 3.273.27

Mean ErrorsMean Errors 0.930.93 1.971.97 4.324.32

MeanMean

Und. StateUnd. State0.190.19 0.420.42 0.950.95

Threat and ErrorThreat and Error

Relationship between number of Relationship between number of threats and errors is relatively threats and errors is relatively weak, although statistically weak, although statistically significant (r=.19)significant (r=.19)

A model of Threat and Error A model of Threat and Error Management derived from Management derived from

LOSA dataLOSA data

The model is being used as The model is being used as framework for the analysis of framework for the analysis of ASAP reports, incidents, and ASAP reports, incidents, and

accidentsaccidents

Threat and Error Management Threat and Error Management ModelModel

Th re ats:Late n t an d Ov e rt

Th re at M an ag e me n t

In co n se q ue n tial In cide n t / Accide n t

Erro rs

Erro r M an age me n t

Un de sire d AircraftState

Un de sire d AircraftState M an age me n t

LOSA and CRMLOSA and CRM

Interpreting LOSA Data for Interpreting LOSA Data for CRMCRM

Procedural errorsProcedural errors - may indicate poor - may indicate poor workload management and/or poor workload management and/or poor proceduresprocedures

Communications errorsCommunications errors - may reflect - may reflect inadequate CRM (monitoring and challenging) inadequate CRM (monitoring and challenging) or complacencyor complacency

Proficiency errorsProficiency errors - suggest pressures to train - suggest pressures to train and/or need for higher standards and better and/or need for higher standards and better monitoring by check airmenmonitoring by check airmen

Decision errorsDecision errors - may indicate need for more - may indicate need for more CRM training on expert decision making and CRM training on expert decision making and risk assessmentrisk assessment

ViolationsViolations - suggest poor procedures, weak - suggest poor procedures, weak captain leadership and/or a culture of non-captain leadership and/or a culture of non-compliancecompliance

““[LOSA] acquires direct, first-hand data on the successful [LOSA] acquires direct, first-hand data on the successful recovery from errors by flight crews during normal line recovery from errors by flight crews during normal line flights. [It] is aimed at collecting data on successful flights. [It] is aimed at collecting data on successful human performance; and this is indeed a first in our human performance; and this is indeed a first in our industry, since aviation has traditionally collected data industry, since aviation has traditionally collected data on failed human performance, such as an accident or on failed human performance, such as an accident or incident investigation.” incident investigation.”

- - Costa Pereira, ICAO Secretary General in May 8, 2000 correspondence Costa Pereira, ICAO Secretary General in May 8, 2000 correspondence to Jane F. Garvey, FAA Administrator to Jane F. Garvey, FAA Administrator

““It is recommended that, IATA and Operators together It is recommended that, IATA and Operators together further promote the use of Flight Operations Quality further promote the use of Flight Operations Quality Assurance, the Line Operations Safety Audit (LOSA)Assurance, the Line Operations Safety Audit (LOSA)……”……”

--IATA Safety Report (Jet) 2000IATA Safety Report (Jet) 2000 Executive Summary Executive Summary, p. vi, p. vi

Endorsed as safety priority by ALPA and IFALPAEndorsed as safety priority by ALPA and IFALPA

Support for LOSASupport for LOSA

Recognition and Extension of Recognition and Extension of LOSALOSA

Flight Safety Foundation/Aviation Week and Flight Safety Foundation/Aviation Week and Space Technology Laurels, 2001Space Technology Laurels, 2001

The UT Threat and Error Management Model The UT Threat and Error Management Model adopted as framework for analysis of adopted as framework for analysis of accidents by IATAaccidents by IATA

Dispatch LOSA (with Continental) underwayDispatch LOSA (with Continental) underway

The University of Texas The University of Texas Human Factors Research ProjectHuman Factors Research Project

www.psy.utexas.edu/HumanFactors