Safety MeasureChapter 4

mentsand Data Resources

Photo credit: Federal Aviation Administration

Air Safety databases are maintained at FAA’s computer center in Oklahoma.

CONTENTS

Measuring Transportation Safety. . . . . . . . . . . . . . . . . . . . . . . . . . . . .Safety Factors. . . . .. . . . . . . . . . . . . . . . . . . ....... ....Exposure Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ......Nonaccident Safety Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Primary Safety Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Secondary and Tertiary Safety Factors: Industry Safety Posture

Data Resources . . . . . . . . . . . . . . . . . . . . . .Federal Safety Data Resources . . . . . . . .FAA . . . . . . . . . . . . . . . . . . . . . . . ● . . . . . .National Transportation Safety BoardNASA . . . . . . . . . . . . . . . . . . . . . .RSPA . . . . . . . . . . . . . . . . . . . . .

Conclusions and Options . . . . . .

Box4-A.

4-1.Commercial

Table

of Transportation

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . .

..... . . . . . . . . .....

. . . . . . . . . . . . . . . .....

. . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . ● ✎✎✎✎☛✎

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

Box

Safety: Definitions . . . . . .

Accident

4-1. Safety Measures and Exposure Parameters

Figure

Causal and

Tables

4-2. Federal Aviation Safety-Databases. . . . . . . .

Preventive

. . . . . . . . . .

. . . . . . . . . .

. . . . . .

. . . . . . . . . . . . . . . ●✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ●

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎

69697071737577777784858687

Page. . . . . . . . . . . . . . . . 73

PageFactors . . . . . . . . . . . . . . . 72

Page. . . . . . . . . . . . . . . . . . . . . . 71. . . . . . . . . . . . . . . . . . . . . . 78

Chapter 4

Safety Measurements and Data Resources

Transportation accidents account for only 2 per-cent of all deaths from any cause in the UnitedStates annually, and the public readily accepts theexistence of travel risk. However, public concernvaries for different kinds of risk, and intense atten-tion focuses on air transportation, even though thefatality rate is very low. One reason maybe the rela-tive perceptions of being in control of one’s destiny—the operator of an automobile feels responsible forhis own fate; the passenger on board a public con-veyance does not. Nonetheless, more people die inprivate automobile accidents in an average monthin the United States than have died in commercialaircraft accidents during the past 10 years.

A commercial aircraft crash, though a relativelyrare event, can result in the simultaneous deathsof hundreds of people and often receives immensepublic attention, while a similar number of isolatedfatalities is hardly noticed. The perceived loss to so-ciety is said to be proportional to the square of thenumber of people killed in a single incident, imply-ing that 10,000 individual deaths are the same as100 at once, and that public preventive effortsshould follow accordingly.1

While sometimes irrational about safety, societiesdo attempt to minimize risk to the extent feasibleand at an acceptable cost. Jimmy Doolittle expressedthis well in a report in 1952 to President Truman:2

The ‘Calculated Risk’ is an American conceptwhich gives mobility to the whole structure. Thephrase simply means a willingness to embark delib-erately on a course of action which offers prospec-tive rewards outweighing its estimated dangers. TheAmerican public accepts the calculated risk of trans-portation accidents as an inescapable condition tothe enjoyment of life in a mechanical age. However,the public expects and cooperates to . . . narrow thegap between relative and absolute safety.

To know if risk is being reduced, one must be ableto measure it, and the first half of this chapter out-lines a theoretical framework for nonaccident safetydata analysis. Collecting and analyzing many ofthese data may not be practical or feasible, how-ever, and the discussion is presented as a guide tocurrent safety data systems and their capabilities.The last sections of the chapter present analyses ofexisting safety databases and assess their utility andlimitations.

I Albert L. Nichols and Richard J. Zechhauser, “The perils of Pru-dence: How Conservative Risk Assessments Distort Regulation,” Reg-ulation, November/December 1986, pp. 13-24.

‘Jerome Lederer, “Aviation Safety Perspectives: Hindsight, Insight,Foresight,” Nineteenth Wings Club ‘Sight’ Lecrure, presented at theWings Club, New York City, Apr. 21, 1982, p. 3.

MEASURING TRANSPORTATION SAFETY

Safety Factors

In passenger transportation, safety factors areevents or procedures that are associated with or in-fluence fatality rates. The probability of death (orinjury) as a result of traveling on a given mode, ifit can be quantified, is the primary benchmark ofpassenger transportation safety. To be useful, alter-native safety indicators must ultimately be correlatedto this benchmark. Vehicle accident rates are alsocommonly used as safety indicators, since most pas-senger fatalities occur as a result of vehicle accidents.

If risk is defined as the probability of death, pastrisk in travling can be empirically determined from

fatality rates. Commercial aviation accidents involv-ing large jets can result in the deaths of hundredsof people; thus, a single accident can significantlyinfluence fatality rates. Consequently, trend anal-yses of fatality rates require data from time periodsof roughly 5 years or more, and these rates give poorindications of short-term changes in risk.

Accident rates can be an alternative to fatality

rates as indicators of safety levels. While fatalitiesare often associated with aviation accidents, thenumber of fatalities, even for a specific type of acci-dent, fluctuates considerably with each crash. Thenumber of accidents may have a smaller range ofyearly variance than the number of fatalities, but

69

70

‘k

Photo credit: National Transportation Safety Board

Major accidents result in tragic losses. Fortunately, such catastrophes are rare.

poses similar analysis problems. For example, midaircollisions involving large, commercial jets haveoccurred twice in the United States during the last10 years—little can be inferred from these numbersregarding changes in collision risk. Since the num-ber of accidents is small and can vary significantly

from one year to the next, accident rates are alsopoor indicators of short-term changes in risk.

Safety factors other than fatalities or accidentsshould be considered for prompt feedback on pol-icy decisions or changes in the aviation operatingenvironment. The Federal Government and theaviation industry maintain a wide assortment ofsafety-related information. However, without con-sideration of the accuracy, completeness, and origi-nal purpose of these databases, safety trend analy-ses based on this information are meaningless.

Exposure Data

Understanding the measures that are the denomi-nators of transportation accident or fatality ratesis necessary for safety analysis. The choice of whichexposure data type to use affects how the rates canbe compared across and within the transportationmodes. Passenger-miles (the number of passengersmultiplied by the miles traveled) are the best avail-able exposure parameters for comparing air trans-portation with other modes and allow broad sys-tem comparisons.3 Risk per passenger-mile is notuniform over a trip, and may vary by routing or

‘Trips between specific city pairs would be a better measure, sincethe relative risks among different modes of travel between two pointsis the primary safety concern. However, the total number of city pairsin the United States is too large for comparative analysis and passen-ger data in this form are not readily available for some modes.

71

time of day. For example, the probability of an ac-cident is significantly higher during takeoff or land-ing than while flying enroute; thus, most commer-cial aviation passenger-miles occur during muchlower than overall average risk conditions. (For fur-ther information, see chapter 5.)

Since the number of passengers per vehicle canvary, vehicle-miles are often used to show exposurewhen comparing accident rates. Risk may not beuniform over each vehicle-mile traveled, and vehi-cle size and speed do not affect accident risk exposureindicated by vehicle-miles.

An aviation accident fatal to one passenger islikely to be fatal to many on board.4 Since most ofthe risk involved with air transportation is associ-ated with takeoff and landing, a 2,000-mile trip issimilar to a 200-mile trip when compared for safety.Therefore, the number of trips (departures) is a validexposure parameter for air transportation, and bothpassenger-departures (or -enplanements) and aircraft-departures can be used.

Finally, time is a common measure of exposurein many types of risk analyses. Flight-hour data arenecessary for economic, operational, and mainte-nance requirements of aircraft and airlines. Sinceaccurate data are kept, they are readily available asexposure information.

No single measurement provides the completesafety picture (see table 4-1). Passenger exposure dataare used when passenger risk is to be indicated, whilemiles are used when it is important that the exposuredata not be influenced by vehicle size or speed.Departure exposure data account for non-uniformrisk over a trip. Time is a generic exposure measurein many fields, and data in that form are often read-ily available.

40n average, 50 percent of the passengers on board aircraft involvedin fatal accidents perish. (See chs. 5 and 7.)

Nonaccident Safety Data

Accident investigations often uncover pervasive,but unrecognized, causal factors and can help pre-vent similar accidents from occurring. However,since commercial aviation accidents are so rare,other measures are needed for identifying short-termchanges in safety. The goal of nonaccident data anal-ysis is to help prevent the first accident from hap-pening.

Potential safety indicators are measurable factorsassociated with or causally related to accidents, fa-talities, or injuries. Ideally, the amount of data avail-able will be large enough, unlike accident or fatal-ity data, so that random events will have a smalleffect on yearly trends. The diagram of aviation ac-cident causal and preventive factors (see figure 4-1)identifies sources for some nonaccident safety indi-cators.

In the diagram, items closely associated with ac-cidents appear near the “accident” box. These of-fer the greatest potential as safety indicators and areexplained in the following sections. Factors moreremoved from “accidents” have a correspondingly

long causal link to them. These factors are meas-ured against more subjective standards and may bemore difficult to quantify -’’industry policy, ” forexample.

Measurement Methodology

Clear and precise definitions exist for aviation ac-cidents (see box 4-A); the consistent and accurateaccident databases pose no problems for analysisfrom a measurement standpoint. Moreover, in theUnited States, every commercial aviation accidentis tracked by the National Transportation SafetyBoard (NTSB), providing a complete set of aviationaccident and fatality data. However, other indica-tors require consideration of the measurement meth-

Table 4.1 .—Safety Measures and Exposure Parameters

Critical events Exposure parameters

Injuries Passenger-miles Vehicle-miles

Fatalities Passenger-hours Vehicle-hours

Accidents Passenger-departures Vehicle-departures

Fatal accidents (or -enplanements) (or -trips)

SOURCE: Office of Technology Assessment, 1988.

72

.

*

fm#IL

. . . . . . . . . . . . .

\—/

>

. . . . . .

\

. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

\

F .. . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . /

7

\/

I

!

I3

. . . . . .

73

odology, because subjective influences and incom- is difficult. Additionally, many of the databases wereplete data affect analyses. designed for administrative support functions, not

While most of the potential nonaccident indica- as safety analysis tools.

tors discussed in this section can be extracted fromFederal and industry databases, in practice, they are Primary Safety Factorsnot very useful for safety analysis. Much of the data Primary safety factors are those most closely cor-come from voluntary reports submitted by pilots, related with accidents, and include incidents andmechanics, or controllers. Despite safety reporting accident causal factors. Theoretically, they are therequirements, ensuring compliance or consistency best substitutes or alternatives to accident data.

74

IncidentsIncidents are events that can be defined loosely

as “near-accidents.”5 Causal factors leading to ac-cidents also lead to incidents, and all accidents be-gin as near accidents. The various combinations ofpossibly unsafe acts and conditions that occur eachday usually end as incidents rather than accidents,and the larger number of incidents offers wider op-portunities for safety trend analyses and for suggest-ing potential accident prevention measures. How-ever, for an aviation incident to be widely known,it must be reported by at least one of the people in-volved. Yet, the definition of an incident is subjectto the interpretation of the observer, and what ap-pears to be an incident to one person may not toanother. Thus, some information may be lost andmeasurement error may occur. Similar errors willresult from incidents that are recognized, but notreported. Various sampling techniques can be em-ployed for testing database consistency, and validtrend analyses are possible if errors in the data canbe estimated. Incident types include:

Near Midair Collision (NMAC).-An incidentassociated with the operation of an aircraft in whichthe possibility of collision occurs as a result of prox-imity of less than 500 feet to another aircraft, or anofficial report is received from an aircrew memberstating that a collision hazard existed between twoor more aircraft.

Runway Incursion.–An occurrence at an airportinvolving an aircraft, vehicle, person, or object onthe ground that creates a collision hazard or resultsin loss of separation with an aircraft taking off, in-tending to take off, landing, or intending to land.

In-flight Fire. –A fire that occurs aboard an air-craft, whether or not damage occurs. Fire is ex-tremely dangerous to aircraft and passengers becauseof the confined nature of cockpits and cabins, theamount and flammability of fuel, and the time in-volved in landing and evacuating an aircraft. Flightcrews are required to report occurrences of in-flightfires to NTSB.

5The National Transportation Safety Board considers an incidentto be “. . . an occurrence other than an accident, associated with theoperation of an aircrafi, which affects or could affect the safety of oper-ations.” 49 CFR 830.2 (Oct. 1, 1987).

Flight-critical Equipment Failure.—’’Flight-criti-cal” is subject to various interpretations. Some ex-amples are control system malfunctions and enginefailures.

Accident Causal FactorsAviation accident investigations attempt to de-

termine and understand the causes leading to acci-dents, in the hope of preventing future mishaps. Thefindings can be grouped into five broad categories,as shown in the causal factors diagram. Few acci-dents (or incidents) result from a single, isolatedcause—a combination of factors is usually involved.An examination of these causal factors points to pos-sible indicators for monitoring safety levels. The fiveprimary causal factor categories are discussed below.

Personnel Capabilities.–Human errors are fac-tors in over two-thirds of commercial aviation ac-cidents; they include lapses in attention, judgment,or perception and deficiencies in knowledge or mo-tor skills. Such errors may be caused by vehicle, envi-ronmental, or health factors, including cockpit lay-out, workload, fatigue, or stress. Aviation personnelmost subject to these errors include flight crewmem-bers, dispatchers, mechanics, and air traffic con-trollers.

In the operating environment, human errors aredifficult to identify for a variety of reasons, includ-ing privacy and sensitivity; for example, possiblemeasurements could include the results of periodicor continuous monitoring of operating personnel.However, human errors need to be understood tobe prevented. Some indicators of personnel capa-bilities which are presently measured and used ineither Federal or industry standards include em-ployee duty hours, work hours, age, training, andexperience levels.

Traffic Environment.–The structure of the air-ways and airports and the level and compositionof air traffic heavily influence safety. Difficulties withfacilities or traffic routing are usually discoveredthrough incidents before an accident occurs. How-ever, high traffic density puts continuous strains onmany aspects of the air traffic control (ATC) system.

For a given air traffic infrastructure, increased traf-fic density most likely correlates with an increasedrisk of midair collisions. While the number of flight

75

operations can be accurately counted or estimated,collisions occur too infrequently to correlate, andNMAC statistics are not as precise. Operational er-ror, operational deviation, and pilot deviation sta-tistics are also potential air traffic safety indicators,but have similar consistency problems. Controllerworkload, the ratio of operations to controllers,might provide insight on air traffic safety if the typeof ATC equipment being used and the nature ofthe traffic mix are considered.

Aircraft Capabilities.—The failure of an aircraftcomponent is a factor in over 40 percent of jetlineraccidents. Examples of components include engines,structural members, landing gear, control systems,and instruments. Mechanical failures can result fromimproper maintenance, design flaws, or operatorerror.

Replacement or repair trends, especially for flight-critical components, are possible indicators of safety,although the severity, along with the frequency ofthe component failure must be considered in quan-tifying risk. The Federal Aviation Administration(FAA), air carriers, and aircraft manufacturers main-tain detailed databases of mechanical reliability data.Analysis and communication of observed trends pre-vent most problems from becoming critical. Otherbroad indicators include engine shutdown rates andunscheduled landings due to mechanical difficulties.

Weather.—Modern aircraft can operate in virtu-ally all kinds of weather, but unpredicted severe con-ditions, such as wind shear or heavy icing, can provedeadly. Poor weather, compounded by mechanicaldifficulties or errors in judgment, provides a com-

dwilliam R. Hendricks, dirwtor, Aviation Safety, Federal AviationAdministration, attachment to letter t. OTA, Dec. 18, 1987. The Fed.eral Aviation Administration defines an “operational error” as”, . . anoccurrence attributable to an element of the air traffic control systemwhich results in less than applicable separation minima between twoor more aircraft, or between an aircraft and terrain or obstacles andobstructions as required by FAA Handbook 7110.65 and supplemen-tal instructions. ”

An “operational deviation” is “. . . an occurrence where applicableseparation minima were maintained but loss in separation minima ex-isted between an aircraft and protected airspace, an aircraft penetratedairspace that was delegated to another position of operation or anotherfacility without prior approval, or an aircraft or controlled vehicle en-croached upon a landing area that was delegated to another positionof operation without prior approval. ”

A “pilot deviation” is “. . . the action of a pilot that results in theviolation of a Federal Aviation Regulation or a North American Aero-space Defense Command (NORAD) Air Defense Identification Zone(ADIZ) tolerance.”

mon scenario for aviation accidents. An understand-ing and timely monitoring of weather conditions isrequired for safe operation of aircraft, as shown infigure 4-1.

Unpredictable Events.—These are factors not in-cluded in the above categories, such as sabotage orterrorism. By definition, unpredictable or randomevents have no trends. Therefore, no unpredicta-ble event indicators are possible except incidents andaccidents, which will show levels of past risk.

Secondary and Tertiary Safety Factors:Industry Safety Posture

Commercial aviation safety is the dual responsi-bility of FAA and the airlines. Federal Aviation Reg-ulations (FARs) set the framework for establishingcommercial aviation operating practices. Under thecurrent system, many practices tailored to individ-ual carrier needs are allowable through programsapproved by FAA Principal Inspectors and FlightStandards District Offices.

The commercial airline industry’s operatingpractices—flight operations, maintenance, andtraining—are a dominant influence on the trafficenvironment, aircraft capabilities, and personnel ca-pabilities causal categories discussed above.’ Thesepractices, along with the operation of the ATC sys-tem, are the secondary safety factors (see box 4-A).

The tertiary safety factors, furthest removed onthe accident/incident causal chain, affect the indus-try operating practices listed above. Industry* phi-losophy and policy, which differ among airlines, dic-tate operating decisions. Federal regulatory policy

in turn influences industry policy and operating

practices. Qualitative assessments of the way oper-ating practices affect safety performance are bestmade by independent inspectors using objectivestandards. In theory, FAA airline inspection pro-grams are such assessments, although airline man-agement and labor organizations receive relatively

little attention in FARs.

TManufacturers, through aircraft design and production, influenceaircraft capabilities, and noncommercial flyers and Federal policy af-fect the air traffic environment. These are assumed to be beyond thedirect control of the airlines.

8“Industry” includes airline management as well as labor unions.

76

Secondary Safety IndicatorsFARs require the reporting of some data relevant

to operating practices. For example, air carrier traffic,schedule, and financial information must be peri-odically submitted to the U.S. Department of Trans-portation (DOT). These data illustrate differencesamong carriers and over time, but as currently re-ported and reviewed, no correlation with safety hasbeen established. Some examples of potential safetyindicators are given below.

Flight Operations.—With the increased use ofhub and spoke networks and the limits of the ATCsystem, airline flight crews and maintenance oper-ations have felt new demands. While each airlinewill handle similar pressures differently, the trendsin the indicators are important to understand. Someexamples include: aircraft daily utilization (numberof hours per day an aircraft is used); departures peraircraft per day; percent of fleet required for daily

operations; and percent of flights into high densityairspace.

Maintenance.– The aircraft capability indicatorsdiscussed previously are applicable measures of main-tenance quality, though equipment design and man-ufacturing quality are important also. Unit main-tenance costs can be used, but there are manyreasons for variations among carriers and over time,such as productivity and technological changes.

Training.– Possible indicators include the num-ber of hours of a type of instruction per applicableemployee and the use of certain nonmandatory butvaluable options, such as simulators, cockpit re-source management, and wind shear training.

Tertiary Safety Indicators

FAA safety audits for regulatory compliance couldindicate airline management attitude, organizationalskill, and operational safety. While inspection dataare subject to the personal biases of the individualinspectors, the use of objective inspection guidelinesand standards, consistent and periodic audits, andvarying inspection teams make inspection resultsvalid measures of safety trends. Regulatory compli-ance data differ from previously discussed indica-tors in that the exposure parameters will no longerbe miles, departures, or hours. Since FAA inspec-tors examine only a small percentage of an airline’srecords, aircraft, and operations, a measure of thequantity of inspection is needed in order to normal-ize the data used for analysis. For example, an in-spection of 10 percent of the records of a large car-rier would probably find more faults than a 10percent examination of a small carrier. A measureof a carrier’s exposure to inspection, such as thenumber of inspector man-hours performed or thenumber of records or operations examined, wouldbe used as the denominator in the indicator ratio.The number of violations per inspection man-houris an example of a regulatory compliance measure.

With appropriate guidelines, the quality of man-agement practices could be measured by inspectorassessment and ranking of certain aspects of airlineoperations. For example, two airlines may meet allFederal standards, but one may still be noticeably

“safer” than the other. Objective standards areneeded to permit consistent analyses across indus-try and time.

77

DATA RESOURCES

The Federal Government collects vast amountsof aviation data to support its responsibility for over-seeing aviation safety, and automated systems arerequired for effective data processing. However,OTA research indicates that the analytical quali-ties of electronic data management systems and theirdata vary significantly among and within the Fed-eral agencies dealing with aviation safety. Theamount and caliber of safety data are significantlybetter for commercial aviation than for other trans-portation modes, but major barriers prevent effec-tive use of the data. While frequently-cited accidentand fatality statistics reflect past risk, a comprehen-sive program using Federal databases could identifyand monitor changes in commercial aviation safetyin a more timely manner. The central difficultiesof such a program are:

● the consistency and availability of appropriatesafety data,

● the accessibility and compatibility of variousdata systems, and

. an emphasis on administrative purposes in thedesign and use of the databases that makes anal-ysis difficult.

These problems are not new. A 1980 GeneralAccounting Office (GAO) report stated that FAAhad not been effective or timely in developing sys-tems to identify safety hazards.9 The report futherexplains that:

. . . although FAA’s hazard identification effortshave been numerous and varied, they have beenhindered by insufficient information gathering,limited analysis that has not fully employed stateof the art capabilities, and an inadequately plannedand coordinated agency approach. ”10

A “blue-ribbon” committee of the National ResearchCouncil concurred with these findings and recom-mended that “. . . the FAA accelerate its develop-ment of an effective information-gathering and datasystem. ”11

W.S. Congress, General Accounting Office, How To Zmprove rheFederal Aviation Administration’s Ability To Deal With Safety Haz-ards, GAOiRCED 80-66 (Washington, DC: Feb. 29, 1980), pp. 5-17.

‘qbid, p. 5.1 INational Rmearch Council, Assembly of Engineering, Committee

on Federal Aviation Administration Airworthiness Certification Pro-cedures, Improving Aircrafi Safety: FAA Certification of CommercialPassenger Aircraft (Washington, DC: National Academy of Sciences,1980), p. 66.

Using the safety indicator theory developed earlierin this chapter, OTA reviewed a wide assortmentof Federal aviation safety databases in an effort todocument changes in commercial aviation safetyduring the past decade. A number of these data-bases contain ostensible safety indicator data (seetable 4-2), but in practice, the nonaccident infor-mation has numerous shortcomings and is of limiteduse for safety trend analysis. An overview and assess-ment of each of the databases listed in table 4-2 arepresented in this section, and uses and potential usesfor these databases are given. (OTA analyses of theseand other data are presented in chapter 5.)

Federal Safety Data Resources

DOT, which has regulatory responsibility fortransportation safety, maintains the largest amountof aviation data. Within DOT, FAA, which moni-tors all aspects of aviation safety, and the Researchand Special Programs Administration (RSPA) areresponsible for the collection and management ofsafety and economic-related information. NTSB andthe National Aeronautics and Space Administra-tion (NASA) also keep specialized aviation safetydata.

FAA

FAA is responsible for promoting aviation safety,achieving efficient use of airspace, operating an airtraffic control system, and fostering air commerce.In support of these missions, FAA collects a widerange of aviation information and operates over 280automated data systems.12 Three organizationswithin FAA, the Associate Administrator for Avia-tion Standards, the Associate Administrator for AirTraffic, and the Office of Aviation Safety collect andmanage most of the safety-related data.

Associate Administrator forAviation Standards

Aviation Standards (AVS) personnel, workingout of regional and field offices across the UnitedStates, collect and review large quantities of data

12U,S. Department of Transportation, Federal Aviation Adminis-tration, information Resources Management Plan, Volume H: SystemsPlan FY87-FY89 (Washington, DC: December 1986), p. 167.

78

Table 4-2.—Federal Aviation Safety Databases

Federal EarliestData type Database agency year a Storage system for historical data

Accident/incident Aviation Accident DataSystem

Accident/incident Accident Incident DataSystem

Incident Aviation Safety ReportingSystem

Incident Near Midair CollisionDatabase

Incident Operational Error Database

Incident Pilot Deviation Database

Mechanical reliability Service Difficulty ReportingSystem

Air Operator Data System

Traffic levels Air Traffic Activity Database

Operational practices Air Operator Data System

Air Carrier Statistics Database

Inspection results Work Program ManagementSystem

Violation/enforcement Enforcement Informationactions System

NTSB

FAA

NASA

FAA

FAA

FAA

FAA

FAA

FAA

FAA

RSPA

FAA

FAA

1982

1978

1975

1980

1985

1985

1978

1980

Previous18 months

1980

1988

1987

1983

Digital DEC-10; published reports

Boeing Computer Services IBM3084; Data General MV-15000

Battelle Columbus Laboratories:VAX integrated computer cluster

IBM/AT; published reports

IBM/AT; published reports

IBM/AT; published reports

Boeing Computer Services IBM3084; Data General MV-15000;published data

Boeing Computer Services IBM3084; Data General MV-15000;published data

Published reports

Data General MV-15000; publisheddata

Digital DEC-10; published reports

Data General MV-15000

Boeing Computer Services IBM3084 -: Data-General MV-15000

KEY: NTSB - National Transportation Safety Board; FAA = Federal Aviation Administration; NASA = National Aeronautics and Space Administration; RSPA = Researchand Special Programs Administration.

aEarliegt year for data stored electronically.

SOURCE: Office of Technology Assessment, 19S8

as well as certificate aircraft, airmen, and airlines;oversee and enforce Federal Aviation Regulations;and investigate aircraft accidents and incidents.Many of these data are entered into the numerousdatabases maintained in Oklahoma City at the MikeMonroney Aeronautical Center and the AviationStandards National Field Office (AVN).

The AVN and Aeronautical Center databases,used primarily to support administrative AVS tasks,reside on various computer hardware. Most of thedata systems are hosted by the Aeronautical Cen-ter’s IBM-3084 mainframe or AVN’S Data GeneralMV-15000 minicomputer, though some operate onthe MV-8000’S located at each regional office, theBurroughs B20 workstations distributed through-out FAA or the Transportation System Center’sDigital DEC-10. Some of the systems, while requiredfor the daily operation of AVS, are less importantfor analyzing system safety. Examples include data-bases containing airmen and airline certification

records, medical records, aircraft registry and air-worthiness information, and regulatory history.AVN does maintain four data systems which areused, or can be used, for safety analyses. These data-bases, containing information on aviation accidentsand incidents, mechanical difficulties, regulation vio-lations, and aircraft utilization and reliability, willbe discussed in this section.

Some of the limitations of these independent andincompatible safety data systems have been recog-nized by FAA. The FAA Information ResourcesManagement Program Office described the problemsthat arose from the lack of coordination during thedevelopment of the specialized data systems:

Little consideration was given to the informationrequirements of other organizational elementswithin the agency. This approach has resulted ina number of fragmented data systems that containnonstandardized data, having limited access, anddo not satisfy the needs of all users. In addition, the

79

data contained in these systems are not always cur-rent and lack the accuracy necessary to effectivelymeet the agency’s program objectives.13

FAA is developing the Aviation Safety Analysis Sys-tem (ASAS) to integrate and standardize currentand future databases and maintain them on a cen-tral host computer linked via a telecommunicationnetwork to workstations located at all AVS facil-ities. An overview of ASAS is given later in thissection.

The hardware and software compatibility prob-lems limit the ease with which data are transferredbetween field personnel and AVN. With the excep-tion of enforcement and inspection information, atpresent, data can be entered into the systems onlyin Oklahoma City. While this limits input errors(effectively, only one or two people enter data perdatabase), timely responses are impossible. Thoughthe field offices have access electronically to mostof the systems, the databases are so intricate thatdata requests usually require processing by the lim-ited number of AVN personnel.

Another option available to AVS personnel, aswell as to any interested party, is a commercialtimeshare network that presently contains three ofthe safety data systems. Operated by Boeing Com-puter Services, the system enables users to accessthe complete on-line FAA databases for accidentsand incidents, service difficulty reports, and enforce-ment cases. Historical data, from as many as 5 pre-vious calendar years, can be extracted in standardor custom-designed formats.

FAA Accident Incident Data System.–Accidentdata provide the key means of measuring aviationsafety. An understanding of underlying accidentcauses and trends leads to preventive measures.Responsibility for investigating all civil aircraft ac-cidents in the United States rests with NTSB,14

though authority is delegated to DOT and FAA forcertain accidents.15 Both FAA and NTSB officialscollect accident data, but NTSB alone determinesprobable causes. FAA is responsible for ensuringaviation safety, and investigates accidents primar-

1]U.S. Department of Transportation, Federal Aviation Adminis-tration, Information Resource Management Plan, Volume I: StrategicOverview (Washington, DC: December 1986), p. 22.

1449 CFR 800.3 (Oct. 1, 1987).1549 cm 800, app. (Oct. 1, 1987).

ily to assess whether corrective action is requiredin the aviation system. In January 1984, both agen-cies began using common forms, the NTSB series6120, for the reporting of accident data. While ef-forts are underway to develop a joint NTSB/FAAaccident database, both agencies currently maintainseparate data systems. There is considerable, but notcomplete, overlap between the two systems. TheNTSB Aviation Accident Data System contains allU.S. civil aircraft accidents and selected incidents,while the FAA Accident Incident Data System(AIDS) has fewer accident records, but substantiallymore incident data than the NTSB system.

AIDS contains general aviation and air carrier in-cidents dating from 1978, and general aviation ac-cidents from 1973. In 1982, as a step toward thecommon NTSB/FAA accident database, air carrieraccident information was introduced to the system.Though the NTSB database is considered the de-finitive source for aircraft accident data, AIDS ismore accessible to FAA personnel on a daily basis.Copies of completed accident reports are forwardedfrom NTSB to AVN, where the data are enteredinto the Data General MV-15000 minicomputer.

While NTSB investigators also use the commonseries 6120 forms for reporting incidents.16 AVSpersonnel use the less detailed FAA Form 8020-5.The completed FAA reports are sent to Oklahomafor processing and review, where contract person-nel classify the incidents and assign probable causefactors. Other AVN employees encode and enterthe incident information into the data system. How-ever, the reports are not verified and no procedureis in place for ensuring consistent reporting fromthe field. OTA found substantial variation in theincident reporting rates among the FAA regions (seechapter 5).

AIDS data are available to FAA regional officesand headquarters via the commercial computertimeshare system operated by Boeing ComputerServices or by printouts from AVN. OTA finds thatwhile separate analyses of incident or accident data

MB regulation, aircra~ operators must notifi the National Trans-Yportation Safety Board of five types of incidents (49 CFR 830.5), whichmay be investigated depending upon the circumstances and NationalTransportation Safety Board workload. This results in approximately30 air carrier reports per year from the Board, compared with over1,400 reports by Federal Aviation Administration investigators.

80

are possible, comparisons of accidents and incidentsare difficult because FAA uses different terminol-ogy in classifying incidents than NTSB uses in ac-cident/incident reports. OTA devised an algorithmfor reorganizing FAA data into the NTSB format,and requested that AVN use it in extracting com-mercial aircraft accident and incident data. AVNprovided little useful automated support. The al-gorithm required searches and combinations ofAIDS data fields; OTA received unwieldy printoutsof a portion of the data requested. Even if the miss-ing information were available, extensive manualprocessing would be necessary to format the data.

While NTSB and NASA provide detailed analy-ses of the accident and incident data they maintain,FAA examines air traffic incident data only. In 1984,the Safety Analysis Division of AVS was moved tothe newly formed Office of Aviation Safety. Con-sequently, AVS does not have the resources to ana-lyze air carrier incident or other data maintainedin Oklahoma City. While sufficient information,such as causes and factors, is collected, it is not usedin measuring and monitoring aviation system safetyor to assist in setting regulations.

Enforcement Information System.–Theoreti-cally, trends in the airline industry safety posturecould be determined from the results of regulatorycompliance audits performed by FAA inspectors. Toaccomplish this, the number and type of violationsper carrier and some measure, such as inspectorman-hours, of each airline’s exposure to inspectionswould be needed. However, while all enforcementactions are tracked and recorded in the EnforcementInformation System (EIS), little information is avail-able on the number of inspections performed or theamount of time spent on them. *7

EIS, which is managed by AVN on the MV-15000minicomputer in Oklahoma City, was designed andis used primarily for administrative purposes. In sup-port of AVS and General Counsel personnel, EIStracks the complete history of each enforcement caseand keeps copies of all documentation. Electronicrecords are available from 1963 to present. Becauseof the sensitivity of the data, only closed cases areavailable to the public.

17U.S. Congress, General Accounting Office, Aviation Safety:Needed Improvements in FM Airline Inspection Program are Under-way, GA()/RCED 87-62 (Washington, DC: May 1987), pp. 24-38.

EIS is the only AVN system that allows inputdirectly from the field offices; the others require thatthe field personnel send paper copies of the data toOklahoma City for processing by AVN personnel.

Service Difficulty Reporting System.–The me-chanical reliability of aircraft and components ismonitored by AVN analysts through the ServiceDifficulty Reporting System (SDRS). Reports, re-quired by regulation,18 are filed by air carriers, re-pair stations, manufacturers, FAA inspectors, andothers concerning specific types of aircraft failuresor malfunctions. These reports arrive at AVN inpaper form where the data are encoded and enteredinto the MV-15000 minicomputer.

While containing data for over 10 years, SDRSis most useful for detecting short-term safety prob-lems. The SDRS program automatically trackstrends in reports according to aircraft and compo-nent type. If the monthly or annual trend in reportsexceed a pre-set value, then the system automaticallyalerts AVN analysts. An airworthiness directive,warning, or alert is issued to the public if, after re-view, the trend alert proves serious.

SDRS data are rarely used for long-term analy-ses. Due to the nature of the system, long-term ad-verse trends avoid detection since they have suchshallow slopes they do not set off the alerting sys-tem. Also, since mechanical difficulties are often dis-covered during maintenance inspections, the fre-quency and depth of these inspections, along withthe willingness of the airlines to file reports, affectthe SDRS database.

Air Operator Data System.–AVS personnelmust frequently refer to information about air car-riers and other commercial operators and the struc-ture of their organizations, fleets, and facilities. Whilesuch information is available in fragments frommany sources within DOT, the Air Operator DataSystem (AIROPS) attempts to consolidate the vitaldata available from within FAA. Of interest forsafety analysis are data involving aircraft operations,such as utilization and engine reliability.

Unlike other AVS data gathering efforts discussed(accident/incident, enforcement, service difficulties),there is no regulatory requirement for air carrier

’814 CFR 121.703 and 14 CFR 135.415 (Jan. 1, 1987).

81

reporting or FAA collection of “air operator” dataas such. ” Air carrier inspectors, though, followgeneral guidelines for collecting the data monthly.They send air operator data to Oklahoma City bymail for processing. While AVN employees ensureaccurate transcription of data, there are no proce-dures in effect for ensuring accuracy at the source.The National Air Transportation Inspection (NATI)Program, which relied on AIROPS for many of itsactivities, discovered many errors in the data. TheNATI report concluded that the Air Operator DataSystem is “. . . in need of corrections and enhance-ments. “2° Data have been collected and publishedin the monthly Air Carrier Aircraft Utilization andPropulsion Reliability Reports since the 1960s,though due to contractor problems, no reports werereleased between January and August 1987.21

Air operator data provide the opportunity foranalyzing certain air carrier operating practices, byindividual company or industry wide. When usedin conjunction with other system information, dailyutilization data give one view of the amount ofschedule pressure placed on aircraft fleets. Enginereliability data, the basis for overwater flight cer-tification, indicate the final product of equipmentdesign and airline maintenance and operating pro-cedures.

Work Program Management Subsystem.–FAA’sstruggles to modernize its air carrier inspection pro-gram are documented in a recent GAO report.22

FAA senior management and safety analysts knewlittle about the inspections being performed duringthe post-deregulation period. The only attempt atusing inspection results for analysis followed theNATI Program in 1984. The NATI Task Force,comprised of former FAA inspectors, reviewed theNATI reports and found that over 20 percent ofthe carriers analyzed had a “less than desirable com-

lgAir Camiers must repo~ organizational, operational, and financialdata to the Research and Special Programs Administration’s Office ofAviation Information Management (and previously to the Civil Aer-onautics Board) as required by 14 CFR 241 and 14 CFR 298. Certainengine problems must be submitted via mechanical reliability reportsas stated in 14 CFR 121.703 and 14 CFR 135.415.

‘OU.S. Department of Transportation, Federal Aviation Adminis-tration, “National Air Transportation Inspection Program,” report forthe Secretary, Mar. 4-June 5, 1984, p. 36.

~lThe data were consolidated and released in special reports in fall1987.

2JGeneral Accounting Office, op. cit., footnote 17.

pliance posture,”23 and that FAA inspector surveil-lance and enforcement needed improvement.24

The NATI Program also identified FAA problemsin collecting and managing inspection data.25

Even before NATI occurred, FAA was planningan automated system for tracking the inspection pro-gram. However, the Work Program ManagementSubsystem WPMS), implemented in October 1984,has been plagued by problems. The microcomputers,on which the inspection data are entered, have in-sufficient capacity for the system requirements. Ad-ditionally, there are not enough of the computersto go around. Moreover, FAA installed inadequatesoftware in the system, limiting the type and extentof the inspection data available for analysis.

Changed in October 1986, the current softwareprovided some usable data in fiscal year 1987. FAA’sWestern Pacific Region has successfully utilizedWPMS for inspection efforts, though it still cannotaccess the central computer in Oklahoma City.WPMS has aided FAA’s geographic inspection con-cept by allowing field inspectors throughout theUnited States to send inspection results directly tothe carrier’s respective principal inspector.

Though designed primarily as a tool for manag-ing the FAA inspection program, WPMS can po-tentially be used for safety analysis. WPMS data,centrally stored at AVN, enable a compilation ofinspection results and a measure of exposure(inspector-hours).

Using the Data Systems for Analysis

OTA found few presentations, let alone analyses,of the safety data contained in the AVS data sys-tems. Moreover, the systems are difficult to use forsafety analyses for two fundamental reasons. First,AVN exercises little quality control of data collec-tion and reporting, because it has neither the man-power nor the imperative to do so. Furthermore,no plans are underway for ensuring that FAA fieldpersonnel or airlines collect and report accurate data.

23U.S. Department of Transportation, Federal Aviation Administra-tion, “Memorandum on Evaluation of National Air TransportationInspection Program Inspection Reports,” April 1985, p. 37.

241bid, p. 41.15 Federal Aviation Administration, op. cit., footnote 20, p. 23.

82

Second, extracting useful data from an establisheddatabase requires not only an understanding of thesafety problem to be analyzed, but knowledge of thelimitations of the computer systems and the intrica-cies of the data fields. These AVN data systems werenot designed as analytical tools, and AVN person-nel are not trained analysts. FAA plans to addresssome aspects of this problem by implementing theAviation Safety Analysis System (ASAS), which,as envisioned, will consolidate and standardize newand existing safety databases. In contrast to thepresent system, FAA personnel without extensivetraining in computer programming will have accessto a wide range of safety data via desktop work-stations.

ASAS was conceived in 1979 to build upon thegeneral office automation program for regional andfield offices then in development at FAA. New of-fice equipment, proposed as part of the automationprogram, was to have sufficient processing and net-work capabilities for an integrated safety data sys-tem. The numerous compatibility and communica-tion difficulties created by the data systems then inuse (for the most part, still in use) at FAA were tobe addressed by ASAS. An ASAS Program Officewas established in 1982 and a long-term phased de-velopment plan was proposed. The initial phase willintegrate and standardize current data systems. Sub-sequent phases will implement and develop newdatabases.

—

Photo credit: Federal Aviation Administration

FAA inspectors and safety analysts need ready accessto complete and accurate data.

The types of ASAS databases fall into four cate-gories: 1) airworthiness data; 2) regulatory data; 3)operational data; and 4) organizational information.Airworthiness data are mainly historical informa-tion on aircraft, such as mandatory modificationsspecified by FAA. Regulatory data consist of back-ground information, such as Notices of ProposedRulemaking, legal opinions, and previous regula-tions. Data describing the aviation environment areincluded in the operational category. These data-bases track airmen, aircraft, and operators alongwith accidents, incidents, mechanical reliabilityreports, and enforcement actions. The work man-agement subsystems to monitor AVS tasks, suchas airline inspections, fall into the category of or-ganizational information.

ASAS will alter many of the tasks currently per-formed by AVS personnel. Data will be entered andvalidated where it is collected and generated, at thefield office level. This increase in employee exposureto automated systems implies a need for substan-tial training and for user-friendly equipment andsoftware. The problems with WPMS, discussedearlier, illustrate the need for proper training andtechnology. It is also proposed that field personnelwill be able to perform their own data analyses usinginformation from several databases through analyti-cal software packages.

Associate Administrator for Air Traffic

In managing the National Airspace System(NAS), Air Traffic (AAT) personnel control traf-fic, operate facilities, and develop procedures andstandards for airways, airspace, and flight opera-tions. On a daily basis, information is collected andreviewed concerning air traffic levels, NAS status,system errors, controller errors, pilot deviations, anddelays, although most of the data are entered intoautomated systems only after reaching specific officeswithin FAA headquarters. Other offices, regions,or field facilities within AAT do not have ready ac-cess to many of these systems.26 However, Office ofAir Traffic Evaluations and Analysis specialists mon-itor every report on operational errors, NMACs,

‘%J.S. Department of Transportation, Federal Aviation Adminis-tration, Information Resource Management Plan, Volume 1: StrategicOverview (Washington, DC: October 1985), p. 14.

83

and pilot deviations and communicate findings tothe field facilities.27

While AAT tracks and analyzes air traffic safetydata, it does not manage the data systems dealingwith incidents or system-wide operational informa-tion of interest to this study. The Office of Avia-tion Safety (discussed in the next section), handlesthe incident data while the air traffic activity dataare processed by the FAA Office of ManagementSystems. The Office of Air Traffic Evaluations andAnalysis is developing its own data system, theOperational Error Reporting System, to receive andtrack operational error reports in a timely fashion.The system has been on-line, linking a number ofregional offices with headquarters, since June 1987.

Air Traffic Activity Database.–An essential ex-posure measure for air safety analysis is the level oftraffic. One parameter, departures, is the best ex-posure reference for general safety comparisons.While departure data are available for specific car-riers from Civil Aeronautics Board records andRSPA, system-wide traffic data, including depar-tures, are available from the Air Traffic ActivityDatabase.

Air traffic control personnel keep track of the dailyactivity at ATC facilities. Monthly summaries ofvarious operations, including the number of takeoffsand landings at airports with control towers and thenumber of aircraft handled by radar control facil-ities, are submitted to the Office of ManagementSystems in FAA headquarters. There the data areencoded for entry into the Boeing Computer Serv-ices System, where they are processed and cross-checked. Due to the large volume of monthly data,the Boeing system is not used for analysis or stor-age, but as a tool for preparing summary reports.Annual Air Traffic Activity Reports are publishedand are available to the public.

Facility, region, or system-total data are available,with tables categorizing information by aircraft oper-ator (air carrier, air taxi, general aviation, and mil-itary). This study used historical tower activity datato illustrate the growth of hubs and as the exposurereference for air traffic incidents. The number of air-craft handled by en route radar controllers is analternate measure of traffic trends.

27B Keith pott~, ~~~miate administrator, Air Traffic, Federal Avia-

tion Administration, personnel communication, Dec. 22, 1987.

Office of Aviation Safety

Reporting directly to the FAA Administrator, theOffice of Aviation Safety conducts accident inves-tigations, safety analyses, and special programs. Inthis role, it monitors or manages several databases.The Office of Aviation Safety operates the NationalAirspace Incident Monitoring System, an automatedsystem containing NMAC, operational error, andpilot deviation databases. FAA maintains contactwith the NASA-administered, but FAA-funded,Aviation Safety Reporting System through the theSafety Analysis Division within the Office of Avia-tion Safety.

Near Midair Collision Database.–FAA learnsabout NMACs primarily from pilot reports, thoughair traffic controllers, passengers, and ground ob-servers also serve as notifiers. In each case, a pre-liminary report is filed and must be investigated byFAA within 90 days.

Although the AVS Accident Incident Data Sys-tem tracks NMACs, they are not included in itsdatabase. All incident reports involving air trafficoperations, including NMACs, end up in the Of-fice of Aviation Safety. There, the data are encodedand entered into an IBM/AT personal computersystem located at FAA headquarters. NMAC infor-mation from 1980 to the present is available in thesystem. FAA has had widely publicized difficultieswith its NMAC data, and instituted a monitoringprocedure in 1985 to ensure proper handling ofNMAC reports. An interagency task group consist-ing of FAA, NASA, and the Department of De-fense was formed in 1986 to review existing NMACdata and recommend ways to reduce the midair col-lision threat. The recommendations cover equip-ment, airspace structure, data reporting, and pilottraining. Additionally, the Office of Aviation Safetyis presently conducting a number of NMAC studies.

Operational Error Database.–The loss of legalflight separation around an aircraft which is at-tributed to the ATC system is an operational error(see footnote 6). For example, during en route oper-ations, controllers are required to keep aircraft apartby 5 miles horizontally and 1,000 feet vertically forflights below 29,000 feet and 2,000 feet vertically forflights above. Operational deviations, generally lessserious than operational errors, do not involve lossof separation between two aircraft, but result from

84

an aircraft passing too close to a restricted airspaceor landing area.

From 1983 to 1985, FAA instituted two changes.First, the enroute ATC computers were reconfiguredwith the Operational Error Detection Programwhich automatically records and reports any lossof proper separation for aircraft in the system. Sec-ond, the responsibility for maintaining an opera-tional error report database was shifted to the Of-fice of Aviation Safety. Preliminary reports ofoperational errors and deviations are filed from theATC facility within 48 hours after the event’s occur-rence. All reported operational errors and deviationsare investigated, and depending on the outcome,a final report is submitted. Personnel from the Of-fice of Aviation Safety encode and enter prelimi-nary and final report data into the IBM/AT.

Pilot Deviation Database.–An ATC facilitythat observes a pilot deviation is responsible forreporting it to the appropriate Flight Standards of-fice for investigation. Prior to 1985, incidents involv-ing pilot deviations were entered into AVN’S Ac-cident Incident Data System, though they were notspecifically categorized as pilot deviations. Presently,the results of pilot deviation investigations are sentdirectly to the Office of Aviation Safety where thedata are entered into an IBM PC. The Office ofAviation Safety is responsible for tracking andreporting trends in pilot deviations, and publishedits first statistical report of pilot deviations in Oc-tober 1987.28 Similar to the operational error data,pilot deviation information stored electronically ex-tend back only to 1985.

NTSB, in a special investigation of runway incur-sions, found that as with operational errors, manypilot deviations are not being formally reported butare resolved informally at the ATC facility in-volved. 29 Additionally, prior to 1985, reportsreaching Flight Standards were investigated primar-ily to determine violation and enforcement actionsagainst the pilot involved, not for safety analysis.30

The number of pilot deviation reports processed by

28U.S. Department of Transportation, Federal Aviation Adminis-tration, Selected Statistics Concerning Reported Pilot Deviations (1985-]986) (Washington, DC: October 1987).

z~atlona] Transportation Safety Board, special ~nVeS@arrOn ‘e-port-Runway Incursions at Controlled Airports in the United States,NTSB/SIR-86/01 (Washington, DC: May 6, 1986), p. 8.

‘Tbid, p. 8.

the Office of Aviation Safety is increasing every year(over 2,500 in 1986), though how much of thatgrowth should be attributed to changing reportingpractices is open to question.

National Transportation Safety Board

NTSB is responsible for investigating all aircraftaccidents and certain incidents, determining theirprobable causes, and making recommendations toFAA. It keeps an extensive database of accident in-formation in an automated system and publishesaccident reports and the results of other special in-vestigations.

Aviation Accident Data System.–Since its in-ception in 1967, NTSB has kept records of civil air-craft accidents.31 The current automated database,the Aviation Accident Data System, contains in-formation on aviation accidents and incidents. Pri-marily designed for administrative purposes, the sys-tem does have analytical capabilities. NTSBpublishes Annual Reviews of Aircraft Accident Dataand occasional Special Studies, which are supportedby statistical analyses accomplished with the datasystem.

The NTSB Aviation Accident Data System con-tains information on every known civil aviationaccident 32 in the United States. Selected incidents,as listed in 49 CFR 830.5, are also included in thedatabase. The system encompasses data from 1962to the present, though changes were made in report-ing methods during this period. A single format wasused until 1982, when the procedure and report formwas revised. The documentation was again changedin 1983, when NTSB accident investigators begansubmitting data in the format that was eventually

adopted as NTSB series 6120.4. The data from thereports are entered into the computer, along withthe findings of probable cause and contributing fac-tors. Computer searches are possible with any datablock or group of blocks as selection criteria.

Differences in data formats impose some restric-tions on possible computer-assisted analyses. For ex-ample, in 1982, NTSB changed its method of clas-

JIFrom 1940 t. 1967, the Civil Aeronautics Board investigated ac-

cidents.IZAccidents involving only militaq or public-use airCrafi are not usu-

ally investigated by the National Transportation Safety Board.

85

sifying accidents. Accidents are now categorized bythe first “occurrence” in the sequence of events thatled to the accident. Earlier, groupings were madeby the accident “type.” NTSB has developed a ma-trix for comparing occurrences and types. For broadsafety studies, the effect of the format changes issmall. While the collection of data has essentiallyremained the same, the latter format allows a moredetailed analysis of accident circumstances.33

NASA

NASA, which provides and supports aviation re-search and development, administers the confiden-tial and voluntary Aviation Safety Reporting Sys-tem (ASRS). ASRS is designed to encourage reportsby pilots and air traffic controllers concerning er-rors and operational problems in the aviation sys-tem, by guaranteeing anonymity and immunity fromprosecution for all reporters. ASRS data can pro-vide an alternate Federal insight into the nature andtrends of aviation incidents.

NASA Aviation Safety Reporting System.–ASRS is a joint effort by FAA, NASA, and the Bat-telle Memorial Institute to provide a voluntaryreporting system where pilots, controllers, andothers can submit accounts of safety-related avia-tion incidents. The system is funded mainly byFAA, administered by NASA, and maintained byBattelle. Reports are sent to the ASRS office atNASA Ames Research Center and the data are ana-lyzed and entered into a computer by employees ofBattelle. The database is maintained at Battelle Co-lumbus Laboratories in Ohio.

Prior to the establishment of ASRS in 1976, at-tempts at providing voluntary incident reportingprograms met with little success. Potential reportersfeared liability and disciplinary consequences. Evenafter FAA introduced its Aviation Safety Report-ing Program (ASRP), which offered limited immu-nity and anonymity to participants, few reports weresubmitted. The aviation community feared thatFAA, responsible for setting and enforcing regula-tions, would misuse the data. FAA acknowledgedthese concerns and transferred control of ASRP to

‘]National Transportation Safety Board, Annual Review of AircrafiAccident Data, U.S. Air Carrier Operations, Calendar Year 1982,NTSB/ARC-86/01 (Washington, DC: n.d.), p. 1.

a neutral third party, NASA. A Memorandum ofAgreement was executed between FAA and NASAin August 1975, establishing ASRS. The Agreementprovided for a limited waiver of disciplinary action,confidentiality of reporting sources, and an Advi-sory Committee comprised of representatives of theaviation community. ASRS became operational onApril 15, 1976.34

Voluntary reports, useful for understanding thenature of incidents, are somewhat deficient in in-dicating prevalence or frequency. Therefore, ASRSwas planned as an “analytical rather than a descrip-tive system. ”35 The ASRS report form (NASAForm ARC 277) was designed to gather the maxi-mum amount of information without discouragingthe reporter. Structured information blocks and keywords are provided, not only to guide the reporter,but to aid subsequent data retrieval and research.Narrative descriptions are encouraged. Space is pro-vided for the reporter’s name, address, and tele-phone number. This permits NASA to acknowl-edge the report’s receipt by return mail, and alsoallows the Battelle analyst to contact the reporterfor followup data. Information that identifies thereporter is deleted before being entered into thecomputer.

Under the guidance of NASA, Battelle receivesthe incident reports, processes and analyzes the data,and publishes reports of the findings. Human fac-tors in aviation safety, a continuing concern at theNASA Ames Research Center, were a major consid-eration in ASRS development. The data analysts,primarily experts in aircraft operations and air trafficcontrol, provide insight into the nature of the hu-man error or other underlying factors in the inci-dents. Although the reports are encoded in detail,the complete narrative text of each report is retainedfor later re-evaluation.

Because ASRS is voluntary and reporters aredeidentified, a concerted effort among a number of

J+ William D. Reynard, Aviation Safety Reporting System, in U.S.Congress, House Committet on Science and Technology, Subcommit-tee on Investigations and Oversight, Aircrafi Safety Technologies(Washington, DC: U.S. Government Printing Office, Nov. 23, 1985),p. 29.

‘5Charles E. Billings, M. D., the National Aeronautics and SpaceAdministration Ames Research Center, “Human Factors in AircraftIncidents: Results of a 7-Year Study,” Aviation, Space, and Environ-mental Medicine, October 1984, p. 961.

86

individuals can distort the database. For example,air traffic controllers at certain facilities increasedtheir reporting of incidents associated with a dis-play system that they wanted upgraded. This report-ing campaign ended with the air traffic controllersstrike in August 1981.36

RSPA

The Office of Aviation Information Managementof RSPA assumed the former Civil AeronauticsBoard’s responsibility for collecting data on airlineoperations, traffic, and finances beginning in 1985.Airlines submit data periodically in accordance with14 CFR Parts 217, 234, 241, 291, and 298. Whilethese data do not directly indicate safety, they doprovide measures of exposure such as departures,hours, and miles. However, the airline categories forexposure data reporting do not correspond to theoperating categories used by NTSB for classifyingaccidents, resulting in some gaps and inaccuraciesin statistics. Financial statistics also have potentialuses in analyses, since many in industry and gov-ernment believe that economics influence safety tosome degree.

Air Carrier Statistics Database.–Part 217 Re-porting Data Pertaining to Civil Aircraft Chartersperformed by U.S. and Foreign Air Carriers (14CFR 217) requires U.S. and foreign air carriers tofile traffic data on any civilian international char-ter flight flown to or from the United States in largeaircraft (over 60 seats or 24,000 pounds of payload).The information reported quarterly shows the char-ter passengers or tons of cargo flown between theorigin and the destination point of the charter. Theinformation is reported by aircraft type by month.

Part 234 Airline Service Quality Reports (14 CFR234) requires 14 certificated U.S. air carriers (a car-rier with more than 1 percent of total domesticscheduled passenger revenues) to file monthly flightperformance information for every domestic non-stop scheduled passenger operation to or from the27 largest U.S. airports (airports with more than Ipercent of domestic scheduled passenger enplane-ments). Carriers are voluntarily reporting data foreach domestic scheduled flight instead of limiting

~Wi]liam D, Reynard, chief, Aviation Safety Reporting SYstem, Per-

sonal communciation, Feb. 23, 1988.

their reporting to the 27 airports. For the origin air-port of each nonstop segment, the carrier reportspublished departure times versus actual departuretimes; for the destination airport, the published ar-rival times versus the actual arrival times are re-ported. This information is reported by date andday. Flights delayed because of mechanical reasons,as defined by FAA, are not reported.

Part 241 Uniform System of Accounts andReports for Large Certificated Air Carriers (14 CFR241) prescribes the accounting and reporting regu-lations for large U.S. certificated air carriers (Sec-tion 401 certificate). A large carrier is defined as acarrier operating aircraft which are designed to ac-commodate more than 60 seats or a cargo payloadof more than 18,000 pounds. All large carriers,according to the level of their operations, as meas-ured by annual operating revenues, are placed intoone of four groups: Group I Small ($10 million andunder), Group 11 Large ($10,000,001 to $75 million),Group III ($75,000,001 to $200 million) and GroupIV (over $200 million), The amount and detail ofreporting increases with carrier size. Data are sub-mitted on individual schedules of the DOT Form41 Report or by electronic media. In general, car-riers report exposure data such as aircraft departures,hours, miles, and passenger enplanements in totaland by aircraft types. A broad range of financialdata including categories of revenues and expensesare also reported, with those related to operationsbeing indexed by aircraft type.

Part 291 Domestic Cargo Transportation (14 CFR291) prescribes the reporting required of carriers pro-viding domestic all-cargo operations exclusively un-der Section 418 certificates. These carriers are re-quired to file Form 291-A, a one page annual report,which contains seven profit and loss items, andseven traffic and capacity items. The data are notreported by aircraft type.

Part 298 Exemptions for Air Taxi Operations (14CFR 298) prescribes the reporting for small certifi-cated air carriers (Section 401 certificate) and com-muter air carriers. Both classes of carriers operateaircraft which are designed for 60 seats or fewer orfor 18,000 pounds of cargo capacity or less. A com-muter air carrier is defined as a special classificationof air taxi operator that provides passenger serviceconsisting of at least five roundtrips per week be-

87

tween two or more points. Commuters report onlytraffic exposure data totals with no indexing by air-craft type. Small certificated air carriers submit thesame information as commuters plus revenue andexpense data. The direct expense data and threeoperational items (block hours, departures, and gal-lons of fuel issued) are indexed by aircraft type onsmall certificated air carrier reports. Air taxi opera-tors which are not commuters have no reporting

Various reports including electronic submissionsare sent monthly, quarterly, semiannually, and an-nuall y to the Office of Aviation Information Man-agement, where the data are entered into the Am-dahl computer located in the DOT headquartersbuilding in Washington, DC. Most of these dataare published or loaded on magnetic tapes and areavailable to the general public by subscription.

requirements.

CONCLUSIONS AND OPTIONS

No single measurement or statistic provides a com-plete picture of commercial aviation safety. Whileaccident and fatality statistics are the best measuresof long-term past risk in commercial aviation, theyare of limited value over short periods of time andare not suitable monitors of short-term effects of pol-icy decisions. For example, the consequences of re-cent rulings requiring collision avoidance systemson commercial transports and transponders onmany general aviation aircraft may not be appar-ent in the accident data for 5 years or more.

Nonaccident safety data, while not substitutes foraccident and fatality data, are valuable supplements.If properly collected and maintained, nonaccidentdata can help identify and estimate the magnitudeof safety problems and permit the monitoring ofsafety programs. OTA concludes that nonaccidentdata must be used in short-term safety analyses.

FAA has made great strides in recent years in col-lecting and analyzing air traffic incident data. In-deed, OTA found FAA’s air traffic data to be themost useful nonaccident indicators of system safety.However, since the air traffic system is so safe, onlya fraction of the commercial aviation accidents andfatalities are caused by the air traffic environment.Consequently, additional nonaccident data are re-quired for tracking changes in commercial aviationsafety. OTA finds that FAA programs to identifyand monitor changes in the commercial aviationsafety system need upgrading.

With the exception of airline inspection records,sufficient data for better monitoring and assessingof commercial aviation safety are collected by or areavailable to Federal aviation authorities. However,

FAA quality control programs need improvementto ensure accurate and consistent data collection andreporting. For example, the FAA computer centerin Oklahoma City, which maintains most of the aircarrier-specific information, does not verify incom-ing data. Furthermore, most of the databases aredesigned primarily for recordkeeping; this constrains,but does not prohibit, analysis,

OTA found the analytical capabilities of both thepersonnel and the data system at NTSB to be valu-able resources. NTSB could readily provide pub-lished reports or customized computer printouts;much of the accident data used in this study wassupplied by NTSB. OTA found that while bothNTSB and FAA maintain accident/incident aircarrier specific databases and are under tight staffrestrictions, the close coordination among NTSBdata system managers, analysts, and field person.nel enables NTNB to use its data system effectivelyfor analysis in contrast to FAA’s system in Okla-homa City.

The FAA electronic systems required for process-ing the vast amount of data collected are adequatestorage media, but their flexibility and utility forsafety analysis vary widely. Experienced safety ana-lysts, the eventual system users, took part in the de-sign of the NASA ASRS and remain involved inthe processing and encoding of data. OTA foundASRS data, along with FAA air traffic informa-tion, to be the most valuable incident data on com-mercial aviation that it reviewed. ASRS stands asan excellent example of how to develop and man-age an aviation safety data system. OTA foundthat the close working relationship among data

86-680 - 88 - 4 : CL 3

88

managers and analysts allows ASRS to be used fora wide range of accident prevention efforts. Thesystem could serve as a useful model for AviationStandards data systems, which were configured toaccept data from report forms poorly designed forcomputer input. Additionally, the FAA computercenter staff, while knowledgeable about and com-petent in using the systems, are not trained analysts.To compound problems, FAA management struc-ture reflects the fragmented nature of the FAA datasystems. Three separate FAA organizations havesafety data responsibilities, and databases, data ter-minology, and automated systems are often incom-patible within and among these organizations.

The few FAA studies that use nonaccident dataappropriately have come primarily from the Officeof Aviation Safety. Recent studies by this office fo-cused on ATC system difficulties, such as nearmidair collisions, air traffic controller errors, andpilot deviations. On the other hand, the Office ofAviation Safety has had little success in using theAVS data systems and their air carrier information.For example, the Office of Aviation Safety preparesthe Annual Report on the Effect of the Airline De-regulation Act on the Level of Air Safety, whichdoes not present or appropriately analyze availablenonaccident statistics. The effect of airline oper-ating or management practices, or changes inthose practices, on commercial aviation safety arerarely addressed in FAA studies. Air carrier-spe-cific information systems, such as the Work ProgramManagement Subsystem and the Air Operator DataSystem, are essential tools for properly trained fieldoffice personnel in support of AVS’s commercialaviation oversight role. OTA finds that improvedaccess to these databases is needed at regional andfield offices, a key consideration for future FAA in-formation systems and enhancements currently be-ing developed.

The advent of airline deregulation raised concernsthat economic pressures could force airline manage-ments to cut back on safety practices. The Officeof Flight Standards, responsible for periodically in-specting all airlines to ensure regulatory compliance,is the logical choice for resolving this issue, but needsto collect and retain the necessary data. Consist-ent, centralized records on the number, extent,and results of air carrier inspections are vital toensuring the efficacy of FAA’s safety function.

Four data areas (all used to varying degreesthroughout FAA) could provide warning signals fordirecting FAA attention, and with further refine-ment, could allow quantified estimates of changesin risk. They include:

. aicraft mechanical reliability, including unsche-duled landings due to mechanical problems;

. airline operating practices, including aircraftscheduling and flightcrew work and duty shifts;

. inspection results, including quality assessmentsof airline practices and violation rates; and

• financial condition of airlines, and how that re-lates to any of the other safety indicators.

Airlines themselves keep crucial safety informa-tion and FAA could benefit from working moreclosely with airline data. For example, many air car-riers maintain large internal databases that couldbe used to validate FAA databases. However, en-suring the confidentiality of the air carrier data iscritical. FAA could encourage improved air car-rier reporting of sensitive safety data, such as in-cidents, by guaranteeing that no penalties will re-suit from reported information and couldconsider making nonreporting a violation. Addi-tionally, access to airline computer systems, suchas maintenance management systems,37 could en-hance FAA’s monitoring capabilities.

While airlines share safety information throughindustry and government sponsored workshops,committees, and forums, no formal, centralized in-dustry process is in place for collecting and evalu-ating these data. The airlines, as a group, mightconsider developing a data system to serve as a co-operative industry clearinghouse for safety-relatedmaintenance, training, and operating information.The system could be established independently orin conduction with FAA, and ideally would tap thepotential of the airlines’ extensive automated infor-mation systems.

OTA concludes that all current FAA data sys-terns could benefit from a thorough, coordinated,agency-wide review, although enough shortcom-ings are known now to effect significant improve-ments in the system. Data managers, analysts, andfield personnel should be involved collectively in allnew data system development projects.

~70ne major airline recently provided the Federal Aviation Admin-istration direct access to its computerized maintenance records.

89

Furthermore, OTA finds that an agency-wide, sys-tem safety management approach for data respon-sibility is needed, and that immediate coordinationof Aviation Standards, Air Traffic, and the Avia-tion Safety Office efforts could bring major bene-fits providing support of policy development andplanning, and permitting more focused allocationof agency resources. The current fragmented ap-

proach creates inconsistencies, nonstandardization,poor quality control, incompatible electronic sys-tems, and insufficient data and data analyses. In thelong term, FAA could establish a consistent moni-toring and analysis program to refine the selectionof safety indicators and the procedures for collect-ing and processing information. Safety management,including data managing, is an iterative process.