AD-A091 229 DAYTON UNIV ON RESEARCH INST F/6 5/9POP-UP WEAPON-DELIVERY MANEUVER: USE OF PILOT SELF-ASSESSMENT D--ETC(U)

OCT 80 D R LYON. ,I L EUBANKS, T H KILLION F33615-77-C-005UNCLASSIFIED AFNRL-TR-8O-33 NL

.SNuIIuIIIIIuIIIIIIIIIIIIIIfflfflf

=llll FD

AFtR-H-80-33

AIR FORCE .,POP-UP APON-DELIVRY MANEUVER:USE OF pll,)T SFLF-ASSESSMENT DATA

IN ANALYSS OF CRrICAL COMPONENTS

By

M ion R. LyonJames L. Eubanks"Thomas 1-. Killion.,,Rbetl T. Nullmever L

UItivemit) of Iayhon Reseanrh hIstdtuleI314)O olhge Parik Avenue

N Edward E. Eddowes

OPERATIONS IRAININ(; DIVISIONR Williams Air Fore Base. Aizona 8522

Ec~e 1980)~SFinal Report

0U %ppro ,d for public relas,,: dimtrihution unliniied.

RcS LABORATORY

* AIR FORCE SYSTEMS COMMANDBROOKS AIR FORCE BASETEXAS 78235

NOTICE

When U.S. Government drawings. specifications, or other data are used for any purposeother than a definitely related Government procurement operation, the Governmentthereby incurs no responsibility nor any obligation whatsoever, and the fact that theGovernment may have formulated, furnished, or in any way supplied the said drawings,specifications, or other data is not to be regarded by implication or otherwise, as in anymanner licensing the holder or any other person or corporation, or conveying any rightsor permission to manufacture. use, or sell any patented invention that may in any way berelated thereto.

This final report was submitted by the University of Dayton Research Institute, 300College Park Avenue. Dayton, Ohio 45469, under Contract F33615-77-C-0054, Project1123. with the Operations Training Division. Air Force Human Resources Laboratory(AFSC). Williams Air Force Base, Arizona 85224. Dr. Edward E. Eddowes was theContract Monitor for the Laboratory.

This report has been reviewed by the Office of Public Affairs (PA) and is releasable to theNational Technical Information Service (NTIS). At NTIS, it will be available to thegeneral public, including foreign nations.

This technical report has been reviewed and is approved for publication.

MARTY R. ROCKWAY, Technical DirectorOperations Training Division

RONALD W. TERRY, Colonel. USAFCommander

SUBJECT TO EXPORT CONTROL LAWS

This document contains information for manufacturing or using munitions of war.Export of the information contained herein, or release to foreign nationals within theUnited States. without first obtaining an export license, is a violation of the InternationalTraffic in Arms Regulations. Such violation is subject to a penalty of up to 2 yearsimprisonment and a fine of $100.000 under 22 U.S.C. 2778.

- .....-.-..-.-

U nclassi fiedSEC URITY C ICATIOtA OF THIS PAGE (W~hen Datt. Emotred) _________________

EPOR DOCMENTTIONPAGEREAD INSTRUCTIONSC) V PORTDOCUENTAION AGEBEFORE COMPLETING FORM

REP MU2. GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER

4. TITLE (and Subtitle) EE

- . Final) / '

SILOTjEF FSS SMFNT [ATA IN ANALYSIS /.

11. CONTRACT OR GRANT NUJMBER(&)

James Lft) ibanks F33015-77-(:#54 j--I'----Thomas 11' 'Killion d til, ~ o 'j 3657~A~

.IiON NAME AND ADDRESS I0. PROGRAM ELEMENT, PROJECT, TASKAREA 8 WORK UNIT NUMBERSUniversity of D~ayton Research Institute 62S

300 Colle'ge Park Avenue 625Daytn, Oio 446911230236 and 11230337

1I. CON4TROLLING OFFICE NAME AND ADDRESS F/ ai

HQ Air Force Human Resources Laboratory ( AFSC) AGtbe ES0Brooks Air Force Base. Texas 78235PAE

30)14. MONITORING AGENCY NAME & ADDRESS(if different from, Contra1"&g Offi.ce) IS. SECURITY CLASS. (of tib report)

Operations Training D~ivision Unclassified

WillamsAir orc 881', rizna 8224- / IS&. DECLASSIFICATION'DOWNGRADINGWillamsAirForc Bae. rizoa 8224SCHEDULE

16. DISTRIBUTION STATEMENT (of tht. Report)

A pproved for pubic release: d ist ribtin Un~l imIlited.

17. DISTRIBUTION STATEMENT (of the ealect entered in Block 20, If differvlv" -

It. SUPPLEMEN4TARY NOTES

19. KEY WORDS (Continue on rerse side it neciteary end identify by block nurtber)

flying training % capol deli% ervpilot trainling task analysisskill nmea-luremewl

skill Iflaintenallce

%elf-asqestnlen I

20. ABSTRACT (Continue or, reoeree aIde If necee..ry end Identify by block number)

~Perform ance I)( ill pop~-lap N eaponilli v i r% mnialeu ve1r byI F-1 pilots anld W eaponsil S%~ stem, O)fficers wasanalyzed by clllect ing det'Iailed rating, ani repolrie e'lrro r- on segni en is of lilt, Inlaneu% erI. I'hle Iiiajtr rI-soih ofIthis ana151si,4 was t hat ra te-i per fo rmatn ce on1 lilt, filnal fewN seconlds o~ft Ile IIIauetuver. ditng wi ic it ec crew istrying it) e st-c iII a ton stanlt anle. hi hlIspeed di% v. is rlearl ilt best l pdjredlictor ofuNlillpllns dhlls% en tl'lrlilTraining imica tStills a1ml issties re-lated to, fl~e u~se of sel f-Ilssessllien t dat 1 are- lriel d isciisetl.

DD I 1473 GEWnDeeFIrd/

"N 7 1-l-if.

2 0 6~~~~~~~~~ ~ ~ ~~~ ,-,,JT LASFCTO F MSAE(hn 31Ftrd

-- - m

SECURITY CLASSIFICATION OF THIS PAGE(Wha Date Entered)

SECURITY CL A.SIFICATION OF THIS PAGE(Whon Del Enlt.ed)

aw

SUMMARY

F-4 Pilots and Weapons Systems Officers (WSOs) from nine missionready squadrons across the United States provided rating and errorevaluation data on their performance of the pop-up weapon-deliverymaneuver. The maneuver was subdivided into eight components.Aircrews assessed their performance on each component of 545 scoredmaneuvers. They also listed causes of less-than-optimal performanceon particular components of each delivery. This information wascompiled and analyzed to determine the relative contribution of eachof the components to the accuracy of weapon delivery.

The major result of these analyses was that rated performance onthe final few seconds of the maneuver, during which the crew is tryingto execute a constant angle, high-speed dive, is clearly the bestpredictor of weapon-delivery accuracy. Several aspects of the datashow this finding is robust and not artifactual. The same result wasobtained for both pilots and WSOs at each wing and for both ratingsand reported error frequencies.

The outcome of this study indicates the analysis ofself-assessment data can, under some conditions, yield importantinformation about the components of a skill. The use ofself-assessment data, however, is not advocated in situations wherereasonably economical and unobtrusive direct measures of performanceare available.

For

B i -- -. -- --

C rv.

1 \or!I

-.--- .--.-.---.. A

PREFACE

This study was conducted by the Operations Training Division, Air ForceHuman Resources Laboratory, Air Force Systems Command. The study supportsProject 1123, Flying Training Development, Mr. James F. Smith, projectmonitor; Task 112302, Training Methods and Media, Dr. Bernell J. Edwards,task scientist; and 11230236, Project SMART, Dr. Edward E. Eddowes, workunit manager.

The authors extend their appreciation to the pilots and weapon systemsofficers of the 334th, 335th, and 336th Tactical Fighter Squadrons of the4th Tactical Fighter Wing, Seymour Johnson AFB, NC; the 68th, 70th, and339th Tactical Fighter Squadrons of the 374th Tactical Fighter Wing, MoodyAFB, GA; and the 428th, 429th, and 430th Tactical Fighter Squadrons of the474th Tactical Fighter Wing, Nellis AFB, NV who provided the information ontactical fighter pilot performance on which this study was based.

-Ti

TABLE OF CONTENTS

Section Page

INTRODUCTION ............................................... 3

USE OF PILOT SELF-ASSESSMENT ................................. 4

TASK DESCRIPTION ........................................... 5

METHOD..................................................... 5

Participants.........................................

Procedure .............................................. 5

Analysis ............................................... 9

RESULTS AND DISCUSSION...................................... 10

CONCLUSIONS................................................ 21

REFERENCES................................................. 23

APPENDIX A................................................. 25

LIST OF FIGURES

Figure P age

1. Pop-Up Weapon Delivery Profile (From Pierce, et al., 1979).. 6

2. Self-Assessment Form for Pop-Up Weapon Delivery ............. 7

3. Instructions Accompanying Self-Assessment Form .............. 8

4. Relationship of Segment Ratings to Bombing Accuracy,All Pilots ................................................... 11

5. Relationship of Segment Ratings to Bombing Accuracy, WSOs... 12

6. Low Angle (100 and 150) Deliveries .......................... 13

7. High Angle (300) Deliveries ................................. 14

8. Relationship of Reported Errors on Parts of the Maneuverto Delivery Failures ........................................ 16

9. Relative Frequency of Track Point and Bomb Run Errors ....... 17

10. Proportion of Marginal and Unsatisfactory Ratings onDifferent Segments of Pop-Up Delivery ...................... 19

2

POP-UP WEAPON-DELIVERY MANEUVER: USE OF PILOT DATAIN ANALYSIS OF CRITICAL COMPONENTS

INTRODUCTION

This report is based on data collected during a large scale testof methodology developed for the Air Force Skill Maintenance andReacquisition Training Program (Project SMART), which has the goal ofdefining and measuring the basic skills which support aircrew missionreadiness. The test was requested by Tactical Air CommnandHeadquarters during a preliminary evaluation of Project SMART. Thepurposes of the report are twofold: (a) present the results of aself-assessment methodology explored by Project SMART to analyze skillin pop-up weapon delivery and (b) document some of the strengths andweaknesses found with this approach. The present report is concernedwith the results and effectiveness of the methodology and a detaileddescription of preliminary development efforts presented in Pierce,DeMalo, Eddowes, and Yates (1979).

In order to adequately assess mission readiness, Project SMART hasfocused its data collection efforts on continuation training. Initialefforts consisted of interviewing pilots at operational squadrons todetermine (a) the tasks most critical to their mission and (b) theimportant parameters underlying performance of these tasks. On the[basis of the interviews, the pop-up weapon delivery (Pierce et al.,1979) and low altitude tactical formation (DeMaio, Eddowes, 1980)tasks were selected for further study. Aircrews were then asked tomake detailed assessments of their own performance on these tasks.This report concerns the analysis of pop-up weapon delivery skillbased on these data.

Collecting subjective data from experts might be expected to havesome inherent advantages. It should have no impact on operationalequipment and does not require long data collection sessions thatmight interfere with scheduling. Because of this, it is sometimesconsidered an acceptable last resort in situations where actualperformance data cannot be collected. Unfortunately, aircrewself-assessment may also be viewed as a potentially misleading sourceof data. Nisbett and Wilson (1977) reviewed a number of studies thatdemonstrate the difficulty of making correct inferences fromsubjective reports, even when such reports do not necessarily implyself-evaluations.

Can pilot self-assessment be a useful source of data forunderstanding the components of mission readiness? Before thespecific case examined in this report is discussed, the general issuesinvolved will be reviewed a little more closely.

43

USE OF PILOT SELF-ASSESSMENT

Consider the following common situation. A pilot performs somemaneuver that can be objectively scored (for example, navigation to apoint, or weapon delivery). The pilot then is informed of the scorereceived. After returning from the sortie, the pilot may be asked for Van evaluation of how well the maneuver was performed. There areseveral ways in which the pilot may arrive at this evaluation. Atbest, the pilot might try to recall the particular characteristics ofthe performance and compare them to some performance ideal. However,the pilot might instead give the answer implied by the score withoutconsidering other aspects of the performance. Or worse, the pilot [

could simply give the evaluation believed to be the most pragmatic,without referring to the actual performance at all.

One way to distinguish between these possibilities is to ask forevaluations of the components of the performance instead of just anoverall evaluation. If the pilot is referring only to informationabout the desirability of the evaluation or the known success of themaneuver, then the evaluation of all the components should covarystrongly, or receive roughly the same evaluation. However, to theextent that the evaluation of different task components showsdifferent characteristics, confidence increases that somecomponent-specific knowledge is being tapped; that the pilot is makingan effort to accurately assess the performance.

Unfortunately, even if different components of performance receivedifferent ratings, this does not prove that the pilot's judgments arebased on accurate memory for performance. Component ratings couldalso be based on r.,constructions of plausible performance. Forexample, if it is part of general pilot lore or accumulated personalexperience that a particular part of a maneuver is the most difficult,self-evaluation of this part might be adjusted accordingly.

Fortunately, there are some non-experimental techniques fordistinguishing between memory for actual performance and plausiblereconstruction. One way is to ask the pilot to recall aircraftparameters (airspeed, altitude, etc.) which characterized eachcomponent. Ideally, these could be compared to the parameters thatthe pilot was trying to achieve. If consistent differences exist,then evidence is gained supporting accurate memory for performance.

Another approach, involving still more detail, is to require thepilot to recall the specific errors made. The lack of reported errorswill not be very revealing, but their presence provides data tocompare with low self-evaluations. As a jeneral principle, one shouldrequire as much detail in self-assessment as the constraints of thesituation will permit.

In the study described here, both component ratings and reportederror data were collected on performance of a short duration,relatively difficult maneuver (pop-up weapon delivery).

4

Wil 4..

TASK DESCRIPTION

The pop-up weapon delivery was identified by pilots very early inthe Project SMART effort as a critical air-to-ground maneuver in thetactical fighter pilot's repertoire. The maneuver is designed to beused after the pilot has flown a low altitude ioute to a target.After reaching a preplanned pulu onthe pilot climbs quicklyand rolls over to an apex altitude and then dives from a prespecif ledposition (called the track point) at a constant dive angle toward therees pit from wiha weapon is released. For-purposes ofanalysiisthis maneuver was broken down into meaningful discretesegments on the basis of interviews with pilots (Pierce et al.,1979). Figure 1, taken from Pierce et al.., shows the resultingsegmentation of the maneuver.

In the present study, mission-ready F-4 pilots were asked to makedetailed assessments of the crew's performance on the aforementionedcomponents of the pop-up delivery. An examination of therelationships between component assessments and accuracy was expectedto isolate the critical components on the maneuver.

METHOD

Participants

Pilots and WSOs from nine operational F-4 squadrons participatedin the study. Squadrons participating were those of the 474thTactical Fighter Wing (TFW) at Nellis AFB, the 347th TFW of MoodyAFB, and the 4th TFW at Seymour Johnson AFB.

Procedure

Pilots and WSOs were asked to make detailed assessments of thecrew's performance on each of the aforementioned segments of thepop-up each time the maneuver was performed. These assessments tookthe form of ratings on a four-point scale (excellent, satisfactory,marginal, and unsatisfactory) of performance on each segment andwritten commnents consisting largely of explanations of errors made oneach segment. In addition, bomb scores (miss distances) were recordedfor each controlled range delivery, and the delivery outcome (bull,hit, miss, dry, or abort) was recorded for tactical range deliveries.

These assessments were gathered on the form shown in Figure 2according to the detailed instructions (Figure 3). During thedebriefing session of daily sorties, approximately 80 percent of theseforms were collected by members of the Project SMART staff, who werepresent to answer questions about the procedure. The remainder of theforms were filled out in the absence of a project researcher by crewswho had nevertheless been formally briefed on the assessment procedure.

w

z M r-

- (

-J '~-C)

A uz

Oz/ m__ __ _ / LUJ

-Ji-

0.~

______ _ U L-its m

POP-UP EVALUATION FOR14r

PILOT 0: __ ____SOUADRON 0- ____ RANGE 11: ________

EVENT:________ BLOCK:_______ DATE-___________

PASS *1st 2nd 3rd 4th

BOMB SCORES

Task Evaluation COMM4ENTSIINDICATE PAS-#.

1. Approach to PUP _____________

2. PUP ___________

3. Climb Leg - - - _ _ _ _ _ _ _ _ _-

4. Target Acquisition - - - -

5. Pull Down Point ___________

6. Apex - - - - _ _ _ _ _

7. Track Point_______________

B. Bomb Run___________ _____

9. Recovery-- _ _ _ _ -

10. Rtn to Low Alt ____________

11. Exposure Time_______ _______

Legend: E - Excellent S -Satisfactory M Marginal U -Unsatisfactory

FIG. 2. SELF-ASSESSMENT FORM FOR POP-UP WEAPON DELIVERY.

7

INSTRUCTIONS

1. COMPLETE THE PILOT IDENTIFICATION PORTION OF THE FORM

2. RECORD BO48 SCORES.

3. GRADE THE TASK EVALUATION SECTION AS FOLLOWS:

E - Excellent. Task performance met criteria with no errorreflecting an-unusually high degree of ability; no compensationswere required.

S - Satisfactory. Task performance met criteria with minimal error;minimal compensations were required.

N - Marginal. Task performance met criteria with error; coepen-

sations were required to salvage the pass delivery.

U - Unsatisfactory. Task performance did not meet criteria; grosserrors in performance led to either an unsafe or aborted pass.

Task evaluations are to be based on 1) proficiency to maneuver theaircraft. ?) situation awareness, 3) aqrensiveness and 4) survivability.

Any item graded as either M or U requires an appropriate explanation under theComments section.

The following indicates those requirements identified with each Itemincluded in the Task Evaluation section:

1. Approach to PUP: (a) Acquisition of PUP; (b) Altitude control;

(c) Airspeed control; and (d) Heading control.

2. PUP: (a) Heading correction; (b) "G" application; (c) Airspeed

correction; and (d) timing/distance error.

3. Climb Leg: (a) Climb angle corrections; and (b) airspeed corrections.

4. Target Acquisition: Self-explanatory

S. Pull Down Point: (a) Roll; (b) Airspeed corrections; (c) *G*application; and (d) Altitude/position control.

6. Apex: (a) Pattern/position correction; and (b) Airspeed corrections.

7. Track Point: (a) Aim off point; (b) Roll out; (c) Initial windcorrection; (d) Angle, azimuth, and position check; and (e) Initialpipper placement.

8. Bomb Run: (a) Aiming error corrections; (b) Airspeed control;

(c) Tracking time control; and (d) Altitude, azimuth, and dive anglecorrections.

g. Recovery: (a) "G" application; (b) Jinking; and (c) Altitude and timing

control

10. Return to Low Altitude: Transition to low altitude.

I1. Exposure Time: Minimizattoi, of total time spent out of low altitudeenvironment.

FIG. 3. INSTRUCTIONS ACCOMPANYING SELF-ASSESSMENT FORM.

8

A very similar self-assessment procedure has been shown to producesegment ratings which, when combined into an overall score for eachdelivery, are related to bombing accuracy of A-7 pilots (Pierce etal., 1979). However, due to a Tactical Air Command requirement thatthis research minimize interference with the normal training routine,it was not possible in either the Pierce et al. (1979) study or thecurrent one to withhold from the crewmembers knowledge of their actualbomb scores. Thus, the simplest hypothesis to explain any overalltendency for higher ratings and low bomb scores to be related is thatthe ratings do not reflect actual performance on individual segments,but rather are based on retrospective inferences from the success orfailure of the delivery. This explanation predicts that there shouldbe no systematic differences in the characteristics of the ratings fordifferent segments. However, the present report contains analyses ofthe segment ratings which show very sharp differences betweensegments. Such differences reflect the use of segment-specificinformation in producing segment ratings.

Analysis

To look directly at the extent to which rated performance on eachpop-up segment was related to bombing accuracy, the following analysiswas performed. First, bomb scores for the controlled range deliverieswere categorized as bulls, hits, or misses according to the criteriaused for this categorization by the squadrons themselves. Thesecriteria (given in Appendix A) differ for different events and are thesame as the criteria by which the tactical range scores arecategorized. Such categorization allows scores for tactical andcontrol ranges and scores from different events to be analyzedtogether, as well as allowing separate analyses to be directlycompared. Next, categorized bomb scores and ratings for each of thefirst eight segments of the pop-up (approach to pull-up point throughbomb run) were cross-classified. In order to obtain sufficient cellfrequencies for a chi-squared test, bomb score categories werecollapsed so that bull and hits were in one category and the misses,dry passes, and aborted runs were in another. Also, ratings werecollapsed into two categories with ratings which do not reflectsignificant error (excellent and satisfactory) in one category andmarginal and unsatisfactory ratings in the other. Chi-squared valuesand values of the chi-squared contingency coefficient (o) werecomputed on the resulting ? x 2 matrices for each of the first eightsegments of the pop-up (the segments that precede the bomb delivery).

Another source of data on the question of the criticality ofpop-up segments is the comments which supplemented the segment ratingson about one-third of the deliveries. Many of these commentsmentioned specific errors made on a particular segment, and there werea sufficient number of errors mentioned to allow computation of theproportion of deliveries which resulted in misses or dry or abortedpasses, given that a particular kind of error was reported. This

9

0 =-

....................................................... ***L-

proportion indicates the severity of the error. It can be comparedwith errors introduced on different parts of the maneuver. Errorswhich were clear consequences of errors made earlier in a deliverywere not included in this analysis.

A number of other analyses, conducted to examine more specificissues, are described in the results section. Because of the numberof statistical tests conducted, the criteria for significance used arep .0025 for each of four tests of entire distributions and p .001for each of 37 tests of specific segment/bomb score relationships.These values yield an overall probability (of accepting a chanceresult as significant) of less than 0.05.

RESULTS AND DISCUSSION

The pilots' ratings of segment performance are considered first.Figure 4 shows the obtained values for the strength of associationbetween segment ratings and bombing accuracy for each segment of thepop-up, based on 545 scorable deliveries from nine operational F-4squadrons. The overall pattern shows that performance ratings on thefinal two segments before ordnance delivery, i.e., track point andbomb run, have by far the strongest relationship to bobaccuracy.TRatITnby the WSOs for the same deliveries show substantially thesame pattern, though the association of ratings with accuracy issomewhat lower. These data are shown in Figure 5. WSOs do not get asclear a view of the maneuver as does the pilot; nor are they directlyinvolved in controlling the aircraft. Therefore, the more detailedanalyses reported below were conducted on the pilot data only.Nevertheless, the WSO data confirm that track point and bomb runratings are the best predictors of accuracy.

Much the same pattern of results holds for maneuvers with bothhigh (300) and low (100 to 150) delivery angles. Figures 6 and7 show the results of a breakdown of the data into high versus lowangle deliveries. The figures show that for low angle deliveriesonly, the predictivity of the tage acquisition rating (segment 4) issignificant. Unfortunately, on this one segment for this partitioningof the data, there are too few observations in one of the cells of thematrix to ensure that the chi-squared estimate of significance isaccurate. However, the main result is clear: track point and bombrun segments were the best predictors of accuracy for both kinds ofdeliveries.

10

C-J

CO-

00

0 C',

I.-.

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 0 O

CQ 0

.. . . .. ... 00c

51c uj.

162

.............. ....... Cl

o CDCL,

I 0:.

0

CHl

CDwU,

zi 0CL

0 .0

C0 0 0

Aminflov SVWd0 01 dIHSN0lLV13H 40 IONUIS

400

CL

0

CLJ

Ir 0...... ..... z

CLL

0

00 0 %.0 d

O0) AmViflov OV409 01 dINSNOliV-13W JO HJ.9N31S

13

Jig,

....... .. .. ....

01 W

OD w

Ii z40 (9

CL I .. . . .

((9

Q 1 CL

0 0 to

(0)vm3) AQWf0VSII 01 dIH4SNlIV13jj JO HION381S

14

There are several possible causes of the main results of thisstudy. An immediate concern is that the pattern is an artifact of theease with which different segments can be rated. It might besupposed, for example, that track point and bomb run performanceratings are more accurate and internally consistent than are theratings of performance on other segments and that it is these ratingcharacteristics, rather than actual segment performance, which causethe observed pattern. However, the importance of the track point andbomb run to overall accuracy appears in an analysis of reported errorsas well as in the analysis of ratings. Figure 8 gives the proportionof deliveries that resulted in misses or dry or aborted passes when anerror was reported for a particular part of the delivery. Since therewere few errors reported for some segments, segments were pooled inways that preserved a meaningful partition of the maneuver. Thenumber of errors in each of the pooled segments are noted on thefigure, which also shows that errors reported as occurring during thetrack point and bomb run are associated with an increase in theproportion of bombs off target as compared t?2 earlier stages of thedelivery. This difference is significant (=10.0, p .0025).The severity of track point and bomb run errors can be appreciated bynoting that, while the overall probability of failing to hit thetarget is 0.37, this probability rises to 0.80 when a specific erroron track point or bomb run is reported.

These data can also give us some idea of the factors which accountfor less than ideal track point and bomb run execution. Figure 9shows the relative frequency of various kinds of reported errors intrack point/bomb run. This distribution of errors differssignificantly from an equal-frequency distribution ( 2 = 36.8,p .001). The tab'.e shows the failure to correct properly for windon the bomb run was the most common error, accounting for 30% of theerrors reported. By contrast, wind correction was mentioned only onceamong the 169 errors reported for the segments preceding track pointand bomb run.

The data, both ratings and errors, indicate that the track pointand bomb run segments of the maneuver are the most predictive of bombaccuracy. There are at least three possible reasons for this result.

One possibility is that the greater predictivity of the finalsegments is due in part to earlier segments. It is possible that poorratings on track point and bomb run reflect errors made on earliersegments as well as track point/bomb run errors. This hypothesisimplies the predictivity of ratings of track point/bomb runperformance should drop when only deliveries in which high ratingswere given to all earlier segments are considered. Table 1 comparesthe predictivity of track point and bomb run ratings for all the datawith that of a subset of the data which meets this condition. As thistable indicates, the relationship between track point and bomb runratings and accuracy does not decrease for runs in which earlyperformance is good. If anything, the relationship decreases whenearlier segments were poorly executed. Thus, early errors are not thesource of the predictivity of track point and bomb run ratings.

0 .:.............. ........................... . . . . . .

a0...........

. .. . . . . . . . . . . . . .. .. C.... .... ... .... ...

w

.... .... ....... .. ... .. ... ..... .. .. ... .. ..... .. ... .. ... .0..........

.... .... .... ...-

.... ... ... ... ...... ... ... ... ............ (1) . . .. 0U...... ...... ...... .. .. .. .. .. .. :..... .... ... 0..... .....

Z 0

....... .... ~....L LL

.. . . . .. . . . . O ..w Z > z0" 0

cr z0..... .... .... ....

... .. .. ... .. ..J~O9W ~O '.....................O~

.. . . .. . . . ... 6

Ur0

wwLAJ 0 >

Doa

> 0W 0

cr w(I. Z c , .-

0 w

> 0

0 LL

* ___0

4 1''c-

Table I

Strength of Relationship to Bomb Accuracy (o) of Track Point andBomb Run Ratings (Segments 7 & 8) for Different Subsets of Deliveries

Track Point Bomb Run

All Deliveries 0.30 0.40

Deliveries in which all earliersegments were rated satisfactoryor excellent. 0.34 0.44

Deliveries in which at least oneearlier segment was rated marginalor unsatisfactory. 0.19 0.30

Two other possible reasons for the greater predictivity of thesefinal segments are (a) they might be the most difficult parts of themaneuver to perform, and (b) independent of their relative difficulty,errors made in the final few seconds of the maneuver necessarily leaveless time available for corrective action than do errors made earlierand, therefore, may be the most likely to go uncorrected and producemisses.

There is no direct index of the relative difficulty of the partsof the maneuver. Figure 10 shows the proportion of marginal andunsatisfactory ratings given for each segment. The final two segmentsreceived a significantly larger proportion of marginal andunsatisfactory ratings than did the first six segments ( 2 = 144,p .001). If this proportion is taken as an index of difficulty(Pierce et al., 1979) then the match between predictivity anddifficulty is quite good. However, these ratings might be reflectingboth differences in the ease with which segments are performed anddifferences in the pilot's criterion for acceptable performance, whichin turn could be affected by the knowledge that errors in the bomb runand track point segments are difficult to correct before delivery.

Another possible index of difficulty is the number of reportederrors for a given segment. Track point and bomb run account for 39%of all reported errors, significantly more than their share( 2 31.5, p .001). This suggests that this part of themaneuver may in fact be the most difficult.

18 .. ,

In-

z0

. .... ..... V

0

N- z

SONIIVJ~~~~~~~ C).~4IV~lUN ~I~V*

CL9

Wt-----------------------------------------

A final piece of relevant information is the degree ofindependence of ratings of track point and bomb run performance. Dothe contributions of these two segments to bomb accuracy reflect acommon factor (such as being near the end of the maneuver) orrelatively independent skills? This question can be addressed bymaking use of the fact that the index of association between twocategorical variables, each with two levels, is equivalent to thecorrelation between those variables when the categories within eachvariable are assigned arbitrary "scores," i.e., 1 and 0 (Hays, 1973,p. 744). The particular scores chosen have no effect on thecorrelation, since it is impossible to produce a nonlineartransformation on two scores.

Using this equivalence, a multiple regression analysis can beperformed with those variables which have already been shown to have asignificant association with bomb accuracy via the chi-squared test.Table ? presents the results of this analysis. The analysis showsthat the predictive utility of track point rating is almost entirely aresult of covariation with bomb run ratings. The partial correlationbetween track point ratings and accuracy with bomb run ratings heldconstant is insignificant, and the addition of track point rating tothe prediction of accuracy increases the multiple correlation onlynegligibly. The results of this analysis are consistent with thehypothesis that both track point position and bomb run performancedepend on a common factor.

Table 2

Joint and Partial Prediction of AccuracyUsing Track Point and Bomb Run Ratings

TP BR A

Track Point Rating (TP)

Bomb Run Rating (BR) 0.52*

Bomb Accuracy (A) 0.30* 0.40*

Multiple R (TP and BR predicting A) = 0.41*

Partial R (TP predicting A holding BR constant) = 0.11*

Partial R (BR predicting A holding TP constant) = 0.29*

*Fnc iiesfqi-cantcoreati -ion-.

: 20

CONCLUSIONS

The goal of the research reported here was to determine theeffectiveness of the self-assessemnt methodology explored by ProjectSMART. Results of a large scale test indicate that the methodologywas successful in isolating the most critical elements in determiningweapon delivery accuracy for the pop-up maneuver, i.e., track pointposition and bomb run.

The primacy of the track point/bomb run portion of the maneuverwas shown both in the rating data and in reported errors (used topredict actual bomb scores). This result was highly significant andwas obtained for both pilots and WSOs in each of three different F-4wings across the country. The data also allowed the relativeFrequencies of occurrence of the major reported track point/bomb runerrors to be categorized and compared.

It should be emphasized that these results apply only toexperienced pilots on familiar training ranges, where navigationproblems are minimal. It remains to be seen whether they aresustained under "high-threat," novel environments. Also, data arerequired from a wider range of tactical tasks, such as air combatmaneuvers, to determine how skill on the pop-up maneuver fits into theoverall proficiency of the misc ion-ready pilot.

It is clear that pilot self-assessment can be a useful source ofdata in identifying critical aircrew skills. The data obtained heresuggest that pilots, when appropriately briefed to reduce extraneousdemand characteristics of the self-assessment task, will make diligentattempts at reporting their performance accurately. And, theconsistent trend across both ratings and error data suggests thatpilots do reliably' discriminate among individual segment performancesof the pop-up maneuver.

The present results, when combined with the conclusions of anearlier examination of student weapon delivery skills using a similarmethodology (Pierce et al., 1979), provide a reasonable picture of theparts of the maneuver which contribute most to overall skill atdifferent stages of mastery. Pierce et al. concluded that, withpractice, F-4 students improved their execution of the initialsegments of the maneuver most. The final track point andJ oib runsegments, however, remained the most difficult parts throughouttraining. The data suggest that as aircrews become accurate enough toqualify as mission ready, they can execute the initial parts of themaneuver well enough so that. !errors in those segments do not greatlyaffect bomb scores. For th,?;e experienced pilots, most of the valueof pop-up training drills consists of the opportunity to practice thecritical final seconds of the maneuver. Therefore, it is possiblethat a device which simulites the rapid correction of position andaiming errors required in :he final run could be of some benefit forcontinuation training.

21

The pilot self-assessment approach yielded a stable and importantresult in this case; however, some of the advantages that might beexpected from self-evaluation techniques were not evident. Forexample, it was mentioned earlier that getting subjective assessmentsought to be a relatively economical way to collect data since onlyminimal equipment is required. However, if a researcher must bepresent after each flight to collect the data, the expense is stillconsiderable. During this test of the methodology, some squadronswere designated to receive full-time researcher coverage, while otherswere briefed on the data collection method and were asked to collectdata on their own for half or all of the one month test period. Thegeneral result is that about three times the number of data forms werereceived when a researcher was present as compared to voluntaryparticipation by the squadrons.

A related problem is that, even though the data forms used couldbe filled out in a minute or so, some pilots appeared to resent thisaddition to their paperwork load. This might be a problem in usingthe forms on a day to day basis; an even less obtrusive method wouldbe desirable.

To conclude, the present analysis demonstrates that usefulinferences about the components of task performance can be made frompilot self-assessment data; however, self-assessment may not be themost feasible method when reasonably economical and unobtrusive directmeasires of performance are available.

22

REFERENCES

DeMaio, J. C., & Eddowes, E. E. Airborne performance measurementassessments: Low altitude tactical formation in two operatingenvironments. AFHRL-TR-79-44. Williams AFB, AZ: OperationsTraining Division, Air Force Human Resources Laboratory, July 1980.

Hays, W. L. Statistics for the social sciences (2nd ed.).New York: Holt, Rinehart & Winston, Inc., 1973.

Nisbett, R. E. & Wilson, T. C. Telling more than we can know. Verbalreports on mental processes. Psychological Review, 1977, 84,231-259.

Pierce, B. J., DeMalo, J. C., Eddowes, E. E., & Yates, 0. Airborneperformance measurement methodology application and validation:

pop-ue training evaluation. AFHRL-TR-79-7, AD-A072 611.Williams AFB, AZ: Flying Training Division, Air Force HumanResources Laboratory, June 1978.

23

APPENDIX A

MISS DISTANCE CRITERIA FOR CATEGORIZING BOMB SCORESFOR DIFFERENT EVENTS (METERS)

EVENTLow Angle Low Angle Dive Dive

Category Bomb Low Drag Bomb Toss

Bull 4.6 m6m 4.6 m 4.6 m

Hit 32 m 53 m 44 m 50 m

25 *U S owwm'ei, ?I~~~' 7~h/

L A A 9 2 9 D A Y T O N U N I V O N R E S E A R C m I N S T A -

POP-UP WEAPON-DELIVERY MANEUVERi USE OF PILOT SELF-ASSESS,UN'L AG 0 R LYON. J L EUaANKS, T I RILION F331IB. ... S.E,,UNCLASSIV7ED MAX R LyN.N .

F-NL D.R.CNL

UPPLEMENTARY

INFORMATION

I o

DEPARTMENT OF THE AIR FORCEAliR FORCE HUMAN RESOURCES LABORATORY IAFSCf

BROOKS AIR FORCE BASE, TEXAS 78235

ftert 10Ann OF: TSR 16 JAN 1981SwancT. Removal of Export Control Statement

To: Defense Technical Information CenterAttn: OTIC/ODA (Mrs Crumbacker)Cameron StationAlexandria VA 22314

S 1. Please remove the Export Control Statement which erroneously appears onthe Notice Page of the reports listed no"~.This statement isintended for application to Statement B reports only.

~Z 2. Please direct any auestions to AFHRL/TSR, AUTOVON 240-3877.

FOR THE COMMANDER

WENDELL L. ANDERSON, Lt Col, USAF I AtchChief, Technical Services Division List of Reports

Cy to: AFHRL/TSE