[american institute of aeronautics and astronautics aiaa atmospheric flight mechanics conference and...

Piloted Evaluation of Degraded-Mode Handling Qualities

David G. Mitchell *Hoh Aeronautics, Inc., Lomita, CA 90717

David H. Mason †The Boeing Company, Philadelphia, PA

Jerry M. Weakley ‡ and Kristi M. Kleinhesselink §Naval Air Systems Command, Patuxent River, MD

Methods for the piloted evaluation of aircraft handling qualities are becoming better defined and more standardized. Typically, such evaluations are limited to the fully operational vehicle. Assessment of the effects of degraded modes (failure states) has historically been limited to estimating the expected degradation in handling qualities resulting from the failures. Little simulation or flight test effort has been dedicated to verifying the degradation in handling qualities. This paper discusses the justification for performing a formal pilot-in-the-loop simulation program, following the protocols developed for fully-operational systems, to determine the handling qualities of a degraded aircraft. Such a program must include mission tasks representative of those required for the operational aircraft, even if such tasks are not expected to be performed by the degraded aircraft. Experience with a series of piloted simulations using the V-22 Osprey tiltrotor aircraft are used to illustrate the process.

I. Introduction he typical approach to handling qualities testing is to assume an unfailed aircraft. When there are no failures, the aircraft can be evaluated using standardized piloting tasks and pilot rating methodology. T

A. Piloting Tasks For more than a decade, work has been underway to develop and define a standard set of piloting tasks for the

evaluation of aircraft handling qualities. Starting in the mid-1980s, a U.S. Army-led effort resulted in a new rotorcraft military specification, Airworthiness Design Standard ADS-33.1 A critical part of ADS-33 has been a set of tasks, referred to as Mission-Task-Elements (MTEs), specific to rotorcraft.

For fixed-wing airplanes, initial work consisted of a McDonnell Douglas-led effort in the early 1990s to develop a Standard Evaluation Maneuver Set2 (STEMS). The STEMS maneuvers are geared primarily toward fighter airplanes and high-angle-of-attack maneuvering, however, so further U.S. Air Force-sponsored work was carried out by Systems Technology, Inc., and Hoh Aeronautics, Inc., in the late 1990s. The result of these companies’ work was a compilation of tasks from research organizations such as Calspan, classical closed-loop tasks such as attitude captures, and several new tasks. The Demonstration Maneuvers catalog3 consists of 36 maneuvers, again focused on fixed-wing airplanes.

For both the fixed-wing Demo Maneuvers and the rotary-wing MTEs, there is a description of the maneuver itself, including requirements for outside visual cues if appropriate. Performance requirements are defined for desired and adequate performance, as required for application of the Cooper-Harper Handling Qualities Rating scale (Figure 1).4

* Technical Director, Associate Fellow AIAA. † Boeing V-22 Flying Qualities IPT Lead. ‡ Aerospace Engineer, AIR 4.3.2.4; currently assigned to Naval Air Mediterranean Repair Facility, Naples, Italy. § Aerospace Engineer, AIR 4.3.2.4.

American Institute of Aeronautics and Astronautics

1

AIAA Atmospheric Flight Mechanics Conference and Exhibit16 - 19 August 2004, Providence, Rhode Island

AIAA 2004-4704

Copyright © 2004 by Hoh Aeronautics, Inc. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission.

B. Piloting Tasks in the Presence of Failures In the course of task development, the principal parties responsible for both the fixed-wing Demo Maneuvers

and the rotary-wing MTEs explicitly considered only fully operational (unfailed) aircraft. For determination of pilot performance requirements, however, it was common in both programs to employ aircraft designs (using variable-stability aircraft or simulation models, not actual operational vehicles) that are representative of failed aircraft.

As an example, the only way to assure that adequate performance limits (meaning the aircraft is no worse than an HQR of 6) are properly defined is by having several pilots fly the task in question with a simulated aircraft that has poor inherent flying qualities characteristics. In such a case, it is not important to determine why a comparable actual aircraft would have such poor flying qualities characteristics – whether by bad basic design or as a result of some failure – but only to know how bad the handling qualities have become.

C. V-22 FQDM Simulation Starting in 2001, a series of piloted simulations was performed to evaluate the pilot-in-the-loop handling

qualities of the V-22 Osprey following one or more failures in the flight control system. The details of these simulations for Flying Qualities Degraded Modes (FQDM) are documented elsewhere5,6 and will be mentioned only briefly in this paper. The interest here is in the process for degraded-mode testing that was developed for the V-22 FQDM simulations.

At the initial planning stages, the technical question to be answered seemed simple enough: what is the impact of failures on flying qualities? There is little in the open literature about the proper method for such flying qualities evaluations. The focus, after all, would normally turn from traditional “flying qualities” to more general concerns about controllability and suitability for safe flight to a landing. Continuation of the primary mission is neither required nor expected, so are “flying qualities” even relevant?

The initial technical question, then, became a key focus for the V-22 FQDM simulation. By using flying qualities testing to evaluate a degraded aircraft, a new approach had to be developed. This new approach to flying qualities testing proved to be very successful and well-received by the piloting and engineering communities.

II. Handling Qualities Testing with Degraded Modes Perhaps not surprisingly, there is considerable controversy about the validity of formal handling qualities testing

for an aircraft with known failures, especially if those failures are expected from the outset to degrade flying qualities to the extent that controllability would be in question. This section outlines the philosophy for handling qualities testing in general, for both fully operational and failed-state aircraft.

A. Philosophy for Handling Qualities Testing As defined by Cooper and Harper, handling qualities are “those qualities or characteristics of an aircraft that

govern the ease and precision with which a pilot is able to perform the tasks required in support of an aircraft role.” Handling qualities (or more generally, flying qualities) of an aircraft may be evaluated by two methods:

1. Obtain parameters that define the response of the aircraft to pilot control inputs, and compare these parameters with established flying qualities criteria. (The military rotorcraft Aeronautical Design Standard ADS-33E-PRF refers to the estimates resulting from this process as “predicted handling qualities.”1)

2. Have test pilots perform defined tasks and assign HQRs, using the scale in Figure 1. (ADS-33E-PRF refers to these as “assigned flying qualities.”1) Ideally, aircraft receive HQRs that are Level 1 by the definitions of the military specifications: HQRs of 1, 2, or 3. Occasional Level 2 HQRs – 4, 5, or 6 – are accepted, depending upon the operating environment and MTE. Aircraft that are operating normally should not receive HQRs of 7 or 8 (Level 3), and never require intense pilot compensation just to retain control (HQR of 9) or be uncontrollable (HQR of 10).

Neither of these methods is meant to specifically exclude the assessment of handling qualities in the presence of failures. Indeed, Cooper and Harper recognized that handling qualities evaluations may routinely include failed-state mission tasks: “Unless … alternatives are spelled out in the task definition, … the pilot must always treat a failure state as having to be coped with for the duration of the task…. Normal and emergency states then will require separate evaluations.”

In practice, application of the HQR scale requires a clear definition of the task and expectations for performance of that task. Limits on desired and adequate performance are specified so that the pilots can properly interpret the Figure 1 scale. It is clear that the handling qualities assessment tasks must contain good task and performance definitions.

There is little controversy, within or outside the handling-qualities community, about the value of using well-defined MTEs. The specifics of those MTEs, however, can prove to be the major issue: given the possibly severely


2

impaired state of the aircraft, is it reasonable to impose task requirements that are difficult, if not sometimes impossible, to achieve? Surely, in the event of extremely improbable failures, it is not fair to demand the successful performance of MTEs that are normally required of the unfailed aircraft? As a practical example, an offset approach to a precision landing on a runway is an accepted, well-defined task for handling qualities evaluations, but most pilots are unlikely to attempt a such a task following one or more failures.

There are at least six clear advantages to performing the same well-defined handling qualities tasks for the unfailed and failed aircraft:

1. Task consistency. It is a way to attain a consistent level of performance from the pilots. Without a well-defined task, there is a risk of getting widely varying answers. Tightly defined tasks impel the desired sense of urgency (precision or aggressiveness). Pilots are made aware of what is expected of the aircraft, even if desired performance cannot be attained. All pilots will know the target performance level; attempting to attain desired performance helps guarantee a high level of urgency.

2. Real-world variability. The preci-sion and aggressiveness defined in handling qualities tasks serves as a surrogate for some of the variables that may be encountered in the real world. For example, an experienced pilot will try to stay out of the control loop with a failed aircraft, but may inadvertently tighten up in turbulence. Rather than try to replicate all possible levels of turbulence, structured tasks serve to induce a similar high-gain activity.

3. Consistency with the military specifications. The rotorcraft specification ADS-33E-PRF, the now-retired fixed-wing airplane specification MIL-F-8785C,7 and the vertical/short takeoff and landing aircraft specification MIL-F-833008 have explicit requirements to determine the flying qualities of the aircraft with failures. (The latter two documents allow a degradation in flying qualities

4. Quantification of the degradation. By obtaiquantify the relative degradation in handling quaMTEs. HQRs for the baseline aircraft can be usesame MTE – to determine the change in aircraft h

5. Relation to task definitions. The knowledgWhen handling qualities tasks are developed inrange of aircraft dynamics from known Level 1models evaluated in the task development procewith a tolerable pilot workload. If a different aircperformance with a tolerable workload – but conaircraft.

6. Relation to task difficulty. Relaxing task rethe severely crippled aircraft could receive HQRthose for the unfailed aircraft. For the precision la“touch down within an area that is X feet on eacdifficult in the simulator than in flight, so Level 2performing the tighter task. Artificially opening tcould lead a pilot to assign a comparable – configuration.

American Institu

pilot decisions

Moderately objectionabledeficiencies

ExcellentHighly desirable

GoodNegligible deficiencies

Fair - Some mildlyunpleasant deficiencies

Pilot compensation not a factor fordesired performance

Pilot compensation not a factor fordesired performance

Minimal pilot compensation required fordesired performance

1

2

3

4

5

6

7

8

9

10

Minor but annoyingdeficiencies

Very objectionable buttolerable deficiencies

Desired performance requires moderatepilot compensation

Adequate performance requiresconsiderable pilot compensation

Adequate performance requiresextensive pilot compensation

Major deficiencies

Major deficiencies

Major deficiencies

Adequate performance not attainable withmaximum tolerable pilot compensation.Controllability not in question

Considerable pilot compensation is requiredfor control

Intense pilot compensation is required toretain control

Major deficiencies Control will be lost during some portionof required operation

Yes

No

Yes

No

Yes

No

ADEQUACY FOR SELECTED TASK OR REQUIRED OPERATION

AIRCRAFTCHARACTERISTICS

DEMANDS ON THE PILOTIN SELECTED TASK OR REQUIRED OPERATION

PILOTRATING

Is adequateperformance

attainable with a tolerablepilot workload?

Deficiencieswarrant

improvement

Deficienciesrequire

improvement

Is itsatisfactory without

improvement ?

Is itcontrollable?

Improvementmandatory

Figure 1. Cooper-Harper Handling Qualities Rating (HQR) Scale4

based on the probability of occurrence of the failure.) ning HQRs for both the unfailed and failed aircraft, it is possible to lities. HQRs can be obtained only through performance of defined d to compare against the HQRs of the degraded aircraft – for the andling qualities due to the failure or failure combination. e that originally went into the development of the task is retained. a simulator (either ground-based or in-flight), there is usually a through expected Level 3, based on analytical criteria. Level 3 ss were not able to meet the adequate performance requirements raft is flown through the same task and is unable to meet adequate trollability is not in question – then from experience it is a Level 3

quirements can open the possibility of a very undesirable dilemma: s that are not much worse than, or possibly are even better than, nding example, suppose the performance limits were relaxed from h side” to “land on a runway.” Precision landing is typically more HQRs (4, 5, or 6) would not be surprising for the unfailed aircraft he performance limits, to accommodate the presence of the failure, or better – HQR for what might be an almost uncontrollable

te of Aeronautics and Astronautics

3

B. Interpretation of Handling Qualities Ratings The use of established handling qualities tasks means that the Cooper-Harper HQR scale of Figure 1 can be

retained for piloted assessment. Depending upon the severity of the failure or failures, however, it must be expected that occasional HQRs assigned by the pilots would be in the Level 3 region, or worse (HQRs of 7, 8, 9, or 10). As a result, it is necessary to interpret the structure of the HQR scale in a rather novel manner.

If the probabilities of occurrence of the failures are extremely remote, a rating of 7 may actually be considered to be a “passing” grade: controllability is not in question. The wording for 8 or 9 is not as clear, though either rating is indicative of an extremely impaired aircraft: The aircraft is controllable, but just barely. The Cooper-Harper scale makes a clear distinction for a rating of 10: the aircraft is not controllable.

If it is expected that some of the worst-case failures might result in HQRs in the range of 7 or worse, a formal process is needed to determine the consequences of such ratings. For the V-22 degraded-mode simulation, a flowchart was devised that defined all possible permutations of HQRs from two pilots, and indicated if an evaluation by a third pilot would be required. This flowchart is shown in Figure 2. The most significant path in the flowchart is for HQR of 10: any rating of 10 required a third evaluation. The bottom of the flowchart indicates the resulting recommendation for real-world operations.

Pilot #1 MTEEvaluation

HQR1-6

HQR7-9

HQR10

HQR1-6

HQR7-10

HQR1-6

HQR7-9

HQR10



HQR1-6

HQR7-9

HQR10


HQR1-6

HQR7-10


HQR1-6

HQR7-10


HQR1-9

HQR10


HQR1-9

HQR10


Acceptable

with W

arnings

Not

Acceptable

Acceptable

Acceptable

Acceptable

Acceptable

with W

arnings

Not

Acceptable

Not

Acceptable

Acceptable

with W

arnings

K

Figure 2. A Flowchart for Interpretation of HQRs5

III. Challenges and Resolutions Not surprisingly, the novel approach outlined in this paper leads to a number of significant challenges, beyond

the fundamental issues discussed above. This section reviews the most interesting of those challenges and outlines the methods that were developed in the V-22 degraded-modes simulation for their resolution.


4

A. Failure Insertion Point While the primary interest in a formal handling qualities evaluation is in the residual handling qualities for the

degraded aircraft, it is valuable to introduce the failures during normal piloted operations, if at all possible. This provides two benefits. 1) It allows an assessment of the transients following the failures. 2) It allows the pilots to experience the specifics of the failures, including not only the effects of the failures on aircraft response characteristics but also the formats of the aural and visual cockpit indications of the failures.

The problem is, how could failures be safely inserted in a ground-base simulation, whose focus is on degraded modes, and have it be a surprise to the pilots? Also, how confident can we be that the failures are introduced at the critical flight condition?

The potential matrix of failures, critical flight conditions, pilot operations, etc., makes it difficult to define a set of insertion points that will guarantee an absolute worst-case scenario during any and all flight scenarios. To optimize the coverage of expected worst-case conditions while minimizing the number of evaluations, for the V-22 investigation the failure insertion evaluation was conducted at a predetermined portion of the flight profile that was expected to be critical for that failure by varying gross weight, center of gravity, nacelle, altitude, airspeed, load factor, climb/descent, etc. For example, if the failure included loss of the rudder, the most critical condition might be at high altitude, heavy weight, and aft CG.

With the exception of failure insertions during level acceleration maneuvers or precision hover, failure insertions during dynamic maneuvers were not evaluated because they were not relevant for the specific failures evaluated. Obviously, if dynamic maneuvering is a critical factor in the failure under investigation, the introduction of the failure should be during such maneuvering.

B. Assessment of Failure Transients Even with a high-fidelity aircraft model and a large-motion simulator, it would be difficult to impart to the pilot

the transient effects resulting from failure insertion in the real aircraft. With a fixed-base simulator, it is even more challenging.

Researchers at NASA Ames Research Center developed a transient effects rating scale several years ago.9 This scale was adopted for the V-22 degraded-modes simulation, but with some modifications. The modified scale is shown in Figure 3. Changes to the original scale consisted of the following:

1. The questions in the left-hand decision tree were re-worded for consistency with the Cooper-Harper scale. The bottom question originally read, “Was recovery impossible?” and a “No” answer led the user further up the scale. The second question read, “Was safety of flight compromised?” It was felt that, since most pilots would consider a failure to be a compromise to the safety of flight, it would be difficult to get a “No” answer to this question, so it was changed to, “Were transients and recovery safe?” This also allowed a “Yes” answer to proceed up the scale, again in the fashion of the Cooper-Harper scale.

2. References to the Operational Flight Envelope (OFE) in the original scale were changed to the Service Flight Envelope (SFE) in Figure 3. This was justified since the concern for the OFE is to maintain Level 1 flying qualities in the absence of failures. For the SFE, the concern is to maintain Level 2 flying qualities, and more importantly to not exceed the limits of the SFE for any extended period of time. The SFE limits are defined in terms of aircraft performance rather than flying qualities.

Following insertion of the failure or failures, the evaluation pilot uses the Figure 3 scale to assign two letter ratings: a rating for the effect of the failure (left column) and a separate rating for the ability to recover (right column). In some cases the two ratings may be the same, but they do not have to be the same.

As with the Cooper-Harper HQR scale, the Failure Transient/Recovery Rating Scale in Figure 3 has some inconsistencies, but also like the HQR scale, it serves to elicit comments from the pilots that help quantify the observed transients resulting from the failures. This scale should be used in future simulation and flight programs.

C. Minimum Pilot Population As with more conventional handling qualities experiments, there is always a concern about the minimum number

of pilots. Ideally, a meaningful population (in a flying qualities sense – not a human factors sense!) is five or more. Realistically, many projects have been performed with just two or three pilots, and occasionally a single pilot has been used for critical evaluations of handling qualities.

For the V-22 study, it was decided from the beginning that the goal would be to have every failure and its associated tasks evaluated by a minimum of two pilots. Depending upon the ratings from those pilots, a third pilot could be required, as dictated by the flowchart in Figure 2. In total, nine different pilots participated in the simulation, but the flowchart process was followed in all cases.


5

Failure occurs

Was recoverypossible?

Weretransients and

recoverysafe?

Catastrophic loss of control, encounter with obstacles,or structural failure

No possibility of avertingcatastrophe

Excursions in aircraft statesmay result in encounter withobstacles, unintentionallanding, or exceedence ofSFE boundaries

Recovery marginal; saferecovery cannot beassured even with maximumpilot attention

Successful recovery verydependent on immediatecritical control action withmaximum pilot attention

Excursions in aircraft statesor controls very objection-able, or aircraft SFE limitsapproached

Objectionable excursions inaircraft states or controls --SFE exceedence not a factor

Excursions in aircraft statesor controls moderate butnot objectionable

Minor excursions inaircraft states

Minimal excursions inaircraft states

Corrective control actionnot required

Corrective control inputsaccomplished withminimal urgency

Corrective control inputsaccomplished with moderatesense of urgency

Corrective control actionrequires immediate andconsiderable pilot effort

Corrective control actionrequires immediate andextensive pilot effort

Effect of Failure Ability to Recover Rating

A

B

C

D

E

F

G

H

Yes

No

Yes

No

SFE: Service Flight Envelope

TolerableIntolerable

Figure 3. Failure Transient/Recovery Rating Scale (Modified from Hindson et al.9)


6

D. Apparent Conflict in the HQR Scale The novel approach outlined in this paper led to exposure of an apparent conflict in the Cooper-Harper HQR

scale (Figure 1): what if the aircraft is so severely degraded that the task simply cannot be performed? A prime example for the V-22 study was a drogue tracking task that required the pilot to join up with a simulated KC-130 tanker and stabilize behind the refueling drogue. With some of the most severe failures it was not possible to control the aircraft sufficiently to move into position, even if real loss of control was not in question.

The HQR scale was not meant to be applied in such situations. While it might be possible that the “required operation” for a degraded aircraft could someday include in-flight refueling, the severity of some failures will dictate a requirement to land as soon as possible; if in-flight refueling is essential for survival, the pilots of the crippled aircraft and the tanker are likely to attempt extreme changes in flight condition, configuration, etc., in order to achieve a successful tanking. If the objectives of this paper are to be met, none of these changes can be allowed – the pilots will attempt to rendezvous with the tanker, at a predefined set of flight conditions and task tolerances, no matter what the failure.

Early in the V-22 simulation program, several pilots argued that since the airplane was not sufficiently controllable to even perform the task – much less to try to achieve at least adequate performance – the assigned HQR must be a 10. These pilots did not actually lose control of the aircraft at any time, but in the context of the mission the aircraft was not controllable.

After a fair amount of discussion, it was decided that this interpretation of the HQR scale is not entirely correct: inability to perform the MTE, because the aircraft does not have the appropriate effectors, is not akin to loss of control. It is, instead, in keeping with the second question on the left side of the HQR scale: Adequate performance cannot be attained with a tolerable pilot workload. The appropriate HQR, then, is a 7, 8, or 9 – not a 10.

IV. Summary and Recommendations Piloted evaluation of controllability following failures can adopt the procedures used in standard handling

qualities tests. Following are recommendations for performing a handling qualities simulation for degraded modes: 1. Define handling qualities evaluation tasks for the unfailed aircraft, then use the same tasks for both the

unfailed and failed aircraft. The objective is to quantify the severity of degradation in handling qualities. Consistent task requirements are necessary for a fair quantification.

2. Provide the evaluation pilots with ample opportunity to practice the selected evaluation tasks with the unfailed aircraft. Handling Qualities Ratings (Figure 1) should be obtained from formal assessment of the handling qualities of the unfailed aircraft. Verify that these ratings are consistent with expectations (from either flight test or simulation) and use them to judge the severity of the degradation in handling qualities once the failures are introduced.

3. Be prepared for HQRs of 7 or worse. Such ratings may be permissible depending upon the probability of occurrence of the failure(s).

4. An HQR of 10 by one pilot is justification for continued study. More pilots should be used to evaluate the same failure and verify the rating.

5. Depending upon the form of the failure, certain evaluation tasks may be impossible to perform. As long as this is caused by loss of performance or loss of necessary control effectors – and not by an extreme degradation in handling qualities – and as long as loss of aircraft control does not occur, the pilots should be instructed to assign HQRs no worse than 9.

6. Transients following failures and ability to recover from the transients should be evaluated using the Failure Transient/Recovery Rating Scale in Figure 3. Two separate ratings are allowed and should be encouraged.

References 1 “Aeronautical Design Standard, Performance Specification, Handling Qualities Requirements for Military Rotorcraft,” US

Army Aviation and Missile Command, ADS-33E-PRF, Mar. 2000. 2 Wilson, D.J., Riley, D.R., and Citurs, K.D., “Aircraft Maneuvers for the Evaluation of Flying Qualities and Agility, Vol. 2:

Maneuver Descriptions and Selection Guide,” WL-TR-93-3082, Aug. 1993. 3 Klyde, D.H., and Mitchell, D.G., “Handling Qualities Demonstration Maneuvers for Fixed Wing Aircraft, Vol. II:

Maneuver Catalog,” WL-TR-97-3100, Oct. 1997. 4 Cooper, G.E., and Harper, R.P., Jr., “The Use of Pilot Rating in the Evaluation of Aircraft Handling Qualities,” NASA TN

D-5153, Apr. 1969.


7

5 Weakley, J.M., Kleinhesselink, K.M., Mason, D.H., and Mitchell, D.G., “Simulation Evaluation of V-22 Degraded-Mode

Flying Qualities,” American Helicopter Society 59th Annual Forum, Phoenix, AZ, May 2003. 6 Mitchell, D.G., Weakley, J.M., Kleinhesselink, K.M., and Mason, D.H., “Development of Mission Task Elements for

V/STOLs,” American Helicopter Society 59th Annual Forum, Phoenix, AZ, May 2003. 7 “Military Specification, Flying Qualities of Piloted Airplanes,” MIL-F-8785C, Nov. 1980. 8 “Military Specification, Flying Qualities of Piloted V/STOL Aircraft,” MIL-F-83300, Dec. 1970. 9 Hindson, W.S., Eshow, M.M., and Schroeder, J.A., “A Pilot Rating Scale for Evaluating Failure Transients in Electronic

Flight Control Systems,” AIAA-90-2827-CP, AIAA Atmospheric Flight Mechanics Conference Proceedings, Portland, OR, August 1990, pp. 270-284.


8

[american institute of aeronautics and astronautics aiaa atmospheric flight mechanics conference and...

Documents