1
Flight Testing Of Fused Reality
Visual Simulation System
Justin Gray, Systems Technology, Inc.
13th Annual AIAA Southern California Aerospace Systems and Technology (ASAT) Conference April 30th 2016, Santa Ana, CA
2
• General Objective
• Fused Reality
• Inflight System Description
• Test Methods and Conditions
• Test Results
• Outcomes and Observations
• Lessons Learned
• Conclusions
Overview
3
General Objective• To assess handling qualities, there is no substitute for flight test.
~ Simulators have improved fidelity, but the human physical response to the dynamic response of the test aircraft can still not be adequately replicated in a simulator.
~ Flight testing comes at a much higher cost than simulation, especially when additional flight assets, such as aerial refueling tankers, are required for the testing.
• What if important handling qualities evaluations tasks that include aerial refueling, formation flying, etc. could be accomplished without the support aircraft?
~ This will not only save valuable flight test dollars, but it will also provide enhanced safety, since additional flight test assets will not be placed in close proximity to the test aircraft.
~ A flight test evaluation system based on a novel head mounted display visual system that combines virtual and real world elements was developed by Systems Technology, Inc.
• This presentation describes a flight test evaluation of such a system known as Fused Reality.
4
Fused Reality: Helicopter Cabin Gun Trainer
• Fused Reality combines real time video via a camera on a head mounted display and virtual elements to provide an overall immersive environment.
• The original concept used a real-time blue screen technique as the keying technology that distinguishes real from virtual:
~ Wherever the keying color is seen in the video, the color is replaced by the virtual scene..
~ IR markers are used to track the position of real world objects that allows for a seamless interaction between the real and the virtual world.
5
Fused Reality: Initial Inflight System• Initially, a cockpit window border keying technology using two color bands
was employed:
~ Under consistent lighting conditions, the approach worked well and led to an effective demonstration.
~ As the program advanced, limitations of the border technology were exposed as integration with an actual aircraft was attempted.
~ Sensed electromagnetic properties in the out-the-window scene were used to successfully key out the cockpit window area.
• This allowed for an inflight technology demonstration and evaluation using the Calspan Learjet Inflight Simulator.
8
Test Methods and Conditions: Evaluation Aircraft – GA8 Airvan
• The GippsAero GA-8 Airvan (N215AV) is a strut braced, high wing, eight seat, Federal Aviation Administration (FAA) type certified aircraft.
• The GA-8 has fully reversible flight controls driven by conventional mechanical pulleys and pushrods.
• Because of modifications, the aircraft has a special airworthiness certificate in the experimental category for research and development.
9
Test Methods and Conditions: Safety & Evaluation Pilots
• Approximately ten hours of flight time in the NTPS Airvan were used in support of this program:
~ Roughly half of those hours dedicated to system integration and checkout
~ The remaining hours to formal evaluation sorties
• Four NASA Armstrong test pilots participated in the formal evaluation flights over several days in mid-January 2015.
10
Test Methods and Conditions: Evaluation Tasks
Drogue Tracking Formation Flying – Stencil Mode
Formation Flying – Virtual ModePrecision Landing
11
Test Methods and Conditions: Evaluation Procedures
• Precision Landing
~ 500’ AGL Virtual Altitude
~ Centerline approach and landing
~ Offset approach and landing (200’ offset, correction at 200’ AGL)
~ Desired/Adequate Landing Boxes: ±25’/±50’ lat, ±250’/±500’ lon
~ Desired/Adequate Airspeed: ±5/±10 KIAS
• Drogue Tracking
~ 5 minutes of practice followed by 30 second evaluation
~ Desired/Adequate (Lateral/Vertical): ±6’/±12’
• Formation Flying – Virtual & Stencil Mode
~ 5 minutes of practice followed by 30 second evaluation
~ Desired/Adequate (Station Keeping Guides): ±5’/±10’
• NVG Mode Demonstration, time permitting
13
Test Results: Precision Offset Landing
• Example Pilot 4 offset landing run shown here received HQR 3
• Pilot 4: “Lag is not a factor at all… …what I am seeing is the airplane through the system and the system is not in the way.”
• Successful “landings” achieved by all four evaluation pilots
• Indicates that the FR system can be used to conduct repeatable precision approach and landing evaluations at altitude.
0 10 20 30 40 50 60 70 80-100
0
100
200
300
400
500
Time (sec)
La
tera
l an
d V
ert
ica
l P
ath
to
Ru
nw
ay
(ft
)
Lateral Offset
Virtual Altitude
0 10 20 30 40 50 60 70 800
10
20
30
40
50
60
70
80
90
100
Time (sec)
Air
sp
ee
d (
KIA
S)
14
Test Results: Drogue Tracking
• Example Pilot 4 drogue tracking run shown here received HQR 8
• Pilot 4: “Very loose laterally.” “This is really hard.” “A little more field-of-view would help the fore/aft cues.”
• The Airvan was never intended to perform such a demanding task and control limitations, particularly in the lateral axis, were challenging for the pilots.
140 145 150 155 160 165 170-10
-8
-6
-4
-2
0
2
4
6
8
10
Time (sec)
Tra
ck
ing
Err
or
(ft)
Lateral Tracking Error
Vertical Tracking Error
140 145 150 155 160 165 170-60
-50
-40
-30
-20
-10
0
Time (sec)
Fo
re/A
ft L
on
git
ud
inal
Err
or
(ft)
15
Test Results: Formation Flying
• Example virtual mode formation flying task shown here received HQR 4
• Pilot 4: “The fore/aft was the hardest thing for me to learn, but once I got a better sense for that, it made it a little easier.”
• The evaluation pilots were able to successfully formation fly with a virtual target aircraft using the FR system in full virtual and stencil modes.
150 155 160 165 170 175 180-5
-4
-3
-2
-1
0
1
2
3
4
5
Time (sec)
Tra
ck
ing
Err
or
(ft)
Lateral Tracking Error
Vertical Tracking Error
150 155 160 165 170 175 180-10
-8
-6
-4
-2
0
2
4
6
8
10
Time (sec)
Fo
re/A
ft L
on
git
ud
inal
Err
or
(ft)
16
Lessons Learned
• Checkout flights were vital to insure that the Fused Reality system was properly integrated with the test aircraft.
• For flight test projects with limited flight hours, inflight trouble shooting should only be done when absolutely necessary.
~ Better to land and fix the problem on the ground than burn flight hours trying to solve the issue in the air.
• Different flight test organizations have different rules, processes, procedures and philosophies, but the common goal is safety first.
~ Communication, respect, an open mind, and concentration on that common goal enable successful collaborations.
• When at a standstill, the FR navigation can drift because it keys off of ground track, but as soon as the aircraft is moving, the system locks on to the correct position.
17
Lessons Learned (continued)
• FR system needs to initialize from a stabilized flight condition at the start of each evaluation run.
• Sun angle was at times a factor for the IR cameras used for the evaluation pilot head tracking and for the video camera attached to the HMD.
~ Best results were achieved when the cameras were not pointed directly at the sun.
• Regarding the sun angle, when the sun angle was to the evaluation pilot’s right, the right eye display could wash out.
~ NVG-like blinders attached to the HMD could alleviate this issue.
• Regarding the evaluation tasks:
~ Markings that provided next to the virtual runway that provided an aim point for the lateral offset would aid the precision offset landing.
~ Field-of-view of the HMD can limit longitudinal closure cues used in the drogue tracking and formation flying tasks.
18
Conclusions
• Using a NTPS Gippsland GA-8 Airvan research aircraft, a new stand-alone system for closed-loop handling qualities assessments was successfully demonstrated that allows virtual out-of-the-window elements to be combined with the real-world cockpit visual environment.
• As a result of these flight tests, all of the issues uncovered in the initial flight test campaign on the Calspan Learjet Inflight Simulator and during the initial GA-8 checkout flights have been corrected and tested in flight.
• Many enhancements for Fused Reality as well as other potential training uses were identified by the team which will aid in improving and commercializing this technology.