georgia tech uav research facility: navigation and control ... · june 06 enj - georgia tech -...
TRANSCRIPT
![Page 1: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/1.jpg)
Eric N. JohnsonLockheed Martin Assistant Professor of Avionics Integration
Daniel Guggenheim School of Aerospace EngineeringGeorgia Institute of Technology
Seminar at DLR, Braunschweig, GermanyMay 23, 2006
Georgia Tech UAV Research Facility:Navigation and Control Research Highlights
![Page 2: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/2.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 2
Outline
• Adaptive Control– GTMax research UAV– GTSpy small ducted fan– Edge aerobatic research aircraft
• Active Vision– Multi-University Research Initiative (MURI), Active-vision control
systems for complex adversarial 3-D environments– International Aerial Robotics Competition
![Page 3: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/3.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 3
Neural-Network Adaptive Control
ApproximateDynamicInversion
ApproximateDynamicInversion
Pseudo-Control
νPlantPlant
δ
Plant Inputs (Actual Controls)
+Reference Model
Reference Model
Command
rmν
PDControl
PDControl
NeuralNetworkNeural
Network
-
+ TrackingError
pdνadν−
Doesn’t require high fidelity nonlinear flight dynamics model
![Page 4: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/4.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 4
( ) azez −+=
11σ
( )xVW TTσyad ==νIn matrix form:In matrix form:
• Feedforward neural networks with a single hidden layer are universal approximators
Single Hidden Layer Neural Network
( )σ ⋅
( )σ ⋅
( )σ ⋅
( )σ ⋅
MM
x1
x2
xN1
M
y1
y2
yN3
V W
N2
N1 N3
![Page 5: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/5.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 5
Neural Network Adaptive Control
PDControl
PDControl
DynamicInversionDynamicInversion
NeuralNetworkNeural
Network
PlantPlantReference Model
Reference Model
-
++
TrackingError
Command
cmdδ ActuatorActuatorδν
rmν
pdνadν−
![Page 6: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/6.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 6
Neural Network Adaptive Control with Pseudo-Control Hedging
PDControl
PDControl
DynamicInversionDynamicInversion
NeuralNetworkNeural
Network
PlantPlantReference Model
Reference Model
-
++
TrackingError
CommandEstimateHedge
EstimateHedge
ActuatorActuatorcmdδ
hedgeν
δν
x
rmν
pdνadν−
![Page 7: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/7.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 7
Implications
• “Shelter” adaptive element from the adverse effects of plant input characteristics: – Linear dynamics, latency, actuator saturation, rate saturation, etc.
• Achievable adaptation performance is increased dramatically
• Adaptation is correct during saturation– Adaptive element can recover from a period of “faulty” adaptation
![Page 8: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/8.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 8
Georgia Tech R-Max: GTMax
• Yamaha R-Max, – 66kg – 3m Rotor
• Instrumented as a Research VTOL UAV
• More than 300 research test flights since March 2002
• Platform for DARPA Software Enabled Control program rotary wing final experiments and demonstrations in Summer 2004
GTMax Automatic landing during November 2002 SEC Meeting
![Page 9: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/9.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 9
GTMax Avionics HardwareManual pilot
antenna
Datalink antennae
Vibration isolated avionics rack
GPS antenna
Sonar
• Flight Computers– 266MHz & 800 MHz
embedded PCs, Ethernet, flash drives
• Sensors– Inertial Measurement Units
(x2)– Differential GPS– Magnetometer, sonar
altimeter– RPM, voltage/fuel warnings– Various camera systems
• Data Links– 11 Mbps Ethernet data link– RS-232 serial data link
• Flight Computers– 266MHz & 800 MHz
embedded PCs, Ethernet, flash drives
• Sensors– Inertial Measurement Units
(x2)– Differential GPS– Magnetometer, sonar
altimeter– RPM, voltage/fuel warnings– Various camera systems
• Data Links– 11 Mbps Ethernet data link– RS-232 serial data link
![Page 10: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/10.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 10
Ground Control StationLaptopsInside
LaptopsInside
Datalink 2Datalink 2
Datalink 1Datalink 1
Ref. GPSRef. GPS
![Page 11: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/11.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 11
GTMax Avionics Installation and Wiring
Primary Flight Computer
Serial Extension Board
FreewaveDGR-115
DC/DC
5V
5V
Battery 12V
Aux
iliar
yM
odul
e
AironetMC4800
EthernetHub
NovAtel RT-2 GPS Receiver
Secondary Computer(Image Processor)
FCS20 Small Autopilot
ISIS-IMU
12V
DC/DC
DC/DC
HMR-2300Magnetometer
DC/DC
Sonar Altimeter
Power DistributionModule
Generator
Yamaha AttitudeControl System
RC Receiver YACS IMU
12V5V
RS-232 SerialEthernetDC Power
Dat
a Li
nkM
odul
eG
PS M
odul
eFC
S20
GPS
Flig
ht C
ompu
ter
Mod
ule
5V
12V
Pan/tilt/roll hardware(power wiring not shown)
Video Transmitter
Camera(several types)
Servos
EncodersEncoders
Encoders
ServosServos
![Page 12: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/12.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 12
GTMax Onboard Baseline Software
• Navigation– 17 State Extended Kalman
Filter Navigation System• Position, velocity, attitude• Accelerometer/gyro biases• Terrain height error
– All Attitude Capable– 100 Hz updates
• Control– Adaptive Neural Network
Trajectory Following Controller – Neural Network: 7 Outputs for
7 Degrees of Freedom– Full speed envelope– High performance position
tracking
• Datalink and Comms– Handles all serial devices – Reroute data over Ethernet,
save to file, playback– Software reconfigurable
• Trajectory Generation– Generates smooth
position/velocity/accel/attitude commands as a function of time from waypoints
![Page 13: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/13.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13
Neural Network Adaptive Flight Control
• Simultaneous adaptation in 7 degrees of freedom• Address large model errors and disturbances
– GTMax model is based on a linear model for hover - adaptation used on for flight over the entire flight envelope of the helicopter
– Different configurations, damage
• Accounts for actuator saturation and “two-loop” design
• Tested over full speedenvelope, aggressive maneuvers
InnerLoop
External Command
OuterLoop
Online trainedNeural
Network
Attitude corrections
![Page 14: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/14.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 14
Automatic Takeoff
![Page 15: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/15.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 15
20 ft. lateral step response
Automatic Rapid Reposition
![Page 16: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/16.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 16
Automatic Landing
![Page 17: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/17.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 17
Auto Pirouette
![Page 18: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/18.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 18
Auto 180o Velocity Change
![Page 19: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/19.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 19
Automatic Flight at 95 ft/sec
9160 9180 9200 92200
20
40
60
80
Time (sec)
Gro
und
Spe
ed (f
t/sec
)
Upwind Downwind
![Page 20: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/20.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 20
Many Configurations…
![Page 21: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/21.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 21
Rotor synchronizes with video at 900RPM
Using RPM to Accommodate Frozen Collective Actuator (10 ft Climb)
10ft
![Page 22: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/22.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 22
Flying 50 ft square
Frozen “Right” Swash-Plate Actuator
![Page 23: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/23.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 23
Small Autopilot Development (FCS20)
• 60mm x 90mm x 32mm, 120 grams– DSP for floating point operations (1.3Gflops)– FPGA IO interface– MEMS rate gyros, accelerometers– Air data– GPS
• Same software as baseline GTMax– 17 state extended Kalman filter navigation– Neural network adaptive trajectory-following flight control– Trajectory generator
![Page 24: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/24.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 24
GTSpy Description
• Modified from MASS Helispy• FCS20 Small autopilot• Weight - 5 lbs• Height - 27 in• Duct Diameter - 11 in• Hover Endurance - 25 min
![Page 25: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/25.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 25
Autopilot Tests on GTSpy, Takeoff
5 pounds11 inch duct
(movie) http://uav.ae.gatech.edu/videos/s040807k1_takeoff.mpg
![Page 26: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/26.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 26
Autopilot Tests on GTSpy
5 pounds11 inch duct
(movie) http://uav.ae.gatech.edu/videos/s040807k3_right40ft.mpg
![Page 27: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/27.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 27
Autopilot Tests on GTSpy, Landing
5 pounds11 inch duct
(movie) http://uav.ae.gatech.edu/videos/s040807k5_landing.mpg
![Page 28: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/28.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 28
First Air Launching of a Hovering AircraftNovember 20, 2004
• GTMax at 400 feet• GTSpy Launched, then
hovers 80 feet below• Both vehicles utilizing the
same NN adaptive controller software
(movie 1) http://uav.ae.gatech.edu/videos/fs041120c1_launchHeliSpyLong.mpg (movie 2) http://uav.ae.gatech.edu/videos/fs041120c1ob_launchHeliSpy.mpg
![Page 29: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/29.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 29
Agile Maneuvering Test Aircraft: GTEdge
• 33% scale Edge 540T
• Avionics– FCS20 Small Autopilot– NovAtel DGPS– FreeWave wireless serial
• Flights began June 2005
60 ft/sec Autoflight(movie) http://uav.ae.gatech.edu/videos/e050624b3_60fps.mpg
![Page 30: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/30.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 30
Automatic Transition to Zero Speed• First transition to near-vertical attitude
and near-zero airspeed, July 2005– Video begins at about 880 sec– Slow level-flight envelope
expansion from forward flight to hover
0
500
1000
-500
0
500
10000
500
1000
1500
North (ft)
trajectory
West (ft)
heig
ht (f
t)
measuredcommandend
800 850 900 9500
20
40
60
time (sec)
spee
d (ft
/sec
)
800 850 900 9500
20
40
60
80
time (sec)
pitc
h an
gle
(deg
)
800 850 900 950-100
-50
0
50
100
time (sec)
elev
ator
(% o
f max
)
800 850 900 9500
20
40
60
80
100
time (sec)
thro
ttle
(%)
800 850 900 950-100
-50
0
50
100
time (sec)
aile
ron
(% o
f max
)
800 850 900 950-100
-50
0
50
100
time (sec)
rudd
er (
% o
f max
)
(movie) http://uav.ae.gatech.edu/videos/e050723c3_vertical.mpgand e050723c2_30fps20fps.mpg
![Page 31: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/31.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 31
High Speed: 160 ft/sec (95 knots)
(movie) http://uav.ae.gatech.edu/videos/e060128a2_160fps.mpg
![Page 32: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/32.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 32
Rapid Transition to Hover & Return to Forward Flight
(movie) http://uav.ae.gatech.edu/videos/e060509a1_rapidTransition.mpg
![Page 33: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/33.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 33
Other Collaborations
AFRL & Aerovironment, Skytote
DARPA & Frontier Systems / Boeing, Maverick/Renegade
D6
![Page 34: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/34.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 34
Active-Vision Control Systems for Complex Adversarial 3-D Environments• AFOSR/AFRL MURI: Georgia Tech, MIT, UCLA,
Virginia Tech• Development of sound methods that utilize 2-D and
3-D imagery to– Enable aerial vehicles to autonomously detect and prosecute targets in
uncertain complex 3-D adversarial environments– Do these things without relying upon highly accurate 3-D models of the
environment– Include capabilities and approaches inspired by those found in nature
• New strategies of – Target recognition/tracking– Obstacle/hazard avoidance– Navigation, guidance, and control
Active-Vision Control
![Page 35: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/35.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 35
Tracking with Region-Based Deformotion
• Simultaneous segmentation, registration flow
• Registration using energy minimization with gradient descent
• Handle occlusions
![Page 36: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/36.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 36
Vision-Only Guidance, Navigation, and Control
![Page 37: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/37.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 37
The Measurements
Available measurements from a candidate window• Size• Rotation• Horizontal pos. in camera• Vertical pos. in camera
![Page 38: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/38.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 38
Flight Test Results
![Page 39: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/39.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 39
Vision-Aided Inertial Navigation
• Inertial Navigation System (INS) aided by 2-D vision sensor looking at a selected image target
Autopilot UAV
IMU + Navigation
Camera Image Processing+Computer Vision
![Page 40: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/40.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 40
Vision-Aided Inertial Navigation
• Approach– Use assumed range (altitude), position in image, size of object in
image, and aircraft state to estimate object position, size, andorientation
– Use subsequent measurement of object image position and size in image to update INS
– Utilize inertial data to maintain lock on target
– Operate without GPS (or any other position aiding)
Last Object LocationLast Object Location
Actual NewObjectLocation
ExtrapolatedNew Object Location∆z
∆y
Actual NewObjectLocation
ExtrapolatedNew Object Location∆z
∆y
![Page 41: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/41.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 41
Vision-Aided INS Trials
Special Purpose Optical Target
Using Existing Features(in this case a window)
![Page 42: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/42.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 42
2GT
Close Approach to Building, no GPS
![Page 43: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/43.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 43
Flight Test Results: Error Plots
error = |estimate – GPS|
Max Errors:9.6 ft (x)8.3 ft (y)4.0 ft (z)
![Page 44: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/44.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 44
Air-to-Air Vision-Based Tracking
Guidance Autopilot UAV
IMU + Navigation
Camera
Estimated:LOS RateRangeObject Size
Acc Com
PositionVelocityAccelerationRates
Kalman Filter Image Processing+Computer Vision
No information is communicated between aircraft, and only passive 2-D vision information is available to maintain formation
Camera
Y
X
y
x
Camera Image Plane
Target
Image frame
Z
u = (uX, uY, uZ) : unit vector α : subtended angle
r : range b : target size
Estimation Stateu, u, , , b. 1 r
r r .Estimation State
u, u, , , b. 1 r r r
.u, u, , , b. 1 r
r r u, u, , , b. 1 r r r
.
Measurementu, α
![Page 45: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/45.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 45
• Fast marching methods + target acquisition process= 10 frames/sec
Video 1 Video 2
Tests on Recorded Videos
![Page 46: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/46.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 46
Estimator DesignApply Extended Kalman Filter (EKF) to• Measurement
calculated by using image processor outputs
• Estimation state
Unit vector expressed in a camera frame
Unit vector expressed in a NED frame
Leader
Camera frame
Xc
Yc
Zc
North
EastDown
Follower
u : unit vector
α : subtended angle
r : range
leader’ssize : b
alat : leader’s lateral acceleration
Measurement and Estimates
Leader’s lateral accelerationIn the leader’s wind frame( perpendicular to leader’s velocity vector )
![Page 47: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/47.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 47
6-DOF Image-in-the-Loop Sim Results
• Relative position • Relative velocity
ImageProcessor EKF Controllerz x̂
Simulator
![Page 48: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/48.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 48
Simulation Results: Image Processor• Image source : recorded video in flight test• Simulated camera motion : circling with a constant speed• Commanded camera position : 100ft behind, 20ft below the leader
• Image processor outputs
Leader’s positions in image Unit vector and subtended angle
![Page 49: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/49.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 49
Simulation Results: 3D EstimatorPosition
Velocity Acceleration
![Page 50: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/50.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 50
Flight Test Results• Image processor outputs and EKF estimates
Unit vector and subtended angle
0.94 0.96 0.98 1 1.02 1.04 1.06
x 104
-0.5
0
0.5
1
navTime
u
u1u2u3
0.94 0.96 0.98 1 1.02 1.04 1.06
104
0.05
0.1
0.15
0.2
0.25
navTime
α
Leader position
0.94 0.96 0.98 1 1.02 1.04 1.06
x 104
0
1000
2000
navTime
X
VISIONGPS
0.94 0.96 0.98 1 1.02 1.04 1.06
x 104
0
500
1000
1500
navTimeY
VISIONGPS
0.94 0.96 0.98 1 1.02 1.04 1.06
104
-500
-400
-300
-200
navTime
Z
VISIONGPS
![Page 51: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/51.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 51
Formation Flight Tests (GTMax/GTEdge)
![Page 52: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/52.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 52
Vision-Based Formation Flight Tests (GTMax/GTEdge)
![Page 53: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/53.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 53
Status
• Currently “closing the loop” on guidance –true vision-based formation flight
• First attempts were earlier this week• Longest run 22 seconds• Current efforts include more advanced vision
processing, estimation, and guidance• Moving into other intended scenarios:
see/avoid, pursuit
![Page 54: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/54.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 54
Box Target Maneuver Sinusoidal Target Maneuver
Neural Network augmenting
estimator corrects bias
}}
Estimating Dynamics of Maneuvering Target
![Page 55: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/55.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 55
International Aerial Robotics Competition
Launch area
Two lights near building
Vehicle or subvehicle(s) enter building
Building and an Entry Point Found
>1m
Transmit an image of “point of interest”inside building
In less than 15 minutes:
Image receiver(& other ground components)Sign on
building
3 km
http://avdil.gtri.gatech.edu/AUVS/CurrentIARC/2001CollegiateRules.html
![Page 56: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/56.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 56
Levels
Level 1: Follow prescribed waypoints for 3kmLevel 2: Locate building and find an entry
• Level 3: Operate inside the building – Can be a different vehicle or subvehicle than used above– Can launch near target structure
• Level 4: All of the above– Complete in < 15 minutes
Contest gets new rules once somebody does level 4
![Page 57: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/57.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 57
AUVSI InternationalAerial Robotics Competition
Launch AreaIdentify building and map entry point usingonboard systems
>1m
Image receiver(For judges)
Sign on correctbuilding
2003 ApproachGoal: Level 2 Mission
![Page 58: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/58.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 58
Simulation and Flight Tests in Preparation for 2003 Competition
![Page 59: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/59.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 59
2003 Results, Level 2
• Allowed Four Mission Attempts• On Three of the Attempts:
– Building Located by Recognizing Symbol• 1m x 1m sign• Fifteen buildings to chose from
– Valid Opening Found and Automatically Reported to Judges– Took 16-17 minutes each time
![Page 60: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/60.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 60
Future Plans
• Continue to expand on air-to-air tracking– Cluttered background– Flight testing and facilitate technology transition
• Air-to-ground tracking (by airplanes)– Leverage air-to-air work (particularly cluttered background and
guidance work)
• Obstacle avoidance– Initial emphasis on unknown fixed obstacles, ownship state known– Using passive monocular 2-D sensor
• Requires considerable work in image processing, 3-D estimation, and guidance
![Page 61: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/61.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 61
Expected Outcomes and Transitions
• New capabilities of autonomous sensing and control, enabling operations: – In the presence of uncertainty– In a clandestine/covert manner– In close proximity to hazards, structures, and/or terrain
• Relevant flight test validation• Enable more capable/reliable existing air vehicles
and guided munitions• Enable entirely new systems to be developed
(for example, capable of operating in urban environments)
![Page 62: Georgia Tech UAV Research Facility: Navigation and Control ... · June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13 Neural Network Adaptive Flight Control • Simultaneous adaptation](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f687a7b90e3a7222c249445/html5/thumbnails/62.jpg)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 62
Acknowledgements
• Sponsors: DARPA, AFRL, AFOSR, NSF, NASA, Lockheed Martin• Partners: GST, Boeing, Draper, Lockheed Martin, Northrop Grumman,
Honeywell, McKenna MOUT, SSCI, OGI, Virginia Tech, MIT, UCLA• Georgia Tech Participants:
Thomas Apker, Anthony Arkwright, Mike Baldwin, Sean Barnes, Anthony Calise, Cesar Carrillo, Jane Case, Claus Christmann, Henrik Christophersen, Scott Clements, Mike Curry, Joerg Dittrich, Graham Drozeski, Jason File, Nate Fisher, Lucas Fortier, Stewart Geyer, Richard Giuly, Luis Gutierrez, Bryan Gwin, Jincheol Ha, Greg Ivey, Mat Hart, Bonnie Heck, Jeong Hur, Anna Ivester, Ole Jakobsen, Eric Johnson, Phillip Jones, Suresh Kannan, Adrian Koller, Dan Laszweski, Tom Linden, Sumit Mishra, Alex Moodie, Kyungjin Moon, Hiong Chair Naik, Wayne Pickell, Seung-Min Oh, J.V.R. Prasad, Alison Proctor, Nimrod Rooz, Bhaskar Saha, Daniel Schrage, Liang Tang, Allen Tannenbaum, Mike Turbe, Shannon Twigg, Suraj Unnikrishnan, George Vachtsevanos, Patricio Vela, Yoko Watanabe, Linda Wills, Allen Wu, Ilkay Yavrucuk, and many others…