machine learning for control of morphing air vehicles john valasek, amanda lampton, adam niksch...
TRANSCRIPT
![Page 1: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/1.jpg)
Machine Learning for Control
of Morphing Air Vehicles
John Valasek, Amanda Lampton, Adam Niksch
Aerospace Engineering DepartmentTexas A&M University
SAE Aerospace Control and Guidance Systems Committee Meeting 99
1 March 2007Boulder, CO
![Page 2: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/2.jpg)
Valasek, Lampton, Niksch - 2
Student Research Team
2006 - 2007
![Page 3: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/3.jpg)
Valasek, Lampton, Niksch - 3
Briefing Agenda
Shape Changing / Morphing for Micro Air Vehicles
Learning
Control
Learning and Control– Adaptive-Reinforcement Learning Control
Some Results
Some Conclusions
Research Issues
Extensions
![Page 4: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/4.jpg)
Valasek, Lampton, Niksch - 4
Biomimetic Example 1
![Page 5: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/5.jpg)
Valasek, Lampton, Niksch - 5
300 million year old design
97% success rate in capturing prey
Flight relies entirely on 3-D unsteady aerodynamics
Unstable in all axes
Can hover as well as fly from a forward speed of +100 body
length/sec (~ 60 mph) to - 3 bl/sec within 5 bl
Can alter heading by 90 degs in less than 0.1 secs
Dragonflies and other flying insects have the fastest visual processing
system in the animal kingdom
Biomimetic Example 2
These feats are accomplished with the neuro-circuitryof a brain smaller than a sesame seed.
Andres Meade, Rice University
![Page 6: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/6.jpg)
Valasek, Lampton, Niksch - 6
Biomimetic Example 3
Courtesy Graham Taylor, Oxford University
Cossack
![Page 7: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/7.jpg)
Valasek, Lampton, Niksch - 7
Biomimetic Example 3
Courtesy Graham Taylor, Oxford University
![Page 8: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/8.jpg)
Valasek, Lampton, Niksch - 8
Biomimetic Example 3Cossack sysid using OKID
![Page 9: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/9.jpg)
Valasek, Lampton, Niksch - 9
Biomimetic Example 3
Courtesy Graham Taylor, Oxford University
Kinematic Engineering Model of Eagle Wing
![Page 10: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/10.jpg)
Valasek, Lampton, Niksch - 10
Reconfigurable; shape changing (morphing); rigid or elastic body plants Multi-Input Multi-Output (MIMO)systems Distributed sensing, actuation, and propulsion
– Multiple, possibly non co-located sensors– Can have large number of actuators control allocation problem– High bandwidth actuators gust alleviation
Robust Adaptive Control– Flow control– Exogenous inputs (turbulence and gusts)– Damage and fault tolerance
Distributed and limited processing capability– Volumetric limit– Mass limit– Power limit– Real-time operation
Integrated guidance, navigation, and control– Self learning, self adapting, self tuning
Biomimetic Flight Summary
Information processing system
![Page 11: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/11.jpg)
Valasek, Lampton, Niksch - 11
A Multi-Role Platform that:
Changes its state substantially to adapt to changing mission environments.
Provides superior system capability not possible without reconfiguration.
Uses a design that integrates innovative combinations of advanced materials, actuators, flow controllers, and mechanisms to achieve the state change.
Morphing Aircraft
Aerospace America, Feb 2004
DARPA’s definition
Changes On The Order Of 50% More Wing Area Or Wing Span And Chord
![Page 12: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/12.jpg)
Valasek, Lampton, Niksch - 12
Morphing for Mission Adaptation
– Large scale, relatively slow, in-flight shape change to enable a single vehicle to perform multiple diverse mission profiles
or:
Morphing for Control
– In-flight physical or virtual shape change to achieve multiple control objectives (maneuvering, flutter suppression, load alleviation, active separation control)
MissionAdaptation
Control
John Davidson, NASA Langley, AFRL Morphing Controls Workshop – Feb 2004
Which Morphing?
![Page 13: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/13.jpg)
Valasek, Lampton, Niksch - 13
Lockheed Martin
NextGen&
Barron Associates
Large Morphing Air Vehicle Models
![Page 14: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/14.jpg)
Valasek, Lampton, Niksch - 14
Small Morphing Air Vehicle Models
Cornell (collaborator) Garcia & Lipson
– Morphing dynamical model and simulation that incorporates aerodynamic and structural effects
– Validated with experimental data– Basic morphing parameters (incidence angle, dihedral angle)
Maryland Hubbard
– Actuating a flapping wing structure with SMA’s– Structure, material, distribution of actuators
Florida Lind
– Morphing flight demonstrator vehicle– Basic morphing parameters (dihedral angle, sweep angle)– H-inf control
lots of modeling, very little control
![Page 15: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/15.jpg)
Valasek, Lampton, Niksch - 15
• Use biologically inspired approach to understand and control the: • morphing (shape changing),
• flight control (stability, flight path, gust tolerance, stall)
• maneuvering (perching and hovering)
of multi-mission micro air vehicles.
• Control the physics, in concert with nonlinearities.• Not as a robustness bandaid for aerodynamic and structural
uncertainties & lack of understanding
• Shape change the entire vehicle, not a component on the vehicle• DARPA definition
Technical Approach and Scope
![Page 16: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/16.jpg)
Valasek, Lampton, Niksch - 16
• Address:
•WHEN to morph, perch, etc.
•HOW to morph, perch, etc.
•LEARNING to morph, perch, etc.
All while keeping the (pointy?) end facing forward.
Big Picture Research Goals
![Page 17: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/17.jpg)
Valasek, Lampton, Niksch - 17
![Page 18: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/18.jpg)
Valasek, Lampton, Niksch - 18
Common Mathematical Domain Helps
To Avoid Ad-hoc Approaches
??
Machine Learning
Reconfiguration Policy
State based methods
learning
Adaptive Control
Parameters in a Known
Functional Relationship
State based methods
adapting
The Mathematical Domain Issue
![Page 19: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/19.jpg)
Valasek, Lampton, Niksch - 19
Adaptive-ReinforcementLearning Control (A-RLC)
Conceptual Control Architecture
for Reconfigurable or Morphing Aircraft
SAMIStructured Adaptive Model Inversion
(Traditional Control)
Flight controller to handle wide variation in dynamic properties
due to shape change
ML Machine Learning(Intelligent Control)
Learn the morphing dynamics and the optimal shape at every flight
condition in real-time
![Page 20: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/20.jpg)
Valasek, Lampton, Niksch - 20
Synthetic Jets for Virtual Shaping and Separation Control
MultiSensor MEMS Arrays for Flow Control Feedback
Adaptive Controller
Sensed Information Aggregation
Control Information Distribution
Reconfiguration Command
Generation
System Performance Evaluation
Knowledge
Base
Environment
Control Architecture
![Page 21: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/21.jpg)
Valasek, Lampton, Niksch - 21
2-D Plate Rectangular Block Ellipsoid Delta Wing Final 2004 2005 2006 2007 Objective
Morphing Air Vehicle Evolution
![Page 22: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/22.jpg)
Valasek, Lampton, Niksch - 22
Morphing Air Vehicle Model - TiiMY
Shape
Ellipsoidal shape with varying axis dimensions.
Constant volume (V) during morphing
2 independent variables: y and z, dependent dimension
Morphing Dynamics
Smart material: shape memory alloy (SMA)
Morphing Dynamics : Simple Nonlinear Differential Equations
6Vx
yz
y-dimension 2
z-dimension 3y
z
y yy V
z zz V
![Page 23: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/23.jpg)
Valasek, Lampton, Niksch - 23
Morphing Dynamics
Y-morphing Z-morphing
![Page 24: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/24.jpg)
Valasek, Lampton, Niksch - 24
Shape Morphing Simulation TiiMY
![Page 25: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/25.jpg)
Valasek, Lampton, Niksch - 25
Optimal Shapes atVarious Flight Conditions
Optimality is defined by identifying a cost function.
J=J (Current shape, Flight condition)
2 2
0.5
( ( )) ( ( ))
3 cos( ) and 2 22
y z y z
Fy z
J J J y S F z S F
S F S e
![Page 26: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/26.jpg)
Valasek, Lampton, Niksch - 26
6-DOF Mathematical Modelfor Dynamic Behavior
Variables
Nonlinear 6–DOF Equations
– Kinematic level:
– Acceleration level:
Drag Force
– Function of air density, square of velocity along axis, and projected area of the vehicle perpendicular to the axis
[ ] [ ]
[ ] [ ]
T Tc x y z c
T T
p d d d v u v w
p q r
;
c l c a
c c d
d
p J v J
mv mv F F
I I I M M
�
�
additional dynamics due to morphing
![Page 27: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/27.jpg)
Valasek, Lampton, Niksch - 27
Learning
![Page 28: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/28.jpg)
Valasek, Lampton, Niksch - 28
– Machine Learning
• Learn the optimal control mechanism by wind-tunnel experiments & flight tests
• Possible learning algorithms include Artificial Neural Networks (ANN) , Explanation-Based Learning (EBL), and Reinforcement Learning (RL)
Candidates to develop inference mechanism– Rules-Based Expert System
• Model the knowledge of human experts
• Imitate the natural behaviour of birds
Question: How Many Control Theorists Does It Take To Change A Using
Artificial Neural Networks ?
Knowledge Based Control
– Biologically inspired control process• Mimic the behaviour of birds
Question: How Many Control Theorists Does It Take To Change A Using
Reinforcement Learning ?
![Page 29: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/29.jpg)
Valasek, Lampton, Niksch - 29
Simple Example
![Page 30: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/30.jpg)
Valasek, Lampton, Niksch - 30
Reinforcement Learning - 1
Supervised or unsupervised learning? Sequential decision making.– Knowledge is based on experience and interaction with the environment, not
on input-output data supplied by an external supervisor
Achieves a specific goal by learning from interactions with the environment.– Considers state information (s)– Performs sequences of actions, (a), observing the consequences– Attempts to maximize rewards (r) over time
• These specify what is to be achieved, not how to achieve it
– Constructs a state value function (V)• Learns an optimal control policy
Memory is contained in the state value function
* *arg max[ ( , ) ( ( , ))]a
r s a V s a
21 2 3
0
... it t t t t i
i
R r r r r
*( ) max ( )V s V s for all s
![Page 31: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/31.jpg)
Valasek, Lampton, Niksch - 31
Reinforcement Learning - 2
Learning is done repetitively, by subjecting to different scenarios
Learning is cumulative and lifelong
Formulations are generally based on Finite Markov Decision Processes (MDP)
• 3 major candidate algorithms: – Dynamic programming
– Monte Carlo methods
– Temporal Difference Learning
![Page 32: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/32.jpg)
Valasek, Lampton, Niksch - 32
1. Actor takes action based upon states and preference function2. Critic updates state value function, and evaluates action3. Actor updates preference function
Learning is done repetitively, by subjecting to different scenarios
Reinforcement Learningself training
![Page 33: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/33.jpg)
Valasek, Lampton, Niksch - 34
Two Illustrative Examples
![Page 34: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/34.jpg)
Valasek, Lampton, Niksch - 35
State: – Gain for δa = Kφ (φcmd - φ)
Actions: – Increase gain by small amount– Decrease gain by small amount
Constraints/Boundaries– Max overshoot– Rise time– Settling time
Interesting Features– e-greedy policy incorporated– Upward annealing of γ
incorporated
max os = 2%Tr = 8sTs = 10s
Cessna 208B Super Cargomaster
3 2
4 3 2
3 15 9.0 6.0 19.7
4.8 5.2 11.1 0.01a
e S S SS
S S S S
Matlab: ~250 sec real-time for 1000 learning episodes
familiar example
Reinforcement Learning
![Page 35: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/35.jpg)
Valasek, Lampton, Niksch - 36
Reinforcement Learningfamiliar example
![Page 36: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/36.jpg)
Valasek, Lampton, Niksch - 37
Reinforcement Learningfamiliar example
![Page 37: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/37.jpg)
Valasek, Lampton, Niksch - 38
0
20
40
60
80
100
120
-4-2024
-505
x
z
y
Start
Finish
Smart Block Demo 1
aerial obstacle course
![Page 38: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/38.jpg)
Valasek, Lampton, Niksch - 40
Smart Block: First Try
![Page 39: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/39.jpg)
Valasek, Lampton, Niksch - 41
Smart Block: Second Try
![Page 40: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/40.jpg)
Valasek, Lampton, Niksch - 42
Smart Block: New Course
![Page 41: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/41.jpg)
Valasek, Lampton, Niksch - 51
Adaptive–Reinforcement Learning Control
Valasek, John, Tandale, Monish D., and Rong, Jie, "A Reinforcement Learning Adaptive Control Architecture for Morphing,” Journal of Aerospace Computing, Information, and Communication, Volume 2, Number 4, pp. 174-195, April 2005.
Valasek, John, Doebbler, James, Tandale, Monish D., and Meade, Andrew J., "Improved Adaptive-Reinforcement Learning Control for Morphing Unmanned Air Vehicles,”Journal of Aerospace Computing, Information, and Communication (in review).
Tandale, Monish D., Rong, Jie, and Valasek, John, "Preliminary Results of Adaptive-Reinforcement Learning Control for Morphing Aircraft,” AIAA-2004-5358, Proceedings of the AIAA Guidance, Navigation, and Control Conference, Providence, RI, 16-19 August 2004.
![Page 42: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/42.jpg)
Valasek, Lampton, Niksch - 52
A-RLC Architecture
![Page 43: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/43.jpg)
Valasek, Lampton, Niksch - 53
Air Vehicle Example
![Page 44: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/44.jpg)
Valasek, Lampton, Niksch - 54
Objective– Demonstrate optimal
shape morphing for multiplespecified flight conditions
Method– For every flight condition, learn
optimal policy that commands
voltage producing the optimal shape– Minimize total cost over the entire flight trajectory– Evaluate the learning performance after 200 learning episodes
Air Vehicle Example
RL Module is Completely Ignorant of Optimality Relations and Morphing Control Functions:
It Must Learn On Its Own, From Scratch
![Page 45: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/45.jpg)
Valasek, Lampton, Niksch - 55
Agent: Morphing Air Vehicle Reinforcement Learning Module
Environment: Various flight conditions
Goal: Fly in optimal shape that minimizes cost
States: Flight condition; shape of vehicle
Actions: Discrete voltages applied to change shape of vehicle– Action set:
Rewards: Determined by cost functions
Optimal control policy: Mapping of the state to the voltage leading to the optimal shape
Timmy Demo
reinforcement learning definitions
( , ) [2,4] [0,5]
( , ) [2,4] [0,5]
y FS
z F
( , ) [0, 0.5, 1, 5] ; [0, 0.5, 1, 4]y zA V V
![Page 46: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/46.jpg)
Valasek, Lampton, Niksch - 56
Episodic Learning
Unsupervised learning episode
Single pass through 100 meter flight path in 200 seconds
Reference trajectory is generated arbitrarily
The flight condition changes twice during each episode
Shape change iteration after every 1 second
Exploration-exploitation dilemma:– Explorative early, exploitative later -policy with decreasing
Limited training examples– Only 6 discrete flight conditions:
2000 samples for KNNPI) rand(aa
asQ a
i
a
else
),(maxarg
) -1 ty (probabili if policy greedy-ε
![Page 47: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/47.jpg)
Valasek, Lampton, Niksch - 57
Demo
![Page 48: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/48.jpg)
Valasek, Lampton, Niksch - 58
Comparison of True Optimal Shape and Learned Shape
KNN learns poorly for
several flight conditions
![Page 49: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/49.jpg)
Valasek, Lampton, Niksch - 59
Function Approximation– Errors remained which could not be eliminated with additional training.
– Use Galerkin-based Sequential Function Approximation (SFA) to approximate the action-value function Q(s,a)
What Happened?
![Page 50: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/50.jpg)
Valasek, Lampton, Niksch - 60
New SFA approach learns optimal shape well
Comparison of True Optimal Shape and Learned Shape
![Page 51: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/51.jpg)
Valasek, Lampton, Niksch - 61
Comparison
Normalized RMS error
Y dimension Z dimension
K N N 1.42 0.821
S F A 1.27 0.661
10% reduction 20% reduction
![Page 52: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/52.jpg)
Valasek, Lampton, Niksch - 68
Reinforcement Learning successfully learns the optimal control policy that results in the optimal shape at every flight condition. – Can function in real-time, leading to better performance as system
operates over the long term.
Adaptive-Reinforcement Learning Control is a promising candidate for control of Mission Morphing.– Maintains asymptotic tracking in the presence of parametric
uncertainties and initial condition errors.
Shape Changes for “Mission Morphing” can be treated as piecewise constant parameter changes– SAMI is a favorable method for trajectory tracking control
“Morphing for Control” will require different control strategy– Piecewise constant approximation no longer valid
What Does This Show?
![Page 53: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/53.jpg)
Valasek, Lampton, Niksch - 69
Issues & Future Directions
Realistic structural response effects – Aeroelastic behaviour– SMA models and hysteretic behaviour
• Priesach model is algebraic, only has major hysterisis loops• Solution: roll your own with R-L
SMA Hysteresis
-0.005
0
0.005
0.01
0.015
0.02
0.025
0.03
0.035
0 20 40 60 80 100
Temperature (C)
Str
ain
Experimental Major Math Model Experimental Minor
![Page 54: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/54.jpg)
Valasek, Lampton, Niksch - 70
Issues & Future Directions
Time scale problem: control methodologies to handle faster shape changes– Hovakimyan’s Adaptive Control– Linear Parameter Varying (LPV) control
Novel distributed sensing and distributed actuation on a large(!) scale– Insect and avian inspired sensing
Learning on a continuous domain– Continuous versus discrete
Modify the simulation to include a more advanced aircraft model– Wing-Body, Wing-Body-Empennage, etc.
Build and fly R/C class morphing demonstrator UAV
![Page 55: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/55.jpg)
Valasek, Lampton, Niksch - 71
Issues & Future Directions
Incorporate aerodynamic and structural effects due to large shape changing
Cost Function– Potential components
• Specified CL
• Minimum drag• Minimum peak stress
Degrees of Freedom– Thickness
• 6% to 24%– Camber
• 0% to 10%– Max camber location
• 0.2c to 0.8c– Chord
• 1 unit to ? units– Angle-of-attack
• -5° to 10 °– Within linear range
R-L For Morphing Airfoils & Wings
![Page 56: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/56.jpg)
Valasek, Lampton, Niksch - 72
Morphing Airfoil Demonstration
![Page 57: Machine Learning for Control of Morphing Air Vehicles John Valasek, Amanda Lampton, Adam Niksch Aerospace Engineering Department Texas A&M University SAE](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ea85503460f94bac1f6/html5/thumbnails/57.jpg)
Valasek, Lampton, Niksch - 73
Questions?
John Valasek
Aerospace Engineering Department
Texas A&M University
3141 TAMU
College Station, TX 77843-3141
(979) 845-1685
FSL Web Page
– http://flutie.tamu.edu/~fsl