prospects of gradient methods for nonlinear control ivo bukovskÝ 1 jiří bÍla 1 noriasu homma 2...

PROSPECTS OF GRADIENT METHODS FOR NONLINEAR CONTROL

Ivo BUKOVSKÝ1

Jiří BÍLA1

Noriasu HOMMA2

Ricardo RODRIGUEZ1

1Czech Technical University in Prague

2Tohoku University, Japan

PROSPECTS OF GRADIENT METHODS FOR NONLINEAR CONTROL

• We consider sample-by-sample adaptation of discrete-time models and controllers by gradient descent

2( )( 1) ( ) ;

... adaptable parameter of a model or controler

kk ki i

i

thi

Qw w i

w

w i

weight update system

• Stability monitoring and maintenance of weight update system of adaptively tuned models and controllers significantly contributes to a stable and convergent control loop

PROSPECTS OF GRADIENT METHODS FOR NONLINEAR CONTROL

PROSPECTS OF GRADIENT METHODS FOR NONLINEAR CONTROL

• In the paper, we introduce derivation of stability condition for gradient-descent tuned models and controllers

• The approach is valid for models and controllers that are nonlinear (incl. linear), but they are linear in parameters– Not suitable for conventional neural networks (MLP,

RBF)– Suitable for Higher-Order Neural Units (HONU, also

known as polynomial neural networks) (not limited to)

PROSPECTS OF GRADIENT METHODS FOR NONLINEAR CONTROL

Further in this presentation

• Fundamental gradient descent schemes for adaptive identification and control

• Static or dynamic Higher Order Neural Units (HONU)

• Stability conditions for static and dynamic HONU and its maintenance at every adaptation step

• Demonstration of achievements with ONU( NOx prediction – EME I, lung motion prediction, nonlinear control loop of a laboratory system)

Plant

Adaptive model-linear

- neural network,

+-

2( )( 1) ( ) ;

kk ki i

i

ew w i

w

( )ku( )krealy

Fundamental gradient descent schemes for adaptive identification and control

Plant Identification by Gradient Descent

( )ke

( )ky

... neural weights

(adaptable parameter)

... control variableu

w

weight update system

(Základní schemata adaptivní identifikace a řízení gradientovými metodami)

Automatické ladění adaptivního stavového regulátoru

Regulovanásoustava

Adaptivní regulátor-lineární

- polynomiální-- klasická neuronová síť

2( )( 1) ( ) ;

kk ki i

i

ew w i

w

( )kv ( )krealy

Referenční model (požadované chování regulované soustavy)

+-

Žádaná hodnota

+-

( )kdesiredy

... adaptovatelný parametr

(váhy u neuronových sítí)

... žádaná hodnota

w

v

Systém adaptovaných

vah

( )ke

Žádaný průběh chování

Fundamental gradient descent schemes for adaptive identification and control (continue)

Tuning of Adaptive Controller in a Feedback Control Loop with Gradient Descent

Plantadaptive controller

- linear PID - neural network,

2( )( 1) ( ) ;

kk ki i

i

ew w i

w

( )kv

( )krealy

Model of desired behavior

+-

( )kdesiredy

... neural weight ,

(adaptable parameter)

... desired value

w

v

+-

2( )( 1) ( ) ;

kk ki i

i

ew w i

w

Plant

Adaptive model-linear

- neural network,

+-

( )ku( )krealy

Fundamental gradient descent schemes for adaptive identification and control (continue)

Updating Control Inputs Directly by Gradient Descent

( )ke

( )ky

2( )( 1) ( ) c

kck ki

eu u

w

+-

( )kdesiredy

eC(k)

The question is:

• How do we assure stability of nonlinear adaptive control loop?• The ways is to assure stability and convergence of adaptive

components in a control loop (plant model + controller)• What nonlinear model to use?

• MLP or RBF networks as models and controllers– Not linear in parameters– Guaranteeing stability is complicated (not

suitable for undergraduate level, difficult for PhD students from non-heavy-math schools)

– Guaranteeing stability is complicated and theoretically heavy for practicioners (thus not attractive for practice)

Static & Dynamic Higher-Order Neural Units

How do we assure stability of the nonlinear adaptive control loop? What model to choose?

Static & Dynamic Higher-Order Neural Units

How do we assure stability of the nonlinear adaptive control loop? What model to choose?

2( )( 1) ( ) ;

kk ki i

i

ew w i

w

Weight-update system:

Example of 2nd-order HONU: 1

( ...)

( ...)

( )

k

k

k

y

y

u

( )kx

( )sk ny

0

r rn n

i j iji j i

x x w

20 0 0 1 0 2 i j ny x x x x x x x x x 0,0 0,1 0,2 i,j n,nw w w w w

“axis of adapted neural weights”

LNU

HONU

convetional NN

2( )k

k

e

0

Approximation strength of neural networks can be improved by adding more neurons or even layers, GA, PSO,…

Static & Dynamic Higher-Order Neural Units (continue)

Sketch of optimization error surfacesLinear x MLP Networks x HONU

Static & Dynamic Higher-Order Neural Units (continue)

Static MLP vs. QNU as MISO models of hot steam turbine averaged data (“steady states”, batch training by Levenberg-Marquardt)

• double hidden layer FFNN

• single hidden layer FFNN

• static QNU• measured data

Static & Dynamic Higher-Order Neural Units (continue)Respiration time series: Training Accuracy for Predicting Exhalation Time -Instances of trained neural architectures trained from different initial conditions by L-M algorithm

2-hidden-layer static MLPs (static feedforward networks)

1-hidden-layer static MLPs (static feedforward networks)

static

QNUs

0 50 100 150

0

20

40

60

80

100

trénovacích epoch

JRNN

JDLNU

JDQNU

trénovacích epoch

Trénování predikce Mackey-

Glass

0 50 100 150

0

20

40

60

80

100

trénovacích epoch

JDQNU

JDLNU

JRNN

Trénování predikce polohy plic0 20 40 60

0

20

40

60

80

trénovacích epoch

JRNN

JRNN

JDQNU

Trénování predikce nelineárního periodického

signálu

0 50 100 150

0

20

40

60

80

100

trénovacích epoch

JRNN

JDLNU

JDQNU

trénovacích epoch

Static & Dynamic Higher-Order Neural Units (continue)

0,0 0 0 0,1 0 1 0,2 0 2

2, ,... i j i j n n n

y w x x w x x w x x

w x x w x

0,0

0,1

0,20 0 0 1 0 2

,

n n

n n

w

w

wy x x x x x x x x

w

rowx colW

1( ...)

( ...)

( )

k

k

k

y

y

u

( )kx

( )sk ny

0

r rn n

i j iji j i

x x w

Static & Dynamic Higher-Order Neural Units (continue)

Stability of weight-update system

• Condition for STATIC HONU

• Condition for DYNAMICAL HONU

( ) ( ) 1k k 1 M colx rowx

( )( ) ( ) ( ) 1

kk n k kse

rowx1 M colx rowx

colW

,

HONU

1( ...)

( ...)

( )

k

k

k

y

y

u

( )kx

0 50 100 150 200 250 300 350 400-2

-1

0

1One Epoch of GD Adaptation of Recurrent QNU to Predict MacKey-Glass Equation (training data vs. neural output)

0 50 100 150 200 250 300 350 400-0.5

0

0.5Prediciton Error during the Epoch of Adaptation

0 50 100 150 200 250 300 350 4000.98

1

1.02

1.04

k

Spectral Radius during the Epoch (stability of weight update system at each adaptation step)

0 100 200 300 400 500 600 700 800

-2

0

2

k

GD Adaptation of Recurrent QNU to Predict MacKey-Glass Equation (training data vs. neural output)

0 100 200 300 400 500 600 700 800-10

0

10

k

Prediciton Error during Adaptation

0 100 200 300 400 500 600 700 800

1

1.5

2

k

Spectral Radius during Adaptation (stability of weight update system at each adaptation step)

600 620 640 660 680 700 720 740 760 780 800

-2

0

2

k

GD Adaptation of Recurrent QNU to Predict MacKey-Glass Equation (training data vs. neural output)

600 620 640 660 680 700 720 740 760 780 800-10

0

10

k

Prediciton Error during Adaptation

600 620 640 660 680 700 720 740 760 780 800

1

1.05

1.1

k

Spectral Radius during Adaptation (stability of weight update system at each adaptation step)

Achievements with QNU

250 300 350 400

-1

-0.5

0

0.5

1

1.5

t [min]

NOx,CO prediction – EME I

trénování testování

Obr. 1: Dobře natrénovaná síť TptRNN pro 3-minutovou predikci klouzavých 3-minutových průměrů NOx, externí měřené vstupy jsou klapky a výkon , (klouzavé průměry se počítají jako průměry předchozích, současných a následujících hodnot, při intervalu predikce 3 minuty to znamená, že externí vstupy jsou již dostupné ale model v principu predikuje 3-minutový průměr který má být za 2 minuty), včase cca 415 ignoruje výpadek měření NOx a výstup modelu dobře nahrazuje měření.

Lung Tumor Motion Prediction

0 500 1000 1500 2000 2500 3000 3500-2

-1

0

1

2

Late

ral a

xis

[mm

]

0 500 1000 1500 2000 2500 3000 3500-10

-5

0

5

Ceph

aloc

auda

l axi

s [m

m]

0 500 1000 1500 2000 2500 3000 3500-2

-1

0

1

2

k

Ante

ropo

ster

ior A

xis

[mm

]

y1

y2

y3

20 40 60 80 100 120

t [sec]

-8

-6

-4

-2

0

2

4

6testing MAE= 0.853120295578 [mm], RMSE= 1.14143756682, treatment time = 86[sec], computing time= 83.385[sec]

20 40 60 80 100 120t [sec]

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0absolute value of prediction error

Lung Tumor Motion Prediction by static QNU

sampling 15 Hz, epochs=100, Ntrain=360, 492 neural weights

Lung Tumor Motion Prediction by static QNU

10^0 10^1 10^2

epochs

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040 Averaged normalized SSE of Retrainings

Nonlinear Control Loop of a Laboratory System

[ ] Ladislav Smetana: Nonlinear Neuro-Controller for Automatic Control,Laboratory System, Master’s Thesis, Czech Tech. Univ. in Prague, 2008.

Nonlinear Control Loop of a Laboratory System

PID Control and Nonlinearity of the Plant

0 100 200 300 400 500 600-30

-25

-20

-15

-10

-5

0

t [s]

y [c

m]

Prubeh PID regulace v zavislosti na hloubce ponoru batyskafu, serizeno na hloubku 20 cm

5 cm

10 cm

15 cm20 cm

25 cm

0 100 200 300 400 500 600-40

-35

-30

-25

-20

-15

-10

-5

0

t [s]

y [c

m]

Prubeh PID regulace v zavislosti na hloubce ponoru batyskafu, serizeno na hloubku 10 cm

5 cm

10 cm

15 cm20 cm

25 cm

Tunned PID controller for 10 cm

30

Tunned PID controller for 20 cm

Nonlinear Control Loop of a Laboratory System

0 100 200 300 400 500 600-30

-25

-20

-15

-10

-5

0

t [s]

y [c

m]

Prubeh PID regulace v zavislosti na hloubce ponoru batyskafu, serizeno na hloubku 20 cm

5 cm

10 cm

15 cm20 cm

25 cm

0 100 200 300 400 500 600-40

-35

-30

-25

-20

-15

-10

-5

0

t [s]

y [c

m]

Prubeh PID regulace v zavislosti na hloubce ponoru batyskafu, serizeno na hloubku 10 cm

5 cm

10 cm

15 cm20 cm

25 cm

310 20 40 60 80 100 120 140

-25

-20

-15

-10

-5

0

t [s]

y [c

m]

Prubeh regulace neuro-regulatoru zavislosti na hloubce ponoru batyskafu

5 cm

10 cm

15 cm20 cm

25 cm

QNU as Adaptive Controller (simplest gradient descent)

Linear PID

Nonlinear Control Loop of a Laboratory System

False Neighbor Analysis is a single-scale analysis

x yyf )(x

( )

( ) ( )

i

j i

x

x x

( )

( ) ( )

i

j i

y

y y

To train neural networks , input (state) vector must be estimated to minimize uncertainty in training data

Děkuji za pozornost

y=f(x)x input data y output data

False Neighbors

1 2 IF AND

THEN and are False Neighbors

=> How much is correct Rx and Ry? - we do not know

=> Let's characterize false neighbors over whole intervals

of Rx and Ry, an

x yR y y R 1 2

1 2

x x

x x

d not just for their single setup

False Neighbor Analysis is a single-scale analysis

Slope of FN in Log-Log plot

FN = 4.2239*log2(id) - 4.5879

-2

0

2

4

6

8

1 1.5 2 2.5 3

log2(id)

log2(FN) Linear (log2(FN))

q(k ) c r (k )H

( )

( )

log

log log

k

k

q

c H r

MULTI-SCALE ANALYSIS approach (MSA)

number of false neighbours on a main diagonal

0

50

100

150

1 2 3 4 5 6

id...index of a diagonal cell

FN

• To characterize a system over the range of setups

• Power law

•What is the fundamental idea?

MULTI-SCALE ANALYSIS approach (MSA)• What is the fundamental idea?

q(k ) c r (k )H

q … quantityH … characterizing exponentr(k) … discretely growing radius

r(k)=2,4,8

•To characterize a system over the range of intervals•The power-law concept

MULTI-SCALE ANALYSIS approach (MSA)• What is the fundamental idea?

q(k ) c r (k )H

q … quantityH … characterizing exponentr(k) … discretely growing radius

r(k)=2,4,8

•To characterize a system over the range of intervals

•The power-law concept

MULTI-SCALE ANALYSIS approach (MSA)• What is the fundamental idea?

k r(k) q A q B

1 2 4 22 4 13 113 8 44 44

r(k)=2,4,8

log2(qB) = 2.2297*log2(r) - 1.1531

log2(qA) = 1.7297*log2(r) + 0.2605

0.9

1.9

2.9

3.9

4.9

1 1.5 2 2.5 3log2(r(k))

q(k ) c r (k )H

MULTI-SCALE ANALYSIS approach (MSA) (cont.)

• How can MSA help to create better neural network models?

j =1 j =2 j =3 j =4 j =5

i=1

max FN (highest chance that y1≠y2

when x1=x2 )

i=2 FN (2,2)

i=3 FN (3,3)

i=4 FN (4,4,)

i=5

min FN (lowest chance that y1≠y2

when x1=x2 )

FN (i ,j ) … count of False Neighbors for Rx (i ) and Ry ( j )

Rx(i

)

Ry ( j )

Smallest Rx - maximum of different states of a system

Largest Rx - minimum of different states of a system

Smallest Ry - maximum of recognized different outputs

Largest Ry - minimum of recognized different outputs

FN decrease

FN decreases

ffecf F

N d

ecre

ase

ffecf F

N d

ecre

ase

( )f yxj

i

False Neighbors Matrix:

Multiscale False Neighbor Approach

MULTI-SCALE ANALYSIS approach (MSA) (cont.)

• What are other potentrials for MSA for signal processing?

• MSA based signal processing

• Variance Fractal Dimension Trajectory (VFDT)

• Mutual Information

– Multiscale approach to calculate mutual information itself

– Mutual information of VFDT processed signals

• Everywhere, where a common analysis is subject to a

single-parameter setup and changing the setup disqualifies

the analysis results.