colloquium on optimization for control 1 dynamic optimization and automatic differentiation yi cao...

Colloquium on Optimization for Control

1

Dynamic Optimization and Dynamic Optimization and Automatic DifferentiationAutomatic Differentiation

Yi CaoSchool of EngineeringCranfield University

Colloquium on Optimization for Control2

OutlineOutline

• Dynamic optimization problems• Parameterization• Recursive high-order Taylor series• ODE and sensitivity solver• Dynamic optimization solver• Differential recurrent neural

network• Continuous-time NMPC• Conclusions


Dynamic optimization problemDynamic optimization problem

• Controller design, =const controller parameter • Adaptive control, =variable controller

parameter• System identification, =model parameter• Predictive control, =control action, variable• State estimation, =initial state• ……

0

min ( , ( ), ( )) ( , , )

( , , , ) 0

. . ( , , ) 0

( , ( ), ( )) 0

ft

f f f t

f f f

t x t t t x dt

f t x x

s t g t x

h t x t t


Solving dynamic optimizationSolving dynamic optimization

• Optimal control theory well established in 1960s.

• Challenge for numerical solutions:– Complex / large scale problems– Efficiency for realtime optimization– Global optimization– Stability– Robustness


Differentiation and dynamic Differentiation and dynamic optimizationoptimization

• H=Φ(t,x,)+λ’f(t,x,)• Optimal conditions:

dx/dt= Hλ=f(t,x,),

dλ/dt=–Hx=–Φx(t,x,)–fx(t,x,)λ

H=Φ(t,x,)+f(t,x,)λ=0

• Efficient solution requires efficient differentiation


Differentiation approachesDifferentiation approaches

• Analytic differentiation manuallyNot a trivial task for large scale problem

• Analytic differentiation using symbolic computing softwareVery complicated results even for a small problem

• Numerical finite differenceInefficient and inaccurate


Automatic Differentiation (1)Automatic Differentiation (1)

• Techniques use computer programs to get derivatives of any functions represented in other computer programs with the same accuracy and efficiency as the function.

• Synonym: Algorithmic Differentiation• First proposed by Johnannes Joos,

1976 in his PhD Thesis, ETH, Zurich



• Factor: every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations such as additions or elementary functions such as exp().

• By applying the chain rule of derivative calculus repeatedly to these operations, derivatives of arbitrary order can be computed automatically, and accurate to working precision.



Two modes to calculate derivatives:• Forward mode: y=f(x(t))

• Reverse mode (adjoint): y=f(x(t))

'( )dx

x y f x xdt

'( ) 'dy

v x f x y t x xdv



Two ways to implement AD• Operator overloading

Each elementary operation is replaced by a new one, working on pairs of value and its derivative (doublet).

• Source transformationProduce new code to calculate derivative based on original code of a function


AD Example: Forward ModeAD Example: Forward Mode

x1’=1, x2’=0v1’=(x1’x2-x1x2’)/x2/

x2=2v2’=cos(v1)*v1’= –1.98 v3’=exp(x2)*x2’=0v4’=v1’-v3’=2y’=v2’+v4’=0.02

x1=1.5, x2=0.5v1=x1/x2=3 v2=sin(v1)=0.1411v3=exp(x2)=1.649v4=v1–v3=1.3513y=v2+v4=1.4924

y=sin(x1/x2)+x1/x2–exp(x2)


x1=1.5, x2=0.5v1=x1/x2=3v2=sin(v1)=0.1411v3=exp(x2)=1.649v4=v1–v3=1.3513y=v2+v4=1.4924

AD Example: Reverse ModeAD Example: Reverse Mode

x2=x2+v1*x1/x2/x2=-1.589

x1=v1/x2=0.02v1=v2*cos(v1)+v1=0.01 x2=v3*exp(x2)=-1.649v1=v4=1, v3=–v4=–1y=1, v2=y=1, v4=y=1

y=sin(x1/x2)+x1/x2–exp(x2)


Automatic Taylor expansionAutomatic Taylor expansion

• z(t)=f(x(t)), t scalar, x and z vectors• TS: x(t)=x0+x1t+x2t2+…+xdtd • TS: z(t)=z0+z1t+z2t2+…+zdtd

• AD forward (TS of z): zk = zk(x0,x1,…,xk)

• AD reverse (TS of sensitivity): zk /xj = zk-j / x0 := Ak-j

• f’x=A0+A1t+A2t2+…+Adtd


Recursively solving ODE using TSRecursively solving ODE using TS

• dx/dt=f(x)• dx/dt=z(t)• Recursive relation: xk+1=zk/(k+1).• x0=x(t0),

• x1=z0(x

0),

• x2=z1(x0,x1), …• x(t0+h)=d

i=0 xihi • Next step: t1=t0+ht0


Solving ODE with sensitivity Solving ODE with sensitivity equationequation

• Sensitivity equation:dx/dt=fx(t,x,)x+f(t,x,)

• v={x0,}

• Avk:=zk/v • Bvk:=dxk/dv• Bvk+1= (Avk+k

j=0Axk-jBvj)/(k+1)• Bv:=dx(t0+h)/dv

=Bv0+Bv1h+Bv2h2+…+Bvdhd


Solving dynamic optimization using Solving dynamic optimization using TSTS

• Convert cost to terminal cost:min F(tf,x(tf),(tf))

• Initial guass: 0={0(t0),…, 0(tf)}• Step: hi=ti+1-ti>0, i=0,1,…,n-1, tn=tf

• Integrate ODE & sensitivity from t0 to tf • x(t0), x(t1), …, x(tf) and F(tf,x(tf),0(tf))• Bx(t1),…,Bx(tf), B(t1),…,B(tf) and Fx, F • dF/d0(ti)=FxBx(tf)…Bx(ti+2)B(ti+1)• For constant , dF/d0= dF/d0(ti)

• Update i+1

=i- dF/di


Error ControlError Control

• h=step, d=order of TS• (h,d)=C(h/r)d+1

• r ≈ rd=||xd-1||/||xd|| for large d

• (h,d-1) ≈ (h,d)(rd/h)≤(h,d)+||xd||

• (h,d)≤h||xd||2/(||xd-1||-h||xd||)

• Given tolerance and d, determine h• Given tolerance, determine optimal d, h to

minimize computation.

• Global error: ≥||Bx||i and g global tolerance

• local tolerance: =g(-1)/(n-1)


NMPC using differential recurrent NNNMPC using differential recurrent NN

• Continuous-time nonlinear identification

• Efficient algorithm to train DRNN• Training performance and

efficiency• DRNN as internal model for NMPC• Efficient algorithm for NMPC• NMPC performance and efficiency


Differential recurrent neural Differential recurrent neural networksnetworks

C

W uW 2

W x

b 1 b 2

u

y

x

Input

Hidden

Output

2 1 1 1 2( )( , )

( )x ux W W x W u b bz f z u

y g z y Cx


DRNN TrainingDRNN Training

• ={b1,b2,vec(W2),vec(W1x),vec(W1u)}

• N=Nh+Nx+Nh×Nu+2Nh×Nx

• Training data: u(t), y(t) at t=0,h,…,Nh• Solving DRNN + sensitivity using TS• Assume t=0 is steady-state, apply u(t)

• y(t) at t=h,…,Nh and ek=y(kh)-y(kh)

• min keTkek/2=ETE/2, E=vec(e1,…,eN)

• Nonlinear least square (NLSQ) optimization


Continuous time NMPCContinuous time NMPC

• minu φ=∫0T(y-yr)TQ(y-yr)+(u-ur)TR(u-ur)dt/2

• s.t. dx/dt=f(x,u), y=g(x)

• umin ≤ u ≤ umax, 0=t0 ≤ … ≤ tP=T

• Parameterize u: piecewise polynomial

• u(t)=qi=0uki(t-tk)i tk ≤ t ≤ tk+1, k=0,…,P-1

• U=[uT00,…,uT

0q,…,uTP-10,…,uT

P-1q]T

• Y=[yT00,…,yT

0d,…,yTP-10,…,yT

P-1d]T

• φ=T(YeTHQYe+Ue

THRUe)/2

• φ=ETE/2 NLSQ


NMPC algorithm (with DRNN)NMPC algorithm (with DRNN)

• x0 (steady-state or estimated), ym

• d=ym-Cx0 (constant for 0 ≤ t ≤ T)

• U0 Ue=U0-Ur X Y Ye=Y-Yr+d

• Jacobian, J=E/U

• Unconstrained: Uk+1=Uk-(JTJ)-1JTE

• Input constrained: lsqnonlin

• Other constraints: fmincon/SQP

• Apply [uT00,…,uT

0q]T

• repeat


Tank Reactor ExampleTank Reactor Example

• 2-CSTR in series, reaction A+B C

• 2-output: T1 and T2 ,2-input: Qcw1 and Qcw2

• Distuabnce: Tcw

• 6 states

CSTR1 CSTR2

Mixer

Feed 1

WaterIn

WaterOut

WaterOut

Water In

Feed 2

Cooling Cooling

Product

CoolingCooling


DRNN identificationDRNN identification

• Nh=6, Nx=6, Nu=2, N=96

• 600-s data with h=0.1 s, N=6000• Total sensitivity 3456000• Validation set sampled at 0.02 s• Advantage: change sampling rate

does not need re-training


Training and ValidatingTraining and Validating

0 200 400 600350

360

370

380T O

1, o K

Time, sec

(a)

0 200 400 600355

360

365

370

T O2,

o K

Time, sec

(b)

0 200 400 600350

360

370

380

390

T O1,

o K

Time, sec

(c)

0 200 400 600350

360

370

380

T O2,

o K

Time, sec

(d)


Training efficiency (one epochTraining efficiency (one epoch))

ODE23 AD

tol t, ms error d t, ms error

103 40.61 1.590 3 2.437 0.001

106 162.28 0.074 6 4.078 1.6×10-7

108 272.66 4.6×10-

4

8 5.391 2.6×10-

10

101

0

316.37 1.9×10-

6

10 6.937 1.1×10-

12


NMPC performance, setpoint changeNMPC performance, setpoint change

0 50 100 150 200360

365

370

To1

, o K

0 50 100 150 200360

365

370

To2

, o K

0 50 100 150 2000

0.5

1

Qcw

1, m

3 /s

time, sec

0 50 100 150 2000

0.5

1

Qcw

2, m

3 /s

time, sec


NMPC performance, disturbance NMPC performance, disturbance rejectionrejection

0 10 20 30362.94

363

363.02

T o1 (

o K)

0 10 20 30362.94

363

363.02

T o2 (

o K)

0 10 20 300

0.2

0.4

0.6

0.8

Qcw

1 ( m

3 /s)

Time, sec0 10 20 30

0

0.2

0.4

0.6

0.8

Qcw

2 ( m

3 /s)

Time, sec


ConclusionsConclusions

• High-order TS using AD• Efficiently solving ODE + sensitivity• Efficient algorithm for general

dynamic optimization• Efficient algorithm for RDNN training• Efficient algorithm for continuous-

time NMPC• Demonstrated with 2-CSTR example.

colloquium on optimization for control 1 dynamic optimization and automatic differentiation yi cao...

Documents

f x b x t f b x t i

t f xt

x d t d ts

d t d slide

t scalar

t i update i

t f initial guass

z d t d ad forward ts