higher derivatives in matlab using mad euroad workshop... · requests for higher derivatives of...

Higher Derivatives in Matlab Using MAD

Shaun Forth

Engineering Systems DepartmentCranfield University (DCMT Shrivenham)

Shrivenham, Swindon SN6 8LA, U.K.email: [email protected]

www.amorg.co.uk/AD/staff_SAF.html

5th European Workshop on Automatic DifferentiationUniversity of Hertfordshire, UK, May 21-22 2007

1/ 29 Higher Derivatives in Matlab Using MAD

www.amorg.co.uk/AD/staff_SAF.html

Plan

1 Introduction

2 MAD’s Forward Modefmad classderivvec class

3 Higher DerivativesSecond DerivativesHow The Heck Does This Work?Required Changes to MAD

4 Reverse Mode AD in Matlab

5 ResultsMinpack Elastic-Plastic Torsion (EPT) ProblemObject-Oriented Test Case

6 Conclusions & Future Work


Introduction

MAD’s overloaded forward mode is efficient for firstderivatives [For06].

Requests for higher derivatives of order 2 to 4 for uncertainty analysisand robust optimisation.

Could implement a Taylor series class a la ADOL-C [GJU96] -manpower expensive.

OR use MAD to recursively differentiate itself (c.f. EuroAD2 - BarakPerlmutter and Jeffrey Siskind, [Gri00, p109]).


MAD’s Forward Mode

Based on two classes:

fmad class - forward mode AD by operator overloading

derivvec class - for storage and combination of multiple directionalderivatives.


fmad Class Constructor

e.g. x=fmad([1.1 2 3],[4 5 6]);

Defines fmad object withI value component - row vector [1.1 2 3]I deriv component - single directional derivative [4 5 6]

Perform overloaded operations, e.g., element-wise multiplication viatimes

z=x.*xvalue = 1.2100 4.0000 9.0000derivatives = 8.8000 20.0000 36.0000

AD enabled by the times.m function of the fmad class.


fmad times function for z=x.*y

function z=times(x,y)if isa(x,’fmad’)

z=x; % deep copy to avoid constructor, no refs. to xif isa(y,’fmad’)

z.deriv=y.value.*z.deriv+z.value.*y.deriv;z.value=z.value.*y.value;

elsez.value=z.value.*y;z.deriv=y.*z.deriv;

endelse

z=y; % deep copy to avoid constructor, no refs. to yz.value=x.*z.value;z.deriv=x.*z.deriv;

end


Working with Multiple Directional Derivatives

What if we want the Jacobian?

Seed derivatives with identity I3

x=fmad([1.1 2 3],eye(3));

Overloaded operation with same times function gives

value = 1.2100 4.0000 9.0000DerivativesSize = 1 3No. of derivs = 3derivs(:,:,1) = 2.2000 0 0derivs(:,:,2) = 0 4 0derivs(:,:,3) = 0 0 6


The fmad and derivvec classes

The fmad constructor function

function xad=fmad(x,dx)% FUNCTION: FMAD% SYNOPSIS: Class constructor for forward Matlab AD objectsxad.value=x;sx=size(xad.value);sd=size(dx);

if prod(sx)==prod(sd)% #values=#derivs => single directional derivativexad.deriv=reshape(dx,sx);

else% #values~=#derivs => multiple directional derivatives% Pass derivatives to derivvec constructorxad.deriv=derivvec(dx,size(xad.value));

end


The derivvec class

Store derivatives as a matrix with each directional derivative unrolledinto a column.

e.g. derivvec(eye(3),[1 3]) derivatives conceptualised as as,[direc 1 direc 2 direc 3[1, 0, 0] [0, 1, 0] [0, 0, 1]

]But stored as,

direc 1 direc 2 direc 3 100

010

001

=

1 0 00 1 00 0 1


The derivvec class



]

But stored as,direc 1 direc 2 direc 3 1

00

010

001

=

1 0 00 1 00 0 1


The derivvec class



]But stored as,


010

001

=

1 0 00 1 00 0 1


The times operation of the derivvec class

e.g. Need to calculate,

[1.1 2 3

]. ∗


00

010

001

with multiplication of each of the 3 directional derivatives.

Convert value to column matrix and replicate 3 times 1.1 1.1 1.12 2 23 3 3

. ∗

1 0 00 1 00 0 1

=

1.1 0 00 2 00 0 3

columns give required directional derivatives.


Accessor Functions

Getting the value

getvalue(z)ans = 1.2100 4.0000 9.0000

Getting external representation of derivatives

getderivs(z)ans(:,:,1) = 2.2000 0 0ans(:,:,2) = 0 4 0ans(:,:,3) = 0 0 6

Getting unrolled internal representation

getinternalderivs(z)ans = 2.2000 0 0

0 4 00 0 6


Higher Derivatives

Basic idea is to be able to use MAD recursively.

For first derivatives:

x=fmad([1.1 2 3],eye(3));z=x.*xzvalue=getvalue(z)

zvalue = 1.2100 4.0000 9.0000zderivs=getinternalderivs(z)

zderivs = 2.2000 0 00 4.0000 00 0 6.0000

Can we just differentiate the above process a second time with MAD?


Second Derivatives

xx=fmad([1.1 2 3],eye(3)); % xx’s value and derivs. D1=I_3x=fmad(xx,eye(3)); % x’s value =xx, x’s derivs. D2=I_3z=x.*x;zvalue=getvalue(getvalue(z)) % z’s value’s value

zvalue = 1.2100 4.0000 9.0000zderivs=getinternalderivs(getvalue(z)) % z’s value’s derivs

% in D1 direcs.zderivs = 2.2000 0 0

0 4.0000 00 0 6.0000

zderivs=getvalue(getinternalderivs(z)) % value of z’s derivs% in D2 direc.

zderivs = 2.2000 0 00 4.0000 00 0 6.0000


Second Derivatives (ctd)

z2ndderivs=reshape(...getinternalderivs(getinternalderivs(z)),[3 3 3])% derivs. in D1 direc. of z’s derivs. in D2 direc.z2ndderivs(:,:,1) = 2 0 0

0 0 00 0 0

z2ndderivs(:,:,2) = 0 0 00 2 00 0 0

z2ndderivs(:,:,3) = 0 0 00 0 00 0 2


Why The Heck Does This Work?

xx=fmad([1.1 2 3],eye(3)); creates object,

xx

[1.1 2 3] eye(3)

value deriv

x=fmad(xx,eye(3)); creates object,

x

eye(3)xx

[1.1 2 3] eye(3)

value deriv

value deriv


How The Heck (ctd.)

z=x.*x; forms object,z

2.*x.value.*x.derivx.value.*x.value

value deriv

xx

[1.1 2 3] eye(3)

value deriv.*

xx

[1.1 2 3] eye(3)

value deriv[1.21, 4.0, 9.0]

2.2 0 00 4 00 0 6

value

deriv

xx

[1.1 2 3] eye(3)

value deriv.* 2 eye(3)

xx

1.1 1.1 1.12 2 23 3 3

[I3, I3, I3]

value deriv.* 2 I32.2 0 0

0 4 00 0 6

[I3, I3, I3] .* 2

e1

e2

e3

value

deriv2.2 0 00 4 00 0 6

I3I3I3

.* 2

e1 e1 e1

e2 e2 e2

e3 e3 e3

value deriv2.2 0 0

0 4 00 0 6

e1 0 00 e2 00 0 e3

value deriv

deriv


How The Heck (ctd.)


2.*x.value.*x.deriv

x.value.*x.value

value deriv

xx

[1.1 2 3] eye(3)

value deriv.*

xx

[1.1 2 3] eye(3)

value deriv

[1.21, 4.0, 9.0]

2.2 0 00 4 00 0 6

value

deriv

xx

[1.1 2 3] eye(3)


xx

1.1 1.1 1.12 2 23 3 3

[I3, I3, I3]


0 4 00 0 6

[I3, I3, I3] .* 2

e1

e2

e3

value

deriv2.2 0 00 4 00 0 6

I3I3I3

.* 2

e1 e1 e1

e2 e2 e2

e3 e3 e3

value deriv2.2 0 0

0 4 00 0 6

e1 0 00 e2 00 0 e3

value deriv

deriv


How The Heck (ctd.)



value deriv

xx

[1.1 2 3] eye(3)

value deriv.*

xx

[1.1 2 3] eye(3)

value deriv

[1.21, 4.0, 9.0]

2.2 0 00 4 00 0 6

value

deriv

xx

[1.1 2 3] eye(3)


xx

1.1 1.1 1.12 2 23 3 3

[I3, I3, I3]


0 4 00 0 6

[I3, I3, I3] .* 2

e1

e2

e3

value

deriv2.2 0 00 4 00 0 6

I3I3I3

.* 2

e1 e1 e1

e2 e2 e2

e3 e3 e3

value deriv2.2 0 0

0 4 00 0 6

e1 0 00 e2 00 0 e3

value deriv

deriv


How The Heck (ctd.)



value deriv

xx

[1.1 2 3] eye(3)

value deriv.*

xx

[1.1 2 3] eye(3)

value deriv

[1.21, 4.0, 9.0]

2.2 0 00 4 00 0 6

value

deriv

xx

[1.1 2 3] eye(3)


xx

1.1 1.1 1.12 2 23 3 3

[I3, I3, I3]

value deriv.* 2 I3

2.2 0 00 4 00 0 6

[I3, I3, I3] .* 2

e1

e2

e3

value

deriv2.2 0 00 4 00 0 6

I3I3I3

.* 2

e1 e1 e1

e2 e2 e2

e3 e3 e3

value deriv2.2 0 0

0 4 00 0 6

e1 0 00 e2 00 0 e3

value deriv

deriv


How The Heck (ctd.)



value deriv

xx

[1.1 2 3] eye(3)

value deriv.*

xx

[1.1 2 3] eye(3)

value deriv

[1.21, 4.0, 9.0]

2.2 0 00 4 00 0 6

value

deriv

xx

[1.1 2 3] eye(3)


xx

1.1 1.1 1.12 2 23 3 3

[I3, I3, I3]

value deriv.* 2 I3

2.2 0 00 4 00 0 6

[I3, I3, I3] .* 2

e1

e2

e3

value

deriv

2.2 0 00 4 00 0 6

I3I3I3

.* 2

e1 e1 e1

e2 e2 e2

e3 e3 e3

value deriv2.2 0 0

0 4 00 0 6

e1 0 00 e2 00 0 e3

value deriv

deriv


How The Heck (ctd.)



value deriv

xx

[1.1 2 3] eye(3)

value deriv.*

xx

[1.1 2 3] eye(3)

value deriv

[1.21, 4.0, 9.0]

2.2 0 00 4 00 0 6

value

deriv

xx

[1.1 2 3] eye(3)


xx

1.1 1.1 1.12 2 23 3 3

[I3, I3, I3]


0 4 00 0 6

[I3, I3, I3] .* 2

e1

e2

e3

value

deriv

2.2 0 00 4 00 0 6

I3I3I3

.* 2

e1 e1 e1

e2 e2 e2

e3 e3 e3

value deriv

2.2 0 00 4 00 0 6

e1 0 00 e2 00 0 e3

value deriv

deriv


How The Heck (ctd.)



value

deriv

xx

[1.1 2 3] eye(3)

value deriv.*

xx

[1.1 2 3] eye(3)

value deriv

[1.21, 4.0, 9.0]

2.2 0 00 4 00 0 6

value

deriv

xx

[1.1 2 3] eye(3)


xx

1.1 1.1 1.12 2 23 3 3

[I3, I3, I3]


0 4 00 0 6

[I3, I3, I3] .* 2

e1

e2

e3

value

deriv2.2 0 00 4 00 0 6

I3I3I3

.* 2

e1 e1 e1

e2 e2 e2

e3 e3 e3

value deriv

2.2 0 00 4 00 0 6

e1 0 00 e2 00 0 e3

value deriv

deriv


Required Changes to MAD

Few changes needed

Made derivvec class superiorto(’fmad’).

Subscript Referencing - make via direct calls to fmad’s subsreffunction.

For efficiency - eliminate unnecessary temporary fmad objects created.


Made derivvec class superiorto(fmad)

derivvec Constructor Function

function dv=derivvec(x,shape,nd)..otherwise

error(’derivvec must have 1 or 2 inputs’)endsuperiorto(’fmad’)

Required for binary operations between fmad and derivvec objects.

e.g. xfmad.*yderivvec

Ensures object precedence is such that derivvec operation calledand not the fmad operation.

Enables differentiation through the derivvec class.


Subscript Referencing

Cannot make subscript references such as x(1:5,2) within fmadclass if x is fmad class.

Have to replace overloaded subscripting with direct calls to fmad’ssubsref function.

In fmad’s prod.m Product Function

localderivs=pval(ones(s(1),1),:)./Aval;

replaced with

S.type=’()’;S.subs=ones(s(1),1),’:’;localderivs=subsref(pval,S)./Aval;


Eliminate Unnecessary Temporary fmad Objects

A.value of fmad class:

Original Code in prod.m (fmad objects highlighted)

Aval=A.value;s=size(Aval);S.subs={ones(s(1),1),’:’};localderivs=subsref(pval,S)./Aval;

Modified Code in prod.m (fmad objects highlighted)

Aval=A.value;s=size(deactivate(Aval));S.subs={ones(s(1),1),’:’};localderivs=subsref(pval,S)./Aval;

Reduces number of fmad operations performed.


Reverse Mode AD in Matlab

Uses a madtape class to build up 2 tapes:1 operations tape of all arithmetic operations, including subscripting.2 value tape of sizes of values involved in calculation, values of those

involved nonlinearly.

Functions to set adjoints and propagate back through thecomputation.

Ensure madtape class superior to fmad to enable forward-over-reverse.


Use of Reverse Mode - Minpack EPT Gradient

x=madtape(x0); % initialise madtape object

f=MinpackEPT_F(x,Prob); % Evaluate the function% and create tapes

fmadtape=getvalue(f) % Extract the objective function value

setadjoint(f,1); % initialise adjoint of dependent var.

MADCalcAdjoints % propagate adjoints back% through the operations tape

gmadtape=getadjoint(x); % extract adjoint of independent


Results

Very preliminary

Minpack EPT case.

Objected oriented application


Minpack EPT Problem

Objective function with sparse Hessian


Minpack EPT Problem

Ratio of Hessian time to function time.n = nx ∗ nx

4 16 64 256

hand-coded 12. 13. 12. 14.fmad(sparse) on fmad(sparse) 662. 664. 1154. 18553.fmad(sparse) on reverse 1160. 1147. 1235. 2897.

Modify Source Codehand-coded 13. 14. 14. 15.fmad(sparse) on fmad(sparse) 569. 559. 757. 4778.fmad(sparse) on reverse 1114. 1094. 1194. 2727.

n2

16 256 4096 65536

All times averaged over a minimum of 5s.


Source Code Modification

[f,fgrad,fhess]=MinpackEPT_FGH(x,Prob)nx=Prob.user.nx; % nx is class doubleny=length(x)./nx; % x active => ny active with zero derivs.if ny~=Prob.user.ny

error(’Must have length(x)==nx*ny’)endhx = 1/(nx+1); % hx is inactivehy = 1/(ny+1); % hy is active with zero derivs.v = zeros(nx+2,ny+2); % ny active => v activev(2:nx+1,2:ny+1) = reshape(x,nx,ny);

% v active allows subscripted adssignment% computer dvdx and dvdy on each edge of the griddvdx = (v(2:nx+2,:)-v(1:nx+1,:))/hx;dvdy = (v(:,2:ny+2)-v(:,1:ny+1))/hy

% unnecessary differentiated division op.


Source Code Modification

[f,fgrad,fhess]=MinpackEPT_FGH(x,Prob)nx=Prob.user.nx; % nx is class doubleny=length(x)./nx; % x active => ny active with zero derivs.if ny~=Prob.user.ny

error(’Must have length(x)==nx*ny’)endhx = 1/(nx+1); % hx is inactivehy = 1/(ny+1); % hy is active with zero derivs.v = zeros(nx+2,ny+2); % ny active => v activev(2:nx+1,2:ny+1) = reshape(x,nx,ny);

% v active allows subscripted adssignment% computer dvdx and dvdy on each edge of the griddvdx = (v(2:nx+2,:)-v(1:nx+1,:))/hx;dvdy = (v(:,2:ny+2)-v(:,1:ny+1))/deactivate(hy);

% deactivate hy => cheaper division


Object-Oriented Test Case Preliminary Results

Test case with 4 classes from aerospace sector n = 21,m = 101.

max. 11 nonzeros/row of J.

Require derivatives for uncertainty propagation.

Users require derivatives to order 3 - results here to order 2 - workedfirst time!

technique cpu(JF)/cpu(F) cpu(∇2F)/cpu(F)

FD 22.0 486.7fmad 1.09 -fmad on fmad - 2.41

Very little computation performed - nearly all time creating objectsand some trivial computation - highlights benefits of OO!!

OO destroys performance of FD.


Jacobian Sparsity


Conclusions & Future Work

MAD can differentiate object-oriented code - including itself!

Further performance optimisation of fmad class needed:I Inactive fmad object is needed.I Store size of fmad object directly - don’t take size of it’s value.

Sparsity detection needed for first and higher derivatives - some proofof concept done using sparse logical matrices - summer 2007.

madtape class needs improvements:I Generate tape once and do multiple adjoints for same inputs x - need

to extend to different x with same control flow.I Performance analysis and optimisation.


References

Shaun A. Forth.An efficient overloaded implementation of forward mode automatic differentiation inMATLAB.ACM Trans. Math. Softw., 32(2):195–222, June 2006.

Andreas Griewank, David Juedes, and Jean Utke.Algorithm 755: ADOL-C: A package for the automatic differentiation of algorithms writtenin C/C++.ACM Transactions on Mathematical Software, 22(2):131–167, 1996.

Andreas Griewank.Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation.Number 19 in Frontiers in Applied Mathematics. SIAM, Philadelphia, PA, 2000.


higher derivatives in matlab using mad euroad workshop... · requests for higher derivatives of...

Documents