branch detection and sparsity estimation in matlab euroad workshop - marina... · branch detection...
Post on 18-Mar-2018
217 Views
Preview:
TRANSCRIPT
Branch Detection and Sparsity Estimation in Matlab
Marina Menshikova & Shaun Forth
Engineering Systems DepartmentCranfield University (DCMT Shrivenham)
Shrivenham, Swindon SN6 8LA, U.K.email: M.Menshikova@cranfield.ac.uk/S.A.Forth@cranfield.ac.uk
6th European Workshop on Automatic DifferentiationINRIA Sophia-Antipolis, France, November 15-16 2007
1/ 28 Branch Detection and Sparsity Estimation in Matlab
Plan
1 Branch Detection - MotivationSparsity estimation for JacobianReverse mode AD implemented via tapingUncertainty propagation using Taylor series
2 Branch detection principlesMain ideaImplementation
3 Examples
4 Sparsity EstimationSparsity Detection in MatlabThe sparsemad ClassWhy do we Propagate Entries (Value Sparsity) and DerivativeSparsity?Test Cases
5 Conclusions
2/ 28 Branch Detection and Sparsity Estimation in Matlab
Branch Detection - Motivation
Branch Detection
Branch Detection - detection of data-dependant control flow.
Some examples illustrate necessity of this.
3/ 28 Branch Detection and Sparsity Estimation in Matlab
Sparsity estimation for Jacobian
function y = y(x(1),x(2),x(3))
if x(1) > 0y = [x(1); x(2)*x(3)];
elsey = [x(1) + x(2); x(3)];
end
J =
[
1 0 00 x(3) x(2)
], x(1) > 0;[
1 1 00 0 1
], x(1) ≤ 0.
SJ =
[
1 0 00 1 1
], x(1) > 0;[
1 1 00 0 1
], x(1) ≤ 0.
4/ 28 Branch Detection and Sparsity Estimation in Matlab
Reverse mode AD implemented via taping
function y = y(x(1),x(2),x(3))
if x(1) > 0y = x(1) + x(2)*x(3);
elsey = x(1) + x(2) + x(3);
end
Tapes of operations:* x(2) x(3) v+ v x(1) y
, x(1) > 0;
+ x(1) x(2) v+ v x(3) y
, x(1) ≤ 0.
5/ 28 Branch Detection and Sparsity Estimation in Matlab
Uncertainty propagation using Taylor series
Taylor series for function y with single input x is
y(x) = y(µx ) +∂y
∂x(x − µx ) +
1
2!
∂2y
∂x2(x − µx )2
+1
3!
∂3y
∂x3(x − µx )3 + O
((x − µx )4
)For x distributed with mean µx , variance σ2
x , skewness S(x) and KurtosisK (x), then expectation and variance are approximated by
µy = y(µx ) +1
2
∂2y
∂x2σ2
x + O(σ3x )
σ2y =
(∂y
∂xσx
)2
+∂2y
∂x2
∂y
∂xS(x)σ3
x +1
4
(∂2y
∂x2σ2
x
)2 (K (x)− 1
)+
1
3
∂3y
∂x3
∂y
∂xK (x)σ4
x + O(σ5x )
provided Taylor series valid for range of x - no change of control-flow.
6/ 28 Branch Detection and Sparsity Estimation in Matlab
Main idea
Overloading comparison functions in fmad== >= > <= < ∼=eq.m ge.m gt.m le.m lt.m ne.m
Use of global MADBRANCHES with components.switch.new.info{:, 1:5}.old
function y=eq(a,b)% overloaded equality test for one or more fmad variables
y=deactivate(a)==deactivate(b);MADStoreBranches(a,b,y,’eq’);
7/ 28 Branch Detection and Sparsity Estimation in Matlab
function MADStoreBranches: part 1
global MADBRANCHES% Check if branch detection is requiredif isfield(MADBRANCHES, ’switch’) && MADBRANCHES.switch
. . . && (isactive(a) || isactive(b))MADBRANCHES.new = [MADBRANCHES.new; y(:)];ST = dbstack(’-completenames’);% Get call stackchange = false;i = MADBRANCHES.i;
for j = 2:i % See if present branch has been located beforeif strcmp(MADBRANCHES.info{j,1},ST(3).file) &&. . . MADBRANCHES.info{j,3} == ST(3).line
MADBRANCHES.info{j,5} = MADBRANCHES.info{j,5} + 1;change = true;break;
endend =⇒ . . .
8/ 28 Branch Detection and Sparsity Estimation in Matlab
function MADStoreBranches: part 2
% Check if this location has been seen before.if change
i = i + 1;% Record the information about new locationMADBRANCHES.info{i,1} = ST(3).file;MADBRANCHES.info{i,2} = ST(3).name;MADBRANCHES.info{i,3} = ST(3).line;MADBRANCHES.info{i,4} = op;MADBRANCHES.info{i,5} = 1;MADBRANCHES.i = i;
end
9/ 28 Branch Detection and Sparsity Estimation in Matlab
Example
1 function y = func(a,b,c)
2 v = a + b;3 if v > 24 y = 2*c;5 else6 y = v.*c;7 end
>> a = fmad(2, [1 0 0]); b = fmad(1, [0 1 0]); c =fmad(-3, [0 0 1]);>> MADSetBranches(’BranchDetection’,’ON’);The branch detection is switched on.>> func(a,b,c);>> branch = MADGetBranches(’Branches’)branch =
1
10/ 28 Branch Detection and Sparsity Estimation in Matlab
Example
1 function y = func(a,b,c)
2 v = a + b;3 if v > 24 y = 2*c;5 else6 y = v.*c;7 end
>> MADSetBranches(’BranchDetection’,’AGAIN’);The result of previous branch detection is recorded.>> func(c,a,b);>> branch = MADGetBranches(’AllBranches’)branch =
[1] [0]
11/ 28 Branch Detection and Sparsity Estimation in Matlab
Example
>> branch = MADGetBranches(’HTML’)fid=fopen(’MADBRANCHES.html’,’w’);fprintf(fid,’%s\n’,’<HTML>’);. . .fprintf(fid,’%s\n’,’</HTML>’);fclose(fid);branches = web([’file:///’ which(’MADBRANCHES.html’)]);
12/ 28 Branch Detection and Sparsity Estimation in Matlab
Example
1 function y = func2(a,b,c)
2 y = 0;3 while abs(a) <= 104 y = y + 1;5 a = a + 1;6 end
>> MADSetBranches(’BranchDetection’,’ON’);The branch detection is switched on.>> func2(a,b,c);>> MADGetBranches(’HTML’);
13/ 28 Branch Detection and Sparsity Estimation in Matlab
Example
1 function y = func2(a,b,c)
2 y = 0;3 while abs(a) <= 104 y = y + 1;5 a = a + 1;6 end
14/ 28 Branch Detection and Sparsity Estimation in Matlab
Sparsity Estimation
Compression Techniques [Gri00, Chap. 7] can greatly improve efficiency forcalculating sparse Jacobians or Hessians.
15/ 28 Branch Detection and Sparsity Estimation in Matlab
Example (Tridiagonal Jacobian)
1 Suppose Jf(x) is tridiagonalJf(x) =
b1 c1
a2 b2 c2
a3 b3 c3
a4 b4 c4
a5 b5
.2 Choose 5× 3 seed matrix
S =
1 0 00 1 00 0 11 0 00 1 0
.
3 Then calculating Jf(x) · S byforward mode AD (or FDapprox.) gives,
Jf(x) ·S =
b1 c1 0a2 b2 c2
c3 a3 b3
b4 c4 a4
a5 b5 0
,i.e., all the Jacobian entries.
4 Tridiagonal Jacobianevaluated using 3Jacobian-vector productsnot n.
16/ 28 Branch Detection and Sparsity Estimation in Matlab
Compression Process
1 Obtain the sparsity pattern - the (i , j) locations of all possiblynon-zero entries in the Jacobian.
2 Construct a seed matrix (or matrices) to initiate forward (and/orreverse) mode [Gri00, Chap. 7], [CV98], [GMP05].
3 Calculate Jacobian products with seed(s) and extract all Jacobianentries.
17/ 28 Branch Detection and Sparsity Estimation in Matlab
Compression Process
1 Obtain the sparsity pattern - the (i , j) locations of all possiblynon-zero entries in the Jacobian.
2 Construct a seed matrix (or matrices) to initiate forward (and/orreverse) mode [Gri00, Chap. 7], [CV98], [GMP05].
3 Calculate Jacobian products with seed(s) and extract all Jacobianentries.
Here we concentrate on step 1 - sparsity estimation.
17/ 28 Branch Detection and Sparsity Estimation in Matlab
AD and Sparsity Estimation
ADIFOR’s SparselinC package [BKBC96] allows for sparse storage ofderivatives - can be used to give Jacobian sparsity for a givenindependent x.
Griewank and Mitev have considered using Bayesian update strategiesto make use of Jacobian-vector and vector-Jacobian products toiteratively deduce sparsity patterns - probing.
Giering and Kaminski [GK06] - source-transformation implementationusing bit-vectors to indicate dependence of active variable on eachindependent (or dependent) in a pure (no values) calculation.
ADOL-C features overloaded bit-vector propagation.
18/ 28 Branch Detection and Sparsity Estimation in Matlab
Sparsity Detection in Matlab
More recent versions of Matlab do support some bit operations(bitand, bitor and bitxor) on unsigned integers - not investigatedto date since MAD tool [For06] relies on use of high-level (sparse)matrix operations.
Matlab also offers sparse logical matrices - can perform all thehigh-level matrix operations of the MAD package on them.
19/ 28 Branch Detection and Sparsity Estimation in Matlab
The sparsemad Class
Three components:
value - stores the numerical value of the object.
entries - logical array identifying the (nonzero) entries inthe value component.
sparsity - a derivvec object holding the sparsity patternfor each component of the value.
20/ 28 Branch Detection and Sparsity Estimation in Matlab
Initialising the sparsemad object
x=sparsemad([1 2 3 4 5],speye(5)>0)
value = 1 2 3 4 5entries = 1 1 1 1 1derivvec object sparsitySize = 1 5No. of derivs = 5directional derivative 1d = (1,1) 1directional derivative 2d = (1,2) 1:directional derivative 5d = (1,5) 1
21/ 28 Branch Detection and Sparsity Estimation in Matlab
The plus(+) Operation
function z=plus(x,y)if isa(x,’sparsemad’)
z=x;if isa(y,’sparsemad’)
z.value=z.value+y.value;z.sparsity=z.sparsity|y.sparsity;% | = logical orz.entries=z.entries|y.entries;% derivvec logical or
elsez.value=x.value+y;
endelse
z=y;z.value=x+y.value;
end
22/ 28 Branch Detection and Sparsity Estimation in Matlab
Why do we Propagate Entries (Value Sparsity) andDerivative Sparsity?
Consider,
x=[1 2 3 4]’; % column vectory=diag(x); % 4x4 diagonal matrixz=y*x; % matrix multiplication
z’s sparsity pattern is the 4× 4 identity.
Key here is sparsity propagation for matrix-multiplication,
z.sparsity = y.entries*x.sparsity + y.sparsity*x.entries
with above applied over each of the 4 directions for sparsity patterns.
If y’s entries are 4× 4 identity then sparsity pattern is correct.
23/ 28 Branch Detection and Sparsity Estimation in Matlab
Why do we Propagate Entries (Value Sparsity) andDerivative Sparsity? (ctd.)
If we didn’t store the entries then for y we could either:
1 Assume y’s entries were the non-zeros of its value component,
z.sparsity= (y.value~=0)*x.sparsity ...+ y.sparsity*(x.value~=0)
this fails (underestimates sparsity) if a component of x happens to bezero.
2 Assume all x’s and y’s entries are full,
z.sparsity= ones(size(y.value))*x.sparsity ...+ y.sparsity*ones(size(x.value))
which overestimates sparsity and would indicate that Jacobian wasfull.
24/ 28 Branch Detection and Sparsity Estimation in Matlab
Brusselator Problem
Semi-discretised ODE - 4 non-zeros per row of sparse Jacobian.
cpu(Jf)/cpu(f) with problem size n10 20 40 80 160 320
numjac(sparse) 7.5 7.4 8.1 8.8 9.6 9.9fmadsparse 96.8 87.6 78.5 68.5 57.5 48.6fmadcmp 88.6 78.0 67.1 55.0 39.6 25.5
cpu(Sf)/cpu(f) with problem size n10 20 40 80 160 320
hand-coded 2.0 2.2 2.9 3.7 4.8 5.0sparsemad 93.5 82.3 75.7 60.8 46.4 33.9
25/ 28 Branch Detection and Sparsity Estimation in Matlab
Finite Element ODE Problem
Finite-element semi-discretised of PDE - f (t, y) = Ay with A tri-diagonal.
cpu(Jf)/cpu(f) with problem size n10 20 40 80 160 320
numjac(sparse) 11.0 10.4 10.9 11.6 13.6 17.7fmadsparse 11.0 10.1 9.9 10.0 10.0 10.0fmadcmp 11.6 11.1 10.5 10.8 11.2 11.9
cpu(Sf)/cpu(f) with problem size n10 20 40 80 160 320
hand-coded 3.8 3.9 4.0 4.6 6.0 8.5sparsemad 11.6 11.3 11.2 11.2 11.4 11.3
Linearity of f (t, y) detected - a global MADsparsemadNonlinear set totrue if nonlinear function encountered by sparsemad class.
26/ 28 Branch Detection and Sparsity Estimation in Matlab
Conclusions
Branch detection allows us to check if sparsity pattern, tape or PDFTaylor series is safe to use for all inputs x.
Sparsity Detection allows us to determine Jacobian sparsity patterns.
Future Work
Discontinuous derivative testing to ensure validity of Taylor series.
Can we estimate how much of a pdf passes along currentcontrol-flow?
Adaptation of Taylor order for accurate statistical moments.
Sparsity detection based on bit-wise operations - can it beat thecurrent sparse matrix implementation?
27/ 28 Branch Detection and Sparsity Estimation in Matlab
References
Christian H. Bischof, Peyvand M. Khademi, Ali Bouaricha, and Alan Carle.
Efficient computations of gradients and Jacobians by dynamic exploitation of sparsity in automatic differentiation.Optimization Methods and Software, 7:1–39, 1996.
Thomas F. Coleman and Arun Verma.
The efficient computation of sparse Jacobian matrices using automatic differentiation.SIAM J. Sci. Comput., 19(4):1210–1233, 1998.
Shaun A. Forth.
An efficient overloaded implementation of forward mode automatic differentiation in MATLAB.ACM Trans. Math. Softw., 32(2):195–222, June 2006.DOI: http://doi.acm.org/10.1145/1141885.1141888.
Ralf Giering and Thomas Kaminski.
Automatic sparsity detection implemented as a source-to-source transformation.In Vassil N. Alexandrov, Geert Dick van Albada, Peter M. A. Sloot, and Jack Dongarra, editors, Computational Science –ICCS 2006, volume 3994 of Lecture Notes in Computer Science, pages 591–598, Heidelberg, 2006. Springer.dx.doi.org/10.1007/11758549_81.
Assefaw Hadish Gebremedhin, Fredrik Manne, and Alex Pothen.
What color is your Jacobian? Graph coloring for computing derivatives.SIAM Review, 47(4):629–705, 2005.
Andreas Griewank.
Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation.Number 19 in Frontiers in Applied Mathematics. SIAM, Philadelphia, PA, 2000.
28/ 28 Branch Detection and Sparsity Estimation in Matlab
top related