a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear...
TRANSCRIPT
![Page 1: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/1.jpg)
a residual-minimizing nonlinear optimization method applied to
tensor approximation
hans de sterck
sabbatical visitor, mpi-inf ag1 (up to august 2011)
department of applied mathematics university of waterloo
![Page 2: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/2.jpg)
1. introduction
• tensor = N-dimensional array • N=3:
• canonical decomposition: decompose tensor in sum of R rank-one terms (approximately)
(from “Tensor Decompositions and Applications”, Kolda and Bader, SIAM Rev., 2009 [1])
![Page 3: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/3.jpg)
introduction
(from “Tensor Decompositions and Applications”, Kolda and Bader, SIAM Rev., 2009 [1])
(problem is non-convex, multiple (local) minima, solution may not exist, ... ; but smooth, and assume there is a local minimum)
(de Silva and Lim, SIMAX, 2009)
![Page 4: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/4.jpg)
link with singular value decomposition
• SVD of
• canonical decomposition of tensor
(from “Tensor Decompositions and Applications”, Kolda and Bader, SIAM Rev., 2009 [1])
![Page 5: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/5.jpg)
2. tensor approximation applications (1) “Discussion Tracking in Enron Email Using PARAFAC” by
Bader, Berry and Browne (2008) (sparse, nonnegative)
![Page 6: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/6.jpg)
tensor approximation applications (2) “All-at-once Optimization for Coupled Matrix and Tensor
Factorizations” by Acar, Kolda and Dunlavy (2011)
![Page 7: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/7.jpg)
tensor approximation applications (3) chemometrics: analyze
spectrofluorometer data (dense) (Bro et al., http://www.models.life.ku.dk/nwaydata1)
• 5 x 201 x 61 tensor: 5 samples (with different mixtures of three amino acids), 61 excitation wavelengths, 201 emission wavelengths
• goal: recover emission spectra of the three amino acids (to determine what was in each sample, and in which concentration)
• also: psychometrics, ...
(from [1])
![Page 8: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/8.jpg)
3. alternating least squares (ALS)
(1) freeze all ar(2), ar
(3), compute optimal ar(1) via a
least-squares solution (linear, overdetermined) (2) freeze ar
(1), ar(3), compute ar
(2)
(3) freeze ar(1), ar
(2), compute ar(3)
• repeat
(from [1])
![Page 9: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/9.jpg)
alternating least squares (ALS)
• ALS is monotone
• ALS is sometimes fast, but can also be extremely slow (depending on problem and initial condition)
![Page 10: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/10.jpg)
alternating least squares (ALS)
fast case slow case
(we used Matlab with Tensor Toolbox (Bader and Kolda) and Poblano Toolbox (Dunlavy et al.) for all computations)
![Page 11: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/11.jpg)
alternating least squares (ALS)
• for linear systems , when a simple iterative method is slow, we accelerate it with – GMRES (generalized minimal residual method) – CG (conjugate gradient method), etc.
• for optimization problems, general approaches to accelerate simple iterative methods are uncommon (do not exist?)
• let’s try to accelerate ALS for the tensor optimization problem • issues: nonlinear, optimization context
![Page 12: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/12.jpg)
4. nonlinear GMRES acceleration of ALS
(Moré-Thuente line search, satisfies Wolfe conditions)
![Page 13: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/13.jpg)
step II: N-GMRES acceleration:
![Page 14: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/14.jpg)
history of nonlinear acceleration mechanism for nonlinear systems (step II)
• Washio and Oosterlee, ETNA, 1997 • GMRES, Saad and Schultz, 1986 (also flexible GMRES, Saad, 1993) • Anderson mixing, 1965; DIIS (direct inversion in the iterative subspace), Pulay, 1980 • can be interpreted as a specific Broyden-type multi-secant method for (see Fang and Saad, 2009; Walker and Ni, 2011)
• BUT: apparently not used systematically yet for optimization (or not common) • this looks like a generally applicable continuous optimization method ...
![Page 15: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/15.jpg)
5. numerical results for ALS-preconditioned N-GMRES applied to tensor problem
• dense test problem (from Tomasi and Bro; Acar et al.): random rank-R tensor modified to obtain specific column collinearity, with added noise
![Page 16: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/16.jpg)
numerical results: dense test problem
![Page 17: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/17.jpg)
dense test problem: optimal window size
![Page 18: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/18.jpg)
6. why does this work: linear case
GMRES for linear systems: • stationary iterative method
(preconditioning process) • preconditioner • define residual and error:
• exact update equation: • approximate update equation:
![Page 19: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/19.jpg)
comparing N-GMRES to GMRES
GMRES for linear systems: • stationary iterative method • generates residuals recursively:
• define (Washio and Oosterlee, ETNA, 1997)
![Page 20: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/20.jpg)
comparing N-GMRES to GMRES
GMRES for linear systems: • stationary iterative process
generates preconditioned residuals that build Krylov space
• GMRES: take optimal linear combination of residuals in Krylov space to minimize the residual
(Washio and Oosterlee, ETNA, 1997)
![Page 21: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/21.jpg)
comparing N-GMRES to GMRES
• GMRES: minimize • seek optimal approximation
same as for N-GMRES
![Page 22: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/22.jpg)
convergence speed of GMRES
• GMRES: minimize • polynomial method: convergence determined by
optimal polynomial (diagonalizable matrix, A=VΛV-1)
![Page 23: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/23.jpg)
convergence speed of N-GMRES
• GMRES (linear case): convergence determined by optimal polynomial
• convergence speed of N-GMRES for optimization: open problem
![Page 24: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/24.jpg)
7. general N-GMRES optimization method general methods for nonlinear optimization (smooth, unconstrained)
(“Numerical Optimization”, Nocedal and Wright, 2006)
1. steepest descent with line search 2. Newton with line search 3. nonlinear conjugate gradient (N-CG) with line search 4. trust-region methods 5. quasi-Newton methods (includes Broyden–Fletcher–Goldfarb–
Shanno (BFGS) and limited memory version L-BFGS)
6. N-GMRES as a general optimization method?
![Page 25: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/25.jpg)
general N-GMRES optimization method • first question: what would be a general preconditioner?
• idea: general N-GMRES preconditioner = update in direction of steepest descent (or: use N-GMRES to accelerate steepest descent)
![Page 26: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/26.jpg)
8. steepest-descent preconditioning
• option A: steepest descent with line search • option B: steepest descent with predefined small step • claim: steepest descent is the ‘natural’ preconditioner for
N-GMRES
![Page 27: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/27.jpg)
steepest-descent preconditioning • claim: steepest descent is the ‘natural’ preconditioner for
N-GMRES • example: consider simple quadratic optimization problem
• we know so becomes
• this gives the same residuals as with : steepest-descent N-GMRES preconditioner corresponds to identity preconditioner for linear GMRES (and: small step is sufficient)
![Page 28: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/28.jpg)
9. numerical results: steepest-descent preconditioning
• steepest descent by itself is slow
• N-GMRES with steepest descent preconditioning is competitive with N-CG and L-BFGS
• option A slower than option B (small step)
![Page 29: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/29.jpg)
numerical results: steepest-descent preconditioning
• extended Rosenbrock function
• steepest descent by itself is slow
• N-GMRES with steepest descent preconditioning is competitive with N-CG and L-BFGS
![Page 30: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/30.jpg)
10. convergence of steepest-descent preconditioned N-GMRES optimization
• assume line searches give solutions that satisfy Wolfe conditions:
(Nocedal and Wright, 2006)
![Page 31: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/31.jpg)
convergence of steepest-descent preconditioned N-GMRES optimization
![Page 32: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/32.jpg)
convergence of steepest-descent preconditioned N-GMRES optimization
sketch of (simple!) proof •
• use Zoutendijk’s theorem: with and thus
• all ui are followed by a steepest descent step, so
• global convergence to a stationary point for general f(u)
![Page 33: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/33.jpg)
the power of N-GMRES optimization (tensor problem)
![Page 34: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/34.jpg)
the power of N-GMRES optimization (tensor problem)
![Page 35: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/35.jpg)
• thank you • questions?
Hans De Sterck, ‘A Nonlinear GMRES Optimization Algorithm for Canonical Tensor Decomposition’, submitted, May 2011, arXiv:1105.5331
Hans De Sterck, ‘Steepest Descent Preconditioning for Nonlinear GMRES Optimization’, submitted, June 2011, arXiv:1106.4426
![Page 36: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/36.jpg)
BACKUP SLIDES
![Page 37: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/37.jpg)
comparing N-GMRES to GMRES • non-preconditioned GMRES for linear systems:
• apply non-preconditioned GMRES to preconditioned linear system or
• preconditioner changes the spectrum of the operator such that (non-preconditioned) GMRES applied to the preconditioned operator converges better
• this alternative viewpoint of preconditioned GMRES leads to the same formulas as what we derived in the previous slides
![Page 38: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/38.jpg)
conjugate gradient (CG)
(Nocedal and Wright, 2006)
![Page 39: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/39.jpg)
preconditioned conjugate gradient (PCG)
(Nocedal and Wright, 2006) (Nocedal and Wright, 2006)
![Page 40: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/40.jpg)
nonlinear conjugate gradient (N-CG)
(Nocedal and Wright, 2006)
![Page 41: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/41.jpg)
9. numerical results: steepest-descent preconditioning
• steepest descent by itself is slow
• N-GMRES with steepest descent preconditioning is competitive with N-CG and L-BFGS
• option A slower than option B (small step)
![Page 42: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/42.jpg)
numerical results: steepest-descent preconditioning
• extended Rosenbrock function
• steepest descent by itself is slow
• N-GMRES with steepest descent preconditioning is competitive with N-CG and L-BFGS
![Page 43: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/43.jpg)
Applied Mathematics Department, University of Waterloo, Canada
“Scalable Scientific Computing” research group
-2 postdocs -5 PhD students -Master’s, undergraduate
research students
Ads, Mobile, and ChromeOS
![Page 44: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/44.jpg)
Scalable Scientific Computing group • numerical PDEs
– compressible fluid dynamics and MHD, space physics applications, HPC
– GPU, finite volume element method, capillarity, ... • numerical linear algebra, iterative methods
– AMG for Markov chains – AMG for eigenproblems and SVD today’s talk – ‘graph applications’, clustering (images), ...
• grid/cloud/hadoop/database, spin systems, inverse problems, ...
![Page 45: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/45.jpg)
general N-GMRES optimization method general methods for nonlinear optimization (smooth, unconstrained)
(“Numerical Optimization”, Nocedal and Wright, 2006)
1. steepest descent with line search 2. Newton with line search 3. nonlinear conjugate gradient (N-CG) with line search 4. trust-region methods 5. quasi-Newton methods (includes Broyden–Fletcher–Goldfarb–
Shanno (BFGS) and limited memory version L-BFGS)
6. N-GMRES as a general optimization method
![Page 46: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/46.jpg)
11. the power of N-GMRES optimization
• N-GMRES optimization method is a general, convergent method (steepest-descent preconditioning)
• its real power: N-GMRES optimization framework can employ sophisticated nonlinear preconditioners
![Page 47: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/47.jpg)
N-GMRES optimization algorithm to accelerate ALS
![Page 48: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/48.jpg)
differences with SVD
1. truncated SVD is best rank-R approximation:
BUT best rank-R tensor cannot be obtained by truncation: different optimization problems for different R!
![Page 49: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/49.jpg)
differences with SVD
2. SVD factor matrices are orthogonal
BUT best rank-R tensor factor matrices are not orthogonal
(from “Tensor Decompositions and Applications”, Kolda and Bader, SIAM Rev., 2009 [1])
![Page 50: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/50.jpg)
tensor approximation applications (3) chemometrics: analyze
spectrofluorometer data (dense) (Bro et al., http://www.models.life.ku.dk/nwaydata1)
• 5 x 201 x 61 tensor: 5 samples (with different mixtures of three amino acids), 61 excitation wavelengths, 201 emission wavelengths
• goal: recover emission spectra of the three amino acids (to determine what was in each sample, and in which concentration)
(from [1])
![Page 51: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/51.jpg)
step II: N-GMRES acceleration:
![Page 52: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/52.jpg)
dense test problem: comparison
(gradients, test case and N-CG from “A scalable optimization approach for fitting canonical tensor decompositions” by Acar, Dunlavy and Kolda, Chemometrics, 2011)
![Page 53: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/53.jpg)
dense test problem: comparison
![Page 54: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/54.jpg)
numerical results: sparse test problem • sparse test problem: d-dimensional finite difference
Laplacian (2 d-way tensor)
![Page 55: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/55.jpg)
sparse test problem: comparison
![Page 56: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/56.jpg)
6. why does this work: GMRES
• N-GMRES step II reduces to preconditioned GMRES in the linear case
• ‘nonlinear Krylov space’ • in step I is a nonlinear preconditioner
for N-GMRES (ALS)
(Washio and Oosterlee, ETNA, 1997)
![Page 57: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/57.jpg)
numerical results: steepest-descent preconditioning
• standard test problems, 10 random initial guesses • N-GMRES with steepest descent preconditioning is
competitive with N-CG and L-BFGS • N-GMRES preconditioner option A (line search) slower than
option B (small step)
![Page 58: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/58.jpg)
12. conclusions • we have proposed the 3-step preconditioned N-GMRES
optimization algorithm as a general nonlinear optimization method (smooth f(u), unconstrained) (uncommon approach, new in optimization?)
• steepest descent preconditioning is the natural ‘default’ preconditioner, it makes N-GMRES competitive with N-CG and L-BFGS, and we have proved global convergence
![Page 59: a residual-minimizing nonlinear optimization method ...€¦ · a residual-minimizing nonlinear optimization method applied to tensor approximation hans de sterck sabbatical visitor,](https://reader030.vdocuments.us/reader030/viewer/2022040613/5f08538a7e708231d42174e5/html5/thumbnails/59.jpg)
conclusions
• the real power of the N-GMRES optimization framework is that advanced nonlinear preconditioners can be used
• ALS-preconditioned N-GMRES optimization performs very well for tensor optimization problem