optimization methods in machine learning...optimization methods for convex problems • interior...
TRANSCRIPT
![Page 1: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/1.jpg)
Optimization Methods in Machine Learning
TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAAAAA
Katya Scheinberg Lehigh University
![Page 2: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/2.jpg)
Primal Semidefinite Programming Problem
![Page 3: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/3.jpg)
Dual Semidefinite Programming Problem
Primal Semidefinite Programming Problem
![Page 4: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/4.jpg)
Duality gap and complementarity
HW: prove the last statement
![Page 5: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/5.jpg)
Complementarity of eignevalues
![Page 6: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/6.jpg)
Complementarity of eigenvalues
![Page 7: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/7.jpg)
Optimality conditions
Convex QP with linear equality constraints.
Closed form solution via solving a linear system
![Page 8: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/8.jpg)
Optimality conditions
Convex QP with linear inequality constraints.
No closed form solution
![Page 9: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/9.jpg)
Nonlinear Constraints, linear objective:
Convex Quadratically Constrained Quadratic Problems
Feasible set can be described as a convex cone Å affine set
![Page 10: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/10.jpg)
Second Order Cone
x= (x0, x1, . . . , xn ), x̄= (x1, . . . , xn )
K2 Rn+1 is a second order cone:
x1
x2
x0
![Page 11: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/11.jpg)
Discovering SOCP cone
A convex quadratic constraint:
Factorize and rewrite:
Norm constraint
More general form
Variable substitution
SOCP:
![Page 12: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/12.jpg)
Second Order Cone Programming
![Page 13: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/13.jpg)
Complementarity Conditions
![Page 14: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/14.jpg)
Formulating SOCPs
Rotated SOCP cone Equivalent to SOCP cone Example:
![Page 15: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/15.jpg)
Unconstrained Optimization
![Page 16: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/16.jpg)
Traditional methods
• Gradient descent • Newton method • Quazi-Newton method • Conjugate gradient method
![Page 17: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/17.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 18: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/18.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 19: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/19.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 20: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/20.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 21: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/21.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 22: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/22.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 23: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/23.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 24: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/24.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 25: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/25.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 26: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/26.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 27: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/27.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 28: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/28.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 29: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/29.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 30: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/30.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 31: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/31.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 32: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/32.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 33: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/33.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 34: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/34.jpg)
Slides from L. Vandenberghe http://www.ee.ucla.edu/~vandenbe/ee236c.html
![Page 35: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/35.jpg)
Interior Point Methods
![Page 36: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/36.jpg)
Interior Point Methods: a history
² Ellipsoid Method, Nemirovskii, 1970’s. No complexity result.
² Polynomial Ellipsoid Method for LP, Khachian 1979. Not practical.
² Karmarkar’s method, 1984, first “efficient” interior point method.
² Primal-dual path following methods and others late 1980’s. Very efficentpractical methods.
² Extensions to other classes of convex problems. Early 1990’s.
² General theory of interior point methods, self-concordant barriers, Nes-terov and Nemirovskii, 1990’s.
![Page 37: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/37.jpg)
Self-concordant barrier
![Page 38: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/38.jpg)
Log barrier for LP
![Page 39: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/39.jpg)
Log-barrier for SDP
![Page 40: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/40.jpg)
Log barrier for SOCP
![Page 41: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/41.jpg)
Dual Linear Programming Problem
Primal Linear Programming Problem
![Page 42: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/42.jpg)
Optimality (KKT) conditions
![Page 43: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/43.jpg)
Central Path
Apply Newton method to the (self-concordant) barrier problem (i.e. to its optimality conditions)
Apply one or two steps of Newton method for a given µ and then reduce µ
![Page 44: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/44.jpg)
KKT conditions for primal central path
![Page 45: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/45.jpg)
Central Path
Optimality conditions for the barrier problem
Apply Newton method to the system of nonlinear equations
![Page 46: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/46.jpg)
Central Path
-c
It exists iff there is nonempty interior for the primal and dual problems.
![Page 47: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/47.jpg)
Interior point methods, the main idea
• Each point on the central path can be approximated by
applying Newton method to the perturbed KKT system.
• Start at some point near the central path for some value of µ, reduce µ.
• Make one or more Newton steps toward the solution with the new value of µ.
• Keep driving µ to 0, always staying close to the solutions
of the central path.
• This prevents the iterates from getting trapped near the boundary and keeps them nicely central.
![Page 48: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/48.jpg)
-c
![Page 49: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/49.jpg)
KKT conditions for dual and primal-dual central paths
![Page 50: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/50.jpg)
Newton step
Primal method
Primal-dual method
Dual method
-c
![Page 51: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/51.jpg)
Predictor-Corrector steps
¾ = 0 for predictor step and ¾ > 0 for corrector step.
Solve the system of linear equations twice with the same matrix
![Page 52: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/52.jpg)
Predictor-Corrector steps
Augmented system
![Page 53: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/53.jpg)
Solving the augmented system
x x
x x
x x
x
x
x
x x
x
x x
x
x
x
x
x
x
x x x x
x x x
x x
x
x x
x
x
Normal equation
x
x x
x x
x x
![Page 54: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/54.jpg)
Cholesky Factorization
x
x x
x x
x
x
x x
x
x x
x
x x x
x x
x x
x x
x x
x x
x
x x
x x
x x
x x
x x
x x
x x
x
x x
=
• Numerically very stable!
• The sparsity pattern of L remains the same at each iteration
• Depends on sparsity pattern of A and ordering of rows of A
• Can compute the pattern in advance (symbolic factorization)
• The work for each factorization depends on sparsity pattern, can be as little as O(n) if very sparse and as much as O(n^3) (if dense).
![Page 55: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/55.jpg)
Complexity per iteration
![Page 56: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/56.jpg)
Complexity and performance
![Page 57: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/57.jpg)
Optimality conditions
Convex QP with linear inequality constraints.
![Page 58: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/58.jpg)
Interior Point method
![Page 59: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/59.jpg)
Newton Step
![Page 60: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/60.jpg)
Complexity per iteration
![Page 61: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/61.jpg)
Dual Semidefinite Programming Problem
Primal Semidefinite Programming Problem
![Page 62: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/62.jpg)
Duality gap and complementarity
![Page 63: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/63.jpg)
Central Path
![Page 64: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/64.jpg)
Central Path
![Page 65: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/65.jpg)
Central Path
Dual CP
Primal-Dual CP
Symmetric Primal-Dual
![Page 66: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/66.jpg)
Computing a step
![Page 67: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/67.jpg)
Computing a step
![Page 68: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/68.jpg)
Cholesky factorization
Each iteration may require O(n6) operations and O(n4) memory.
![Page 69: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/69.jpg)
Second Order Cone Programming
![Page 70: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/70.jpg)
Complementarity Conditions
![Page 71: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/71.jpg)
Log-barrier formulation
![Page 72: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/72.jpg)
Perturbed optimality conditions
![Page 73: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/73.jpg)
Newton step
![Page 74: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/74.jpg)
Optimization methods for convex problems • Interior Point methods
– Best iteration complexity O(log(1/²)), in practice <50. – Worst per-iteration complexity (sometimes prohibitive)
• Active set methods – Exponential complexity in theory, often linear in practice. – Better per iteration complexity.
• Gradient based methods – or O(1/²) iterations – Matrix/vector multiplication per iteration
• Nonsmooth gradient based methods – O(1/²) or O(1/²2) iterations – Matrix/vector multiplication per iteration
• Block coordinate descent – Iteration complexity ranges from unknown to similar to FOMs. – Per iteration complexity can be constant.
![Page 75: Optimization Methods in Machine Learning...Optimization methods for convex problems • Interior Point methods – Best iteration complexity O(log(1/²)), in practice](https://reader035.vdocuments.us/reader035/viewer/2022081523/5fe473db2674fe305a7cfcdf/html5/thumbnails/75.jpg)
Homework 1.
2.
3.