optimization methods in machine learning lecture 15 › katyas › files › optml ›...
TRANSCRIPT
![Page 1: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/1.jpg)
Optimization Methods in Machine Learning Lecture 15
TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAAAAA
![Page 2: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/2.jpg)
Optimization Methods for SVMs
• Stochastic gradient method • Block-coordinate descent • Active set method
![Page 3: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/3.jpg)
Support Vector Machines
+ -
![Page 4: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/4.jpg)
Classification SVM Problem
![Page 5: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/5.jpg)
Classification SVM Problem
![Page 6: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/6.jpg)
Stochastic gradient approach
![Page 7: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/7.jpg)
Unconstrained formulation of the SVM problem
![Page 8: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/8.jpg)
Subgradient step
![Page 9: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/9.jpg)
SVM problem using Huber loss function
![Page 10: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/10.jpg)
Approximate subgradient step
![Page 11: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/11.jpg)
Stochastic Gradient Method
![Page 12: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/12.jpg)
Convergence in expectation
![Page 13: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/13.jpg)
Why does this work?
![Page 14: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/14.jpg)
![Page 15: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/15.jpg)
![Page 16: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/16.jpg)
![Page 17: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/17.jpg)
![Page 18: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/18.jpg)
Optimization Problem
Linear formulation Kernel formulation
![Page 19: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/19.jpg)
Support Vectors
+ -
x x
x
x ®=c
0<®<c
®=0
0<®<c
0<®<c
®=0
![Page 20: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/20.jpg)
Decomposition Methods
![Page 21: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/21.jpg)
Dual Optimization Problem
![Page 22: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/22.jpg)
Decomposition approach
![Page 23: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/23.jpg)
Active set method for convex QP
Solve this “small” QP problem by any method
![Page 24: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/24.jpg)
Decomposition Method
How to determine the next set B? Look for steepest descent direction of size |B|.
![Page 25: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/25.jpg)
Finding the new set B
Ordered vector
Pick the same number of di = yi and di = ¡ yimaking sure that di satisfy the conditionsfrom prev. slide
![Page 26: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/26.jpg)
Workload of a decomposition method
![Page 27: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/27.jpg)
Reducing the cost of finding the new set B
Reduce the size of the vector
by ignoring the elements that are likely to be in the middle (for example because they were in the middle last 100 iterations)
But we do need to compute the entire vector to verify optimality!
![Page 28: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/28.jpg)
Complexity
![Page 29: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/29.jpg)
Interior point methods Cplex, OOQP, OOPS, Mosek
O(k2n)£ O(n log(1/²) O(k2n) in practice, very accurate solutions
Active set method SVM-QP, Cplex
Exponential in theory O(nsn2) in practice, very accurate solutions
Decomposition methods SMO, SVMlight
O(n^2/²), reasonably accurate solutions
Cutting plane methods SVMperf
O(Rns/²), no accurate solutions
Stochastic Gradient Pegasos
O(Rs/²), probabilitsitc results, requires i.i.d samples, no accurate solutions
![Page 30: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/30.jpg)
Optimality Conditions
![Page 31: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/31.jpg)
Active Set
![Page 32: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/32.jpg)
Partitioning of matrix Q
![Page 33: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/33.jpg)
+ -
![Page 34: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/34.jpg)
+ -
![Page 35: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/35.jpg)
+ -
![Page 36: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/36.jpg)
+ -
![Page 37: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/37.jpg)
Active Set Method
![Page 38: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/38.jpg)
Active Set Method
![Page 39: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/39.jpg)
Active Set Method
![Page 40: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/40.jpg)
Active Set Method
![Page 41: Optimization Methods in Machine Learning Lecture 15 › katyas › files › OPTML › Lecture15.pdfOptimization Methods in Machine Learning Lecture 15 TexPoint fonts used in EMF](https://reader033.vdocuments.us/reader033/viewer/2022042315/5f035f897e708231d408e4dc/html5/thumbnails/41.jpg)
Complexity