![Page 1: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/1.jpg)
+
Quadratic Programming and DualitySivaraman Balakrishnan
![Page 2: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/2.jpg)
+Outline
Quadratic Programs
General Lagrangian Duality
Lagrangian Duality in QPs
![Page 3: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/3.jpg)
+Norm approximation
Problem
Interpretation Geometric – try to find projection of b into ran(A) Statistical – try to find solution to b = Ax + v
v is a measurement noise (choose norm so that v is small in that norm)
Several others
![Page 4: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/4.jpg)
+Examples
-- Least Squares Regression
-- Chebyshev
-- Least Median Regression
More generally can use *any* convex penalty function
![Page 5: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/5.jpg)
+Picture from BV
![Page 6: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/6.jpg)
+Least norm
Perfect measurements
Not enough of them
Heart of something known as compressed sensing
Related to regularized regression in the noisy case
![Page 7: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/7.jpg)
+Smooth signal reconstruction
S(x) is a smoothness penalty
Least squares penalty Smooths out noise and sharp transitions
Total variation (peak to valley intuition) Smooths out noise but preserves sharp transitions
![Page 8: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/8.jpg)
+Euclidean Projection
Very fundamental idea in constrained minimization
Efficient algorithms to project onto many many convex sets (norm balls, special polyhedra etc)
More generally finding minimum distance between polyhedra is a QP
![Page 9: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/9.jpg)
+Quadratic Programming Duality
![Page 10: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/10.jpg)
+General recipe
Form Lagrangian
How to figure out signs?
![Page 11: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/11.jpg)
+Primal & Dual Functions
Primal
Dual
![Page 12: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/12.jpg)
+Primal & Dual Programs
Primal Programs
Constraints are now implicit in the primal
Dual Program
![Page 13: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/13.jpg)
+Lagrangian Properties
Can extract primal and dual problem
Dual problem is always concave Proof
Dual problem is always a lower bound on primal Proof
Strong duality gives complementary slackness Proof
![Page 14: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/14.jpg)
+Some examples of QP duality
Consider the example from class
Lets try to derive dual using Lagrangian
![Page 15: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/15.jpg)
+General PSD QP
Primal
Dual
![Page 16: + Quadratic Programming and Duality Sivaraman Balakrishnan](https://reader036.vdocuments.us/reader036/viewer/2022081503/5697c0091a28abf838cc70d3/html5/thumbnails/16.jpg)
+SVM – Lagrange Dual
Primal SVM
Dual SVM
Recovering Primal Variables and Complementary Slackness