+ quadratic programming and duality sivaraman balakrishnan

+

Quadratic Programming and DualitySivaraman Balakrishnan

+Outline

Quadratic Programs

General Lagrangian Duality

Lagrangian Duality in QPs

+Norm approximation

Problem

Interpretation Geometric – try to find projection of b into ran(A) Statistical – try to find solution to b = Ax + v

v is a measurement noise (choose norm so that v is small in that norm)

Several others

+Examples

-- Least Squares Regression

-- Chebyshev

-- Least Median Regression

More generally can use *any* convex penalty function

+Picture from BV

+Least norm

Perfect measurements

Not enough of them

Heart of something known as compressed sensing

Related to regularized regression in the noisy case

+Smooth signal reconstruction

S(x) is a smoothness penalty

Least squares penalty Smooths out noise and sharp transitions

Total variation (peak to valley intuition) Smooths out noise but preserves sharp transitions

+Euclidean Projection

Very fundamental idea in constrained minimization

Efficient algorithms to project onto many many convex sets (norm balls, special polyhedra etc)

More generally finding minimum distance between polyhedra is a QP

+Quadratic Programming Duality

+General recipe

Form Lagrangian

How to figure out signs?

+Primal & Dual Functions

Primal

Dual

+Primal & Dual Programs

Primal Programs

Constraints are now implicit in the primal

Dual Program

+Lagrangian Properties

Can extract primal and dual problem

Dual problem is always concave Proof

Dual problem is always a lower bound on primal Proof

Strong duality gives complementary slackness Proof

+Some examples of QP duality

Consider the example from class

Lets try to derive dual using Lagrangian

+General PSD QP

Primal

Dual

+SVM – Lagrange Dual

Primal SVM

Dual SVM

Recovering Primal Variables and Complementary Slackness