convex programming
DESCRIPTION
Convex Programming. Brookes Vision Reading Group. Huh?. What is convex ??? What is programming ??? What is convex programming ???. Huh?. What is convex ??? What is programming ??? What is convex programming ???. Convex Function. f(t x + (1-t) y)TRANSCRIPT
![Page 1: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/1.jpg)
Convex Programming
Brookes Vision Reading Group
![Page 2: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/2.jpg)
Huh?
• What is convex ???
• What is programming ???
• What is convex programming ???
![Page 3: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/3.jpg)
Huh?
• What is convex ???
• What is programming ???
• What is convex programming ???
![Page 4: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/4.jpg)
Convex Function
f(t x + (1-t) y) <= t f(x) + (1-t) f(y)
![Page 5: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/5.jpg)
Convex Function
Is a linear function convex ???
![Page 6: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/6.jpg)
Convex Set
Region above a convex function is a convex set.
![Page 7: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/7.jpg)
Convex Set
Is the set of all positive semidefinite matrices convex??
![Page 8: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/8.jpg)
Huh?
• What is convex ???
• What is programming ???
• What is convex programming ???
![Page 9: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/9.jpg)
Programming
• Objective function to be minimized/maximized.
• Constraints to be satisfied.
ExampleObjective function
Constraints
![Page 10: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/10.jpg)
Example
Feasible region
Vertices
Objective function
Optimal solution
![Page 11: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/11.jpg)
Huh?
• What is convex ???
• What is programming ???
• What is convex programming ???
![Page 12: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/12.jpg)
Convex Programming• Convex optimization function
• Convex feasible region
• Why is it so important ???
• Global optimum can be found in polynomial time.
• Many practical problems are convex
• Non-convex problems can be relaxed to convex ones.
![Page 13: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/13.jpg)
Convex Programming• Convex optimization function
• Convex feasible region
• Examples ???
• Linear Programming• Refer to Vladimir/Pushmeet’s reading group
• Second Order Cone Programming• What ???
• Semidefinite Programming• All this sounds Greek and Latin !!!!
![Page 14: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/14.jpg)
Outline
• Convex Optimization– Second Order Cone Programming (SOCP)– Semidefinite Programming (SDP)
• Non-convex optimization– SDP relaxations– SOCP relaxations
• Optimization Algorithms– Interior Point Method for SOCP– Interior Point Method for SDP
2 out of 3 is not bad !!!
![Page 15: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/15.jpg)
Outline
• Convex Optimization– Second Order Cone Programming (SOCP)– Semidefinite Programming (SDP)
• Non-convex optimization– SDP relaxations– SOCP relaxations
• Optimization Algorithms– Interior Point Method for SOCP– Interior Point Method for SDP
![Page 16: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/16.jpg)
Second Order Cone• || u || < t
• u - vector of dimension ‘d-1’• t - scalar• Cone lies in ‘d’ dimensions
• Second Order Cone defines a convex set
• Example: Second Order Cone in 3D
x2 + y2 <= z2
![Page 17: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/17.jpg)
x2 + y2 <= z2
HmmmICE CREAM !!
![Page 18: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/18.jpg)
Second Order Cone Programming
Minimize fTx
Subject to || Ai x+ bi || <= ciT x + di
i = 1, … , L
Linear Objective Function
Affine mapping of SOC
Constraints are SOC of ni dimensions
Feasible regions are intersections of conic regions
![Page 19: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/19.jpg)
Example
![Page 20: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/20.jpg)
Why SOCP ??
• A more general convex problem than LP– LP SOCP
• Fast algorithms for finding global optimum– LP - O(n3)– SOCP - O(L1/2) iterations of O(n2∑ni)
• Many standard problems are SOCP-able
![Page 21: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/21.jpg)
SOCP-able Problems• Convex quadratically constrained quadratic programming
• Sum of norms
• Maximum of norms
• Problems with hyperbolic constraints
![Page 22: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/22.jpg)
SOCP-able Problems• Convex quadratically constrained quadratic programming
• Sum of norms
• Maximum of norms
• Problems with hyperbolic constraints
![Page 23: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/23.jpg)
QCQP
Minimize xT P0 x + 2 q0T x + r0
Subject to xT Pi x + 2 qiT x + ri
Pi >= 0
|| P01/2 x + P0
-1/2 x ||2 + r0 -q0TP0
-1p0
![Page 24: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/24.jpg)
QCQP
Minimize xT P0 x + 2 q0T x + r0
Subject to xT Pi x + 2 qiT x + ri
Minimize t
Subject to || P01/2 x + P0
-1/2 x || < = t
|| P01/2 x + P0
-1/2 x || < = (r0 -q0TP0
-1p0)1/2
![Page 25: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/25.jpg)
SOCP-able Problems• Convex quadratically constrained quadratic programming
• Sum of norms
• Maximum of norms
• Problems with hyperbolic constraints
![Page 26: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/26.jpg)
Sum of Norms
Minimize || Fi x + gi ||
Minimize ti
Subject to || Fi x + gi || <= ti
Special Case: L-1 norm minimization
![Page 27: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/27.jpg)
SOCP-able Problems• Convex quadratically constrained quadratic programming
• Sum of norms
• Maximum of norms
• Problems with hyperbolic constraints
![Page 28: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/28.jpg)
Maximum of Norms
Minimize max || Fi x + gi ||
Minimize t
Subject to || Fi x + gi || <= t
Special Case: L-inf norm minimization
![Page 29: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/29.jpg)
You weren’t expecting a question, were you ??
![Page 30: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/30.jpg)
SOCP-able Problems• Convex quadratically constrained quadratic programming
• Sum of norms
• Maximum of norms
• Problems with hyperbolic constraints
![Page 31: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/31.jpg)
Hyperbolic Constraints
w2 <= xy x >= 0 , y >= 0
|| [2w; x-y] || <= x+y
![Page 32: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/32.jpg)
Let’s see if everyone was awake !
![Page 33: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/33.jpg)
Outline
• Convex Optimization– Second Order Cone Programming (SOCP)– Semidefinite Programming (SDP)
• Non-convex optimization– SDP relaxations– SOCP relaxations
• Optimization Algorithms– Interior Point Method for SOCP– Interior Point Method for SDP
![Page 34: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/34.jpg)
Semidefinite Programming
Minimize C XSubject to Ai X = bi
X >= 0Linear Objective Function
Linear Constraints
Linear Programming on Semidefinite Matrices
![Page 35: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/35.jpg)
Why SDP ??
• A more general convex problem than SOCP– LP SOCP SDP
• Generality comes at a cost though– SOCP - O(L1/2) iterations of O(n2∑ni)– SDP - O((∑ni)1/2) iterations of O(n2∑ni
2)
• Many standard problems are SDP-able
![Page 36: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/36.jpg)
SDP-able Problems
• Minimizing the maximum eigenvalue
• Class separation with ellipsoids
![Page 37: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/37.jpg)
SDP-able Problems
• Minimizing the maximum eigenvalue
• Class separation with ellipsoids
![Page 38: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/38.jpg)
Minimizing the Maximum Eigenvalue
Matrix M(z)
To find vector z* such that max is minimized.
Let max(M(z)) <= n
max(M(z)-nI) <= 0
min(nI - M(z)) >= 0
nI - M(z) >= 0
![Page 39: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/39.jpg)
Minimizing the Maximum Eigenvalue
Matrix M(z)
To find vector z* such that max is minimized.
Max -nnI - M(z) >= 0
![Page 40: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/40.jpg)
SDP-able Problems
• Minimizing the maximum eigenvalue
• Class separation with ellipsoids
![Page 41: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/41.jpg)
![Page 42: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/42.jpg)
![Page 43: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/43.jpg)
Outline
• Convex Optimization– Second Order Cone Programming (SOCP)– Semidefinite Programming (SDP)
• Non-convex optimization– SDP relaxations– SOCP relaxations
• Optimization Algorithms– Interior Point Method for SOCP– Interior Point Method for SDP
![Page 44: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/44.jpg)
Non-Convex Problems
Minimize xTQ0x + 2q0Tx + r0
Subject to xTQix + 2qiTx + ri < = 0
Qi >= 0
=> Convex
Redefine x in homogenous coordinates.
y = (1; x)
Non-Convex Quadratic Programming Problem !!!
![Page 45: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/45.jpg)
Non-Convex Problems
Minimize xTQ0x + 2q0Tx + r0
Subject to xTQix + 2qiTx + ri < = 0
Minimize yTM0y
Subject to yTMiy < = 0
Mi = [ ri qiT; qi Qi]
Let’s solve this now !!!
![Page 46: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/46.jpg)
Non-Convex Problems
• Problem is NP-hard.
• Let’s relax the problem to make it convex.
• Pray !!!
![Page 47: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/47.jpg)
Outline
• Convex Optimization– Second Order Cone Programming (SOCP)– Semidefinite Programming (SDP)
• Non-convex optimization– SDP relaxations– SOCP relaxations
• Optimization Algorithms– Interior Point Method for SOCP– Interior Point Method for SDP
![Page 48: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/48.jpg)
SDP Relaxation
Minimize yTM0y
Subject to yTMiy < = 0
Minimize M0 Y
Subject to Mi Y < = 0
Y = yyT
Bad Constraint !!!!
No donut for you !!!
![Page 49: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/49.jpg)
SDP Relaxation
Minimize yTM0y
Subject to yTMiy < = 0
Minimize M0 Y
Subject to Mi Y < = 0
Y >= 0
SDP Problem
Nothing left to do ….but Pray
Note that we have squared the number of variables.
![Page 50: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/50.jpg)
Example - Max Cut
• Graph: G=(V,E)
• Maximum-Cut
![Page 51: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/51.jpg)
• Graph: G=(V,E)
• Maximum-Cut
Example - Max Cut
- xi = -1
- xi = +1
![Page 52: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/52.jpg)
• Graph: G=(V,E)
• Maximum-Cut
Example - Max Cut
Alright !!! So it’s an integer programming problem !!!
Doesn’t look like quadratic programming to me !!!
![Page 53: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/53.jpg)
Max Cut as an IQP
Max Cut problem can be written as
Naah !! Let’s get it into the standard quadratic form.
![Page 54: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/54.jpg)
![Page 55: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/55.jpg)
Max Cut as an IQP
Max Cut problem can be written as
Naah !! Let’s get it into the standard quadratic form.
![Page 56: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/56.jpg)
Solving Max Cut using SDP Relaxations
To the white board.(You didn’t think I’ll prepare slides for this, did you??)
![Page 57: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/57.jpg)
Outline
• Convex Optimization– Second Order Cone Programming (SOCP)– Semidefinite Programming (SDP)
• Non-convex optimization– SDP relaxations– SOCP relaxations
• Optimization Algorithms– Interior Point Method for SOCP– Interior Point Method for SDP
![Page 58: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/58.jpg)
SOCP Relaxation
Minimize yTM0y
Subject to yTMiy < = 0
Minimize M0 Y
Subject to Mi Y < = 0
Y >= 0
Remember
Y = [1 xT; x X]
X - xxT >= 0
![Page 59: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/59.jpg)
SOCP Relaxation
Say you’re given C = { C1, C2, … Cn} such that Cj >= 0
Cj (X - xxT) >= 0
(Ux)T (Ux) <= Cj X
Wait .. Isn’t this a hyperbolic constraint
Therefore, it’s SOCP-able.
![Page 60: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/60.jpg)
SOCP Relaxation
Minimize yTM0y
Subject to yTMiy < = 0
Minimize Q0 X + 2q0Tx + r0
Subject to Qi X + 2qiTx + ri < = 0
Cj (X - xxT) >= 0
Cj C
![Page 61: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/61.jpg)
SOCP Relaxation
If C is the infinite set of all semidefinite matricesSOCP Relaxation = SDP Relaxation
If C is finite, SOCP relaxation is ‘looser’ than SDP relaxation.
Then why SOCP relaxation ???
Efficiency - Accuracy Tradeoff
![Page 62: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/62.jpg)
Choice of C
Remember we had squared the number of variables.
Let’s try to reduce them with our choice of C.
For a general problem - Kim and Kojima
Using the structure of a specific problem - e.g. Muramatsu and Suzuki for Max Cut
![Page 63: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/63.jpg)
Choice of C
Minimize cT x
Subject to Qi X + 2qiTx + ri < = 0
Q X + 2qTx + r <= 0
Q = n i uiuiT
Let 1 >= 2 >= …. k >= 0 >= k+1 >= n
![Page 64: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/64.jpg)
Choice of C
C = Q+ = k i uiui
T
Q X + 2qTx + r <= 0
xT Q+ x - Q+ X <= 0
xT Q+ x + k+1 i uiuiT X + 2qTx + r <= 0
zi
![Page 65: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/65.jpg)
Choice of C
C = Q+ = k i uiui
T
xT Q+ x + k+1 i zi+ 2qTx + r <= 0
uiuiT i = k+1, k+2, … n
xTuiuiTx - uiui
T X <= 0
Q X + 2qTx + r <= 0
![Page 66: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/66.jpg)
Choice of C
C = Q+ = k i uiui
T
xT Q+ x + k+1 i zi+ 2qTx + r <= 0
uiuiT i = k+1, k+2, … n
xTuiuiTx - zi <= 0
Q X + 2qTx + r <= 0
![Page 67: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/67.jpg)
Specific Problem Example - Max Cut
ei = [0 0 …. 1 0 …0]
uij = ei + ej
vij = ei - ej
C = ei ei
T i = 1, … , |V|
uij uijT (i,j) E
vij vijT (i,j) E
![Page 68: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/68.jpg)
Specific Problem Example - Max Cut
Warning: Scary equations to follow.
![Page 69: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/69.jpg)
Outline
• Convex Optimization– Second Order Cone Programming (SOCP)– Semidefinite Programming (SDP)
• Non-convex optimization– SDP relaxations– SOCP relaxations
• Optimization Algorithms– Interior Point Method for SOCP– Interior Point Method for SDP
![Page 70: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/70.jpg)
Outline
• Convex Optimization– Second Order Cone Programming (SOCP)– Semidefinite Programming (SDP)
• Non-convex optimization– SDP relaxations– SOCP relaxations
• Optimization Algorithms– Interior Point Method for SOCP– Interior Point Method for SDP
![Page 71: Convex Programming](https://reader030.vdocuments.us/reader030/viewer/2022020920/56815870550346895dc5d0ba/html5/thumbnails/71.jpg)
Back to work now !!!