discrete optimization lecture 3 – part 1 m. pawan kumar [email protected] slides available online
TRANSCRIPT
![Page 1: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/1.jpg)
Discrete OptimizationLecture 3 – Part 1
M. Pawan Kumar
Slides available online http://cvn.ecp.fr/personnel/pawan/
![Page 2: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/2.jpg)
Energy Minimization
Va Vb Vc Vd
2
5
4
2
6
3
3
7
0
1 1
0
0
2
1
1
4 1
0
3
Q(f; ) = ∑a a;f(a) + ∑(a,b) ab;f(a)f(b)
Label l0
Label l1
![Page 3: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/3.jpg)
Energy Minimization
Va Vb Vc Vd
2
5
4
2
6
3
3
7
0
1 1
0
0
2
1
1
4 1
0
3
Q(f; ) = ∑a a;f(a) + ∑(a,b) ab;f(a)f(b)
2 + 1 + 2 + 1 + 3 + 1 + 3 = 13
Label l0
Label l1
![Page 4: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/4.jpg)
Energy Minimization
Va Vb Vc Vd
2
5
4
2
6
3
3
7
0
1 1
0
0
2
1
1
4 1
0
3
Q(f; ) = ∑a a;f(a) + ∑(a,b) ab;f(a)f(b)
Label l0
Label l1
![Page 5: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/5.jpg)
Energy Minimization
Va Vb Vc Vd
2
5
4
2
6
3
3
7
0
1 1
0
0
2
1
1
4 1
0
3
Q(f; ) = ∑a a;f(a) + ∑(a,b) ab;f(a)f(b)
5 + 1 + 4 + 0 + 6 + 4 + 7 = 27
Label l0
Label l1
![Page 6: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/6.jpg)
Energy Minimization
Va Vb Vc Vd
2
5
4
2
6
3
3
7
0
1 1
0
0
2
1
1
4 1
0
3
Q(f; ) = ∑a a;f(a) + ∑(a,b) ab;f(a)f(b)
f* = arg min Q(f; )
q* = min Q(f; ) = Q(f*; )
Label l0
Label l1
![Page 7: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/7.jpg)
Min-Marginals
Va Vb Vc Vd
2
5
4
2
6
3
3
7
0
1 1
0
0
2
1
1
4 1
0
3
f* = arg min Q(f; ) such that f(a) = i
Min-marginal qa;i
Label l0
Label l1
![Page 8: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/8.jpg)
Min-Marginals and MAP• Minimum min-marginal of any variable = energy of MAP labelling
minf Q(f; ) such that f(a) = i
qa;i mini
mini ( )
Va has to take one label
minf Q(f; )
![Page 9: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/9.jpg)
RecapWe only need to know two sets of equations
General form of Reparameterization
’a;i = a;i
’ab;ik = ab;ik
+ Mab;k
- Mab;k
+ Mba;i
- Mba;i
’b;k = b;k
Reparameterization of (a,b) in Belief Propagation
Mab;k = mini { a;i + ab;ik }
Mba;i = 0
![Page 10: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/10.jpg)
Dynamic Programming
3 variables 2 variables + book-keeping
n variables (n-1) variables + book-keeping
Start from left, go to right
Reparameterize current edge (a,b)
Mab;k = mini { a;i + ab;ik }
’ab;ik = ab;ik+ Mab;k - Mab;k’b;k = b;k
Repeat
![Page 11: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/11.jpg)
• LP Relaxation and its Dual
• TRW Message Passing
Outline
![Page 12: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/12.jpg)
Integer Programming Formulation
min ∑a ∑i a;i ya;i + ∑(a,b) ∑ik ab;ik yab;ik
ya;i {0,1}
∑i ya;i = 1
yab;ik = ya;i yb;k
![Page 13: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/13.jpg)
Integer Programming Formulation
min Ty
ya;i {0,1}
∑i ya;i = 1
yab;ik = ya;i yb;k
= [ … a;i …. ; … ab;ik ….]
y = [ … ya;i …. ; … yab;ik ….]
![Page 14: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/14.jpg)
Linear Programming Relaxation
min Ty
ya;i {0,1}
∑i ya;i = 1
yab;ik = ya;i yb;k
Two reasons why we can’t solve this
![Page 15: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/15.jpg)
Linear Programming Relaxation
min Ty
ya;i [0,1]
∑i ya;i = 1
yab;ik = ya;i yb;k
One reason why we can’t solve this
![Page 16: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/16.jpg)
Linear Programming Relaxation
min Ty
ya;i [0,1]
∑i ya;i = 1
∑k yab;ik = ∑kya;i yb;k
One reason why we can’t solve this
![Page 17: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/17.jpg)
Linear Programming Relaxation
min Ty
ya;i [0,1]
∑i ya;i = 1
One reason why we can’t solve this
= 1∑k yab;ik = ya;i∑k yb;k
![Page 18: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/18.jpg)
Linear Programming Relaxation
min Ty
ya;i [0,1]
∑i ya;i = 1
∑k yab;ik = ya;i
One reason why we can’t solve this
![Page 19: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/19.jpg)
Linear Programming Relaxation
min Ty
ya;i [0,1]
∑i ya;i = 1
∑k yab;ik = ya;i
No reason why we can’t solve this *
*memory requirements, time complexity
![Page 20: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/20.jpg)
Dual
Let’s try to understand it intuitively
![Page 21: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/21.jpg)
Dual of the LP RelaxationWainwright et al., 2001
Va Vb Vc
Vd Ve Vf
Vg Vh Vi
Va Vb Vc
Vd Ve Vf
Vg Vh Vi
Va Vb Vc
Vd Ve Vf
Vg Vh Vi
1
2
3
4 5 6
i
![Page 22: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/22.jpg)
Dual of the LP RelaxationWainwright et al., 2001
q*(1)
i
q*(2)
q*(3)
q*(4) q*(5) q*(6)
q*(i)
Dual of LP
Va Vb Vc
Vd Ve Vf
Vg Vh Vi
Va Vb Vc
Vd Ve Vf
Vg Vh Vi
Va Vb Vc
Vd Ve Vf
Vg Vh Vi
max
![Page 23: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/23.jpg)
Dual of the LP RelaxationWainwright et al., 2001
i
max q*(i)
I can easily compute q*(i)
I can easily maintain reparam constraint
So can I easily solve the dual?
![Page 24: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/24.jpg)
• LP Relaxation and its Dual
• TRW Message Passing
Outline
![Page 25: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/25.jpg)
Things to Remember
• Forward-pass computes min-marginals of root
• BP is exact for trees
• Every iteration provides a reparameterization
![Page 26: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/26.jpg)
TRW Message PassingKolmogorov, 2006
Va Vb Vc
Vd Ve Vf
Vg Vh Vi
VaVb Vc
VdVe Vf
VgVh Vi
1
2
3
4 5 6
i
q*(i)
Pick a variable Va
![Page 27: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/27.jpg)
TRW Message PassingKolmogorov, 2006
i
q*(i)
Vc Vb Va
1c;0
1c;1
1b;0
1b;1
1a;0
1a;1
Va Vd Vg
4a;0
4a;1
4d;0
4d;1
4g;0
4g;1
![Page 28: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/28.jpg)
TRW Message PassingKolmogorov, 2006
1 + 4 + rest
q*(1) + q*(4) + K
Vc Vb Va Va Vd Vg
Reparameterize to obtain min-marginals of Va
1c;0
1c;1
1b;0
1b;1
1a;0
1a;1
4a;0
4a;1
4d;0
4d;1
4g;0
4g;1
![Page 29: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/29.jpg)
TRW Message PassingKolmogorov, 2006
’1 + ’4 + rest
Vc Vb Va
’1c;0
’1c;1
’1b;0
’1b;1
’1a;0
’1a;1
Va Vd Vg
’4a;0
’4a;1
’4d;0
’4d;1
’4g;0
’4g;1
One pass of Belief Propagation
q*(’1) + q*(’4) + K
![Page 30: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/30.jpg)
TRW Message PassingKolmogorov, 2006
’1 + ’4 + rest
Vc Vb Va Va Vd Vg
Remain the same
q*(’1) + q*(’4) + K
’1c;0
’1c;1
’1b;0
’1b;1
’1a;0
’1a;1
’4a;0
’4a;1
’4d;0
’4d;1
’4g;0
’4g;1
![Page 31: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/31.jpg)
TRW Message PassingKolmogorov, 2006
’1 + ’4 + rest
min{’1a;0,’1a;1} + min{’4a;0,’4a;1} + K
Vc Vb Va Va Vd Vg
’1c;0
’1c;1
’1b;0
’1b;1
’1a;0
’1a;1
’4a;0
’4a;1
’4d;0
’4d;1
’4g;0
’4g;1
![Page 32: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/32.jpg)
TRW Message PassingKolmogorov, 2006
’1 + ’4 + rest
Vc Vb Va Va Vd Vg
Compute average of min-marginals of Va
’1c;0
’1c;1
’1b;0
’1b;1
’1a;0
’1a;1
’4a;0
’4a;1
’4d;0
’4d;1
’4g;0
’4g;1
min{’1a;0,’1a;1} + min{’4a;0,’4a;1} + K
![Page 33: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/33.jpg)
TRW Message PassingKolmogorov, 2006
’1 + ’4 + rest
Vc Vb Va Va Vd Vg
’’a;0 = ’1a;0+ ’4a;0
2
’’a;1 = ’1a;1+ ’4a;1
2
’1c;0
’1c;1
’1b;0
’1b;1
’1a;0
’1a;1
’4a;0
’4a;1
’4d;0
’4d;1
’4g;0
’4g;1
min{’1a;0,’1a;1} + min{’4a;0,’4a;1} + K
![Page 34: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/34.jpg)
TRW Message PassingKolmogorov, 2006
’’1 + ’’4 + rest
Vc Vb Va Va Vd Vg
’1c;0
’1c;1
’1b;0
’1b;1
’’a;0
’’a;1
’’a;0
’’a;1
’4d;0
’4d;1
’4g;0
’4g;1
’’a;0 = ’1a;0+ ’4a;0
2
’’a;1 = ’1a;1+ ’4a;1
2
min{’1a;0,’1a;1} + min{’4a;0,’4a;1} + K
![Page 35: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/35.jpg)
TRW Message PassingKolmogorov, 2006
’’1 + ’’4 + rest
Vc Vb Va Va Vd Vg
’1c;0
’1c;1
’1b;0
’1b;1
’’a;0
’’a;1
’’a;0
’’a;1
’4d;0
’4d;1
’4g;0
’4g;1
’’a;0 = ’1a;0+ ’4a;0
2
’’a;1 = ’1a;1+ ’4a;1
2
min{’1a;0,’1a;1} + min{’4a;0,’4a;1} + K
![Page 36: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/36.jpg)
TRW Message PassingKolmogorov, 2006
Vc Vb Va Va Vd Vg
2 min{’’a;0, ’’a;1} + K
’1c;0
’1c;1
’1b;0
’1b;1
’’a;0
’’a;1
’’a;0
’’a;1
’4d;0
’4d;1
’4g;0
’4g;1
’’1 + ’’4 + rest
’’a;0 = ’1a;0+ ’4a;0
2
’’a;1 = ’1a;1+ ’4a;1
2
![Page 37: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/37.jpg)
TRW Message PassingKolmogorov, 2006
Vc Vb Va Va Vd Vg
’1c;0
’1c;1
’1b;0
’1b;1
’’a;0
’’a;1
’’a;0
’’a;1
’4d;0
’4d;1
’4g;0
’4g;1
min {p1+p2, q1+q2} min {p1, q1} + min {p2, q2}≥
2 min{’’a;0, ’’a;1} + K
’’1 + ’’4 + rest
![Page 38: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/38.jpg)
TRW Message PassingKolmogorov, 2006
Vc Vb Va Va Vd Vg
Objective function increases or remains constant
’1c;0
’1c;1
’1b;0
’1b;1
’’a;0
’’a;1
’’a;0
’’a;1
’4d;0
’4d;1
’4g;0
’4g;1
2 min{’’a;0, ’’a;1} + K
’’1 + ’’4 + rest
![Page 39: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/39.jpg)
TRW Message Passing
Initialize i. Take care of reparam constraint
Choose random variable Va
Compute min-marginals of Va for all trees
Node-average the min-marginals
REPEAT
Kolmogorov, 2006
Can also do edge-averaging
![Page 40: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/40.jpg)
• Preliminaries
• LP Relaxation and its Dual
• TRW Message Passing– Examples– Primal Solution– Results
Outline
![Page 41: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/41.jpg)
Example 1
Va Vb
0
1 1
0
2
5
4
2l0
l1
Vb Vc
0
2 3
1
4
2
6
3
Vc Va
1
4 1
0
6
3
6
4
5
6
7
Pick variable Va. Reparameterize.
![Page 42: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/42.jpg)
Example 1
Va Vb
-3
-2 -1
-2
5
7
4
2
Vb Vc
0
2 3
1
4
2
6
3
Vc Va
-3
1 -3
-3
6
3
10
7
5
6
7
Average the min-marginals of Va
l0
l1
![Page 43: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/43.jpg)
Example 1
Va Vb
-3
-2 -1
-2
7.5
7
4
2
Vb Vc
0
2 3
1
4
2
6
3
Vc Va
-3
1 -3
-3
6
3
7.5
7
7
6
7
Pick variable Vb. Reparameterize.
l0
l1
![Page 44: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/44.jpg)
Example 1
Va Vb
-7.5
-7 -5.5
-7
7.5
7
8.5
7
Vb Vc
-5
-3 -1
-3
9
6
6
3
Vc Va
-3
1 -3
-3
6
3
7.5
7
7
6
7
Average the min-marginals of Vb
l0
l1
![Page 45: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/45.jpg)
Example 1
Va Vb
-7.5
-7 -5.5
-7
7.5
7
8.75
6.5
Vb Vc
-5
-3 -1
-3
8.75
6.5
6
3
Vc Va
-3
1 -3
-3
6
3
7.5
7
6.5
6.5
7 Value of dual does not increase
l0
l1
![Page 46: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/46.jpg)
Example 1
Va Vb
-7.5
-7 -5.5
-7
7.5
7
8.75
6.5
Vb Vc
-5
-3 -1
-3
8.75
6.5
6
3
Vc Va
-3
1 -3
-3
6
3
7.5
7
6.5
6.5
7 Maybe it will increase for Vc
NO
l0
l1
![Page 47: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/47.jpg)
Example 1
Va Vb
-7.5
-7 -5.5
-7
7.5
7
8.75
6.5
Vb Vc
-5
-3 -1
-3
8.75
6.5
6
3
Vc Va
-3
1 -3
-3
6
3
7.5
7
Strong Tree Agreement
Exact MAP Estimate
f1(a) = 0 f1(b) = 0 f2(b) = 0 f2(c) = 0 f3(c) = 0 f3(a) = 0
l0
l1
![Page 48: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/48.jpg)
Example 2
Va Vb
0
1 1
0
2
5
2
2
Vb Vc
1
0 0
1
0
0
0
0
Vc Va
0
1 1
0
0
3
4
8
4
0
4
Pick variable Va. Reparameterize.
l0
l1
![Page 49: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/49.jpg)
Example 2
Va Vb
-2
-1 -1
-2
4
7
2
2
Vb Vc
1
0 0
1
0
0
0
0
Vc Va
0
0 1
-1
0
3
4
9
4
0
4
Average the min-marginals of Va
l0
l1
![Page 50: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/50.jpg)
Example 2
Va Vb
-2
-1 -1
-2
4
8
2
2
Vb Vc
1
0 0
1
0
0
0
0
Vc Va
0
0 1
-1
0
3
4
8
4
0
4 Value of dual does not increase
l0
l1
![Page 51: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/51.jpg)
Example 2
Va Vb
-2
-1 -1
-2
4
8
2
2
Vb Vc
1
0 0
1
0
0
0
0
Vc Va
0
0 1
-1
0
3
4
8
4
0
4 Maybe it will decrease for Vb or Vc
NO
l0
l1
![Page 52: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/52.jpg)
Example 2
Va Vb
-2
-1 -1
-2
4
8
2
2
Vb Vc
1
0 0
1
0
0
0
0
Vc Va
0
0 1
-1
0
3
4
8
f1(a) = 1 f1(b) = 1 f2(b) = 1 f2(c) = 0 f3(c) = 1 f3(a) = 1
f2(b) = 0 f2(c) = 1
Weak Tree Agreement
Not Exact MAP Estimate
l0
l1
![Page 53: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/53.jpg)
Example 2
Va Vb
-2
-1 -1
-2
4
8
2
2
Vb Vc
1
0 0
1
0
0
0
0
Vc Va
0
0 1
-1
0
3
4
8
Weak Tree Agreement
Convergence point of TRW
l0
l1
f1(a) = 1 f1(b) = 1 f2(b) = 1 f2(c) = 0 f3(c) = 1 f3(a) = 1
f2(b) = 0 f2(c) = 1
![Page 54: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/54.jpg)
• Preliminaries
• LP Relaxation and its Dual
• TRW Message Passing– Examples– Primal Solution– Results
Outline
![Page 55: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/55.jpg)
Obtaining the Labelling
Only solves the dual. Primal solutions?
Va Vb Vc
Vd Ve Vf
Vg Vh Vi
’ = i
Fix the labelOf Va
![Page 56: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/56.jpg)
Obtaining the Labelling
Only solves the dual. Primal solutions?
Va Vb Vc
Vd Ve Vf
Vg Vh Vi
’ = i
Fix the labelOf Vb
Continue in some fixed order
Meltzer et al., 2006
![Page 57: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/57.jpg)
Computational Issues of TRW
• Speed-ups for some pairwise potentials
Basic Component is Belief Propagation
Felzenszwalb & Huttenlocher, 2004
• Memory requirements cut down by half
Kolmogorov, 2006
• Further speed-ups using monotonic chains
Kolmogorov, 2006
![Page 58: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/58.jpg)
Theoretical Properties of TRW
• Always converges, unlike BP
Kolmogorov, 2006
• Strong tree agreement implies exact MAP
Wainwright et al., 2001
• Optimal MAP for two-label submodular problems
Kolmogorov and Wainwright, 2005
ab;00 + ab;11 ≤ ab;01 + ab;10
![Page 59: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/59.jpg)
• Preliminaries
• LP Relaxation and its Dual
• TRW Message Passing– Examples– Primal Solution– Results
Outline
![Page 60: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/60.jpg)
ResultsBinary Segmentation Szeliski et al. , 2008
Labels - {foreground, background}
Unary Potentials: -log(likelihood) using learnt fg/bg models
Pairwise Potentials: 0, if same labels
1 - exp(|da - db|), if different labels
![Page 61: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/61.jpg)
ResultsBinary Segmentation
Labels - {foreground, background}
Unary Potentials: -log(likelihood) using learnt fg/bg models
Szeliski et al. , 2008
Pairwise Potentials: 0, if same labels
1 - exp(|da - db|), if different labels
TRW
![Page 62: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/62.jpg)
ResultsBinary Segmentation
Labels - {foreground, background}
Unary Potentials: -log(likelihood) using learnt fg/bg models
Szeliski et al. , 2008
Belief Propagation
Pairwise Potentials: 0, if same labels
1 - exp(|da - db|), if different labels
![Page 63: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/63.jpg)
ResultsStereo Correspondence Szeliski et al. , 2008
Labels - {disparities}
Unary Potentials: Similarity of pixel colours
Pairwise Potentials: 0, if same labels
1 - exp(|da - db|), if different labels
![Page 64: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/64.jpg)
ResultsSzeliski et al. , 2008
Labels - {disparities}
Unary Potentials: Similarity of pixel colours
Pairwise Potentials: 0, if same labels
1 - exp(|da - db|), if different labels
TRW
Stereo Correspondence
![Page 65: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/65.jpg)
ResultsSzeliski et al. , 2008
Labels - {disparities}
Unary Potentials: Similarity of pixel colours
Belief Propagation
Pairwise Potentials: 0, if same labels
1 - exp(|da - db|), if different labels
Stereo Correspondence
![Page 66: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/66.jpg)
ResultsNon-submodular problems Kolmogorov, 2006
BP TRW-S
30x30 grid K50
BP TRW-S
BP outperforms TRW-S
![Page 67: Discrete Optimization Lecture 3 – Part 1 M. Pawan Kumar pawan.kumar@ecp.fr Slides available online](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f1c5503460f94c32ae4/html5/thumbnails/67.jpg)
Code + Standard Data
http://vision.middlebury.edu/MRF