finding k-best map solutions using lp relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf ·...
TRANSCRIPT
![Page 1: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/1.jpg)
Finding k-best MAP Solutions Using LP Relaxations
Amir GlobersonSchool of Computer Science and Engineering
The Hebrew University
Joint Work with: Menachem Fromer (Hebrew Univ.)
![Page 2: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/2.jpg)
Prediction ProblemsConsider the following problem:
Observe variables:
Predict variables: xh
xv
![Page 3: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/3.jpg)
Prediction ProblemsConsider the following problem:
Observe variables:
Predict variables:
Noisy Image Source Image
Received bits Code word
Symptoms Disease
Sentence Derivation
Countless applications:
Images:
Error correcting codes
Medical diagnostics
Text
Visible Hidden
xh
xv
![Page 4: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/4.jpg)
Statistical Models for Prediction
![Page 5: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/5.jpg)
Statistical Models for Prediction
One approach:
![Page 6: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/6.jpg)
Statistical Models for Prediction
One approach:
Assume (or learn) a model for p(xh,xv)
![Page 7: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/7.jpg)
Statistical Models for Prediction
One approach:
Assume (or learn) a model for
Predict the most likely hidden values
p(xh,xv)
arg maxxh
p(xh|xv)
![Page 8: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/8.jpg)
Statistical Models for Prediction
One approach:
Assume (or learn) a model for
Predict the most likely hidden values
p(xh,xv)
arg maxxh
p(xh|xv)
This conditional distribution often corresponds to a graphical model
![Page 9: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/9.jpg)
Statistical Models for Prediction
One approach:
Assume (or learn) a model for
Predict the most likely hidden values
p(xh,xv)
arg maxxh
p(xh|xv)
This conditional distribution often corresponds to a graphical model
Need to know how to find an assignment with maximum probability
![Page 10: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/10.jpg)
The MAP ProblemGiven a graphical model over
f(x) =!
ij
!ij(xi, xj)
x1, . . . , xn
Find the most likely assignment:
xi
xj!ij(xi, xj)
p(x) =1Z
ef(x)
arg maxx
f(x)
![Page 11: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/11.jpg)
MAP Approximationsx is discrete so generally NP hard
![Page 12: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/12.jpg)
MAP Approximations
Many approximation approaches:
x is discrete so generally NP hard
![Page 13: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/13.jpg)
MAP Approximations
Many approximation approaches:
Greedy search
x is discrete so generally NP hard
![Page 14: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/14.jpg)
MAP Approximations
Many approximation approaches:
Greedy search
Loopy belief propagation (e.g., max product)
x is discrete so generally NP hard
![Page 15: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/15.jpg)
MAP Approximations
Many approximation approaches:
Greedy search
Loopy belief propagation (e.g., max product)
Linear programming relaxations
x is discrete so generally NP hard
![Page 16: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/16.jpg)
MAP Approximations
Many approximation approaches:
Greedy search
Loopy belief propagation (e.g., max product)
Linear programming relaxations
x is discrete so generally NP hard
![Page 17: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/17.jpg)
MAP Approximations
Many approximation approaches:
Greedy search
Loopy belief propagation (e.g., max product)
Linear programming relaxations
LP approaches
x is discrete so generally NP hard
![Page 18: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/18.jpg)
MAP Approximations
Many approximation approaches:
Greedy search
Loopy belief propagation (e.g., max product)
Linear programming relaxations
LP approaches
Provide optimality certificates
x is discrete so generally NP hard
![Page 19: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/19.jpg)
MAP Approximations
Many approximation approaches:
Greedy search
Loopy belief propagation (e.g., max product)
Linear programming relaxations
LP approaches
Provide optimality certificates
Optimal in some cases (e.g., submodular functions)
x is discrete so generally NP hard
![Page 20: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/20.jpg)
MAP Approximations
Many approximation approaches:
Greedy search
Loopy belief propagation (e.g., max product)
Linear programming relaxations
LP approaches
Provide optimality certificates
Optimal in some cases (e.g., submodular functions)
Can be solved via message passing
x is discrete so generally NP hard
![Page 21: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/21.jpg)
The k-best MAP Problem
![Page 22: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/22.jpg)
The k-best MAP Problem
Find the k best assignments for f(x)
![Page 23: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/23.jpg)
The k-best MAP Problem
Find the k best assignments for f(x)
Denote these by x(1), . . . ,x(k)
![Page 24: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/24.jpg)
The k-best MAP Problem
Find the k best assignments for f(x)
Denote these by
Useful in:
x(1), . . . ,x(k)
![Page 25: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/25.jpg)
The k-best MAP Problem
Find the k best assignments for f(x)
Denote these by
Useful in:
Finding multiple candidate solutions when the energy function is not accurate (e.g., protein design)
x(1), . . . ,x(k)
![Page 26: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/26.jpg)
The k-best MAP Problem
Find the k best assignments for f(x)
Denote these by
Useful in:
Finding multiple candidate solutions when the energy function is not accurate (e.g., protein design)
As a first processing stage before applying more complex methods
x(1), . . . ,x(k)
![Page 27: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/27.jpg)
The k-best MAP Problem
Find the k best assignments for f(x)
Denote these by
Useful in:
Finding multiple candidate solutions when the energy function is not accurate (e.g., protein design)
As a first processing stage before applying more complex methods
Supervised learning
x(1), . . . ,x(k)
![Page 28: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/28.jpg)
From 2 to k best
We can show that given a polynomial algorithm for k=2, the problem can be solved for any k in O(k)
Focus on k=2
Our key question: what is the LP formulation of the problem, and its relaxations?
![Page 29: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/29.jpg)
OutlineLP formulation of the MAP problem
LP for 2nd best
General (intractable) exact formulation
Tractable formulation for tree graphs
Approximations for non-tree graphs
Experiments
![Page 30: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/30.jpg)
MAP and LP
![Page 31: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/31.jpg)
MAP and LPMAP: max
xf(x)
![Page 32: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/32.jpg)
MAP and LPMAP:
MAP as LP:
maxx
f(x)
![Page 33: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/33.jpg)
MAP and LPMAP:
MAP as LP:
maxx
f(x)
maxµ!S
µ · !
![Page 34: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/34.jpg)
MAP and LPMAP:
MAP as LP:
S
maxx
f(x)
maxµ!S
µ · !
![Page 35: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/35.jpg)
MAP and LPMAP:
MAP as LP:
S
Hard
maxx
f(x)
maxµ!S
µ · !
![Page 36: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/36.jpg)
MAP and LPMAP:
MAP as LP:
S
Hard
Approximate MAP via LP
maxx
f(x)
maxµ!S
µ · !
![Page 37: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/37.jpg)
MAP and LPMAP:
MAP as LP:
S
Hard
Approximate MAP via LP
maxx
f(x)
maxµ!S
µ · !
![Page 38: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/38.jpg)
MAP and LPMAP:
MAP as LP:
S
Hard
Approximate MAP via LP
maxx
f(x)
Schlesinger, Deza & Laurent, Boros, Wainwright, Kolmogorov
maxµ!S
µ · !
![Page 39: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/39.jpg)
LP Formulation of MAP
![Page 40: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/40.jpg)
LP Formulation of MAPx! = arg max
x
!
ij"E
!ij(xi, xj)
![Page 41: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/41.jpg)
LP Formulation of MAP
maxq(x)
!
x
q(x)!
ij
!ij(xi, xj)=
x! = arg maxx
!
ij"E
!ij(xi, xj)
![Page 42: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/42.jpg)
LP Formulation of MAP
maxq(x)
!
x
q(x)!
ij
!ij(xi, xj)=
0
1q!(x)
xx!x! = arg max
x
!
ij"E
!ij(xi, xj)
![Page 43: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/43.jpg)
LP Formulation of MAP
maxq(x)
!
x
q(x)!
ij
!ij(xi, xj) maxq(x)
!
ij
!
xi,xj
qij(xi, xj)!ij(xi, xj)= =
0
1q!(x)
xx!x! = arg max
x
!
ij"E
!ij(xi, xj)
![Page 44: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/44.jpg)
LP Formulation of MAP
Objective depends only on pairwise marginals
maxq(x)
!
x
q(x)!
ij
!ij(xi, xj) maxq(x)
!
ij
!
xi,xj
qij(xi, xj)!ij(xi, xj)= =
0
1q!(x)
xx!x! = arg max
x
!
ij"E
!ij(xi, xj)
![Page 45: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/45.jpg)
LP Formulation of MAP
Objective depends only on pairwise marginals
But only those that correspond to some distribution
maxq(x)
!
x
q(x)!
ij
!ij(xi, xj) maxq(x)
!
ij
!
xi,xj
qij(xi, xj)!ij(xi, xj)= =
0
1q!(x)
xx!x! = arg max
x
!
ij"E
!ij(xi, xj)
q(x)
![Page 46: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/46.jpg)
LP Formulation of MAP
Objective depends only on pairwise marginals
But only those that correspond to some distribution
This set is called the Marginal polytope ( Wainwright & Jordan)
maxq(x)
!
x
q(x)!
ij
!ij(xi, xj) maxq(x)
!
ij
!
xi,xj
qij(xi, xj)!ij(xi, xj)= =
0
1q!(x)
xx!x! = arg max
x
!
ij"E
!ij(xi, xj)
q(x)
![Page 47: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/47.jpg)
LP Formulation of MAP
Objective depends only on pairwise marginals
But only those that correspond to some distribution
This set is called the Marginal polytope ( Wainwright & Jordan)
maxq(x)
!
x
q(x)!
ij
!ij(xi, xj) maxq(x)
!
ij
!
xi,xj
qij(xi, xj)!ij(xi, xj)= =
0
1q!(x)
xx!x! = arg max
x
!
ij"E
!ij(xi, xj)
q(x)
maxx
!
ij
!ij(xi, xj) = maxµ!M(G)
!
ij
µij(xi, xj)!ij(xi, xj)
![Page 48: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/48.jpg)
LP Formulation of MAP
Objective depends only on pairwise marginals
But only those that correspond to some distribution
This set is called the Marginal polytope ( Wainwright & Jordan)
maxq(x)
!
x
q(x)!
ij
!ij(xi, xj) maxq(x)
!
ij
!
xi,xj
qij(xi, xj)!ij(xi, xj)= =
0
1q!(x)
xx!x! = arg max
x
!
ij"E
!ij(xi, xj)
q(x)
maxx
!
ij
!ij(xi, xj) = maxµ!M(G)
!
ij
µij(xi, xj)!ij(xi, xj)= maxµ!M(G)
µ · !
![Page 49: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/49.jpg)
LP Formulation of MAP
Objective depends only on pairwise marginals
But only those that correspond to some distribution
This set is called the Marginal polytope ( Wainwright & Jordan)
maxq(x)
!
x
q(x)!
ij
!ij(xi, xj) maxq(x)
!
ij
!
xi,xj
qij(xi, xj)!ij(xi, xj)= =
0
1q!(x)
xx!x! = arg max
x
!
ij"E
!ij(xi, xj)
q(x)
maxx
!
ij
!ij(xi, xj) = maxµ!M(G)
!
ij
µij(xi, xj)!ij(xi, xj)
See: Cut polytope (Deza, Laurent), Quadric polytope (Boros)
= maxµ!M(G)
µ · !
![Page 50: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/50.jpg)
The Marginal Polytope
Marginal Polytope
M(G)max
µ!M(G)
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
![Page 51: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/51.jpg)
The Marginal Polytope
Marginal Polytope
M(G)µmax
µ!M(G)
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
![Page 52: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/52.jpg)
The Marginal Polytope
Marginal Polytope
M(G)µ
There exists a p(x) s.t. p(xi, xj) = µij(xi, xj)
maxµ!M(G)
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
![Page 53: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/53.jpg)
The Marginal Polytope
Marginal Polytope
M(G)µ
There exists a p(x) s.t. p(xi, xj) = µij(xi, xj)
maxµ!M(G)
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
Difficult set to characterize. Easy to outer bound
![Page 54: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/54.jpg)
The Marginal Polytope
Marginal Polytope
M(G)µ
There exists a p(x) s.t. p(xi, xj) = µij(xi, xj)
maxµ!M(G)
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
Difficult set to characterize. Easy to outer bound
The vertices have integral values and correspond to assignments on x
![Page 55: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/55.jpg)
Relaxing the MAP LPmax
x
!
ij
!ij(xi, xj) = maxµ!M(G)
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
M(G)
![Page 56: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/56.jpg)
Relaxing the MAP LPmax
x
!
ij
!ij(xi, xj) = maxµ!M(G)
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
Exact but Hard!M(G)
![Page 57: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/57.jpg)
Relaxing the MAP LPmax
x
!
ij
!ij(xi, xj) ! maxµ!S
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
S
M(G)
![Page 58: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/58.jpg)
Relaxing the MAP LPmax
x
!
ij
!ij(xi, xj) ! maxµ!S
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
If optimum is an integral vertex, MAP is solved
S
M(G)
![Page 59: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/59.jpg)
Relaxing the MAP LPmax
x
!
ij
!ij(xi, xj) ! maxµ!S
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
If optimum is an integral vertex, MAP is solved
Possible outer bound: Pairwise consistencyS
M(G)
![Page 60: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/60.jpg)
Relaxing the MAP LPmax
x
!
ij
!ij(xi, xj) ! maxµ!S
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
If optimum is an integral vertex, MAP is solved
Possible outer bound: Pairwise consistency
j!
i!
k! !
xi
µij(xi, xj) =!
xk
µjk(xj , xk)
S
M(G)
![Page 61: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/61.jpg)
Relaxing the MAP LPmax
x
!
ij
!ij(xi, xj) ! maxµ!S
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
If optimum is an integral vertex, MAP is solved
Possible outer bound: Pairwise consistency
j!
i!
k! !
xi
µij(xi, xj) =!
xk
µjk(xj , xk)Exact for trees
S
M(G)
![Page 62: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/62.jpg)
Relaxing the MAP LPmax
x
!
ij
!ij(xi, xj) ! maxµ!S
!
ij!E
!
xi,xj
µij(xi, xj)!ij(xi, xj)
If optimum is an integral vertex, MAP is solved
Possible outer bound: Pairwise consistency
j!
i!
k! !
xi
µij(xi, xj) =!
xk
µjk(xj , xk)
Efficient message passing schemes for solving the resulting (dual) LP
Exact for trees
S
M(G)
![Page 63: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/63.jpg)
OutlineLP formulation of the MAP problem
LP for 2nd best
General (intractable) exact formulation
Tractable formulation for tree graphs
Approximations for non-tree graphs
Experiments
![Page 64: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/64.jpg)
The 2nd best problem and LP
MAP 2nd best
![Page 65: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/65.jpg)
The 2nd best problem and LP
maxx
f(x)MAP 2nd best
![Page 66: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/66.jpg)
The 2nd best problem and LP
maxx !=x(1)
f(x)maxx
f(x)MAP 2nd best
![Page 67: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/67.jpg)
The 2nd best problem and LP
maxx !=x(1)
f(x)maxx
f(x)
maxµ!M(G)
µ · !
MAP 2nd best
![Page 68: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/68.jpg)
The 2nd best problem and LP
maxx !=x(1)
f(x)maxx
f(x)
maxµ!M(G)
µ · ! maxµ!M(G,x(1))
µ · !
x(1)
MAP 2nd best
![Page 69: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/69.jpg)
The 2nd best problem and LP
maxx !=x(1)
f(x)maxx
f(x)
maxµ!M(G)
µ · ! maxµ!M(G,x(1))
µ · !
x(1)
MAP 2nd best
Approximations:
![Page 70: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/70.jpg)
The 2nd best problem and LP
maxx !=x(1)
f(x)maxx
f(x)
maxµ!M(G)
µ · ! maxµ!M(G,x(1))
µ · !
x(1)
MAP 2nd best
Approximations:
![Page 71: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/71.jpg)
The 2nd best problem and LP
maxx !=x(1)
f(x)maxx
f(x)
maxµ!M(G)
µ · ! maxµ!M(G,x(1))
µ · !
x(1)
MAP 2nd best
Approximations:
![Page 72: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/72.jpg)
A new marginal polytope
Given an assignment z, define the Assignment Excluding Marginal Polytope:M(G, z)
![Page 73: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/73.jpg)
A new marginal polytope
Given an assignment z, define the Assignment Excluding Marginal Polytope:M(G, z)
M(G, z)
![Page 74: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/74.jpg)
A new marginal polytope
Given an assignment z, define the Assignment Excluding Marginal Polytope:M(G, z)
µ
M(G, z)
![Page 75: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/75.jpg)
A new marginal polytope
Given an assignment z, define the Assignment Excluding Marginal Polytope:M(G, z)
µ
There exists a p(x) s.t. p(xi, xj) = µij(xi, xj)
M(G, z)
![Page 76: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/76.jpg)
A new marginal polytope
Given an assignment z, define the Assignment Excluding Marginal Polytope:M(G, z)
µ
There exists a p(x) s.t. p(xi, xj) = µij(xi, xj)
and:
M(G, z)
![Page 77: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/77.jpg)
A new marginal polytope
Given an assignment z, define the Assignment Excluding Marginal Polytope:M(G, z)
µ
There exists a p(x) s.t. p(xi, xj) = µij(xi, xj)
and: p(z) = 0
M(G, z)
![Page 78: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/78.jpg)
A new marginal polytope
Given an assignment z, define the Assignment Excluding Marginal Polytope:M(G, z)
µ
There exists a p(x) s.t. p(xi, xj) = µij(xi, xj)
and: p(z) = 0
M(G, z)
![Page 79: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/79.jpg)
A new marginal polytope
Given an assignment z, define the Assignment Excluding Marginal Polytope:M(G, z)
µ
There exists a p(x) s.t. p(xi, xj) = µij(xi, xj)
and: p(z) = 0
M(G)
M(G, z)
![Page 80: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/80.jpg)
A new marginal polytope
Given an assignment z, define the Assignment Excluding Marginal Polytope:M(G, z)
µ
There exists a p(x) s.t. p(xi, xj) = µij(xi, xj)
and: p(z) = 0
zM(G)
M(G, z)
![Page 81: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/81.jpg)
LP for the 2nd best problem
The 2nd best problem corresponds to the following LP:
maxx !=x(1)
f(x;!) = maxµ"M(G,x(1))
µ · !
x(1)
![Page 82: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/82.jpg)
LP for the 2nd best problem
The 2nd best problem corresponds to the following LP:
maxx !=x(1)
f(x;!) = maxµ"M(G,x(1))
µ · !
Is there a simple characterization of ? M(G, x(1))
x(1)
![Page 83: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/83.jpg)
LP for the 2nd best problem
The 2nd best problem corresponds to the following LP:
maxx !=x(1)
f(x;!) = maxµ"M(G,x(1))
µ · !
Is there a simple characterization of ? M(G, x(1))
Is it plus one inequality?M(G)
x(1)
![Page 84: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/84.jpg)
LP for the 2nd best problem
The 2nd best problem corresponds to the following LP:
maxx !=x(1)
f(x;!) = maxµ"M(G,x(1))
µ · !
Is there a simple characterization of ? M(G, x(1))
Is it plus one inequality?
If so, what inequality?
M(G)
x(1)
![Page 85: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/85.jpg)
OutlineLP formulation of the MAP problem
LP for 2nd best
General (intractable) exact formulation
Tractable formulation for tree graphs
Approximations for non-tree graphs
Experiments
![Page 86: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/86.jpg)
Adding inequalities to z z
M(G)
![Page 87: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/87.jpg)
Adding inequalities to Any valid inequality must separate from the other vertices
z z
M(G)
![Page 88: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/88.jpg)
Adding inequalities to Any valid inequality must separate from the other vertices
How about: (Santos 91)!
i
µi(zi) ! n" 1
z z
M(G)
![Page 89: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/89.jpg)
Adding inequalities to Any valid inequality must separate from the other vertices
How about: (Santos 91)
RHS is n for z and or less for other vertices
!
i
µi(zi) ! n" 1
z z
n! 1
M(G)
![Page 90: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/90.jpg)
Adding inequalities to Any valid inequality must separate from the other vertices
How about: (Santos 91)
RHS is n for z and or less for other vertices
But: Results in fractional vertices, even for trees
!
i
µi(zi) ! n" 1
z z
n! 1
M(G)
![Page 91: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/91.jpg)
Adding inequalities to Any valid inequality must separate from the other vertices
How about: (Santos 91)
RHS is n for z and or less for other vertices
But: Results in fractional vertices, even for trees
!
i
µi(zi) ! n" 1
z z
n! 1
M(G)
![Page 92: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/92.jpg)
Adding inequalities to Any valid inequality must separate from the other vertices
How about: (Santos 91)
RHS is n for z and or less for other vertices
But: Results in fractional vertices, even for trees
Only an outer bound on
!
i
µi(zi) ! n" 1
z z
n! 1
M(G)
M(G, z)
![Page 93: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/93.jpg)
The tree case
![Page 94: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/94.jpg)
The tree caseFocus on the case where G is a tree
![Page 95: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/95.jpg)
The tree caseFocus on the case where G is a tree
is given by pairwise consistencyM(G)
![Page 96: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/96.jpg)
The tree caseFocus on the case where G is a tree
is given by pairwise consistency
Define:
I(µ,z) =!
i
(1! di)µi(zi) +!
ij!G
µij(zi, zj)
M(G)
![Page 97: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/97.jpg)
The tree caseFocus on the case where G is a tree
is given by pairwise consistency
Define:
I(µ,z) =!
i
(1! di)µi(zi) +!
ij!G
µij(zi, zj)
M(G)
H(µ) =!
i
(1! di)Hi(Xi) +!
ij!G
H(Xi, Xj)Bethe:
![Page 98: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/98.jpg)
The tree caseFocus on the case where G is a tree
is given by pairwise consistency
Define:
I(µ,z) =!
i
(1! di)µi(zi) +!
ij!G
µij(zi, zj)
M(G)
![Page 99: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/99.jpg)
The tree caseFocus on the case where G is a tree
is given by pairwise consistency
Define:
I(µ,z) =!
i
(1! di)µi(zi) +!
ij!G
µij(zi, zj)
M(G)
Theorem:
M(G, z) =!µ | µ !M(G), I(µ,z) " 0
"
![Page 100: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/100.jpg)
The tree caseFocus on the case where G is a tree
is given by pairwise consistency
Define:
z
I(µ,z) =!
i
(1! di)µi(zi) +!
ij!G
µij(zi, zj)
M(G)
Theorem:
M(G, z) =!µ | µ !M(G), I(µ,z) " 0
"M(G)
![Page 101: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/101.jpg)
The tree caseFocus on the case where G is a tree
is given by pairwise consistency
Define:
z
I(µ,z) =!
i
(1! di)µi(zi) +!
ij!G
µij(zi, zj)
M(G)
Theorem:
M(G, z) =!µ | µ !M(G), I(µ,z) " 0
"M(G)
I(µ,z) ! 0
![Page 102: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/102.jpg)
The tree caseFocus on the case where G is a tree
is given by pairwise consistency
Define:
z
I(µ,z) =!
i
(1! di)µi(zi) +!
ij!G
µij(zi, zj)
M(G)
Theorem:
M(G, z) =!µ | µ !M(G), I(µ,z) " 0
"M(G)
I(µ,z) ! 0
![Page 103: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/103.jpg)
The tree caseFocus on the case where G is a tree
is given by pairwise consistency
Define:
z
I(µ,z) =!
i
(1! di)µi(zi) +!
ij!G
µij(zi, zj)
M(G)
Theorem:
M(G, z) =!µ | µ !M(G), I(µ,z) " 0
"
I(µ,z) ! 0
M(G, z)
![Page 104: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/104.jpg)
The tree caseFocus on the case where G is a tree
is given by pairwise consistency
Define:
z
I(µ,z) =!
i
(1! di)µi(zi) +!
ij!G
µij(zi, zj)
M(G)
Theorem:
M(G, z) =!µ | µ !M(G), I(µ,z) " 0
"
I(µ,z) ! 0
M(G, z)Proof...
![Page 105: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/105.jpg)
ProofA(G, z) =
!µ | µ !M(G), I(µ,z) " 0
"Define:
![Page 106: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/106.jpg)
ProofA(G, z) =
!µ | µ !M(G), I(µ,z) " 0
"Define:
A(G, z) =M(G, z)Want to show:
![Page 107: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/107.jpg)
Proof
Want to show that if there exists a p(x) that has these marginals and p(z)=0.
µ ! A(G, z)
A(G, z) =!µ | µ !M(G), I(µ,z) " 0
"Define:
![Page 108: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/108.jpg)
Proof
Want to show that if there exists a p(x) that has these marginals and p(z)=0.
µ ! A(G, z)
A(G, z) =!µ | µ !M(G), I(µ,z) " 0
"Define:
Can construct p(x)
![Page 109: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/109.jpg)
Proof
Want to show that if there exists a p(x) that has these marginals and p(z)=0.
µ ! A(G, z)
A(G, z) =!µ | µ !M(G), I(µ,z) " 0
"Define:
![Page 110: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/110.jpg)
Proof
Want to show that if there exists a p(x) that has these marginals and p(z)=0.
µ ! A(G, z)
F (µ) =
!""#
""$
min p(z)s.t. pij(xi, xj) = µij(xi, xj)
pi(xi) = µi(xi)p(x) ! 0
A(G, z) =!µ | µ !M(G), I(µ,z) " 0
"Define:
![Page 111: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/111.jpg)
Proof
Want to show that if there exists a p(x) that has these marginals and p(z)=0.
µ ! A(G, z)
F (µ) =
!""#
""$
min p(z)s.t. pij(xi, xj) = µij(xi, xj)
pi(xi) = µi(xi)p(x) ! 0
A(G, z) =!µ | µ !M(G), I(µ,z) " 0
"
= 0!µ " A(G, z)
Define:
![Page 112: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/112.jpg)
Proof
Want to show that if there exists a p(x) that has these marginals and p(z)=0.
µ ! A(G, z)
F (µ) =
!""#
""$
min p(z)s.t. pij(xi, xj) = µij(xi, xj)
pi(xi) = µi(xi)p(x) ! 0
In fact we can show that for trees:
µ !M(G) F (µ) = max{0, I(µ,z)}
A(G, z) =!µ | µ !M(G), I(µ,z) " 0
"
= 0!µ " A(G, z)
Define:
![Page 113: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/113.jpg)
Proof - key ideas
F (µ) =
!""#
""$
min p(z)s.t. pij(xi, xj) = µij(xi, xj)
pi(xi) = µi(xi)p(x) ! 0
![Page 114: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/114.jpg)
Proof - key ideas
F (µ) =
!""#
""$
min p(z)s.t. pij(xi, xj) = µij(xi, xj)
pi(xi) = µi(xi)p(x) ! 0 !x "= z
![Page 115: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/115.jpg)
Proof - key ideas
F (µ) =
!""#
""$
min p(z)s.t. pij(xi, xj) = µij(xi, xj)
pi(xi) = µi(xi)p(x) ! 0 !x "= z
,
![Page 116: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/116.jpg)
Proof - key ideas
F (µ) =
!""#
""$
min p(z)s.t. pij(xi, xj) = µij(xi, xj)
pi(xi) = µi(xi)p(x) ! 0 !x "= z
,
Dual: max ! · µs.t.
!ij !ij(xi, xj) +
!i !i(xi) ! 0 "x #= z!
ij !ij(zi,zj) +!
i !i(zi) = 1
![Page 117: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/117.jpg)
Proof - key ideas
F (µ) =
!""#
""$
min p(z)s.t. pij(xi, xj) = µij(xi, xj)
pi(xi) = µi(xi)p(x) ! 0 !x "= z
We show that the value of the above is
,
I(µ,z)
Dual: max ! · µs.t.
!ij !ij(xi, xj) +
!i !i(xi) ! 0 "x #= z!
ij !ij(zi,zj) +!
i !i(zi) = 1
![Page 118: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/118.jpg)
Proof - key ideas
F (µ) =
!""#
""$
min p(z)s.t. pij(xi, xj) = µij(xi, xj)
pi(xi) = µi(xi)p(x) ! 0 !x "= z
We show that the value of the above is
From there it’s easy to conclude that
,
I(µ,z)
Dual: max ! · µs.t.
!ij !ij(xi, xj) +
!i !i(xi) ! 0 "x #= z!
ij !ij(zi,zj) +!
i !i(zi) = 1
![Page 119: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/119.jpg)
Proof - key ideas
F (µ) =
!""#
""$
min p(z)s.t. pij(xi, xj) = µij(xi, xj)
pi(xi) = µi(xi)p(x) ! 0 !x "= z
We show that the value of the above is
From there it’s easy to conclude that
F (µ) = max{0, I(µ,z)}
,
I(µ,z)
Dual: max ! · µs.t.
!ij !ij(xi, xj) +
!i !i(xi) ! 0 "x #= z!
ij !ij(zi,zj) +!
i !i(zi) = 1
![Page 120: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/120.jpg)
Proof - Max marginalsmax ! · µs.t. !(x) ! 0 "x #= z
!(z) = 1!(x) =
!
ij
!ij(xi, xj) +!
i
!i(xi)
![Page 121: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/121.jpg)
Proof - Max marginals
Use max-marginals:
max ! · µs.t. !(x) ! 0 "x #= z
!(z) = 1!(x) =
!
ij
!ij(xi, xj) +!
i
!i(xi)
![Page 122: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/122.jpg)
Proof - Max marginals
Use max-marginals:
!̄(xi) = maxx̂:x̂i=xi
!(x)
!̄(xi.xj) = maxx̂:x̂i=xi,x̂j=xj
!(x)
max ! · µs.t. !(x) ! 0 "x #= z
!(z) = 1!(x) =
!
ij
!ij(xi, xj) +!
i
!i(xi)
![Page 123: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/123.jpg)
Proof - Max marginals
Use max-marginals:
!̄(xi) = maxx̂:x̂i=xi
!(x)
!̄(xi.xj) = maxx̂:x̂i=xi,x̂j=xj
!(x)!̄(zi) = 1!̄(xi) ! 0 xi "= zi
max ! · µs.t. !(x) ! 0 "x #= z
!(z) = 1!(x) =
!
ij
!ij(xi, xj) +!
i
!i(xi)
![Page 124: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/124.jpg)
Proof - Max marginals
Use max-marginals:
!̄(xi) = maxx̂:x̂i=xi
!(x)
!̄(xi.xj) = maxx̂:x̂i=xi,x̂j=xj
!(x)!̄(zi) = 1!̄(xi) ! 0 xi "= zi
max ! · µs.t. !(x) ! 0 "x #= z
!(z) = 1!(x) =
!
ij
!ij(xi, xj) +!
i
!i(xi)
Rewrite: !(x) =!
i
(1! di)!̄(xi) +!
ij!T
!̄ij(xi, xj)
![Page 125: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/125.jpg)
Proof - Max marginals
Use max-marginals:
!̄(xi) = maxx̂:x̂i=xi
!(x)
!̄(xi.xj) = maxx̂:x̂i=xi,x̂j=xj
!(x)!̄(zi) = 1!̄(xi) ! 0 xi "= zi
Result follows after some algebra
max ! · µs.t. !(x) ! 0 "x #= z
!(z) = 1!(x) =
!
ij
!ij(xi, xj) +!
i
!i(xi)
Rewrite: !(x) =!
i
(1! di)!̄(xi) +!
ij!T
!̄ij(xi, xj)
![Page 126: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/126.jpg)
Tree Graph - Summary
x(1)
M(G, x(1)) =!µ | µ !M(G), I(µ,x(1)) " 0
"
![Page 127: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/127.jpg)
Tree Graph - Summary
The LP for 2nd best differs from the marginal polytope by one linear inequality constraint
x(1)
M(G, x(1)) =!µ | µ !M(G), I(µ,x(1)) " 0
"
![Page 128: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/128.jpg)
Tree Graph - Summary
The LP for 2nd best differs from the marginal polytope by one linear inequality constraint
The 2nd best satisfies so it cannot be any assignment
x(1)
M(G, x(1)) =!µ | µ !M(G), I(µ,x(1)) " 0
"
I(µ,x(1)) = 0
![Page 129: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/129.jpg)
Tree Graph - Summary
The LP for 2nd best differs from the marginal polytope by one linear inequality constraint
The 2nd best satisfies so it cannot be any assignment
x(1)
x(2)
M(G, x(1)) =!µ | µ !M(G), I(µ,x(1)) " 0
"
I(µ,x(1)) = 0
![Page 130: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/130.jpg)
Tree Graph - Summary
The LP for 2nd best differs from the marginal polytope by one linear inequality constraint
The 2nd best satisfies so it cannot be any assignment
x(1)
x(2)
x(2)
M(G, x(1)) =!µ | µ !M(G), I(µ,x(1)) " 0
"
I(µ,x(1)) = 0
![Page 131: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/131.jpg)
Tree Graph - Summary
The LP for 2nd best differs from the marginal polytope by one linear inequality constraint
The 2nd best satisfies so it cannot be any assignment
x(1)
x(2)
x(2)
x(2)
M(G, x(1)) =!µ | µ !M(G), I(µ,x(1)) " 0
"
I(µ,x(1)) = 0
![Page 132: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/132.jpg)
Tree Graph - Summary
The LP for 2nd best differs from the marginal polytope by one linear inequality constraint
The 2nd best satisfies so it cannot be any assignment
x(1)
x(2)
x(2)
x(2)X
M(G, x(1)) =!µ | µ !M(G), I(µ,x(1)) " 0
"
I(µ,x(1)) = 0
![Page 133: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/133.jpg)
Tree Graph - Summary
The LP for 2nd best differs from the marginal polytope by one linear inequality constraint
The 2nd best satisfies so it cannot be any assignment
x(1)
x(2)
x(2)
M(G, x(1)) =!µ | µ !M(G), I(µ,x(1)) " 0
"
I(µ,x(1)) = 0
![Page 134: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/134.jpg)
Non tree graphsAny graph can be converted into a junction tree
We can apply our tree result there
For a junction tree with cliques C and separators S, the inequality is:
!
S!S(1! dS)µS(zS) +
!
C!CµC(zC) " 0
Specifying the marginal polytope requires a number of variables exponential in the tree width. Not practical.
![Page 135: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/135.jpg)
OutlineLP formulation of the MAP problem
LP for 2nd best
General (intractable) exact formulation
Tractable formulation for tree graphs
Approximations for non-tree graphs
Experiments
![Page 136: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/136.jpg)
Non trees - Approximations
x(1)
TrueM(G, x(1))
![Page 137: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/137.jpg)
Non trees - Approximations
x(1)
TrueM(G, x(1))
![Page 138: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/138.jpg)
Non trees - Approximations
x(1)
TrueM(G, x(1))
Outer bound on M(G)
![Page 139: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/139.jpg)
Non trees - Approximations
x(1)
TrueM(G, x(1))
Outer bound on M(G)
![Page 140: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/140.jpg)
Non trees - Approximations
x(1)
TrueM(G, x(1))
Outer bound on M(G)
![Page 141: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/141.jpg)
Non trees - Approximations
x(1)
TrueM(G, x(1))
Outer bound on M(G)
![Page 142: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/142.jpg)
Spanning tree inequalities
Give a spanning subtree T of G defineIT (µ,z) =
!
i
(1! di)µi(zi) +!
ij!T
µij(zi, zj)
IT (µ,z) ! 0And the constraint:
![Page 143: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/143.jpg)
Spanning tree inequalities
Give a spanning subtree T of G defineIT (µ,z) =
!
i
(1! di)µi(zi) +!
ij!T
µij(zi, zj)
IT (µ,z) ! 0And the constraint:
![Page 144: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/144.jpg)
Spanning tree inequalities
Give a spanning subtree T of G defineIT (µ,z) =
!
i
(1! di)µi(zi) +!
ij!T
µij(zi, zj)
IT (µ,z) ! 0And the constraint:
![Page 145: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/145.jpg)
Spanning tree inequalities
Give a spanning subtree T of G defineIT (µ,z) =
!
i
(1! di)µi(zi) +!
ij!T
µij(zi, zj)
IT (µ,z) ! 0And the constraint:
![Page 146: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/146.jpg)
Spanning tree inequalities
Give a spanning subtree T of G defineIT (µ,z) =
!
i
(1! di)µi(zi) +!
ij!T
µij(zi, zj)
Separates z from the other vertices but might result in fractional vertices
IT (µ,z) ! 0And the constraint:
![Page 147: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/147.jpg)
Spanning tree inequalities
Give a spanning subtree T of G defineIT (µ,z) =
!
i
(1! di)µi(zi) +!
ij!T
µij(zi, zj)
Separates z from the other vertices but might result in fractional vertices
z
IT (µ,z) ! 0And the constraint:
![Page 148: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/148.jpg)
Spanning tree inequalities
Give a spanning subtree T of G defineIT (µ,z) =
!
i
(1! di)µi(zi) +!
ij!T
µij(zi, zj)
Separates z from the other vertices but might result in fractional vertices
z
IT (µ,z) ! 0And the constraint:
IT (µ,z) ! 0
![Page 149: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/149.jpg)
Spanning tree inequalities
Give a spanning subtree T of G defineIT (µ,z) =
!
i
(1! di)µi(zi) +!
ij!T
µij(zi, zj)
Separates z from the other vertices but might result in fractional vertices
zFractional vertex
IT (µ,z) ! 0And the constraint:
IT (µ,z) ! 0
![Page 150: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/150.jpg)
Adding all spanning trees
![Page 151: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/151.jpg)
Adding all spanning trees
Can we add all spanning tree inequalities efficiently?
![Page 152: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/152.jpg)
Adding all spanning trees
Can we add all spanning tree inequalities efficiently?
Yes, via a cutting plane approach:
![Page 153: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/153.jpg)
Adding all spanning trees
Can we add all spanning tree inequalities efficiently?
Yes, via a cutting plane approach:
Start with one inequality
![Page 154: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/154.jpg)
Adding all spanning trees
Can we add all spanning tree inequalities efficiently?
Yes, via a cutting plane approach:
Start with one inequality
Solve LP
![Page 155: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/155.jpg)
Adding all spanning trees
Can we add all spanning tree inequalities efficiently?
Yes, via a cutting plane approach:
Start with one inequality
Solve LP
If solution is fractional, find a violated tree inequality (if exists) and add it
![Page 156: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/156.jpg)
Cutting Plane Algorithm
![Page 157: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/157.jpg)
Cutting Plane Algorithm
z
![Page 158: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/158.jpg)
Cutting Plane Algorithm
zT1
![Page 159: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/159.jpg)
Cutting Plane Algorithm
zµ1
T1
![Page 160: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/160.jpg)
Cutting Plane Algorithm
zµ1 Is there a tree
inequality thatviolates?
µ1
T1
![Page 161: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/161.jpg)
Cutting Plane Algorithm
zµ1 Is there a tree
inequality thatviolates?
µ1
T1
T2
![Page 162: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/162.jpg)
Cutting Plane Algorithm
zµ1 Is there a tree
inequality thatviolates?
µ1
T1
T2
![Page 163: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/163.jpg)
Cutting Plane Algorithm
How do we find a violated tree inequality?
Note: Even all spanning tree inequalities might not suffice
zµ1 Is there a tree
inequality thatviolates?
µ1
T1
T2
![Page 164: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/164.jpg)
Finding a violated spanning tree
For a given find
If it’s positive, add the maximizing tree
µ maxT
IT (µ,z)
![Page 165: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/165.jpg)
Finding a violated spanning tree
For a given find
If it’s positive, add the maximizing tree
µ maxT
IT (µ,z)
How can we maximize over all trees? Note that:
![Page 166: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/166.jpg)
Finding a violated spanning tree
For a given find
If it’s positive, add the maximizing tree
µ maxT
IT (µ,z)
How can we maximize over all trees? Note that:
IT (µ,z) =!
ij!T
"µij(zi, zj)! µi(zi)! µj(zj)
#+
!
i
µi(zi)
![Page 167: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/167.jpg)
Finding a violated spanning tree
For a given find
If it’s positive, add the maximizing tree
µ maxT
IT (µ,z)
How can we maximize over all trees? Note that:
IT (µ,z) =!
ij!T
"µij(zi, zj)! µi(zi)! µj(zj)
#+
!
i
µi(zi)
![Page 168: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/168.jpg)
Finding a violated spanning tree
For a given find
If it’s positive, add the maximizing tree
µ maxT
IT (µ,z)
How can we maximize over all trees? Note that:
IT (µ,z) =!
ij!T
"µij(zi, zj)! µi(zi)! µj(zj)
#+
!
i
µi(zi)
wij
![Page 169: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/169.jpg)
Finding a violated spanning tree
For a given find
If it’s positive, add the maximizing tree
µ maxT
IT (µ,z)
How can we maximize over all trees? Note that:
IT (µ,z) =!
ij!T
"µij(zi, zj)! µi(zi)! µj(zj)
#+
!
i
µi(zi)
wij Fixed
![Page 170: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/170.jpg)
Finding a violated spanning tree
For a given find
If it’s positive, add the maximizing tree
µ maxT
IT (µ,z)
How can we maximize over all trees? Note that:
IT (µ,z) =!
ij!T
"µij(zi, zj)! µi(zi)! µj(zj)
#+
!
i
µi(zi)
Decomposes into edge scores. Maximizing tree can be found using a maximum-weight-spanning-tree algorithm (e.g., Wainwright 02)
wij Fixed
![Page 171: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/171.jpg)
ExperimentsAlternative algorithms for approximate 2nd best:
Using approximate marginals from max-product (BMMF; Yanover and Weiss 04)
Lawler/Nillson (72,80) - Partition assignments :
Maximize over each part approximately. Cost O(n)
Our algorithm: STRIPES
x != x(1)
x1 != x(1)1 x2 = " x3 = " . . . xn = "
x1 = x(1)1 x2 != x(1)
2 x3 = " . . . xn = "...
......
......
x1 = x(1)1 x2 = x(1)
2 x3 = x(3)1 . . . xn != x(n)
1
![Page 172: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/172.jpg)
Attractive GridsIsing models with ferromagnetic interaction
The local-polytope guaranteed to yield exact first best (but not equal to the marginal polytope)
Goal: Find 50 best. Stripes and Nillson find all of them exactly. Up to 19 spanning trees added
S N B0
0.5
1
S N B0
50
Stripes Nillson BMMF
0
50
0Stripes Nillson BMMF
Rank Run Time
![Page 173: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/173.jpg)
Protein Side Chain Prediction
Given protein’s 3D shape (backbone), choose most probable side chain configuration
xi!
xk!
xj !
xh!
G=(V,E)!
Protein backbone!
Side-chains!
(MRFs from Yanover, Meltzer, Weiss ‘06)!
Can be cast as a MAP problem
Important to obtain multiple possible solutions
p(x) ! eP
ij!E !ij(xi,xj)
![Page 174: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/174.jpg)
Protein Side Chain Prediction
Stripes found the exact solutions for all problems studied
In some cases, we used a tighter approximation of the marginal polytope (Sontag et al, UAI 08)
S N B0
50
S N B0
0.5
1
Stripes Nillson BMMF0
50
0Stripes Nillson BMMF
![Page 175: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/175.jpg)
Open Questions
![Page 176: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/176.jpg)
Open QuestionsWhen are spanning trees enough?
![Page 177: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/177.jpg)
Open QuestionsWhen are spanning trees enough?
What is the polytope structure for k-best?
![Page 178: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/178.jpg)
Open QuestionsWhen are spanning trees enough?
What is the polytope structure for k-best?
Finding k-best “different” solutions
![Page 179: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/179.jpg)
Open QuestionsWhen are spanning trees enough?
What is the polytope structure for k-best?
Finding k-best “different” solutions
Scalable algorithms
![Page 180: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/180.jpg)
Open QuestionsWhen are spanning trees enough?
What is the polytope structure for k-best?
Finding k-best “different” solutions
Scalable algorithms
If a given problem is solved with a marginal polytope relaxation, what can we say about the second best?
![Page 181: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/181.jpg)
Open QuestionsWhen are spanning trees enough?
What is the polytope structure for k-best?
Finding k-best “different” solutions
Scalable algorithms
If a given problem is solved with a marginal polytope relaxation, what can we say about the second best?
![Page 182: Finding k-best MAP Solutions Using LP Relaxationscnls.lanl.gov/~jasonj/poa/slides/globerson.pdf · 2014-09-24 · Finding k-best MAP Solutions Using LP Relaxations Amir Globerson](https://reader033.vdocuments.us/reader033/viewer/2022042006/5e6fc92db4f3047ff9410d74/html5/thumbnails/182.jpg)
SummaryThe 2nd best can be posed as a linear program
For trees differs from 1st best by one constraint only
For non-trees, approximation can be devised by adding inequalities for all spanning trees
Empirically effective