![Page 1: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/1.jpg)
Learning to Combine Bottom-Up and Top-Down Segmentation
Anat Levin and Yair Weiss
School of CS&Eng,
The Hebrew University of Jerusalem, Israel
![Page 2: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/2.jpg)
Bottom-up segmentation
• Malik et al, 2000 • Sharon et al, 2001•Comaniciu and Meer, 2002•…
Bottom-up approaches: Use low level cues to group similar pixels
![Page 3: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/3.jpg)
Bottom-up segmentation is ill posed
Some segmentation example (maybe horses from Eran’s paper)
Many possible segmentation are equally good based on low level cues alone.
images from Borenstein and Ullman 02
![Page 4: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/4.jpg)
Top-down segmentation •Class-specific, top-down segmentation (Borenstein & Ullman Eccv02)
•Winn and Jojic 05
•Leibe et al 04
•Yuille and Hallinan 02.
•Liu and Sclaroff 01
•Yu and Shi 03
![Page 5: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/5.jpg)
Combining top-down and bottom-up segmentation
Find a segmentation:
1. Similar to the top-down model
2. Aligns with image edges
+
![Page 6: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/6.jpg)
Previous approaches
• Borenstein et al 04 Combining top-down and bottom up segmentation.
• Tu et al ICCV03 Image parsing: segmentation, detection, and recognition.
• Kumar et al CVPR05 Obj-Cut.
•Shotton et al ECCV06: TextonBoost
Previous approaches: Train top-down and bottom-up models independentlyindependently
![Page 7: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/7.jpg)
Why learning top-down and bottom-up models simultaneously?
•Large number of freedom degrees in tentacles configuration- requires a complex deformable top down model
•On the other hand: rather uniform colors- low level segmentation is easy
![Page 8: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/8.jpg)
•Learn top-down and bottom-up models simultaneouslysimultaneously
•Reduces at run time to energy minimization with binary labels (graph min cut)
Our approach
![Page 9: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/9.jpg)
Energy model
k
IFkij
kxxjxixjiwIxE ,)()(),();(
Consistency with fragments segmentation
Segmentation alignment with image edges
![Page 10: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/10.jpg)
Energy model
k
IFkij
kxxjxixjiwIxE ,)()(),();(
Segmentation alignment with image edges
Consistency with fragments segmentation
![Page 11: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/11.jpg)
Energy model
k
IFkij
kxxjxixjiwIxE ,)()(),();(
Segmentation alignment with image edges
Resulting min-cut segmentation
Consistency with fragments segmentation
![Page 12: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/12.jpg)
Learning from segmented class images
Training data:Ttt
Ttt xI 11 }{ }{
Goal: Learn fragments for an energy function
![Page 13: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/13.jpg)
Learning energy functions using conditional random fields
t
IxE
tttt
tteIZ
IxP ),;(
);(
1);|(
Theory of CRFs:
•Lafferty et al 2001
•LeCun and Huang 2005
x
IxEt
teIZ ),;();(
CRFs For vision:
•Kumar and Hebert 2003
•Ren et al 2006
•He et al 2004, 2006
•Quattoni et al 2005
•Torralba et al 04
![Page 14: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/14.jpg)
tx
E(x)
tx
E(x)
Minimize energy of true segmentation
Maximize energy of all other configurations
t
IxE
tttt
tteIZ
IxP ),;(
);(
1);|(
Learning energy functions using conditional random fields
“It's not enough to succeed. Others must fail.” –Gore Vidal
![Page 15: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/15.jpg)
Minimize energy of true segmentation
Maximize energy of all other configurations
tx
P(x)
tx
P(x)
t
IxE
tttt
tteIZ
IxP ),;(
);(
1);|(
Learning energy functions using conditional random fields
“It's not enough to succeed. Others must fail.” –Gore Vidal
![Page 16: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/16.jpg)
Differentiating CRFs log-likelihood
Log-likelihood gradients with respect to :
Expected feature response minus observed feature response
ObsIFt
CurrentIFt tktk
xxxx ,
,
k
IFkij
kxxjxixjiwFIxE ,)()(),(),,,;(
Log-likelihood is convex with respect to
Yair- in the original version of this slide I had another equation expressing the expectation as a sum of marginals (see next hidden slide). At least for me, it wasn’t originally clear what this expectation means before I saw the other equation. However, I try to delete un necessary equations..
![Page 17: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/17.jpg)
CRFs cost- evaluating partition function
Derivatives- evaluating marginal probabilities
Use approximate estimations:
•Sampling
•Belief Propagation and Bethe free energy
•Used in this work: Tree reweighted belief propagation and Tree reweighted upper bound (Wainwright et al 03)
Conditional random fields-computational challenges
);(log
tIZ
);|(
ti IrxP
![Page 18: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/18.jpg)
Fragments selection
Candidate fragments pool:
Greedy energy design:
ij
jxixjiwIxE )()(),();(
IFxx ,1 1 IFxx ,2 2
IFxx ,3 3
![Page 19: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/19.jpg)
Fragments selection challenges
Straightforward computation of likelihood improvement is impractical
2000 Fragments
50 Training images
10 Fragments selection iterations
1,000,000 inference operations!
![Page 20: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/20.jpg)
Fragments selection
Fragment with low error on
the training set
First order approximation to log-likelihood gain:
ModelCurrentIFt
ObsIFt tt
xxxx
,,
Fragment not accounted for by the
existing model
Similar idea in different contexts:
•Zhu et al 1997
•Lafferty et al 2004
•McCallum 2003
![Page 21: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/21.jpg)
•Requires a single inference process on the previous iteration energy to evaluate approximations with respect to all fragments
•First order approximation evaluation is linear in the fragment size
First order approximation to log-likelihood gain:
ModelCurrentIFt
ObsIFt tt
xxxx
,,
Fragments selection
![Page 22: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/22.jpg)
Fragments selection- summary
Initialization: Low- level term
For k=1:K
•Run TRBP inference using the previous iteration energy.
•Approximate likelihood gain of candidate fragments
•Add to energy the fragment with maximal gain.
ModelCurrentIFt
ObsIFt tt
xxxx
,,
![Page 23: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/23.jpg)
Training horses model
![Page 24: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/24.jpg)
Training horses model-one fragment
![Page 25: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/25.jpg)
Training horses model-two fragments
![Page 26: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/26.jpg)
Training horses model-three fragments
![Page 27: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/27.jpg)
Results- horses dataset
![Page 28: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/28.jpg)
Results- horses dataset
Fragments number
Mis
lab
eled
pix
els
per
cen
t
Comparable to previous results (Kumar et al, Borenstein et al.) but with far fewer fragments
![Page 29: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/29.jpg)
Results- artificial octopi
![Page 30: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/30.jpg)
Results- cows datasetFrom the TU Darmstadt Database
![Page 31: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/31.jpg)
Results- cows dataset
Fragments number
Mis
lab
eled
pix
els
per
cen
t
![Page 32: Learning to Combine Bottom-Up and Top-Down Segmentation](https://reader036.vdocuments.us/reader036/viewer/2022062409/56814eca550346895dbc68c9/html5/thumbnails/32.jpg)
Conclusions
•Simultaneously learning top-down and bottom-up segmentation cues.
•Learning formulated as estimation in Conditional Random Fields
•Novel, efficient fragments selection algorithm
•Algorithm achieves state of the art performance with a significantly smaller number of fragments