on multiple foreground cosegmentation
DESCRIPTION
On Multiple Foreground Cosegmentation. Gunhee Kim Eric P. Xing. School of Computer Science, Carnegie Mellon University. June 18, 2012. Outline. Problem Statement Algorithm Overview Foreground Modeling Region Assignment Experiments Conclusion. - PowerPoint PPT PresentationTRANSCRIPT
1
On Multiple Foreground Cosegmentation
Gunhee Kim Eric P. Xing
School of Computer Science, Carnegie Mellon University
June 18, 2012
2
• Problem Statement • Algorithm
Overview Foreground Modeling Region Assignment
• Experiments• Conclusion
Outline
3
Image Cosegmentation
High-level signal: recurring objects in multiple images
Jointly segment multiple images into K foregrounds and background
• [R06] Rother et al. CVPR2006 • [H09] Hochbaum and Singh, ICCV2009• [J10,J12] Joulin et al, CVPR2010, CVPR2012• [B11] Batra et al, IJCV 2011• [M11] Mukherjee et al, CVPR 2011• [V10,V11] Vincente et al, ECCV 2010, CVPR2011• [K11] Kim et al, ICCV 2011.
[R06] [J10]
[B11]
[K11]
4
Popular Cosegmentation Datasets
CMU-Cornell iCoseg Dataset [Batra et al. IJCV11]
MSRC Dataset [Winn et al. ICCV05]
Synthesized Dataset [Rhemann et al. CVPR09]
5
Popular Cosegmentation Datasets
CMU-Cornell iCoseg Dataset [Batra et al. IJCV11]
MSRC Dataset [Winn et al. ICCV05]
Synthesized Dataset [Rhemann et al. CVPR09]
Input images are carefully prepared by humanso that the objects of interest are salient enough in every single image.
6
General Users’ Photo Sets
A part of Apple+picking photo stream from Flickr
• A series of photos are taken for a specific moment• The number of foregrounds (ie. subjects of interest) are finite
Follow an ordinary user’s photo-taking pattern.
Girl in red (R)Girl in blue (G)Baby (B)Apple bucket (A)(R, G, A) (B, A) (R, A) (G, A)
(R, G, B, A) (R, G, A) (R, G, B, A) (R, A)
• Each image contains an unknown subset of foregrounds
7
General Users’ Photo Sets
A part of Apple+picking photo stream from Flickr
• A series of photos are taken for a specific moment• The number of foregrounds (ie. subjects of interest) are finite
Follows an ordinary users’ photo-taking pattern.
Girl in red (R)Girl in blue (G)Baby (B)Apple bucket (A)(R, G, A) (B, A) (R, A) (G, A)
(R, G, B, A) (R, G, A) (R, G, B, A) (R, A)
• Each image contains an unknown subset of foregrounds
Has NOT yet explicitly addressed
8
Problem StatementMultiple Foreground Cosegmentation
• Optionally, a user may assign some examples of foregrounds.
• Each image contains an unknown subset of foregrounds
Given an image set I, K foregrounds of interestSegment each image into regions of {F1,…,FK+1} (FK+1=B)
Girl in red (R)Girl in blue (G)Baby (B)Apple bucket (A)(R, G, A) (B, A) (R, A) (G, A)
(R, G, B, A) (R, G, A) (R, G, B, A) (R, A)
9
• Problem Statement • Algorithm
Overview Foreground Modeling Region Assignment
• Experiments• Conclusion
Outline
10
Overview of Algorithm
Iteratively solve
ForegroundModeling
RegionAssignment
• Learn appearance models of K+1 foregrounds (FGs)
• Performed in each image separately• Allocate the regions of image into one of K+1 FGs
Initialization
SupervisedAssigned by a user
UnsupervisedApply diversity ranking of [Kim et al. ICCV11]
Iterate until convergence
[Kim&Toralba NIPS09][Rother et al. SIGGRAPH04]
✔
11
• Problem Statement • Algorithm
Overview Foreground Modeling Region Assignment
• Experiments• Conclusion
Outline
12
Foreground (FG) Model
Definition of k-th FG model
Any region classifiers or their combination
Value Assignment (Testing) Foreground learning (Training)
Baby FG model
Baby FG model
A parametric function
Given any region S, return its value (score) of FG k
13
Foreground (FG) Model
Definition of k-th FG model
A parametric function
Given any region S, return its value (score) to FG k
Any region classifiers or their combination
Gaussian Mixture Model (GMM)
Spatial Pyramid + linear SVM (SPM)
• One of most popular in cosegmentation literatures
• One of most popular in image classification
• RGB colors • Gray/HSV SIFT
14
• Problem Statement • Proposed Algorithm
Foreground Modeling Region Assignment
• Experiments• Conclusion
Outline
15
Region Assignment
Given learned (or initialized) FG Models, RA is individually performed in each image
Cow FG Person FG
Background
Oversegment
16
Region Assignment
Given learned (or initialized) FG Models, RA is individually performed in each image
Cow FG Person FG
Background
Region Assignment
17
A Naïve Region Assignment
A naïve way: Assign each segment to the FG whose value is the highest
Cow FG Person FG
BackgroundCow BG BG BG Cow
However, it will NOT work !
18
Why does not A Naïve RA work?
Most naïve way: Assign each segment to FG whose value is the highest
Cow FG Person FGs1
s2
Cow won!
Person won!
My value of {s1,s2} is 18.
{s1,s2}My value of s2 is 5.
My value of s1 is 10.
My value of {s1,s2} is 35.
My value of s2 is 20.
My value of s1 is 7.
19
Why does not A Naïve RA work?
Most naïve way: Assign each segment to FG whose value is the highest
Cow FG Person FGs1
s2
Cow won!
Person won!
My value of {s1,s2} is 18.
{s1,s2}My value of s2 is 5.
My value of s1 is 10.
My value of {s1,s2} is 35.
My value of s2 is 20.
My value of s1 is 7.
Value of {s1,s2} to person FG
Value of s1 to Cow FG
Value of s2 to person FG +>
Have to evaluate the combinations of segments
20
Region Assignment by Combinatorial Auction
Cow FG Person FG
BackgroundItems to sell
Bidder 1 (or buyer)
Bidder 2
Bidder 3
• Each FG is allowed to bid packages of items with their own values.
• Distribute the segments to maximize the values.
21
Region Assignment as Combinatorial Auction
Assign the segments to FGs to maximize the overall values.
Winner determination problem (ie. Welfare maximization)
Unfortunately, NP-complete and Inapproximable
Feasibility: Each segment cannot be assigned more than once
[Cramton et al. 2005].
2 |Si|
subsets
22
✔
Next Goal: Tractable Solution to WDP
There are two different ways…
1. Constraints on value functions
2. Constraints on generating bidding packages
Allow any combinations of FG models (ie. Region classifiers)
• If value functions are submodular or subadditive, …
[Felzenszwalb et al.2008]
23
Procedure of Region Assignment
1. Each FG creates a set of foreground candidates
3. Solve WDP by choosing feasible FG candidates.
Follow a general combinatorial auction scenario.
: n FG candidates by FG k
: a bundle of segments : its value: FG ID
• In this step, each FG does not care their winning chances.
2. Finally,
24
Assumption for FG Candidates
A FG instance = a sub-tree of Gi
A FG instance in an image = a set of adjacent segments.
Any FG candidate = a sub-tree of Gi
25
Generating FG Candidatesby Beam Search
• Beam width D• Computation time: O(D|Si|2) • Number of FG candidates per FG: |Bi
k| = O(D|Si|)
For each size of candidates, we keep only D high valued ones.
2. Ot all possible subgraphs by adding
a single edge to elements of O.
1. O every single segment
3. Compute values vo vk(o) for all and keep only top D high valued ones to O.
4. Iterate (|Si|-1) times.
For each foreground k,
26
Example of Candidate Set
19.74 39.42 29.30
18.22 47.0639.47
39.6119.85 19.88
18.78
16.35
59.49
Back-ground
Apple bucket
Baby
27
Solving WDP
19.74 39.42 29.30
18.22 47.0639.47
39.6119.85 19.88
18.78
16.35
59.49
Back-ground
Apple bucket
Baby Solve WDP !
The number of FG candidates is (K+1)D|Si|.
Even a faster algorithm…
By search in polynomial time
28
Solving WDP
[Theorem] Dynamic program can solve WDP in O(|Bi||Si|)worst time if every candidate in Bi can be represented by a connected subgraph of a tree Ti
*.[Sandholm et al.2003]
Each FG candidate is a tree, but the aggregation is not.
19.74 39.42 29.30
18.22 47.0639.47
39.6119.85 19.88
18.78
16.35
59.49
Back-ground
Apple bucket
Baby
Candidate 2
Candidate 1
29
Inferring the Most Probable Tree from FG Candidate Set
19.74 39.42 29.30
18.22 47.0639.47
39.6119.85 19.88
18.78
16.35
59.49
Back-ground
Apple bucket
Baby
Solve the following MLE solution
: all possible spanning trees
where
w2 = 5Candidate 2
Candidate 1w1 = 20
Reject the bids that are not a subtree of
30
Inferring the Most Probable Tree from Candidate Set
19.74 39.42 29.30
18.22 47.0639.47
39.6119.85 19.88
18.78
16.35
59.49
Back-ground
Apple bucket
Baby
Solve the following MLE solution
: all possible spanning trees
where
MLE solution = MST by Kruskal’s algorithm in O(|Bi||Si|2)
Almost identical to Chow-Liu tree structure learning
31
Finally, Solve WDP
[Theorem] Dynamic program can solve WDP in O(|Bi||Si|)worst time if every candidate in Bi can be represented by a connected subgraph of a tree Ti
*.[Sandholm et al.2003]
optimal assignment segmentation
CABOB algorithm [Sandholm et al.2005]
32
• Problem Statement • Proposed Algorithm
Foreground Modeling Region Assignment
• Experiments• Conclusion
Outline
33
Two Experiments
FlickrMFC dataset
ImageNet dataset
• Goal: To achieve multiple foreground cosegmentation
• New benchmark dataset (14 groups, 20 images, fully-labeled)
ex. Cow group: {person, car, cow (brown, black)}
• Goal: Scalability
ex. green lizard
34
Quantitative Evaluation
Segmentation accuracies
Metric MFC-S: (supervised) our methodMFC-U: (unsupervised) our methodCOS : Submodular optimization [Kim et al. ICCV11]DC: Discriminative clustering [Joulin. CVPR10]LDA: LDA-based localization [Russell et al. CVPR06]MNcut: Normalized cuts [Cour et al. CVPR05]
35
Cosegmentation Examples
FlickrMFC
ImageNet
Australian terrier Lion
36
• Problem Statement • Proposed Algorithm
Foreground Modeling Region Assignment
• Experiments• Conclusion
Outline
37
Conclusion
• Each image contains a unknown subset of foregrounds
• Web-oriented applications
• Code and FlickrMFC dataset
will be available in this month!
• Fast and distributable (ex. Linear with M,K, polynomial in |Si|)
• Can be used for other tasks (ex.detection) beyond segmentation.
Multiple foreground cosegmentation
Combinatorial auction-based region assignment
38
Take-Home Message
Combinatorial optimization for Image Segmentation!
• New region descriptors & classifiers every year
• Multiple fast machines are available.
May avoid difficult ML algorithms or high-order models
Similar line of thought: Our ICCV 2011 paperSubmodular optimizationSimply enumerate the cases
(or hypotheses) not in a brute-force but in a smart way.