learning decompositional shape models from examples
DESCRIPTION
Learning Decompositional Shape Models from Examples. Alex Levinshtein Cristian Sminchisescu Sven Dickinson University of Toronto. Hierarchical Models. Manually built hierarchical model proposed by Marr And Nishihara - PowerPoint PPT PresentationTRANSCRIPT
Learning Decompositional Learning Decompositional Shape Models from Shape Models from
ExamplesExamples
Alex LevinshteinAlex LevinshteinCristian SminchisescuCristian Sminchisescu
Sven DickinsonSven Dickinson
University of TorontoUniversity of Toronto
Hierarchical ModelsHierarchical ModelsManually built hierarchical model proposed by Marr And Nishihara(“Representation and recognition of the spatial organization of three dimensional shapes”, Proc. of Royal Soc. of London, 1978)
Our goalOur goal
Automatically construct a generic hierarchical shape model from exemplars
Challenges:
Cannot assume similar appearance among different exemplars
Generic features are highly ambiguous
Generic features may not be in one-to-one correspondence
Layered Motion SegmentationsLayered Motion Segmentations Kumar, Torr and Zisserman, ICCV 2005Kumar, Torr and Zisserman, ICCV 2005
Models image projection, lighting and motion blurModels image projection, lighting and motion blur
Models spatial continuity, occlusions, and works over multiple Models spatial continuity, occlusions, and works over multiple frames (frames (cf.cf. earlier work by earlier work by Jojic & Frey, CVPR 2001Jojic & Frey, CVPR 2001))
Estimates the number of segments, their mattes, layer Estimates the number of segments, their mattes, layer assignment, appearance, lighting and transformation assignment, appearance, lighting and transformation parameters for each segmentparameters for each segment
Initialization using loopy BP, refinement using graph cutsInitialization using loopy BP, refinement using graph cuts
Fergus, R., Perona, P., and Zisserman, A., “Object Class Recognition by Unsupervised Scale-Invariant Learning”, CVPR 2003
Constellation modelsConstellation models
Categorical featuresCategorical features
Match
Constructing a Hierarchical Constructing a Hierarchical Model from ExamplesModel from Examples
Input:
Question: What is it?
Output:
Overview of the ApproachOverview of the ApproachExemplar imagesExtract Blob
GraphsMatch Blob
Graphs (many-to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model partsModel
decomposition relations
Model attachment relations
Blob Graph ConstructionBlob Graph ConstructionExemplar imagesExtract Blob
GraphsMatch Blob
Graphs (many-to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model decomposition relations
Model attachment relations
Model parts
Blob Graph ConstructionBlob Graph Construction
Edges are invariant to articulation
Choose the largest connected component.
The Representation and Matching of Categorical ShapeA. Shokoufandeh, L. Bretzner, D. Macrini, M.F. Demirci, C. Jonsson, and S. Dickinson, CVIU, Vol. 103, 2006, pp 139--154
Blob Graph ConstructionBlob Graph Construction
Connectivity measure: max{d1/major(A), d2/major(B)}
Perceptual grouping of blobs:
Feature matchingFeature matchingExemplar imagesExtract Blob
GraphsMatch Blob
Graphs (many-to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model decomposition relations
Model attachment relations
Model parts
Feature matchingFeature matchingOne-to-one matching. Rely on shape and context, not appearance!Many-to-many
matching
A Many-to-Many Graph A Many-to-Many Graph Matching FrameworkMatching Framework
Demirci, Shokoufandeh, Keselman, Bretzner, and Dickinson (IJCV 2006)
1. Embed graphs with low distortion to yield weighted point distributions.
2. Compute many-to-many correspondences between the two distributions using EMD.
3. The computed flows yield a many-to-many node correspondence between the two graphs.
Feature embedding and Feature embedding and EMDEMD
Spectral embedding
Returning to our set of Returning to our set of inputsinputs
Many-to-many matching of every pair of exemplars.
Part ExtractionPart ExtractionExemplar imagesExtract Blob
GraphsMatch Blob
Graphs (many-to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model decomposition relations
Model attachment relations
Model parts
Many-to-many matching Many-to-many matching resultsresults100%100%
100%100%100%100%
50%50% 50%50%
Extracting partsExtracting parts
Part – a collection of blobs.Part – a collection of blobs. Ideal partIdeal part
Represents blobs that occur frequently Represents blobs that occur frequently and participate in one-to-one and participate in one-to-one correspondence across many exemplars. correspondence across many exemplars.
Finding partsFinding parts From the pairwise matching results, find From the pairwise matching results, find
clusters (cliques) of blobs matching one-clusters (cliques) of blobs matching one-to-one.to-one.
Results of the part extraction Results of the part extraction stagestage
Extracting attachment Extracting attachment relationsrelations
Exemplar imagesExtract Blob
GraphsMatch Blob
Graphs (many-to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model decomposition relations
Model attachment relations
Model parts
Extracting attachment Extracting attachment relationsrelations
Right arm is typically connected to torso in exemplar images !
TorsoRight Arm
Number of times blobs drawn from the two clusters were attachedNumber of times blobs from the two clusters co-appeared in an image.
is high
Extracting attachment Extracting attachment relationsrelations
Number of times blobs from the two clusters co-appeared in an image.
Number of times blobs drawn from the two clusters were attached
Threshold PA to get part attachment
Extracting decomposition Extracting decomposition relationsrelations
Exemplar imagesExtract Blob
GraphsMatch Blob
Graphs (many-to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model decomposition relations
Model attachment relations
Model parts
Left Arm
Upper Lower
Extracting decomposition Extracting decomposition relationsrelations
Extracting decomposition Extracting decomposition relationsrelations
Sum of all flows between blobs in two clusters
Number of flows between blobs in two clusters
Combine PA and PF to obtain a decomposition score
Model construction stage Model construction stage summarysummary
Model Construction:
Clustering blobs into parts based on one-to-one matching results.
Recovering relations between parts based on individual matching and attachment results.
Assemble Final ModelAssemble Final ModelExemplar imagesExtract Blob
GraphsMatch Blob
Graphs (many-to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model decomposition relations
Model attachment relations
Model parts
ResultsResults
Experiments
List of system parametersList of system parametersParameterParameter DescriptionDescription
Perceptual Perceptual Grouping Grouping ThresholdThreshold
The threshold determining the pairwise The threshold determining the pairwise connectivity between blobs in the exemplar connectivity between blobs in the exemplar imagesimages
DD The dimensionality of the embedding space The dimensionality of the embedding space during matchingduring matching
Embedding Embedding dimensionality dimensionality during part during part extractionextraction
The dimensionality of the embedding space The dimensionality of the embedding space used for blob clustering.used for blob clustering.
KK The maximum number of parts in the modelThe maximum number of parts in the modelTTattachattach The threshold for accepting attachment The threshold for accepting attachment
between parts in the final modelbetween parts in the final modelTTchildchild The threshold for considering a part to be a The threshold for considering a part to be a
potential child of another partpotential child of another partTTdecompdecomp The threshold for accepting a decompositionThe threshold for accepting a decomposition
ConclusionsConclusions Generic models must be defined at multiple levels of Generic models must be defined at multiple levels of
abstraction, as Marr proposed.abstraction, as Marr proposed.
Coarse shape features, such as blobs, are highly ambiguous Coarse shape features, such as blobs, are highly ambiguous and cannot be matched without contextual constraints.and cannot be matched without contextual constraints.
Moreover, features that exist at different levels of Moreover, features that exist at different levels of abstraction must be matched many-to-many in the abstraction must be matched many-to-many in the presence of noise.presence of noise.
The many-to-many matching results can be analyzed to The many-to-many matching results can be analyzed to yield both the parts and relations of a decompositional yield both the parts and relations of a decompositional model.model.
Preliminary results indicate that a limited decompositional Preliminary results indicate that a limited decompositional model can be learned from a set of noisy examples.model can be learned from a set of noisy examples.
Future workFuture work Construct models for objects other than Construct models for objects other than
humans – objects with richer decompositional humans – objects with richer decompositional hierarchies.hierarchies.
Automatically learn perceptual grouping Automatically learn perceptual grouping relations between blobs from labeled relations between blobs from labeled examples.examples.
Develop indexing and matching frameworks Develop indexing and matching frameworks for decompositional models.for decompositional models.