learning decompositional shape models from examples

Learning Decompositional Learning Decompositional Shape Models from Shape Models from

ExamplesExamples

Alex LevinshteinAlex LevinshteinCristian SminchisescuCristian Sminchisescu

Sven DickinsonSven Dickinson

University of TorontoUniversity of Toronto

Hierarchical ModelsHierarchical ModelsManually built hierarchical model proposed by Marr And Nishihara(“Representation and recognition of the spatial organization of three dimensional shapes”, Proc. of Royal Soc. of London, 1978)

Our goalOur goal

Automatically construct a generic hierarchical shape model from exemplars

Challenges:

Cannot assume similar appearance among different exemplars

Generic features are highly ambiguous

Generic features may not be in one-to-one correspondence

Layered Motion SegmentationsLayered Motion Segmentations Kumar, Torr and Zisserman, ICCV 2005Kumar, Torr and Zisserman, ICCV 2005

Models image projection, lighting and motion blurModels image projection, lighting and motion blur

Models spatial continuity, occlusions, and works over multiple Models spatial continuity, occlusions, and works over multiple frames (frames (cf.cf. earlier work by earlier work by Jojic & Frey, CVPR 2001Jojic & Frey, CVPR 2001))

Estimates the number of segments, their mattes, layer Estimates the number of segments, their mattes, layer assignment, appearance, lighting and transformation assignment, appearance, lighting and transformation parameters for each segmentparameters for each segment

Initialization using loopy BP, refinement using graph cutsInitialization using loopy BP, refinement using graph cuts

Fergus, R., Perona, P., and Zisserman, A., “Object Class Recognition by Unsupervised Scale-Invariant Learning”, CVPR 2003

Constellation modelsConstellation models

Categorical featuresCategorical features

Match

Constructing a Hierarchical Constructing a Hierarchical Model from ExamplesModel from Examples

Input:

Question: What is it?

Output:

Overview of the ApproachOverview of the ApproachExemplar imagesExtract Blob

GraphsMatch Blob

Graphs (many-to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model partsModel

decomposition relations

Model attachment relations

Blob Graph ConstructionBlob Graph ConstructionExemplar imagesExtract Blob

GraphsMatch Blob

Graphs (many-to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model decomposition relations

Model attachment relations

Model parts

Blob Graph ConstructionBlob Graph Construction

Edges are invariant to articulation

Choose the largest connected component.

The Representation and Matching of Categorical ShapeA. Shokoufandeh, L. Bretzner, D. Macrini, M.F. Demirci, C. Jonsson, and S. Dickinson, CVIU, Vol. 103, 2006, pp 139--154

Blob Graph ConstructionBlob Graph Construction

Connectivity measure: max{d1/major(A), d2/major(B)}

Perceptual grouping of blobs:

Feature matchingFeature matchingExemplar imagesExtract Blob

GraphsMatch Blob

Graphs (many-to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model decomposition relations

Model attachment relations

Model parts

Feature matchingFeature matchingOne-to-one matching. Rely on shape and context, not appearance!Many-to-many

matching

A Many-to-Many Graph A Many-to-Many Graph Matching FrameworkMatching Framework

Demirci, Shokoufandeh, Keselman, Bretzner, and Dickinson (IJCV 2006)

1. Embed graphs with low distortion to yield weighted point distributions.

2. Compute many-to-many correspondences between the two distributions using EMD.

3. The computed flows yield a many-to-many node correspondence between the two graphs.

Feature embedding and Feature embedding and EMDEMD

Spectral embedding

Returning to our set of Returning to our set of inputsinputs

Many-to-many matching of every pair of exemplars.

Part ExtractionPart ExtractionExemplar imagesExtract Blob

GraphsMatch Blob

Graphs (many-to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model decomposition relations

Model attachment relations

Model parts

Many-to-many matching Many-to-many matching resultsresults100%100%

100%100%100%100%

50%50% 50%50%

Extracting partsExtracting parts

Part – a collection of blobs.Part – a collection of blobs. Ideal partIdeal part

Represents blobs that occur frequently Represents blobs that occur frequently and participate in one-to-one and participate in one-to-one correspondence across many exemplars. correspondence across many exemplars.

Finding partsFinding parts From the pairwise matching results, find From the pairwise matching results, find

clusters (cliques) of blobs matching one-clusters (cliques) of blobs matching one-to-one.to-one.

Results of the part extraction Results of the part extraction stagestage

Extracting attachment Extracting attachment relationsrelations

Exemplar imagesExtract Blob

GraphsMatch Blob

Graphs (many-to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model decomposition relations

Model attachment relations

Model parts

Extracting attachment Extracting attachment relationsrelations

Right arm is typically connected to torso in exemplar images !

TorsoRight Arm

Number of times blobs drawn from the two clusters were attachedNumber of times blobs from the two clusters co-appeared in an image.

is high

Extracting attachment Extracting attachment relationsrelations

Number of times blobs from the two clusters co-appeared in an image.

Number of times blobs drawn from the two clusters were attached

Threshold PA to get part attachment

Extracting decomposition Extracting decomposition relationsrelations

Exemplar imagesExtract Blob

GraphsMatch Blob

Graphs (many-to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model decomposition relations

Model attachment relations

Model parts

Left Arm

Upper Lower

Extracting decomposition Extracting decomposition relationsrelations

Extracting decomposition Extracting decomposition relationsrelations

Sum of all flows between blobs in two clusters

Number of flows between blobs in two clusters

Combine PA and PF to obtain a decomposition score

Model construction stage Model construction stage summarysummary

Model Construction:

Clustering blobs into parts based on one-to-one matching results.

Recovering relations between parts based on individual matching and attachment results.

Assemble Final ModelAssemble Final ModelExemplar imagesExtract Blob

GraphsMatch Blob

Graphs (many-to-many)

Extract PartsExtract

Decomposition Relations

Extract Attachment Relations

Assemble Final Model

Blob graphs

Many-to-many correspondences

Model decomposition relations

Model attachment relations

Model parts

ResultsResults

Experiments

List of system parametersList of system parametersParameterParameter DescriptionDescription

Perceptual Perceptual Grouping Grouping ThresholdThreshold

The threshold determining the pairwise The threshold determining the pairwise connectivity between blobs in the exemplar connectivity between blobs in the exemplar imagesimages

DD The dimensionality of the embedding space The dimensionality of the embedding space during matchingduring matching

Embedding Embedding dimensionality dimensionality during part during part extractionextraction

The dimensionality of the embedding space The dimensionality of the embedding space used for blob clustering.used for blob clustering.

KK The maximum number of parts in the modelThe maximum number of parts in the modelTTattachattach The threshold for accepting attachment The threshold for accepting attachment

between parts in the final modelbetween parts in the final modelTTchildchild The threshold for considering a part to be a The threshold for considering a part to be a

potential child of another partpotential child of another partTTdecompdecomp The threshold for accepting a decompositionThe threshold for accepting a decomposition

ConclusionsConclusions Generic models must be defined at multiple levels of Generic models must be defined at multiple levels of

abstraction, as Marr proposed.abstraction, as Marr proposed.

Coarse shape features, such as blobs, are highly ambiguous Coarse shape features, such as blobs, are highly ambiguous and cannot be matched without contextual constraints.and cannot be matched without contextual constraints.

Moreover, features that exist at different levels of Moreover, features that exist at different levels of abstraction must be matched many-to-many in the abstraction must be matched many-to-many in the presence of noise.presence of noise.

The many-to-many matching results can be analyzed to The many-to-many matching results can be analyzed to yield both the parts and relations of a decompositional yield both the parts and relations of a decompositional model.model.

Preliminary results indicate that a limited decompositional Preliminary results indicate that a limited decompositional model can be learned from a set of noisy examples.model can be learned from a set of noisy examples.

Future workFuture work Construct models for objects other than Construct models for objects other than

humans – objects with richer decompositional humans – objects with richer decompositional hierarchies.hierarchies.

Automatically learn perceptual grouping Automatically learn perceptual grouping relations between blobs from labeled relations between blobs from labeled examples.examples.

Develop indexing and matching frameworks Develop indexing and matching frameworks for decompositional models.for decompositional models.