learning decompositional shape models from examples
DESCRIPTION
Learning Decompositional Shape Models from Examples. Alex Levinshtein Cristian Sminchisescu Sven Dickinson. The Evolution of Object Recognition. Appearance-based models. Automatically built appearance-based model from video sequence - PowerPoint PPT PresentationTRANSCRIPT
Learning Decompositional Learning Decompositional Shape Models from Shape Models from
ExamplesExamples
Alex LevinshteinAlex Levinshtein
Cristian SminchisescuCristian Sminchisescu
Sven DickinsonSven Dickinson
The Evolution of Object Recognition
Appearance-based modelsAppearance-based models
Automatically built appearance-based model from video sequence(Ramanan, D. and Forsyth, D.A., “Using Temporal Coherence to Build Models of Animals”, ICCV, 2003)
Constellation model(Fergus, R., Perona, P., and Zisserman, A., “Object Class Recognition by Unsupervised Scale-Invariant Learning”, CVPR, 2003)
Appearance-based modelsAppearance-based models
Hierarchical ModelsHierarchical ModelsManually built hierarchical model proposed by Marr And Nishihara(“Representation and recognition of the spatial organization of three dimensional shapes”, Proc. of Royal Soc. of London, 1978)
Our goalOur goal
Automatically construct a generic hierarchical shape model from exemplars
Challenges:
Cannot assume similar appearance among different exemplars
Generic features are highly ambiguous
Generic features may not be in one-to-one correspondence
Automatically constructed Automatically constructed Hierarchical ModelsHierarchical Models
Input:
Question: What is it?
Output:
Stages of the systemStages of the systemExemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
Blob Graph ConstructionBlob Graph ConstructionExemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
Blob Graph ConstructionBlob Graph Construction
Choose the largest connected component.On the Representation and Matching of Qualitative Shape at Multiple Scales
A. Shokoufandeh, S. Dickinson, C. Jonsson, L. Bretzner, and T. Lindeberg,ECCV 2002
Blob Graph ConstructionBlob Graph Construction
Connectivity measure: max{d1/major(A), d2/major(B)}
Perceptual grouping of blobs:
Blob Graph ConstructionBlob Graph ConstructionEdge weights between
connected blobs:
Edge weights between disconnected blobs are computed based on shortest path distances.
Edge weights are invariant to articulation.
Feature matchingFeature matchingExemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
Feature matchingFeature matchingOne-to-one matching. Rely on shape and context, not appearance!
?
Many-to-many matching
Feature embeddingFeature embedding
00 171171 202202 230230
171171 00 373373 400400
202202 373373 00 432432
230230 400400 432432 00
Spectral embedding
Matching using Earth Mover’s Matching using Earth Mover’s DistanceDistance
EMD under Transformation EMD under Transformation AlgorithmAlgorithm
Returning to our set of Returning to our set of inputsinputs
Many-to-many matching of every pair of exemplars.
Part ExtractionPart ExtractionExemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
Results of the part extraction Results of the part extraction stagestage
What is next?What is next?
Extracting attachment Extracting attachment relationsrelations
Exemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
Extracting attachment Extracting attachment relationsrelations
Right arm is typically connected to torso in exemplar images !
Extracting attachment Extracting attachment relationsrelations
Number of times blobs from the two clusters co-appeared in an image.
Number of times blobs drawn from the two clusters were attached
Extracting decomposition Extracting decomposition relationsrelations
Exemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
Extracting decomposition Extracting decomposition relationsrelations
Extracting decomposition Extracting decomposition relationsrelations
Sum of all flows between blobs in two clusters
Number of flows between blobs in two clusters
Finding decompositions Finding decompositions (example)(example)
Model construction stage Model construction stage summarysummary
Model Construction:
Clustering blobs into parts based on one-to-one matching results.
Recovering relations between parts based on individual matching and attachment results.
Assemble Final ModelAssemble Final ModelExemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
Experiments
List of system parametersList of system parametersParameterParameter DescriptionDescription
Perceptual Perceptual Grouping Grouping ThresholdThreshold
The threshold determining the pairwise The threshold determining the pairwise connectivity between blobs in the exemplar connectivity between blobs in the exemplar imagesimages
DD The dimensionality of the embedding space The dimensionality of the embedding space during matchingduring matching
Embedding Embedding dimensionality dimensionality during part during part extractionextraction
The dimensionality of the embedding space The dimensionality of the embedding space used for blob clustering.used for blob clustering.
KK The maximum number of parts in the modelThe maximum number of parts in the model
TTattachattach The threshold for accepting attachment The threshold for accepting attachment between parts in the final modelbetween parts in the final model
TTchildchild The threshold for considering a part to be a The threshold for considering a part to be a potential child of another partpotential child of another part
TTdecompdecomp The threshold for accepting a decompositionThe threshold for accepting a decomposition
ConclusionsConclusions General framework for constructing a General framework for constructing a
generic decompositional model from generic decompositional model from different exemplars with dissimilar different exemplars with dissimilar appearance.appearance.
Recovering decompositional relations Recovering decompositional relations requires solving the difficult many-to-requires solving the difficult many-to-many graph matching problem.many graph matching problem.
Preliminary results indicate good model Preliminary results indicate good model recovery from noisy features.recovery from noisy features.
Future workFuture work
Construct models for objects other than Construct models for objects other than humans.humans.
Provide scale invariance during matching.Provide scale invariance during matching.
Automatically learn perceptual grouping Automatically learn perceptual grouping relations from labeled examples.relations from labeled examples.
Develop indexing and matching framework Develop indexing and matching framework for decompositional models.for decompositional models.