1

Extracting Subimages of an Unknown Category from a Set of Images

Sinisa Todorovic and Narendra AhujaBeckman Institute, UIUC

Presented by Tingfan Wu

Objective

Clusters

General Steps

Training Images

Random segmentsfeature vectors

Unseen image

C3

C2C1

ModelsFt1=(x1,x2….xn)Ft2=(x1,x2….xn)Ft3=(x1,x2….xn)Ft4=(x1,x2….xn)feature vectors

F1=(x1,x2….xn)F2=(x1,x2….xn)F3=(x1,x2….xn)F4=(x1,x2….xn)

• Varieties– Segmetation Methods– Feature Spaces– Clustering Methods

= C1

Segment out all the cars

….

fused tree model for cars

Unseen image

Training images

Segmented Cars

Segmentation Trees

OverviewOverview

Multiscale Seg.

Segment out all the cars

….

fused tree model for cars

Unseen image

Training images

Segmented Cars

Segmentation Trees

MultiscaleMultiscaleSegmentation TreeSegmentation Tree

Feature Extraction = Image Segmentation

Multiscale Segmentation Tree

Region Descriptor on Tree Node

Attr(Node) = Description of the region

What are good region descriptors?

• Photometric– Gray level

• Geometric (rotation invariant)– Area– C.M.– Boundary Shape Histogram

• Hybrid – Salient descriptor

• Topology– Recursive containment of regions

(¹ v ; ¾2v )

hv (1 : : : K )

(av )(xv ; yv )

(©v )hv (1)

hv (2)

hv (8)

Can be rotation invariant

hv (3)

Salient Descriptor for a Region

• An outstanding region among siblings?– Brighter/darker?– Noisier /more homogenous– Larger/Smaller– Higher/lower entropy

on boundary shape• Empirical result: best λ=0.5

Photometric Geometric

hv (1)hv (2)

hv (8)

. . .

Salience Contract Flow(microview)

Average Direction and Magnitude

¡!©v

¡!wv=d2+

++

+

++

Salience Contract Flow(macroview)

Match salience contract flow

~¡!©1 ¼

¡!©2

Store Regional Descriptor on Treenode

Photometric Geometric Salient

Segment out all the cars

….

fused tree model for cars

Unseen image

Training images

Segmented Cars

Segmentation Trees

Maximal CommonMaximal CommonSubtreeSubtree Matching Matching

Segment out all the cars

….

fused tree model for cars

Unseen image

Training images

Segmented Cars

Segmentation Trees

How does it works?How does it works?

Inexact Matching: Structural Noise

Use tree edit distance instead

Tree Edit Distance• Editor Operations : costs ~ Dissimilarity(x,y)

– Remove a node– Add a node – Replace a node

-

+r

Metaphor: String Edit Distance• Unifying Editor Operations

– Remove a node– Add a node (removal on partner) – Replace a node (paired removal on both string/tree)

AABBBBCCEdit : Add Y

AABBYBBCC

Edit : Remove Y

AABBXBBCCEdit : Replace X with Y

AABBYBBCC

Edit : Remove X Edit : Remove Y

Tree Edit Distance• Editor Operation (with costs)

– Two way removal only

t t’u=E1(t)∩E2(t’)Dist.(t, t’) = Dist.(t, u) + Dist(u, t’)

E():Sequenceof removal

E1() E1()

Reduce Edit-Distance matching to Non-edit matching

• Transitive Closure• (see animation)

Closure Original

Matching Criteria

Divide and Conquer

Try all pairs of (v, v’) combinations = O(|t| + |t’|)

NP-complete QP approx. O(|Cvv’|)

Segment out all the cars

….

fused tree model for cars

Unseen image

Training images

Segmented Cars

Segmentation Trees

Model Model GenerationGeneration

Model: Union of Subtrees

…

NP-Hard

1.Pairwise matching2.One by one union

Sub-optimalOptimal

= ∪

Next TreeT = T u Tnext

Category Model

Segment out all the cars

….

fused tree model for cars

Unseen image

Training images

Segmented Cars

Segmentation Trees

Testing:Testing:SegmentationSegmentation

Segment out all the cars

fused tree model for cars

Unseen image Segmented Cars

Testing: Detect & SegmentationMaximal CommonMaximal CommonSubtreeSubtree Matching Matching

Match : (Similarity > Thresh) (precision/recall)

Performace Evaluaton

Results (Caltech 101 Face)

Varying Matching Thresh. (precision/recall)

Results (UIUC Car Side View)

#positive/#training: 5/10 vs 10/20(2hr on P4-2.4G/2G)

Results (Caltech 101 Face)

#positive/#training: 3/6 vs 6/12

Rotation Invariant

Caltech (Cars Rear View)

#positive/#training: 10/20

Conclusion

• Contribution– Good Image Representation Seg. Tree

• Small amount of training data– Cf. Statistical Learning/Clustering

• Ex. Visual Words + pLSA

• Allow Non-category Images noise• Allow occlusion (disconnected regions)

Region Descriptor

Annotated Recursive Tree

Salient Flow

Sliced area histogram

Region Area

Graylevel

xx

xxx

x

x

x

TopologicalGeometricPhotmetric

Thank you

• Quicktopic

Cf. Visual Words+pLSA

• Visual Words recognize connected object only• Tree Matching is more conservative due to

intersection

Cf. Visual Words +pLSA

Visual Words/pLSA

Tree matching

Caltech FacesVisual Words/pLSA

Tree matching

ReSPEC(Use Color Histogram)