1

Unsupervised Modeling of Object Categories

Using Link Analysis Techniques

Unsupervised Modeling of Object Categories

Using Link Analysis Techniques

Gunhee KimChristos Faloutsos

Martial Hebert

Computer ScienceCarnegie Mellon University

June 23, 2008, Anchorage, AK

2

OutlineOutline

• Problem Statement & Our Approach

• Network Construction

• Link Analysis Techniques– Ranking of features wrt an image/object

– Structural Similarity

• Unsupervised Modeling– Category Discovery

– Localization

• Complexity

• Conclusion

3

Unsupervised Modeling[1-5]Unsupervised Modeling[1-5]

[1] Sivic et al, ICCV 2005[2] Fritz&Schiele, DAGM 2006[3] Grauman&Darrell, CVPR 2006[4] Todorovic&Ahuja, CVPR 2006[5] Cao&Fei-Fei, ICCV 2007

• Category discovery + Localization• Category discovery + Localization• Category discovery + Localization

4

Previous Work Previous Work

• Topic models based on bag of words[1][2][5]

[1] Sivic et al, ICCV 2005[2] Fritz&Schiele, DAGM 2006[3] Grauman&Darrell, CVPR 2006[4] Todorovic&Ahuja, CVPR 2006[5] Cao&Fei-Fei, ICCV 2007

• Tree matching[4]

• Clustering with partial matching[3]

wN

d z

D

5

IntuitionIntuition

Link Analysis Techniques

Visual Information

Solve Visual tasks

A large-scale Network

+

6

Statistics of Link StructureStatistics of Link Structure

7

Large-Scale NetworksLarge-Scale Networks

WWW[1] Oscars social network[2] Metabolic network[3]

Neural network[4]A food web[5]

(1):

htt

p:/

/ww

w.o

pte

.org

, (2

): T

he

Aca

de

my

of

Mo

tion

Pic

ture

Art

s a

nd

S

cie

nce

s, (

4):

htt

p:/

/ww

w.w

illa

me

tte

.ed

u/~

go

rr/c

lass

es/

cs4

49

/bra

in.h

tml (

5):

M

ark

ew

ma

n

8

OutlineOutline

• Problem Statement & Our Approach

• Network Construction

• Link Analysis Techniques– Ranking of features wrt an image/object

– Structural Similarity

• Unsupervised Modeling– Category Discovery

– Localization

• Complexity

• Conclusion

9

Visual Similarity Network: VerticesVisual Similarity Network: Vertices

• Vertices: Any Local Features

– Harris Affine + SIFT

I1

Im

: Adjacency Matrix of G

M

nnI1

Im

10

Visual Similarity Network: Edges & WeightsVisual Similarity Network: Edges & Weights

• Edges: Correspondences by image matching

– Spectral Matching[1-2]: Appearance affinity + Geometric Consistency

• Weights: Stronger geometric consistency, higher values

M

nn

Ia

[1][2] Leordeanu & Hebert, ICCV05, ICML06.

Ia

Ib

Ib

11

OutlineOutline

• Problem Statement & Our Approach

• Network Construction

• Link Analysis Techniques– Ranking of features wrt an image/object

– Structural Similarity

• Unsupervised Modeling– Category Discovery

– Localization

• Complexity

• Conclusion

12

1. Ranking of the Features1. Ranking of the Features

→“models capture the hubs in the visual network”

[1] Fergus, Perona, Zisserman, IJCV 2007.

Fergus et al[1]: “models capture the essence of categories”

>>

13

Ranking Removes Noisy MatchingRanking Removes Noisy Matching

14

How to Rank the FeaturesHow to Rank the Features

• PageRank[1]

– Recursive Definition

VoteVote

[1] Brin and Page. WWW 1998

15

Rationale: Why Ranking Works?Rationale: Why Ranking Works?

Consistent Matching

Highly varient Matching

Hub Outlier

16

2. Structural Similarity2. Structural Similarity

• Similar vertices → Similar Link structures

Node i Node j

+ (3) similarity of matching behaviors

(1) Appearance Similarity + (2) Geometric consistency

17

How to Mine Structural SimilarityHow to Mine Structural Similarity

• Automatic Extraction of Synonyms[1]

[1] Blondel et al. SIAM review 2004.

Node i Node j

A vertex structural similarity matrix Z

nnu v : If v appears in the definition of u

18

OutlineOutline

• Problem Statement & Our Approach

• Network Construction

• Link Analysis Techniques– Ranking of features wrt an image/object

– Structural Similarity

• Unsupervised Modeling– Category Discovery

– Localization

• Complexity

• Conclusion

19

Compute Ranking wrt Each ImageCompute Ranking wrt Each Image

Ia

PageRank:

vPDMP ))(1(

P-vector for Ia

n1

I1 I2 Im

20

Ia

Less correlated

Meaning of RankingMeaning of Ranking

P-vectorfor Ia

Ranked importance of the other features wrt image a

Ranked importance of features in image a

Valuable for Category discovery

Ia

Ib

Ic

Highly correlated

n1

21

Image Affinity MatrixImage Affinity Matrix

Vertex structural similarity matrix Z

I1 I2 Im

n1

nn

m PageRank vectors

mm

Image affinity matrix A

bjaibj IbIa

jiia

Ib

ja baZaPbPbaA,

),()()(),(

n >> m(Ex. 1mil >> 600)

22

Category Discovery by ClusteringCategory Discovery by Clustering

600 images of 6 Object Classes of Catech-101

k-NN graph [1]

Normalized spectral Clustering [2]

[1] Luxburg. Statistics and Computing, 2007

[2] Shi & Malik, PAMI 2000

k = 10log(m)

23

Results of Category DiscoveryResults of Category Discovery

• TUD/ETHZ dataset

– Experimental Setup follows [1]

– 75 images per object, 10 repetition

95.47%

Motorbikes Cars Giraffes

Motorbikes 93.3 2.7 0.0 6.7 2.7

Cars 4.8 2.6 95.2 2.6 0.0

Giraffes 2.0 1.1 0.1 0.4 97.9 1.4

[1] Grauman & Darrell, CVPR 2006

24

Results of Category DiscoveryResults of Category Discovery

• Caltech-101 Object classes (100 per object)

A C F M WA 98.2 0.7 0.1 0.8 0.2C 0.6 99.3 0.0 0.0 0.1F 2.2 0.1 96.2 0.0 1.5M 1.3 0.9 0.0 97.5 0.3W 2.7 0.8 0.0 1.2 95.3

4 obj:98.55%

[1] Grauman & Darrell, CVPR 2006[2] Sivic et al, ICCV 2005

5 obj:97.30%

6 obj:95.42%

> [2]: 98%, [1]: 86%

A

C

F

M

W

KA C F M W K

A 94.5 0.5 0.0 0.5 0.3 4.2C 1.1 97.1 0.0 0.0 0.0 1.8F 1.5 0.0 95.6 0.0 1.8 1.1M 1.4 0.4 0.0 93.5 0.1 4.6W 2.2 0.3 0.0 0.3 93.4 3.8K 1.5 0.0 0.1 0.0 0.0 98.4

A C F MA 98.4 1.0 0.1 0.5C 0.2 99.8 0.0 0.0F 1.9 0.1 98.0 0.0M 1.4 0.6 0.0 98.0

25

Compute Ranking wrt a CategoryCompute Ranking wrt a Category

P-vector for category C1

PageRank: vPDMP ))(1(

nc11

nc21

nc31

P-vector for category C2

P-vector for category C3

26

Meaning of RankingMeaning of Ranking

Valuable for Localization

Ranked importance of each feature wrt its category

Ia

P-vector for a giraffe class

nc1

27

Localization – Confidence ValuesLocalization – Confidence Values

nc11 nc1nc1

),()()()( ij

cb

jcicic abZbPaPaIj

),()()()( ij

cb

jcicic abZbPaPaIj

),()()()( ij

cb

jcicic abZbPaPaIj

nc21 nc2nc2nc31 nc3nc3

P-vectors and Vertex similarity matrix for each category

28

Examples of LocalizationExamples of Localization

29

Quantitative Results of LocalizationQuantitative Results of Localization

False Positives

[1] Quack et al. ICCV 2007

30

OutlineOutline

• Problem Statement & Our Approach

• Network Construction

• Link Analysis Techniques– Ranking of features wrt an image/object

– Structural Similarity

• Unsupervised Modeling– Category Discovery

– Localization

• Complexity

• Conclusion

31

Complexity IssuesComplexity Issues

• The VSN representation is

• Sparsity of the network

– Power iterations for sparse matrices:

• Scale-free network

)( 2nO

Ex. 6 objects of Caltech-101 900K nodes

Degrees of Vertices

Per

cen

tag

e o

f ve

rtic

es

)()( nOEO

4105

32

OutlineOutline

• Problem Statement & Our Approach

• Network Construction

• Link Analysis Techniques– Ranking of features wrt an image/object

– Structural Similarity

• Unsupervised Modeling– Category Discovery

– Localization

• Complexity

• Conclusion

33

ConclusionConclusion

• A new formulation of unsupervised modeling

– Statistics of the link structure

– Finding communities (categories) and hubs (class representative visual information)

• Link analysis techniques

• Competitive performance

• Future directions

– Statistical framework

– Scalability

34

Comments?Comments?

Thank You

gunhee@cs.cmu.edu

35

Supplementary MaterialSupplementary Material

If any questions and comments, please send me an email at

gunhee@cs.cmu.eduhttp://www.cs.cmu.edu/~gunhee

36

Spectral Matching[1-2]Spectral Matching[1-2]

[1] M. Leordeanu and M. Hebert. A spectral technique for correspondence problems using pairwise constraints, 2005. ICCV.[2] M. Leordeanu and M. Hebert. Efficient map approximation for dense energy functions, 2006. ICML.

OK

Pairs of wrong correspondences are unlikely to preserve geometry

Pairs of correct correspondences are very likely to preserve geometry

37

Weights of EdgesWeights of Edges

• 1. Stronger geometric consistency, higher weights

• 2. 10 matches / 50 features > 10 matches / 100 features

Cij > 0.8Cij > 0.7Cij > 0.6Cij > 0.5Cij > 0.4

wij = 0.0284wij = 0.0144wij = 0.0071wij = 0.0040wij = 0.0018

38

Example of Vertex Similarity[1]Example of Vertex Similarity[1]

• Similarity between Vertices of Directed Graphs

– Only based on link structures

[1] Blondel et al. SIAM review 2004.

39

Ranking for Category DiscoveryRanking for Category Discovery

• Relative Importance wrt each image

O

O

O

O

M Ma

- Only consider the relations between Ia and the others

Ia

Ia

- Why? To avoid Topic Drift

PageRank:

Ia

x x

P-vector for Ia

vPDMp ))(1(

40

Computation of Relative ImportanceComputation of Relative Importance

Ia

Ia

M

• Before “Category Discovery”

O

O

O

O

Why? Topic Drift

PageRank for Ia

41

Relative Importance for ModelingRelative Importance for Modeling

Ia

Ia

M

• Before “Category Discovery”

O

O

O

O

Wait ! In General, majority of images are from different object classes. What if the result is distracted by them?

PageRank: Recursive Definition!!

Ia

The vote by the same object will be much more appreciated !

Ib Ic

42

LocalizationLocalization

• Relative Importance wrt each category

M

P-vector for each category

Mc1 Mc2 Mc3

PageRank: vPDMp ))(1(

43

Computation of Relative ImportanceComputation of Relative Importance

O

OP-vector

• After “Category Discovery”

object category k

M

Valuable for Localization

Ranked importance of features wrt each object category

44

Toy Example: Relative ImportanceToy Example: Relative Importance

• Matching behavior

– Consistent between important features in the same class, Highly variant between backgrounds and different objects

Matching

45

Toy Example: Relative ImportanceToy Example: Relative Importance

Image 2 Image 1

– Consistent between important features in the same class, Highly variant between backgrounds and different objects

46

Toy Example: Relative ImportanceToy Example: Relative Importance

– Consistent between important features in the same class, Highly variant between backgrounds and different objects

Image 3 Image 1

47

Toy Example: Relative ImportanceToy Example: Relative Importance

– Consistent between important features in the same class, Highly variant between backgrounds and different objects

Image 4 Image 1

Consistent

Highly variant

48

Unsupervised ModelingUnsupervised Modeling

• Category Discovery + Localization

Ia

I1

Ia

Ranked importance of the features in I1 wrt Ia

P-vector Pa

Affinity of I1 to Ia

This value is not used !

A vertex similarity matrix Z

I1 + A(a,1)

Im

49

LocalizationLocalization

• Category Discovery + Localization

O

OP-vector

object category k

M

object category 1 O

OZ

P-vector Pc

ai +Zc

ai

=0.8

P-vector Pc