From Partial Shape Matching through Local Deformation
to Robust Global Shape Similarity for Object Detection
Tianyang Ma and Longin Jan Latecki Temple University
Outline Contour
illumination texture
Contour is robust to changes in
Contour-based object detection
[V. Ferrari, F. Jurie, and C. Schmid. IJCV 2009][Q. Zhu, L. Wang, Y. Wu and J. Shi. ECCV 2008] [X. Bai, X. Wang, L. J. Latecki, W. Liu and Z. Tu. ICCV 2009][C. Lu, L. J. Latecki, N. Adluru, X. Yang, and H. Ling. ICCV 2009] [S. Riemenschneider, M. Donoser, and H. Bishof. ECCV 2010][P. Srinivasan, Q. Zhu, and J. Shi. CVPR 2010][A. Toshev, B. Taskar, and K. Daniilidis. CVPR 2010]
Challenges
2 . Part of the true contour of the target object may be wrongly connected to part of a background contour resulting in a single edge fragment part to part matching needed
1 . The contour of the desired object is typically fragmented over several pieces.
This has been addressed by other approaches.
Partial Shape Matching
[S. Riemenschneider, M. Donoser, and H. Bishof. ECCV 2010]
The key ideas:• each contour part is represented as a submatrix• efficient matching with integral image
We utilize these ideas but use different geometric features.Our object detection framework is very different.
Outline
1. Shape Representation2. Partial Matching in Images3. Contour Selection as Maximal Clique Computation4. Experiments
Outline
1. Shape Representation2. Partial Matching in Image3. Contour Selection as Maximal Clique Computation4. Experiments
Shape Representation: Histogramless Shape Context
Shape Context[Belongie et al. 2002]
j
i
P Q
P
Q
j
i
i
j
Distance Matrix:
Angle Matrix:
( , )2( , ) log(1 )P P
i jD i j p p
( , ) ( , ) ,P Pi ji j p p
Histogram
Contour part = dist. submatrix + angle submatrix
( , ) ( , )P PD i j
two matrices are used to represent the entire contour two block diagonal matrices represent the green contour part
MM
M
M
M
M
Single partial contour P
( , ) ( , )P P i j
Relation of any two contour parts is submatrices
( , ) ( , )P QD i j
( , ) ( , )P Q i j
two submatrices are used to represent spatial configuration of a part composed of two contour segments.
M
M
M
M
Partial Contours P and Q
2( , ) ( , )
( , ) 2
( , ) ( , )( , ) exp( )
( ( , ) )
P P T T
D P P
D i j D i jA P T
D i j
2( , ) ( , )
2
( , ) ( , )( , ) exp( )
P P T Ti j i jA P T
( , ) ( , ) ( , )DA P T A P T A P T
21 1
1( , ) ( , )m m
i i
P T A P Tm
( , )P PD ( , )P P
Shape Similarity
P
T
T
( , )T TD ( , )T T
( , )T TD ( , )T T
Shape Similarity
P Q
TU
T
U
( , )P QD ( , )P Q
( , )T UD ( , )T U
( , )T UD ( , )T U
Outline
1. Shape Representation2. Partial Matching in Image3. Contour Selection as Maximal Clique Computation4. Experiments
Partial matching between edge fragments and model contour
1( , ) { ,..., }k j j le j l q q
1( , ) { ,..., }i j lM i l p p i
1i l j
1j l
( , )M i l
( , )ke j l
Partial image edge fragment
Partial Model fragment
Partial matching determines:
( ( , ), ( , ))kM i l e j l
Key advantages: • no model decomposition into parts is needed• no breaking or connecting edge fragments is needed• tolerates missing contour fragments in edge image.
Partial matching between edge fragments and model contour
• 1) Construct a 4D tensor• 2) Take the maximum of the 4D matrix along different length ,
and suppress it tol
j
i
( , , )S i j k
Partial matching: ( ( , ), ( , ))kM i l e j lj - start point on edge contour k
k –
inde
x of
edg
e co
ntou
r
i - start point on model contour
j
i
( , , )S i j k
k
1i
1i
All relevant edge fragments are mapped to their corresponding model fragments.
Key benefits:1. Top down selection of relevant
edge subfragments.2. All selected edge subfragments
are mapped to 1D curve of the model contour.
j
i
( , , )S i j k
k
1i
1i
...
...
How to find the true contours of the target shape in the edge image?
Problem formulation
?
Key idea:Given a minimal required coverage of the model contour,we want to select non overlapping model fragments that maximizethe configuration similarity to the corresponding image fragments.
Outline
1. Shape Representation2. Partial Matching in Image3. Contour Selection as Maximal Clique Computation4. Experiments
Construction of Affinity Matrix
)),(),,(( 1111 ljeliMv mi
)),(),,(),,(),,((),( 22112211 ljeljeliMliMjiA nm
Each vertex of the graph corresponds to a partial match
The affinity between and is
1 1 1 1( ( , ), ( , ))mM i l e j l
2 2 2 2( ( , ), ( , ))nM i l e j l
jv
iv
iv jv
( , )A i j
The weighted affinity graph is denoted as G = (V, A).
Construction of Affinity Matrix
High affinity:
Low affinity:
Problem with Affinity Matrix
Wrong matches may also have high affinity:
Detected Contours are Maximal CliquesA maximal clique is a subset of V with maximal average affinity between all pairs of its vertices.
In this example, the maximal clique has 4 nodes selected from over 500 nodes. Therefore, most clustering based approach may not succeed.
[ M. Pavan and M. Pelillo. PAMI 2007]
In order to solve this combinational problem, we relax it to
A vertex is selected as belonging to a maximal clique iff v V 0vx
Each maximal clique corresponds to a local maximum of:
Each local solution is not a final solution but a detection hypothesis.
Indicator = selected maximal clique of vertices of V.
Computing Maximal Cliques
{0,1}NX
1: 0 and 1NX R X X
( ) Tf X X AX
We employ the new algorithm to find all significant local maxima:[H. Liu , L. J. Latecki, and S. Yan. NIPS 2010]
Furthermore, we locally deform the model, then evaluate and score each detection hypothesis.
Outline
1. Shape Representation2. Partial Matching in Image3. Contour Selection as Maximal Clique Computation4. Experiments
Experiments on ETHZ dataset
Quantitative Evaluation
Mean APOur method 0.877P. Srinivasan, Q. Zhu, and J. Shi. CVPR 2010 0.872S. Maji and J. MalikCVPR 2009 0.771P. Felzenszwalb, D. McAllester, and D. Ramanan CVPR 2008 0.712C. Lu, L. J. Latecki, N. Adluru, X. Yang, and H. Ling. ICCV 2009 0.709
Interpolated Average Precision : AP
Accuracy of boundary localization
Each entry is the coverage/precision for correct detections at 0.4 FPPI.
Thanks!