recognition and matching based on local invariant features cordelia schmid inria, grenoble david...
TRANSCRIPT
![Page 1: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/1.jpg)
Recognition and Matching
based on local invariant features
Cordelia Schmid
INRIA, Grenoble
David Lowe
Univ. of British Columbia
![Page 2: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/2.jpg)
Introduction
Local invariant photometric descriptors
( )local descriptor
Local : robust to occlusion/clutter + no segmentation
Photometric : distinctive
Invariant : to image transformations + illumination changes
![Page 3: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/3.jpg)
History - Matching
Matching based on line segments
Not very discriminant
Solution : matching with interest points & correlation
[ A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry,
Z. Zhang, R. Deriche, O. Faugeras and Q. Luong,
Artificial Intelligence 1995 ]
![Page 4: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/4.jpg)
Approach
• Extraction of interest points with the Harris detector
• Comparison of points with cross-correlation
• Verification with the fundamental matrix
![Page 5: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/5.jpg)
Harris detector
Interest points extracted with Harris (~ 500 points)
![Page 6: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/6.jpg)
Cross-correlation matching
Initial matches (188 pairs)
![Page 7: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/7.jpg)
Global constraints
Robust estimation of the fundamental matrix
99 inliers 89 outliers
![Page 8: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/8.jpg)
Summary of the approach
• Very good results in the presence of occlusion and clutter– local information– discriminant greyvalue information – robust estimation of the global relation between images– for limited view point changes
• Solution for more general view point changes– wide baseline matching (different viewpoint, scale and rotation)– local invariant descriptors based on greyvalue information
![Page 9: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/9.jpg)
History - Recognition
Color histogram [Swain 91]
b
g
r
Each pixel is described
by a color vector
Distribution of color vectors
is described by a histogram
=> not robust to occlusion, not invariant, not distinctive
![Page 10: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/10.jpg)
History - Recognition
Eigenimages [Turk 91]
• Each face vector is represented in the eigenimage space– eigenvectors with the highest eigenvalues = eigenimages
• The new image is projected into the eigenimage space– determine the closest face
. .1v.
3v
2v.
not robust to occlusion, requires segmentation, not invariant,
discriminant
![Page 11: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/11.jpg)
History - Recognition
Geometric invariants [Rothwell 92]
• Function with a value independent of the transformation
• Invariant for image rotation : distance of two points
• Invariant for planar homography : cross-ratio
),(),( yxfyxf tt yxTyx ),(),( where
=> local and invariant, not discriminant, requires sub-pixel extraction of primitives
![Page 12: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/12.jpg)
History - Recognition
Problems : occlusion, clutter, image transformations, distinctiveness
Solution : recognition with local photometric invariants
[ Local greyvalue invariants for image retrieval,
C. Schmid and R. Mohr,
PAMI 1997 ]
![Page 13: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/13.jpg)
Approach
1) Extraction of interest points (characteristic locations)
2) Computation of local descriptors
3) Determining correspondences
4) Selection of similar images
( )local descriptor
![Page 14: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/14.jpg)
Interest points
Geometric featuresrepeatable under transformations
2D characteristics of the signalhigh informational content
Comparison of different detectors [Schmid98] Harris detector
![Page 15: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/15.jpg)
Harris detector
Based on the idea of auto-correlation
Important difference in all directions => interest point
![Page 16: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/16.jpg)
Harris detector
2
),(
)),(),((),( yyxxIyxIyxf kkkWyx
k
kk
y
xyxIyxIyxIyyxxI kkykkxkkkk )),(),((),(),(with
2
),(
),(),(),(
Wyx
kkykkx
kky
xyxIyxIyxf
Auto-correlation function for a point and a shift ),( yx ),( yx
Discret shifts can be avoided with the auto-correlation matrix
![Page 17: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/17.jpg)
Harris detector
y
x
yxIyxIyxI
yxIyxIyxI
yx
Wyxkky
Wyxkkykkx
Wyxkkykkx
Wyxkkx
kkkk
kkkk
),(
2
),(
),(),(
2
)),((),(),(
),(),()),((
Auto-correlation matrix
![Page 18: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/18.jpg)
Harris detection
• Auto-correlation matrix– captures the structure of the local neighborhood– measure based on eigenvalues of this matrix
• 2 strong eigenvalues => interest point
• 1 strong eigenvalue => contour
• 0 eigenvalue => uniform region
• Interest point detection– threshold on the eigenvalues– local maximum for localization
![Page 19: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/19.jpg)
Local descriptors
Descriptors characterize the local neighborhood of a point
( )local descriptor
![Page 20: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/20.jpg)
Local descriptors
)2
),(exp(
2
1),),((
2
2
2
tt yx
yxG
ydxdyyxxIyxGGyxI ),(),()(),(
)(*),(
)(*),(
)(*),(
)(*),(
)(),(
)(),(
),(
yy
xy
xx
y
x
GyxI
GyxI
GyxI
GyxI
GyxI
GyxI
yxv
Greyvalue derivatives
![Page 21: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/21.jpg)
Local descriptors
Invariance to image rotation : differential invariants [Koen87]
nsor epsilon te ricantisymmet theis where
2
2
)(
ij
yyyyxyxyxxxx
yyxx
yyyyyxxyxxxx
yyxx
kjiijk
lkijklij
ljiijkkkjiij
llijkklkijklij
ijij
ii
jiji
ii
LLLLLL
LL
LLLLLLLL
LLLL
L
LLLL
LLLL
LLLLLLLL
LLLLLLLL
LL
L
LLL
LL
L
![Page 22: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/22.jpg)
Local descriptors
Robustness to illumination changes
In case of an affine transformation
baII )()( 21 xx
2
2
2
2
2/1
2/3
)(
)(
)(
)
)(
)(
)(
)(
ii
kjiijk
ii
lkijklij
ii
kjiijkkkjiij
ii
llijkklkijklij
ii
jiij
ii
ii
ii
jiji
LL
LLLL
LL
LLLL
LL
LLLLLLLL
LL
LLLLLLLL
LL
LL
LL
L
LL
LLL
![Page 23: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/23.jpg)
Local descriptors
Robustness to illumination changes
In case of an affine transformation
baII )()( 21 xx
or normalization of the image patch with mean and variance
![Page 24: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/24.jpg)
Determining correspondences
Vector comparison using the Mahalanobis distance
)()(),( 1 qpqpqp TMdist
( ) ( )=?
![Page 25: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/25.jpg)
Selection of similar images
• In a large database – voting algorithm– additional constraints
• Rapid acces with an indexing mechanism
![Page 26: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/26.jpg)
Voting algorithm
local characteristicsvector of
( )1I 1I nI2I2I
![Page 27: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/27.jpg)
Voting algorithm
1I 1I nI2I2I} }
1 1 02 1 1
I is the corresponding model image1
2 1 1
![Page 28: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/28.jpg)
Additional constraints
• Semi-local constraints– neighboring points should match– angles, length ratios should be similar
• Global constraints– robust estimation of the image transformation (homogaphy, epipolar geometry)
1
21
~2
~
11
2
3
2
3
![Page 29: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/29.jpg)
Results
database with ~1000 images
![Page 30: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/30.jpg)
Results
![Page 31: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/31.jpg)
Results
![Page 32: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/32.jpg)
Summary of the approach
• Very good results in the presence of occlusion and clutter– local information– discriminant greyvalue information – invariance to image rotation and illumination
• Not invariance to scale and affine changes
• Solution for more general view point changes– local invariant descriptors to scale and rotation– extraction of invariant points and regions
![Page 33: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/33.jpg)
Approach for Matching and Recognition
• Detection of interest points/regions– Harris detector (extension to scale and affine invariance)– Blob detector based on Laplacian
• Computation of descriptors for each point
• Similarity of descriptors
• Semi-local constraints
• Global verification
![Page 34: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/34.jpg)
Approach for Matching and Recognition
• Detection of interest points/regions
• Computation of descriptors for each point– greyvalue patch, diff. invariants, steerable filter, SIFT descriptor
• Similarity of descriptors– correlation, Mahalanobis distance, Euclidean distance
• Semi-local constraints
• Global verification
![Page 35: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/35.jpg)
Approach for Matching and Recognition
• Detection of interest points/regions
• Computation of descriptors for each point
• Similarity of descriptors
• Semi-local constraints– geometrical or statistical relations between neighborhood points
• Global verification– robust estimation of geometry between images
![Page 36: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/36.jpg)
Overview
8:30-8:45 Scale invariant interest points
8:45-9:00 SIFT descriptors
9:00-9:25 Affine invariance of interest points + applications
9:25-9:45 Evaluation of interest points + descriptors
9:45-10:15 Break
![Page 37: Recognition and Matching based on local invariant features Cordelia Schmid INRIA, Grenoble David Lowe Univ. of British Columbia](https://reader031.vdocuments.us/reader031/viewer/2022032106/56649e695503460f94b65dcb/html5/thumbnails/37.jpg)
Overview
10:15-11:15 Object recognition system, demo, applications
11:15-11:45 Recognition of textures and object classes
11:45-12:00 Future directions + discussion