lecture 2 mei-chen yeh 03/09/2010. outline demos image representation and feature extraction –...
TRANSCRIPT
![Page 1: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/1.jpg)
Lecture 2
Mei-Chen Yeh03/09/2010
![Page 2: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/2.jpg)
Outline
• Demos• Image representation and feature extraction– Global features– Local features: SIFT
• Assignment #2 (due: 03/16)
![Page 3: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/3.jpg)
Demos• Augmented Reality
– http://www.youtube.com/watch?v=P9KPJlA5yds– http://www.youtube.com/watch?v=U2uH-jrsSxs
• Tracking– Traffic– Counting people
• Image search– MyFinder: http://128.111.56.44/myFinder/– Simplicity: http://wang14.ist.psu.edu/cgi-bin/zwang/regionsearch_show.cgi
• Image annotation– ALIPR: http://alipr.com/
• Embedded face detection and recognition• Tiling slide show • Pivot: http://www.technologyreview.com/video/?vid=533
![Page 4: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/4.jpg)
Multimedia Systems: A Multidisciplinary Subject
• Signal Processing• Data Mining• Machine Learning• Pattern Recognition• Networking• … and more!
![Page 5: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/5.jpg)
Topics (1)
• Image/video processing– Feature extraction– Video syntax analysis– Compression
![Page 6: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/6.jpg)
Topics (2)
• Content-based image/video retrieval– Copy detection– Region-based retrieval– Multi-dimensional indexing
![Page 7: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/7.jpg)
Topics (3)
• Multimodal system– Audio processing– Multimodality analysis
![Page 8: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/8.jpg)
Topics (4)
• Semantic concept detection– Object detection– Object recognition
![Page 9: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/9.jpg)
Topics (5)
• Tracking– Motion features– Models– Single-, multiple-object tracking
![Page 10: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/10.jpg)
Topic (6)
• Qualify of Service/Experience– QoE Framework– VoIP System Evaluation– Imaging System Evaluation
![Page 11: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/11.jpg)
Resources of the readings• ACM International Conference on Multimedia– The premier annual event on multimedia research,
technology, and art– Started since 1993– >400 attendees– Program: Content, Systems, Applications, HC tracks– Full papers (16%), short papers (28%) – Technical demonstrations, open source software
competition, the doctoral symposium, tutorials (6), workshops (11), a brave new topic session, panels (2), Multimedia grand challenge
• IEEE Transactions on Multimedia
![Page 12: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/12.jpg)
Image Representations
![Page 13: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/13.jpg)
Multimedia file formats
• A list of some formats used in the popular product “Macromedia Director”
• These formats differ mainly in how data are compressed.
• Features are normally extracted from raw data.
![Page 14: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/14.jpg)
1-bit images
• Each pixel is stored as a single bit (0 or 1), so also referred to as binary image.
• So-called 1-bit monochrome image
No color
![Page 15: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/15.jpg)
8-bit gray-level images
• Each pixel has a gray-value between 0 and 255. (0=>black, 255=>white)
• Image resolution refers to the number of pixels in a digital image
• A 640 x 480 grayscale image requires ??? kB
One byte per pixel640x480 = 307,200 ~ 300 kB
![Page 16: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/16.jpg)
24-bit color images
• Each pixel is represented by three bytes, usually representing RGB.• This format supports
256x256x256 (16,777,216) possible colors.
• A 640x480 24-bit color image would require 921.6 kB!
Lena: 1972Lena: 1997
![Page 17: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/17.jpg)
![Page 18: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/18.jpg)
Image Features
![Page 19: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/19.jpg)
Feature types
• Global features– Color– Shape– Texture
• Local features– SIFT– SURF– Self-similarity descriptor– Shape context descriptor– …
…
……
A fixed-length feature vector
![Page 20: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/20.jpg)
Color histogram
• A color histogram counts pixels with a given pixel value in Red, Green, and Blue (RGB).
• An example of histogram that has 2563 bins, for 24-bit color images:
![Page 21: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/21.jpg)
Color histogram (cont.)
• Quantization
![Page 22: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/22.jpg)
Color histogram (cont.)
• Problems of such a representation
Case 1
Case 2
Case 3
SAME!
SAME!
SAME!
![Page 23: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/23.jpg)
Search by color histograms
![Page 24: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/24.jpg)
Regional color
• Divide the image into regions
• Extract a color histogram for each region
• Put together those color histograms into a long feature vector
![Page 25: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/25.jpg)
Textures
• Many natural and man-made objects are distinguished by their texture.
• Man-made textures– Walls, clothes, rugs…
• Natural textures– Water, clouds, sand, grass, …
What is this?
![Page 26: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/26.jpg)
Examples
More: http://www.ux.uis.no/~tranden/brodatz.html
![Page 27: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/27.jpg)
Texture features
• Structural– Describe arrangement of texture elements– E.g., “texton model”, “texel model”
• Statistical– Characterize texture in terms of statistics– E.g., co-occurrence matrix, Markov random field
• Spectral– Analyze in spatial-frequency domain– E.g., Fourier transform, Gabor filter, wavelets
![Page 28: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/28.jpg)
Textual Properties
• Coarseness: coarse vs. fine• Contrast: high vs. low• Orientation: directional vs. non-directional• Edge: line-like vs. blob-like• Regularity: regular vs. random• Roughness: rough vs. smooth
![Page 29: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/29.jpg)
Shape
• Boundary-based feature– Use only the outer boundary of the shape– E.g. Fourier descriptor, shape context descriptor
• Region-based feature– Use the entire shape region– Local descriptors
![Page 30: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/30.jpg)
Shape: Fourier descriptor
![Page 31: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/31.jpg)
Properties
• Invariant to translation, scale, and rotation
![Page 32: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/32.jpg)
Feature types
• Global features– Color– Shape– Texture
• Local features– SIFT– SURF– Self-similarity descriptor– Shape context descriptor– …
…
……
A fixed-length feature vector
![Page 33: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/33.jpg)
David G. Lowe. Distinctive Image Features from Scale-Invariant Key-points, IJCV, 2004
![Page 34: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/34.jpg)
What is SIFT?
• Scale Invariant Feature Transform (SIFT) is an approach for detecting and extracting local feature descriptors from an image.
• SIFT feature descriptors are reasonably invariant to – scaling– rotation– image noise– changes in illumination– small changes in viewpoint
![Page 35: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/35.jpg)
Types of invariance
illumination scale rotation viewing angle
![Page 36: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/36.jpg)
621 128162.38 155.79 44.30 2.6157 6 0 0 0 0 0 1 58 63 1 0 7 6 1 8 8 9 0 024 42 39 14 0 0 0 0 0 0 7 2 44 7 0 0 23 22 6 69 137 64 0 0 0 0 11 137 55 12 0 0 2 25 137 112 0 0 0 0 3 17 30 6 34 1 0 0 20 51 137 89 137 89 0 0 0 15 115 102137 47 0 0 4 37 26 43 0 0 0 0 19 45 4 0 0 0 0 0 0 16 137 53 33 2 0 0 0 56 137 51 57 2 0 0 0 3 14 35 0 0 0 0 0 2 0 0282.47 185.76 27.80 2.0090 0 0 0 0 0 0 0 1 41 13 1 0 12 4 0 5 17 15 16 17 83 35 16 19 0 0 1 2 13 24 104 0 1 9 0 0 0 0 0 22 127 127 5 0 0 0 1 127 127 75 16 6 0 0 70 55 2 0 1 0 0 25 127 1 1 9 0 0 1 1 2 115 22 49 4 0 0 0 68127 127 30 4 0 0 0 58 67 127 69 0 0 0 5 20 2 0 0 0 4 65 5 2 85 50 6 0 1 15 2 30 56 93 53 19 0 0 4 41 22 127 86 1 0 2 17 20……….
Number of keypointsFeature dimension
![Page 37: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/37.jpg)
Matching two images
![Page 38: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/38.jpg)
• Densely cover the image(an image with 500x500 pixels => 2000 feature vectors)• Distinctive• Invariant to image scale, rotation, and partially invariant to changing
viewpoints and illumination • Perform the best among local descriptors
– K. Mikolajczyk and C. Schmid, “A performance evaluation of local descriptors,” PAMI 05.
![Page 39: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/39.jpg)
Simple test (scale and rotate)• Scale to 60% and rotate 30 degree
693 keypoints
349 keypoints
214 matches!
![Page 40: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/40.jpg)
Simple test (illumination)
693 keypoints
467 matches!
633 keypoints
![Page 41: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/41.jpg)
693 keypoints
728 keypoints
25 matches!Simple test (different appearance)
![Page 42: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/42.jpg)
Simple test (different appearance)693 keypoints
832 keypoints
1 match!
![Page 43: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/43.jpg)
Simple Test (different appearance with occlusion)
693 keypoints
1124 keypoints
0 match!
![Page 44: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/44.jpg)
• How to generate SIFT feature descriptors?• How to use SIFT features descriptors (for
object recognition, image retrieval, etc.) ?
About SIFT…
![Page 45: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/45.jpg)
SIFT: Overview
• Major stages of SIFT computation
Scale-space extrema detection
Keypoint localization
Orientation assignment
Keypoint descriptor
An image
feature vectors (128-d)
Identify potential interest points(location, scale)
Localize candidate keypointsReduced sets of (location, scale)
Identify the dominant orientations (location, scale, orientation)
Build a descriptor based onhistogram of gradients in localneighborhood
Interest point detector+
descriptor
![Page 46: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/46.jpg)
Step 1: Scale-space extrema detection
• How do we detect locations that are invariant to scale change of the image?
• Detecting extrema in scale-space– For a given image I(x,y), its linear scale-space representation:
– Be efficiently implemented by searching for local peaks in a series of DoG (difference-of-Gaussian) images
),(*),,(),,( yxIyxGyxL 222 2/)(
22
1),,(
yxeyxG
),,(),,(
),(*)),,(),,((),,(
yxLkyxL
yxIyxGkyxGyxD
![Page 47: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/47.jpg)
Step 1: Scale-space extrema detection
σ
kσ
k2σ
![Page 48: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/48.jpg)
DoGimages
Gaussianimages
![Page 49: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/49.jpg)
Step 2: Scale-space extrema detection
DoG
DoG
DoG
If X is the largest or the smallest of all of its neighbors, X is called a keypoint.
![Page 50: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/50.jpg)
Why DoG?
• An efficient function to compute• A close approximation to the scale-normalized Laplacian of
Gaussian– Lindeberg showed that the normalization of the Laplacian with the
factor σ2 is required for true scale invariance. (1994)– Mikolajczyk found that the maxima and minima of
produce the most stable image features. (2002)
• DoG v.s.
G22
G22
GkyxGkyxG
k
yxGkyxGGG
22
2
)1(),,(),,(
),,(),,(
![Page 51: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/51.jpg)
Output of Step 1
~ 2000 keypoints in a 500x500 imageToo many keypoints!
![Page 52: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/52.jpg)
Step 2: Accurate keypoint localization
• Reject points that have low contrast or are poorly localized along an edge
Image size:233x189 832
729 536
![Page 53: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/53.jpg)
Step 2: Accurate keypoint localization
• Another example
Extrema of DoG across scales
After removal of lowcontrast points
After removal ofedge responses
![Page 54: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/54.jpg)
Step 2: Accurate keypoint localization
• Simple method (Lowe, ICCV 1999)– Use gradient magnitudes
• More sophisticated method (Brown and Lowe, BMVC 2002)– Use the Taylor expansion of the scale-space function,
compare the function value at the extremum to a threshold (0.03)
– Use the ratio of eigenvalues of a 2x2 Hessian matrix, eliminate keypoints with a ratio greater than 10
)max(1.0 ,, jiji MM
![Page 55: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/55.jpg)
Step 3: Orientation assignment
![Page 56: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/56.jpg)
Step 3: Orientation assignment• To achieve invariance to rotation• Compute gradient magnitude and orientation for each
image sample L(x, y, σ)
• Form an orientation histogram from the gradient orientations of sample points within a region around the keypoint, weighted by its gradient magnitude and a Gaussian-weighted window
• Detect the highest peak
![Page 57: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/57.jpg)
Step 4: Local image descriptor
Use a 4x4 grid computed from a 16x16 sample array128-d = 4 * 4 * 8 (orientations)
Examples: 2x2 grid on a 8x8 sample array
![Page 58: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/58.jpg)
Step 4: Local image descriptor
• Fairly compact (128 values)
![Page 59: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/59.jpg)
Results
![Page 60: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/60.jpg)
Summary
Scale-space extrema detection
Keypoint localization
Orientation assignment
Keypoint descriptor
An image
feature vectors
scale
rotation
illumination changeviewpoint change
Invariant to…
![Page 61: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/61.jpg)
Discussions
• Do local features solve the object recognition problem?
• How do we deal with the false positives outside the object?
• How do we reduce the complexity matching two sets of local features?
![Page 62: Lecture 2 Mei-Chen Yeh 03/09/2010. Outline Demos Image representation and feature extraction – Global features – Local features: SIFT Assignment #2 (due:](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649ed15503460f94be02cb/html5/thumbnails/62.jpg)
Assignment #2• Download SIFT demo program
– http://www.cs.ubc.ca/~lowe/keypoints/– Or
http://www.csie.ntnu.edu.tw/~myeh/courses/s10_ms/Assignments/siftDemoV4.zip
• Prepare at least two pairs of images which you think are similar– 1st set: SIFT can match well– 2nd set: SIFT cannot match well
• Email to TA ([email protected]) your report that includes– Your experimental results– Your observations