![Page 1: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/1.jpg)
CS 1674: Intro to Computer Vision
Edges and Segments
Prof. Adriana KovashkaUniversity of Pittsburgh
October 18, 2016
![Page 2: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/2.jpg)
Midterm exam statistics
90-100% 80-89% 70-79% 60-69% 50-59%
18 14 3 1 1
![Page 3: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/3.jpg)
Edges vs Segments
Figure adapted from J. Hays
![Page 4: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/4.jpg)
Edges vs Segments
• Edges
– More low-level
– Don’t need to be closed
• Segments
– Ideally one segment for each semantic group/object
– Should include closed contours
![Page 5: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/5.jpg)
Edge detection
• Goal: map image from 2d array of pixels to a set of
curves or line segments or contours.
• Why?
• Main idea: look for strong gradients, post-process
Figure from J. Shotton et al., PAMI 2007
Source: K. Grauman
![Page 6: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/6.jpg)
What causes an edge?
Depth discontinuity:
object boundary
Change in surface
orientation: shape
Cast shadows
Reflectance change:
appearance
information, texture
Source: K. Grauman
![Page 7: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/7.jpg)
Designing an edge detector
• Criteria for a good edge detector:– Good detection: find all real edges, ignoring noise or other
artifacts
– Good localization
• detect edges as close as possible to the true edges
• return one point only for each true edge point
• Cues of edge detection– Differences in color, intensity, or texture across the boundary
– Continuity and closure
– High-level knowledge
Source: L. Fei-Fei
![Page 8: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/8.jpg)
Characterizing edges
• An edge is a place of rapid change in the image intensity function
imageintensity function
(along horizontal scanline) first derivative
edges correspond to
extrema of derivative
Source: L. Lazebnik
![Page 9: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/9.jpg)
Intensity profile Intensity
Gradient
Source: D. Hoiem
![Page 10: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/10.jpg)
With a little Gaussian noise
Gradient
Source: D. Hoiem
![Page 11: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/11.jpg)
Effects of noise• Consider a single row or column of the image
– Plotting intensity as a function of position gives a signal
Where is the edge?Source: S. Seitz
![Page 12: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/12.jpg)
Effects of noise
• Difference filters respond strongly to noise
– Image noise results in pixels that look very different from their neighbors
– Generally, the larger the noise the stronger the response
• What can we do about it?
Source: D. Forsyth
![Page 13: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/13.jpg)
Solution: smooth first
• To find edges, look for peaks in )( gfdx
d
f
g
f * g
)( gfdx
d
Source: S. Seitz
![Page 14: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/14.jpg)
• Differentiation is convolution, and convolution is associative:
• This saves us one operation:
gdx
dfgf
dx
d )(
Derivative theorem of convolution
gdx
df
f
gdx
d
Source: S. Seitz
Image with edge
Derivativeof Gaussian
Edge = maxof derivative
![Page 15: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/15.jpg)
Gradients -> edges
Primary edge detection steps:
1. Smoothing: suppress noise
2. Edge enhancement: filter for contrast
3. Edge localization
Determine which local maxima from filter output are actually edges vs. noise
• Threshold, Thin
Source: K. Grauman
![Page 16: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/16.jpg)
Thresholding
• Choose a threshold value t
• Set any pixels less than t to 0 (off)
• Set any pixels greater than or equal to t to 1
(on)
Source: K. Grauman
![Page 17: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/17.jpg)
Original image
Source: K. Grauman
![Page 18: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/18.jpg)
Gradient magnitude image
Source: K. Grauman
![Page 19: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/19.jpg)
Thresholding gradient with a lower threshold
Source: K. Grauman
![Page 20: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/20.jpg)
Thresholding gradient with a higher threshold
Source: K. Grauman
![Page 21: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/21.jpg)
Canny edge detector
• Filter image with derivative of Gaussian
• Find magnitude and orientation of gradient
• Non-maximum suppression:
– Thin wide “ridges” down to single pixel width
• Linking and thresholding (hysteresis):
– Define two thresholds: low and high
– Use the high threshold to start edge curves and
the low threshold to continue them
• MATLAB: edge(image, ‘Canny’);
• >>help edgeSource: D. Lowe, L. Fei-Fei
![Page 22: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/22.jpg)
Example
input image (“Lena”)
![Page 23: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/23.jpg)
Derivative of Gaussian filter
x-direction y-direction
Source: L. Lazebnik
![Page 24: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/24.jpg)
Compute Gradients (DoG)
X-Derivative of Gaussian Y-Derivative of Gaussian Gradient Magnitude
Source: D. Hoiem
![Page 25: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/25.jpg)
The Canny edge detector
norm of the gradient (magnitude)
Source: K. Grauman
![Page 26: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/26.jpg)
The Canny edge detector
thresholding
Source: K. Grauman
![Page 27: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/27.jpg)
The Canny edge detector
thresholding
How to turn
these thick
regions of the
gradient into
curves?
Source: K. Grauman
![Page 28: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/28.jpg)
Non-maximum suppression
Check if pixel is local maximum along gradient direction,
select single max across width of the edge
• requires checking interpolated pixels p and r
Source: K. Grauman
![Page 29: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/29.jpg)
Bilinear Interpolation
http://en.wikipedia.org/wiki/Bilinear_interpolation
![Page 30: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/30.jpg)
The Canny edge detector
thinning
(non-maximum suppression)
Problem:
pixels along
this edge
didn’t
survive the
thresholding
Source: K. Grauman
![Page 31: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/31.jpg)
Hysteresis thresholding
• Use a high threshold to start edge curves,
and a low threshold to continue them.
Source: S. Seitz
![Page 32: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/32.jpg)
Hysteresis thresholding
original image
high threshold
(strong edges)
low threshold
(weak edges)
hysteresis threshold
Source: L. Fei-Fei
![Page 33: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/33.jpg)
Hysteresis thresholding
original image
high threshold
(strong edges)
low threshold
(weak edges)
hysteresis threshold
Source: L. Fei-Fei
![Page 34: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/34.jpg)
Effect of (Gaussian kernel spread/size)
Canny with Canny with original
The choice of depends on desired behavior• large detects large scale edges
• small detects fine features
Source: S. Seitz
![Page 35: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/35.jpg)
Low-level edges vs. perceived contours
• Berkeley segmentation database:http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/
image human segmentation gradient magnitude
Source: L. Lazebnik
![Page 36: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/36.jpg)
How can we do better?
So far, we have only considered change in intensity as a cue for the
existence of an edge.
![Page 37: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/37.jpg)
pB Boundary Detector
Martin, Fowlkes, Malik 2004: Learning to Detection
Natural Boundaries…
http://www.eecs.berkeley.edu/Research/Projects/C
S/vision/grouping/papers/mfm-pami-boundary.pdf
Figure from Fowlkes
![Page 38: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/38.jpg)
pB Boundary Detector
Figure from Fowlkes
![Page 39: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/39.jpg)
Brightness
Color
Texture
Combined
Human
![Page 40: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/40.jpg)
Results
Human (0.95)
Pb (0.88)
Source: D. Hoiem
![Page 41: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/41.jpg)
Results
Human
Pb
Human (0.96)
Global PbPb (0.88)
Source: D. Hoiem
![Page 42: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/42.jpg)
Human (0.95)
Pb (0.63)
Source: D. Hoiem
![Page 43: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/43.jpg)
Human (0.90)
Pb (0.35)
For more:
http://www.eecs.berkeley.edu/Research/Projects
/CS/vision/bsds/bench/html/108082-color.html
![Page 44: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/44.jpg)
The goals of segmentation
Separate image into coherent “objects”image human segmentation
Source: L. Lazebnik
![Page 45: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/45.jpg)
The goals of segmentation
Separate image into coherent “objects”
Group together similar-looking pixels for
efficiency of further processing
X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003.
“superpixels”
Source: L. Lazebnik
![Page 46: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/46.jpg)
Types of segmentations
Oversegmentation Undersegmentation
Multiple Segmentations
Source: D. Hoiem
![Page 47: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/47.jpg)
Major processes for segmentation
Bottom-up: group tokens with similar features
Top-down: group tokens that likely belong to the
same object
[Levin and Weiss 2006]Source: D. Hoiem
![Page 48: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/48.jpg)
[Figure by J. Shi]
[http://poseidon.csd.auth.gr/LAB_RESEARCH/Latest/imgs/S
peakDepVidIndex_img2.jpg]
Determine image regions
Group video frames into shots
Fg / Bg
[Figure by Wang & Suter]
Object-level grouping
Figure-ground
[Figure by Grauman & Darrell]
Source: K. Grauman
Examples of grouping in vision
![Page 49: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/49.jpg)
Goals:
• Gather features that belong together
• Obtain an intermediate representation that compactly
describes key image (video) parts
Hard to measure success
• What is interesting depends on the application
Adapted from K. Grauman
Grouping in vision
![Page 50: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/50.jpg)
Gestalt cues for grouping
Gestalt (psychology):• Whole is greater than sum of its parts
• Relationships among parts can yield new properties/features
Psychologists identified series of factors that
predispose set of elements to be grouped (by
human visual system)
Good intuition and basic principles for grouping
• Some (e.g., symmetry) are difficult to implement in
practice
Adapted from K. Grauman, D. Hoiem
![Page 51: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/51.jpg)
Source: K. Grauman
Gestaltism: We perceive the interpretation
![Page 52: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/52.jpg)
The Muller-Lyer illusion
Gestaltism: We perceive the interpretation
Source: D. Hoiem
![Page 53: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/53.jpg)
Source: K. Grauman
Continuity, explanation by occlusion
Gestaltism: We perceive the interpretation
![Page 54: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/54.jpg)
Source: D. Forsyth
Gestaltism: We perceive the interpretation
![Page 55: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/55.jpg)
Principles of perceptual organization
From Steve Lehar: The Constructive Aspect of Visual PerceptionSource: D. Hoiem
![Page 56: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/56.jpg)
Similarity
http://chicagoist.com/attachments/chicagoist_alicia/GEESE.jpg, http://wwwdelivery.superstock.com/WI/223/1532/PreviewComp/SuperStock_1532R-0831.jpg
![Page 57: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/57.jpg)
Common fate
Image credit: Arthus-Bertrand (via F. Durand)
Source: K. Grauman
![Page 58: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/58.jpg)
Proximity
http://www.capital.edu/Resources/Images/outside6_035.jpg
![Page 59: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/59.jpg)
Segmentation and grouping
• Inspiration from human perception
– Gestalt properties
• Bottom-up segmentation via clustering
– Features: color, texture, …
– Algorithms:
• Mode finding and mean shift: k-means, mean-shift
• Graph-based: normalized cuts
Source: K. Grauman
![Page 60: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/60.jpg)
intensity
pix
el
co
un
t
input image
black pixelsgray
pixels
white
pixels
• These intensities define the three groups.
• We could label every pixel in the image according to
which of these primary intensities it is.
• i.e., segment the image based on the intensity feature.
• What if the image isn’t quite so simple?
1 23
Image segmentation: toy example
Source: K. Grauman
![Page 61: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/61.jpg)
intensity
pix
el
co
un
t
input image
input imageintensity
pix
el
co
un
t
Source: K. Grauman
![Page 62: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/62.jpg)
input imageintensity
pix
el
co
un
t
• Now how to determine the three main intensities that
define our groups?
• We need to cluster.
Source: K. Grauman
![Page 63: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/63.jpg)
0 190 255
• Goal: choose three “centers” as the representative
intensities, and label every pixel according to which of
these centers it is nearest to.
• Best cluster centers are those that minimize SSD
between all points and their nearest cluster center ci:
1 23
intensity
Source: K. Grauman
![Page 64: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/64.jpg)
Clustering
• With this objective, it is a “chicken and egg” problem:
– If we knew the cluster centers, we could allocate
points to groups by assigning each to its closest center.
– If we knew the group memberships, we could get the
centers by computing the mean per group.
Source: K. Grauman
![Page 65: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/65.jpg)
K-means clustering
• Basic idea: randomly initialize the k cluster centers, and
iterate between the two steps we just saw.
1. Randomly initialize the cluster centers, c1, ..., cK
2. Given cluster centers, determine points in each cluster
• For each point p, find the closest ci. Put p into cluster i
3. Given points in each cluster, solve for ci
• Set ci to be the mean of points in cluster i
4. If ci have changed, repeat Step 2
Properties• Will always converge to some solution
• Can be a “local minimum”
• does not always find the global minimum of objective function:
Source: Steve Seitz
![Page 66: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/66.jpg)
Source: A. Moore
![Page 67: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/67.jpg)
Source: A. Moore
![Page 68: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/68.jpg)
Source: A. Moore
![Page 69: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/69.jpg)
Source: A. Moore
![Page 70: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/70.jpg)
Source: A. Moore
![Page 71: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/71.jpg)
K-means converges to a local minimum
James Hays
![Page 72: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/72.jpg)
K-means clustering
• Matlab demo
http://www.cs.pitt.edu/~kovashka/cs1699/kmea
ns_demo.m
• Java demos:
http://kovan.ceng.metu.edu.tr/~maya/kmeans/inde
x.html
http://home.dei.polimi.it/matteucc/Clustering/tutoria
l_html/AppletKM.html
![Page 73: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/73.jpg)
K-means: pros and cons
Pros• Simple, fast to compute
• Converges to local minimum of within-cluster squared error
Cons/issues• Setting k?
• Sensitive to initial centers
• Sensitive to outliers
• Detects spherical clusters
• Assuming means can be computed
Source: K. Grauman
![Page 74: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/74.jpg)
An aside: Smoothing out cluster
assignments• Assigning a cluster label per pixel may yield outliers:
1 2
3
?
original labeled by cluster center’s
intensity
• How to ensure they are
spatially smooth?
Source: K. Grauman
![Page 75: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/75.jpg)
Segmentation as clustering
Depending on what we choose as the feature space, we
can group pixels in different ways.
Grouping pixels based
on intensity similarity
Feature space: intensity value (1-d)
Source: K. Grauman
![Page 76: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/76.jpg)
K=2
K=3
quantization of the feature space;
segmentation label map
Source: K. Grauman
![Page 77: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/77.jpg)
Segmentation as clustering
R=255
G=200
B=250
R=245
G=220
B=248
R=15
G=189
B=2
R=3
G=12
B=2R
G
B
Feature space: color value (3-d) Source: K. Grauman
Depending on what we choose as the feature space, we
can group pixels in different ways.
Grouping pixels based
on color similarity
![Page 78: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/78.jpg)
Segmentation as clustering
Depending on what we choose as the feature space, we
can group pixels in different ways.
Grouping pixels based
on intensity similarity
Clusters based on intensity
similarity don’t have to be spatially
coherent.
Source: K. Grauman
![Page 79: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/79.jpg)
Segmentation as clustering
X
Y
Intensity
Both regions are black, but if we
also include position (x,y), then
we could group the two into
distinct segments; way to encode
both similarity & proximity.Source: K. Grauman
Grouping pixels based
on intensity+position similarity
Depending on what we choose as the feature space, we
can group pixels in different ways.
![Page 80: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/80.jpg)
• Color, brightness, position alone are not
enough to distinguish all regions…
Source: L. Lazebnik
Segmentation as clustering
![Page 81: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/81.jpg)
Segmentation as clustering
Depending on what we choose as the feature space, we
can group pixels in different ways.
F24
Grouping pixels based
on texture similarity
F2
Feature space: filter bank responses (e.g., 24-d)
F1
…
Filter bank
of 24 filters
Source: K. Grauman
![Page 82: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/82.jpg)
Segmentation w/ texture features• Find “textons” by clustering vectors of filter bank outputs
• Describe texture in a window based on texton histogram
Malik, Belongie, Leung and Shi, IJCV 2001
Texton mapImage
Texton index Texton index
Cou
nt
Cou
nt
Source: L. Lazebnik
Cou
nt
Texton index
![Page 83: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/83.jpg)
Image segmentation example
Source: K. Grauman
![Page 84: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/84.jpg)
K-means: pros and cons
Pros• Simple, fast to compute
• Converges to local minimum of within-cluster squared error
Cons/issues• Setting k?
• Sensitive to initial centers
• Sensitive to outliers
• Detects spherical clusters
• Assuming means can be computed
Source: K. Grauman
![Page 85: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/85.jpg)
• The mean shift algorithm seeks modes or local
maxima of density in the feature space
Mean shift algorithm
imageFeature space
(L*u*v* color values)
Source: K. Grauman
![Page 86: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/86.jpg)
Kernel density estimation
Kernel
Data (1-D)
Estimated
density
Source: D. Hoiem
![Page 87: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/87.jpg)
Search
window
Center of
mass
Mean Shift
vector
Mean shift
Slide by Y. Ukrainitz & B. Sarel
![Page 88: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/88.jpg)
Search
window
Center of
mass
Mean Shift
vector
Mean shift
Slide by Y. Ukrainitz & B. Sarel
![Page 89: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/89.jpg)
Search
window
Center of
mass
Mean Shift
vector
Mean shift
Slide by Y. Ukrainitz & B. Sarel
![Page 90: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/90.jpg)
Search
window
Center of
mass
Mean Shift
vector
Mean shift
Slide by Y. Ukrainitz & B. Sarel
![Page 91: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/91.jpg)
Search
window
Center of
mass
Mean Shift
vector
Mean shift
Slide by Y. Ukrainitz & B. Sarel
![Page 92: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/92.jpg)
Search
window
Center of
mass
Mean Shift
vector
Mean shift
Slide by Y. Ukrainitz & B. Sarel
![Page 93: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/93.jpg)
Search
window
Center of
mass
Mean shift
Slide by Y. Ukrainitz & B. Sarel
![Page 94: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/94.jpg)
Points in same cluster converge
Source: D. Hoiem
![Page 95: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/95.jpg)
• Cluster: all data points in the attraction basin
of a mode
• Attraction basin: the region for which all
trajectories lead to the same mode
Mean shift clustering
Slide by Y. Ukrainitz & B. Sarel
![Page 96: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/96.jpg)
• Compute features for each point (color, texture, etc)
• Initialize windows at individual feature points
• Perform mean shift for each window until convergence
• Merge windows that end up near the same “peak” or mode
Mean shift clustering/segmentation
Source: D. Hoiem
![Page 97: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/97.jpg)
http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html
Mean shift segmentation results
![Page 98: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/98.jpg)
Mean shift segmentation results
![Page 99: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/99.jpg)
Mean shift
• Pros:– Does not assume shape on clusters
– One parameter choice (window size)
– Generic technique
– Find multiple modes
– Robust to outliers
• Cons:– Selection of window size
Adapted from K. Grauman
![Page 100: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/100.jpg)
Mean-shift reading
• Nicely written mean-shift explanation (with math)http://saravananthirumuruganathan.wordpress.com/2010/04/01/introduction-to-mean-shift-algorithm/
• Includes .m code for mean-shift clustering
• Mean-shift paper by Comaniciu and Meerhttp://www.caip.rutgers.edu/~comanici/Papers/MsRobustApproach.pdf
• Adaptive mean shift in higher dimensionshttp://mis.hevra.haifa.ac.il/~ishimshoni/papers/chap9.pdf
Source: K. Grauman
![Page 101: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/101.jpg)
q
Images as graphs
Fully-connected graph
• node (vertex) for every pixel
• link between every pair of pixels, p,q
• affinity weight wpq for each link (edge)
– wpq measures similarity
» similarity is inversely proportional to difference (in color and position…)
p
wpq
w
Source: Steve Seitz
![Page 102: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/102.jpg)
Segmentation by Graph Cuts
Break Graph into Segments
• Want to delete links that cross between segments
• Easiest to break links that have low similarity (low weight)
– similar pixels should be in the same segments
– dissimilar pixels should be in different segments
w
A B C
Source: Steve Seitz
q
p
wpq
![Page 103: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/103.jpg)
Cuts in a graph: Min cut
Link Cut
• set of links whose removal makes a graph disconnected
• cost of a cut:
AB
Find minimum cut• gives you a segmentation
• fast algorithms exist for doing this
Source: Steve Seitz
BqAp
qpwBAcut,
,),(
![Page 104: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/104.jpg)
Clustering Strategies
• K-means– Iteratively re-assign points to the nearest cluster
center
• Mean-shift clustering– Estimate modes
• Graph cuts– Split the nodes in a graph based on assigned links with
similarity weights
• Agglomerative clustering– Start with each point as its own cluster and iteratively
merge the closest clusters
![Page 105: CS 1674: Intro to Computer Visionkovashka/cs1674_fa16/vision_12_edges... · 2. Given cluster centers, determine points in each cluster • For each point p, find the closest c i](https://reader036.vdocuments.us/reader036/viewer/2022070111/604e6e2a3b9d6f314a2ab61a/html5/thumbnails/105.jpg)
Why do we cluster?
• Summarizing data– Look at large amounts of data– Represent a large continuous vector with the cluster number
• Counting– Histograms of texture, color, SIFT vectors
• Segmentation– Separate the image into different mid-level regions– Find object boundaries
• Prediction– Images in the same cluster may have the same labels
Slide credit: J. Hays, D. Hoiem