what we didn’t have time for cs664 lecture 26 thursday 12/02/04 some slides c/o dan huttenlocher,...
TRANSCRIPT
![Page 1: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/1.jpg)
What we didn’t have time for
CS664 Lecture 26Thursday 12/02/04
Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun
![Page 2: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/2.jpg)
Administrivia Final project is due at noon on
Friday 12/17 Write-up only (5MB max) Be sure to include some pictures
Send me email if you missed any quiz for a good reason
![Page 3: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/3.jpg)
Outline Geometry Graph-based segmentation Statistics
![Page 4: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/4.jpg)
Geometry
![Page 5: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/5.jpg)
Homogeneous coordinates
Identify a point in the image plane with ray passing through that point (pixel) (x,y) ´ ( x, y, ) for non-zero (X,Y,Z) ´ (X/Z,Y/Z,1) for non-zero Z
![Page 6: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/6.jpg)
Advantages Many non-linear operations
become linear in homogeneous coordinates Example: (X,Y,Z) projects to
(fX/Z,fY/Z)
2D point 3D point
3x4 camera
projection
![Page 7: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/7.jpg)
Camera projection matrix
![Page 8: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/8.jpg)
epipole
Epipolar geometry
epipolarplane
epipolar
line
Stefano Soatto (c) 2002
![Page 9: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/9.jpg)
Pencil of planes Different epipolar planes for
different scene points x Plane defined by camera origins + x
![Page 10: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/10.jpg)
Epipolar lines are important For pixel p in I
there is a corresponding epipolar line in I’ This allows us to
limit the search! Generalization of
stereo to arbitrary camera positions
Classical stereo has parallel cameras
p
![Page 11: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/11.jpg)
Example: verged stereo
![Page 12: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/12.jpg)
Examples: motion
Parallel toImage Plane
Forward
![Page 13: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/13.jpg)
Essential matrix E Ex is perpendicular to x’s epipolar
line in the other image So if x’ corresponds to x then
x’TEx = 0 Captures the scene geometry
We assume the cameras are calibrated
Otherwise we get the fundamental matrix
![Page 14: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/14.jpg)
Estimating the geometry The essential matrix has 5 parameters
Can estimate from 5 corresponding points
Fundamental matrix has 7 The question of “how few perfect
correspondences do you need” has spawned an unfortunately large literature
![Page 15: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/15.jpg)
Yet more optimization We can estimate the essential
matrix from a bunch of point matches
A similar technique can be used to compute structure from motion Bundle adjustment
![Page 16: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/16.jpg)
RANSAC (line fitting) Variant of generate-and-test Pick a small set of points at random Fit them via least squares Points “far” from this line are
outliers Repeat until you find a line with
very few outliers
![Page 17: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/17.jpg)
RANSAC (camera geometry) Pick a small set of corresponding
pixels At least 5 (essential) or 7
(fundamental) Compute the matrix from these See how many corresponding
pixels this matrix explains
![Page 18: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/18.jpg)
Graph-based Segmentation
![Page 19: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/19.jpg)
Segmentation by min cut
ImagePixels
w
SimilarityMeasure
MinimumCut
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 20: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/20.jpg)
Min cuts don’t segment well
Ideal Cut
Cuts with lesser weightthan the ideal cut
* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 21: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/21.jpg)
Normalized cuts Instead of the min cut, minimize
Measure of dis-similarity between the sets A and B NP-hard to minimize Rely on continuous approximation
VyBzVyAx
cut yzw
BAw
yxw
BAwBAN
,,,
,
,
,,
![Page 22: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/22.jpg)
Normalized cuts examples
![Page 23: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/23.jpg)
Limitations of normalized cuts Works by binary partitioning Slow and memory-intensive Textured backgrounds are
problems
![Page 24: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/24.jpg)
Other graph-based methods Many other variants on min cuts
Typical cuts, nested cuts, etc. No clear winner for segmentation
Perhaps mean shift?
![Page 25: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/25.jpg)
MST-based segmentation Minimum spanning tree is the
cheapest way to connect all pixels into a single component (or “region”)
Merge two components when the cheapest edge between them is cheap compared to a measure of the internal variation
Provably good segmentation under a fairly natural definition
Neither too coarse nor too fine
![Page 26: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/26.jpg)
Example output
Solves many problems with normalized cuts
![Page 27: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/27.jpg)
More statistics
![Page 28: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/28.jpg)
Dimensionality reduction
We can represent orange points only by their v1 coordinate
![Page 29: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/29.jpg)
Eigenfaces An n-pixel image
is a point in <n
Find low-dimensional representation of face images (from a training set)
Recognition by finding the closest face in face space
![Page 30: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/30.jpg)
Markov Random Fields
MRF defining property:
Hammersley-Clifford Theorem:
),|(Pr),|(Pr pqpqp Nqffpqff
),(
),( ),(exp~)(Prqp
qpqp ffVf
neighborhood relationships (n-links)
image pixels (vertices)
pf - disparity at pixel p
),...,( 1 mfff - configuration
![Page 31: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/31.jpg)
MAP estimation of an MRF
)Pr()|Pr(maxargˆ ffOff
p qp
qpqppp
f
ffVfOgf),(
),( ),()|(lnexpmaxargˆ
)|(Prmaxargˆ Offf
Observed data
Likelihoodfunction
(sensor noise)
Prior (MRF model)
Bayes rule
![Page 32: What we didn’t have time for CS664 Lecture 26 Thursday 12/02/04 Some slides c/o Dan Huttenlocher, Stefano Soatto, Sebastian Thrun](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e605503460f94b5b9d4/html5/thumbnails/32.jpg)
Energy minimization
),(
),( ),()|(ln)(qp
qpqp
p
pp ffVfOgfE
Data term
(sensor noise)
Smoothness term
(MRF prior)