lec09 stereo mvs - university of california, san...
TRANSCRIPT
![Page 1: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/1.jpg)
Lecture9:StereoandMVS
CSE152:ComputerVisionManmohanChandraker
![Page 2: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/2.jpg)
Recap
![Page 3: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/3.jpg)
epipolarlines
Disparity
(x1,y1) (x2,y1)
x2-x1=thedisparityofpixel(x1,y1)
Twoimagescapturedbyapurelyhorizontal translatingcamera(rectifiedstereopair)
Estimatingthedisparityisequivalenttoestimatingdepth.
![Page 4: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/4.jpg)
Depth from disparity
f
x x’
baseline
z
C C’
X
f
![Page 5: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/5.jpg)
Your basic stereo algorithm
For each epipolar lineFor each pixel in the left image
• compare with every pixel on same epipolar line in right image• pick pixel with minimum match cost
Improvement: match windows
![Page 6: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/6.jpg)
Popular matching scores
• SSD (Sum Squared Distance)
• NCC (Normalized Cross Correlation)
where
• What advantages might NCC have over SSD?
![Page 7: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/7.jpg)
Plane sweep stereoRe-order (pixel and disparity) evaluation loops
for every pixel, for every disparityfor every disparity for every pixelcompute cost compute cost
![Page 8: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/8.jpg)
Stereo matching framework1. For every disparity, compute raw matching
costs
Why use a robust function?• occlusions, other outliers
Can also use alternative match criteria.
![Page 9: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/9.jpg)
Stereo matching framework2. Aggregate costs spatially
• Here, we are using a box filter(efficient moving averageimplementation)
• Can also use weighted average,or other filters
![Page 10: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/10.jpg)
Stereo matching
I(x, y) J(x, y)
y = 141
E(x, y, d) the disparity space image (DSI)x
d
![Page 11: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/11.jpg)
Stereo matching framework3. Choose winning disparity at each pixel
y = 141
E(x, y, d) the disparity space image (DSI)x
d
![Page 12: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/12.jpg)
Stereo matching framework3. Choose winning disparity at each pixel
4. Interpolate to sub-pixel accuracy
d
E(d)
d*
![Page 13: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/13.jpg)
width of a pixel
Choosingthestereobaseline
What’stheoptimalbaseline?
Large Baseline Small Baseline
![Page 14: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/14.jpg)
width of a pixel
Choosingthestereobaseline
What’stheoptimalbaseline?– Toosmall:largedeptherror– Toolarge:difficultsearchproblem
Large Baseline Small Baseline
all of thesepoints projectto the same pair of pixels
![Page 15: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/15.jpg)
Traditional Stereo MatchingAdvantages:
• gives detailed surface estimates• fast algorithms based on moving averages• sub-pixel disparity estimates and confidence
Limitations:• narrow baseline Þ noisy estimates• fails in textureless areas• gets confused near occlusion boundaries
![Page 16: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/16.jpg)
Stereo as energy minimization
What defines a good stereo correspondence?1. Match quality
– Want each pixel to find a good match in the other image2. Smoothness
– If two pixels are adjacent, they should (usually) move about the same amount
![Page 17: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/17.jpg)
Stereo as energy minimization
• Find disparity map d that minimizes an energy function
• Simple pixel or window matching
Match distance between windows I(x, y) and J(x + d(x,y), y)=
![Page 18: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/18.jpg)
Stereo as energy minimizationBetter objective function
match cost smoothness cost
Want each pixel to find a good match in the other image
Adjacent pixels should (usually) move about the same
amount
![Page 19: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/19.jpg)
Stereo as energy minimization
match cost:
smoothness cost:
4-connected neighborhood
8-connected neighborhood
: set of neighboring pixels
![Page 20: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/20.jpg)
Stereo matching 20
Energy minimization1-D example: approximating splines
zx,y
dx,y
Options forsmoothness
![Page 21: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/21.jpg)
RelaxationHow can we get the best solution?Differentiate energy function, set to 0
![Page 22: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/22.jpg)
RelaxationIteratively improve a solution by locally
minimizing the energy: relax to solution
zx,y
dx,ydx-1,y dx+1,y
![Page 23: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/23.jpg)
Graph cutsSolution technique for general 2D problem
![Page 24: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/24.jpg)
Graph cutsTwo different kinds of moves:
Compute best possible match within integer disparity.
![Page 25: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/25.jpg)
Smoothness cost
“Potts model”
L1 distance
How do we choose V?
![Page 26: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/26.jpg)
Depth Map Results
Input image Sum Abs Diff
Graph cuts
![Page 27: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/27.jpg)
CSE576,Spring2008 Stereomatching 27
Stereo evaluation
![Page 28: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/28.jpg)
CSE576,Spring2008 Stereomatching 28
Stereo—best algorithms
![Page 29: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/29.jpg)
Real-time stereo
Used for robot navigation (and other tasks)• Several software-based real-time stereo techniques have
been developed (most based on simple discrete search)
Nomad robot searches for meteorites in Antarticahttp://www.frc.ri.cmu.edu/projects/meteorobot/index.html
![Page 30: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/30.jpg)
Multi-viewStereo
Figures by Carlos Hernandez
Input:calibratedimagesfromseveralviewpointsOutput:3Dobjectmodel
![Page 31: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/31.jpg)
Stereo:anotherviewerror
depth
![Page 32: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/32.jpg)
56 Flickr images taken by 8 photographers
![Page 33: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/33.jpg)
![Page 34: lec09 stereo mvs - University of California, San Diegocseweb.ucsd.edu/~mkchandraker/classes/CSE152/Winter2019/Lectures/lec09_stereo_mvs.pdf• What advantages might NCC have over SSD?](https://reader030.vdocuments.us/reader030/viewer/2022040912/5e8625bc074dcb0419350a56/html5/thumbnails/34.jpg)
State-of-the-artinbinocularstereo
[Zbontar andLeCun,JMLR2016]