stereo class 7 read chapter 7 of tutorial tsukuba dataset
TRANSCRIPT
![Page 1: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/1.jpg)
Stereo Class 7
Read Chapter 7 of tutorial
http://cat.middlebury.edu/stereo/
Tsukuba dataset
![Page 2: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/2.jpg)
Geometric Computer Vision course schedule(tentative)
Lecture Exercise
Sept 16 Introduction -
Sept 23 Geometry & Camera model Camera calibration
Sept 30 Single View Metrology(Changchang Wu)
Measuring in images
Oct. 7 Feature Tracking/Matching Correspondence computation
Oct. 14 Epipolar Geometry F-matrix computation
Oct. 21 Shape-from-Silhouettes Visual-hull computation
Oct. 28 Stereo matching papers
Nov. 4 Structure from motion and visual SLAM
Project proposals
Nov. 11 Multi-view geometry and self-calibration
Papers
Nov. 18 Shape-from-X Papers
Nov. 25 Structured light and active range sensing
Papers
Dec. 2 3D modeling, registration and range/depth fusion
(Christopher Zach?)
Papers
Dec. 9 Appearance modeling and image-based rendering
Papers
Dec. 16 Final project presentations Final project presentations
![Page 3: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/3.jpg)
Stereo
• Standard stereo geometry• Stereo matching
• Aggregation• Optimization (1D, 2D)
• General camera configuration• Rectifications• Plane-sweep
• Multi-view stereo
![Page 4: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/4.jpg)
Stereo
![Page 5: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/5.jpg)
6
Challenges
• Ill-posed inverse problem• Recover 3-D structure from 2-D
information
• Difficulties• Uniform regions• Half-occluded pixels
![Page 6: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/6.jpg)
7
Pixel Dissimilarity• Absolute difference of intensities
c=|I1(x,y)- I2(x-d,y)|• Interval matching [Birchfield 98]
• Considers sensor integration• Represents pixels as intervals
![Page 7: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/7.jpg)
8
Alternative Dissimilarity Measures
• Rank and Census transforms [Zabih ECCV94]
• Rank transform:• Define window containing R pixels around each
pixel• Count the number of pixels with lower
intensities than center pixel in the window• Replace intensity with rank (0..R-1)• Compute SAD on rank-transformed images
• Census transform: • Use bit string, defined by neighbors, instead of
scalar rank
• Robust against illumination changes
![Page 8: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/8.jpg)
9
Rank and Census Transform Results
• Noise free, random dot stereograms• Different gain and bias
![Page 9: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/9.jpg)
10
Systematic Errors of Area-based Stereo
• Ambiguous matches in textureless regions
• Surface over-extension [Okutomi IJCV02]
![Page 10: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/10.jpg)
11
Surface Over-extension• Expected value of E[(x-y)2]
for x in left and y in right image is:
Case A: σF2+ σB
2+(μF- μB)2 for
w/2-λ pixels in each row
Case B: 2 σB2 for w/2+λ pixels in
each row
Right image
Left image
Disparity of back surface
![Page 11: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/11.jpg)
12
Surface Over-extension• Discontinuity
perpendicular to epipolar lines
• Discontinuity parallel to epipolar lines
Right image
Left image
Disparity of back surface
![Page 12: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/12.jpg)
13
Over-extension and shrinkage
• Turns out that:
for discontinuities perpendicular to epipolar lines
• And:
for discontinuities parallel to epipolar lines
26
ww
22
ww
![Page 13: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/13.jpg)
14
Random Dot Stereogram Experiments
![Page 14: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/14.jpg)
15
Random Dot Stereogram Experiments
![Page 15: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/15.jpg)
16
Offset Windows
Equivalent to using minnearby costResult: loss of depth accuracy
![Page 16: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/16.jpg)
17
Discontinuity Detection
• Use offset windows only where appropriate• Bi-modal distribution of SSD • Pixel of interest different than
mode within window
![Page 17: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/17.jpg)
18
Compact Windows
• [Veksler CVPR03]: Adapt windows size based on:• Average matching error per pixel• Variance of matching error• Window size (to bias towards larger
windows)
• Pick window that minimizes cost
![Page 18: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/18.jpg)
19
Integral Image
Sum of shaded part
Compute an integral image for pixel dissimilarity at each possible disparity
A C
DB
Shaded area = A+D-B-CIndependent of size
![Page 19: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/19.jpg)
20
Results using Compact Windows
![Page 20: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/20.jpg)
21
Rod-shaped filters
• Instead of square windows aggregate cost in rod-shaped shiftable windows [Kim CVPR05]
• Search for one that minimizes the cost (assume that it is an iso-disparity curve)
• Typically use 36 orientations
![Page 21: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/21.jpg)
22
Locally Adaptive Support
Apply weights to contributions of neighboringpixels according to similarity and proximity [Yoon CVPR05]
![Page 22: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/22.jpg)
23
Locally Adaptive Support
• Similarity in CIE Lab color space:
• Proximity: Euclidean distance
• Weights:
![Page 23: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/23.jpg)
24
Locally Adaptive Support: Results
Locally Adaptive Support
![Page 24: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/24.jpg)
25
Locally Adaptive Support: Results
![Page 25: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/25.jpg)
Occlusions
(Slide from Pascal Fua)
![Page 26: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/26.jpg)
Exploiting scene constraints
![Page 27: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/27.jpg)
Ordering constraint
11 22 33 4,54,5 66 11 2,32,3 44 55 66
2211 33 4,54,5 6611
2,32,3
44
55
66
surface slicesurface slice surface as a pathsurface as a path
occlusion right
occlusion left
![Page 28: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/28.jpg)
Uniqueness constraint
• In an image pair each pixel has at most one corresponding pixel• In general one corresponding pixel• In case of occlusion there is none
![Page 29: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/29.jpg)
Disparity constraint
surface slicesurface slice surface as a pathsurface as a path
bounding box
dispa
rity b
and
use reconstructed features to determine bounding box
constantdisparitysurfaces
![Page 30: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/30.jpg)
Stereo matching
Optimal path(dynamic programming )
Similarity measure(SSD or NCC)
Constraints• epipolar
• ordering
• uniqueness
• disparity limit
Trade-off
• Matching cost (data)
• Discontinuities (prior)
Consider all paths that satisfy the constraints
pick best using dynamic programming
![Page 31: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/31.jpg)
Hierarchical stereo matching
Dow
nsam
plin
g
(Gau
ssia
n p
yra
mid
)
Dis
pari
ty p
rop
ag
ati
on
Allows faster computation
Deals with large disparity ranges
![Page 32: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/32.jpg)
Disparity map
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
![Page 33: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/33.jpg)
Example: reconstruct image from neighboring
images
![Page 34: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/34.jpg)
![Page 35: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/35.jpg)
36
Semi-global optimization
• Optimize: E=Edata+E(|Dp-Dq|=1)+E(|Dp-Dq|>1) [Hirshmüller CVPR05]• Use mutual information as cost
• NP-hard using graph cuts or belief propagation (2-D optimization)
• Instead do dynamic programming along many directions • Don’t use visibility or ordering constraints• Enforce uniqueness• Add costs
![Page 36: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/36.jpg)
37
Results of Semi-global optimization
![Page 37: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/37.jpg)
38
Results of Semi-global optimization
No. 1 overall in Middlebury evaluation(at 0.5 error threshold as of Sep. 2006)
![Page 38: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/38.jpg)
Energy minimization
(Slide from Pascal Fua)
![Page 39: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/39.jpg)
Graph Cut
(Slide from Pascal Fua)
(general formulation requires multi-way cut!)
![Page 40: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/40.jpg)
(Boykov et al ICCV‘99)
(Roy and Cox ICCV‘98)
Simplified graph cut
![Page 41: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/41.jpg)
Belief Propagation
per pixel +left +right +up +down
2 3 4 5 20...subsequent iterations
(adapted from J. Coughlan slides)
first iteration
Belief of one node about another gets propagated through messages (full pdf, not just most likely state)
![Page 42: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/42.jpg)
![Page 43: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/43.jpg)
Stereo matching with general camera configuration
![Page 44: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/44.jpg)
Image pair rectification
![Page 45: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/45.jpg)
Planar rectification
Bring two views Bring two views to standard stereo setupto standard stereo setup
(moves epipole to )(not possible when in/close to image)
~ image size
(calibrated)(calibrated)
Distortion minimization(uncalibrated)
![Page 46: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/46.jpg)
![Page 47: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/47.jpg)
Polar re-parameterization around epipolesRequires only (oriented) epipolar geometryPreserve length of epipolar linesChoose so that no pixels are compressed
original image rectified image
Polar rectification(Pollefeys et al. ICCV’99)
Works for all relative motionsGuarantees minimal image size
![Page 48: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/48.jpg)
polarrectification
planarrectification
originalimage pair
![Page 49: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/49.jpg)
Example: Béguinage of Leuven
Does not work with standard Homography-based approaches
![Page 50: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/50.jpg)
Example: Béguinage of Leuven
![Page 51: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/51.jpg)
Stereo camera configurations
(Slide from Pascal Fua)
![Page 52: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/52.jpg)
Multi-camera configurations
Okutami and Kanade
(illustration from Pascal Fua)
![Page 53: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/53.jpg)
• Multi-baseline, multi-resolution• At each depth, baseline and
resolution selected proportional to that depth
• Allows to keep depth accuracy constant!
Variable Baseline/Resolution Stereo (Gallup et al., CVPR08)
![Page 54: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/54.jpg)
Variable Baseline/Resolution Stereo: comparison
![Page 55: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/55.jpg)
Multi-view depth fusion
• Compute depth for every pixel of reference image• Triangulation• Use multiple views• Up- and down
sequence• Use Kalman filter
(Koch, Pollefeys and Van Gool. ECCV‘98)
Allows to compute robust texture
![Page 56: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/56.jpg)
Plane-sweep multi-view matching
• Simple algorithm for multiple cameras• no rectification necessary• doesn’t deal with occlusions
Collins’96; Roy and Cox’98 (GC); Yang et al.’02/’03 (GPU)
![Page 57: Stereo Class 7 Read Chapter 7 of tutorial Tsukuba dataset](https://reader037.vdocuments.us/reader037/viewer/2022103101/56649ebf5503460f94bcab16/html5/thumbnails/57.jpg)
Next class: structured light and active scanning