lecture 5. dense reconstruction and tracking with real ... · •compute the photo-metric data-term...
TRANSCRIPT
![Page 1: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/1.jpg)
Lecture 5.Dense Reconstruction and Tracking
with Real-Time Applications
Part 2: Geometric ReconstructionDr Richard Newcombe and Dr Steven Lovegrove
Slide content developed from:
[Newcombe, “Dense Visual SLAM”, 2015] [Lovegrove, “Parametric Dense Parametric SLAM”]
and [Szeliski, Seitz, Zitnick UW CSE576 CV lectures]
![Page 2: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/2.jpg)
Geometric ReconstructionDense reconstruction of scene geometry
![Page 3: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/3.jpg)
Stereo and Constrained Correspondence
Basic Principle: Triangulation• Gives reconstruction as intersection of two rays
Requires
– camera pose (calibration)
– point correspondence (e.g. feature extraction and matching)
![Page 4: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/4.jpg)
Dense Scene Geometry Generative Model
I2 (x,y) = I1( π( T12 K-1 D2 (x,y) [x,y] ) )
I2, D2I1
![Page 5: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/5.jpg)
Special Case for a Rectified Stereo Image Pair
I1 I2Depth Image (D2)
![Page 6: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/6.jpg)
Disparity for Rectified Stereo Pairs
I1 I2Disparity Image (d2)
Rectified Stereo generative model with Brightness Constancy:
I2(x,y) = I1 (x + d2(x,y), y)
![Page 7: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/7.jpg)
Stereo Correspondence as energy minimization
I1(x, y) I2(x, y)
y = 141
C(x, y, d); the disparity space image (DSI)x
d
Pixel Error:e(x, y, d) = I1 (x + d , y) – I2 (x,y)
Cost (with quadratic penalty):C(x, y, d) = ( I1 (x + d , y) – I2 (x,y))2
![Page 8: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/8.jpg)
Stereo as energy minimization
y = 141
x
d
Simple pixel / window matching: choose the minimum of each column in the DSI independently:
![Page 9: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/9.jpg)
Aggregation window, error and cost functions
Ground truth SAD W=11SAD W=3 SAD W=25
Effect of window size (W) for aggregating the photometric cost:
![Page 10: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/10.jpg)
Similarity Measure Formula
Sum of Absolute Differences (SAD)
Sum of Squared Differences (SSD)
Zero-mean SAD
Locally scaled SAD
Normalized Cross Correlation (NCC)
Aggregation window, error and cost functions
The design of the cost function, including window size for aggregation, image error function and penalty can improve quality of correspondence:
![Page 11: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/11.jpg)
More Advanced Aggregation Functions
Adaptive Support-Weight Approach for Correspondence Search
[Yoon and Kweon, 2006]
Ground truth SAD, W=11
![Page 12: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/12.jpg)
Plane Sweep for Multiple view aggregation
• How to Integrate more information from Multiple Views?
[Newcombe, 2013]
![Page 13: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/13.jpg)
Plane Sweep for Multiple view aggregation
• Compute the photo-metric data-term between a reference frame and all available frames
• Integrate photo-metric costs into a single (3D) voxel volume
• Use a Plane Induced Homography to efficiently transfer pixels:
𝑢′𝑣′1
= 𝜋 𝐾 𝑅 −𝑡 ⋅ 𝑛T
𝑑
′
𝐾T𝑢𝑣1
[Extracted from “DTAM: Dense Tracking and Mapping in Real-Time. Newcombe, Lovegrove, Davison, 2011]
![Page 14: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/14.jpg)
Switch to live coding demo of Plane Sweep
● And take a break!
![Page 15: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/15.jpg)
Effect of Stereo Baseline
Recap Stereo Steps• Calibrate cameras
• Compute disparity (d)
• Estimate depth (z)
For Rectified Stereo:
Baseline (b)
Scene Depth (z)
Focal Length (f)
![Page 16: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/16.jpg)
Choosing the Baseline
What’s the optimal baseline?• Too small: large depth error
• Too large: difficult search problem
Large Baseline Small Baseline
Error in Z :
![Page 17: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/17.jpg)
Effect of Baseline on Estimation
[Okutomi 1993]
![Page 18: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/18.jpg)
Effect of Baseline on Estimation
1/z
Matching Score
Small Baseline
1/z
Matching Score
Large Baseline
![Page 19: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/19.jpg)
[Okutomi 1993]
![Page 20: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/20.jpg)
Variable Baseline/Resolution Stereo
[Gallup et al]
![Page 21: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/21.jpg)
Multiple Baseline Stereo
Basic Approach• Choose a reference view• Use your favorite stereo algorithm BUT
• replace two-view SSD with SSD over all baselines
• Optimally chose a set of images to maintain a constant compute or error metric
Limitations• Which is the best reference view?• Visibility: how to select which frames have scene co-visibility?
[Kang, Szeliski, Chai, CVPR’01]
![Page 22: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/22.jpg)
Image Modelling and DenoisingEstimating scene geometry with constraints
![Page 23: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/23.jpg)
Denoising Data
Can we recover D given the noisy version?
![Page 24: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/24.jpg)
How to reduce the noise in the Depth Images?
Independent Gaussian
Noise
![Page 25: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/25.jpg)
Smoothing – e.g. apply image filtering?
• Where are the sharp edges from the buildings?• Mean filtering doesn’t take into expected image structures
Mean Filtered version of (b)
![Page 26: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/26.jpg)
Gradients of Expected Depth Image
Mean Filtered
version of Noisy
Depth Image
Depth Image
![Page 27: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/27.jpg)
Priors: What do the images look like in gradient space?
Generalized Gaussian Distribution:
Histogram of –Log for the Image gradient dI/dx for visible light
Histogram of –Log for the Image gradient dD/dx for Depth Image of the scene
• We can use statistics of image derivatives in expected data• E.G what is the distribution of image gradients in a passive or depth image?
![Page 28: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/28.jpg)
The probability of the depth image?
Data Term Likelihood assuming Independent Gaussian Noise:
Smoothness Prior w/ Gaussian Dist. Over 1st Order Image Gradients:
![Page 29: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/29.jpg)
The probability of the depth image?
Recap Bayesian Inference given a distribution Likelihood and Prior:
Searching for the maximum a posteriori estimate Depth Image:
arg max
arg max
arg max
![Page 30: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/30.jpg)
Depth Denoising by Energy Minimization
Transform to Energy, E(D), minimization problem using –Log:
Energy, E(D):
Solve this minimization problem using: Gradient Descent, Quasi Newton Methods or Discrete Optimization techniques
Where λ combines factors relating to the variances ν2 , σ2.
[Newcombe, 2013]
![Page 31: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/31.jpg)
Comparison of Image Priors in Denoising2 1
![Page 32: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/32.jpg)
Correspondence as Global Energy Minimization• This denoising approach can be directly applied to correspondence (e.g. depth):
{ {
Match Cost (Data Term) Smoothness Cost (Priors over solution space)Want each pixel to find a good
match in the other imageBias the search towards
realistic solutions
• Many matching costs and Priors see [ Scharstein & R. Szeliski ] [D. Scharstein, Middlebury Stereo Evaluation, www.middlebury.edu/stereo] :
• Can be computationally expensive• There are reasonable alternatives to Full Global Optimization, see [Hirshmüller CVPR05]
![Page 33: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/33.jpg)
Scene ReconstructionFrom Image space depth to complete geometric models
![Page 34: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/34.jpg)
Problem: How to Combine Depth Images into a Complete Model?
[Extracted from KinectFusion. Newcombe et al, 2011]
![Page 35: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/35.jpg)
Reconstructed
Surfaces
Merging depth maps
Depth map 1 Depth map 2 Combination (Union)
• Naïve combination (union) produces artifacts
• Better solution: find “average” surface• Surface that minimizes sum (of squared) distances to the depth maps
[From Curless & Levoy, 1996]
![Page 36: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/36.jpg)
Least squares surface solution
E( f ) di
2
i1
N
(x, f ) dx[Slide from Seitz, UW CSEP576]
![Page 37: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/37.jpg)
Representing Geometry Implicitly
Signed Distance Functions
![Page 38: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/38.jpg)
Example: Truncated Signed Distance Function (TSDF)
[Newcombe, 2015]
![Page 39: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/39.jpg)
Representing Scenes with TSDF
[KinectFusion, Newcombe et al, 2011]
![Page 40: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/40.jpg)
A Single Ray Observation in TSDF
![Page 41: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/41.jpg)
Ray Observations in TSDF
True Scene DepthSDF Value
0
1
2
3
-3
-2
-1Distance along ray
![Page 42: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/42.jpg)
Fusing Noisy Ray Observations in TSDF
True Scene DepthSDF Value
0
1
2
3
-3
-2
-1Distance along ray
Noisy Depth Measurements
Average of the Noisy
Measurements
Estimated Scene Depth
![Page 43: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/43.jpg)
VRIP [Curless & Levoy 1996]
depth map 1 depth map 2 combination
signed
distance
function
isosurface
extraction
![Page 44: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/44.jpg)
Michael Goesele
16 images (ring)47 images (ring)
Merging Depth Maps: Temple Model
317 images
(hemisphere)input image ground truth model
Goesele, Curless, Seitz, 2006
![Page 45: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/45.jpg)
Global Photometric Volume Optimization• Instead of fusing noisy depth maps into a volume
• Compute the photo-metric data-term for co-visible pairs of frames
• Integrate the photo-metric costs into a single (3D) voxel volume
• Define the total cost function with a surface regularization term
• Minimize the Global cost of the ‘surface’ that passes through the volume
![Page 46: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/46.jpg)
Application: Multi-view stereo from Internet Collections
[Goesele, Snavely, Curless, Hoppe, Seitz, ICCV 2007]
![Page 47: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/47.jpg)
Voxel Coloring Algorithms [Seitz & Dyer]
• The space of possible Volumetric Scene Reconstructions
• These Approaches obtain voxel Coloring that ‘generate’ the observed images
Discretized
Scene Volume
N3 voxels
C colors
All Scenes (CN3)
Photo-Consistent
Scenes
TrueScene
[Slide from Steve Seitz]
![Page 48: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/48.jpg)
Example: Reconstruction from Silhouettes (C=2)
Binary Images
• Back-project each silhouette
• Intersect back-projected volumes
• How can we get Shape Silhouettes?
[Slide from Steve Seitz]
![Page 49: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/49.jpg)
Volume intersection
Reconstruction Contains the True Scene• But is generally not the same
• In the limit (all views) get visual hull
[Slide from Steve Seitz]
![Page 50: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/50.jpg)
Voxel algorithm for volume intersection
Color voxel black if on silhouette in every image
• for M images, N3 voxels
• Don’t have to search 2N3 possible scenes!
• Useful for reconstructions from Green Screen
O( ? ),
[Slide from Steve Seitz]
![Page 51: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/51.jpg)
Properties of Volume Intersection
Pros• Easy to implement, fast• Accelerated via octrees [Szeliski 1993]
Cons• No concavities• Reconstruction is not photo-consistent• Requires identification of silhouettes
More General Cases (Color images, general cameras):• Voxel Coloring [Seitz and Dyer]• Space Carving [Kutulakos and Seitz]
![Page 52: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/52.jpg)
Applications of Direct Methods:Real-Time Mapping and TrackingUsing Passive and RGB-D sensors
![Page 53: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/53.jpg)
KinectFusion: Dense Surface Tracking and Mapping in Real-Time
• Uses an RGB-D Sensor
• First Dense SLAM System
• Interleaves:1. TSDF Fusion (Map)
2. Projective ICP (Track)
• Efficient to implement on GPU Compute Architecture
• Memory for Scene is O(N^3)
Newcombe, Izadi et al
![Page 54: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/54.jpg)
Real-Time 3D Reconstruction at Scale using Voxel Hashing
[Niesner, Zollhofer, Izadi, Stamminger]
• Extends KinectFusionmethods to work over very large volumes
• Very Efficient <O(N^3) Memory!
![Page 55: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/55.jpg)
ElasticFusion: Dense SLAM Without A Pose Graph
• Uses Surfel scene representation
• RGB-D dense tracking
• Enables Loop Closure with a Deformation Graph
[Whelan et al]
![Page 56: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/56.jpg)
DSO: Direct Sparse Odometry
[Engel, Koltun, Cremers]
• Passive Mono Camera
• Full Direct Formulation:• Jointly optimizes scene
geometry & Camera Motion
• Generative model for accounting for Image Brightness changes
• Works across many more indoor/Outdoor Scenes
![Page 57: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/57.jpg)
DART: Dense Articulated Real-Time Tracking
[Schmidt et al]
• Uses RGB-D Sensor
• Tracking only systems
• Tracks any Piece-wise rigid Articulated Object Model
• Applications in Hand, Human, Robot and Object Tracking
![Page 58: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/58.jpg)
Live RGB-D Image Real-time Non-Rigid Reconstruction
DynamicFusion: Reconstruction and Tracking of Non-rigid Scenes in Real-time
[Newcombe, Fox, Seitz]
• Uses RGB-D Sensor
• Generalizes KinectFusion
• Non-rigid Scene Motion Representation
![Page 59: Lecture 5. Dense Reconstruction and Tracking with Real ... · •Compute the photo-metric data-term between a reference frame and all available frames ... Effect of Baseline on Estimation](https://reader034.vdocuments.us/reader034/viewer/2022042910/5f402a3aafae0e421d7f7ebc/html5/thumbnails/59.jpg)
Live Depth VideoVolumetric
Motion EstimationNon-Rigid
Reconstruction
Volumetric
Surface Fusion
DynamicFusion: Reconstruction and Tracking of Non-rigid Scenes in Real-time
[Newcombe, Fox, Seitz]